Lessons from Building StackSpot AI: A Contextualized AI Coding Assistant
CoRR(2023)
摘要
With their exceptional natural language processing capabilities, tools based
on Large Language Models (LLMs) like ChatGPT and Co-Pilot have swiftly become
indispensable resources in the software developer's toolkit. While recent
studies suggest the potential productivity gains these tools can unlock, users
still encounter drawbacks, such as generic or incorrect answers. Additionally,
the pursuit of improved responses often leads to extensive prompt engineering
efforts, diverting valuable time from writing code that delivers actual value.
To address these challenges, a new breed of tools, built atop LLMs, is
emerging. These tools aim to mitigate drawbacks by employing techniques like
fine-tuning or enriching user prompts with contextualized information.
In this paper, we delve into the lessons learned by a software development
team venturing into the creation of such a contextualized LLM-based
application, using retrieval-based techniques, called CodeBuddy. Over a
four-month period, the team, despite lacking prior professional experience in
LLM-based applications, built the product from scratch. Following the initial
product release, we engaged with the development team responsible for the code
generative components. Through interviews and analysis of the application's
issue tracker, we uncover various intriguing challenges that teams working on
LLM-based applications might encounter. For instance, we found three main group
of lessons: LLM-based lessons, User-based lessons, and Technical lessons. By
understanding these lessons, software development teams could become better
prepared to build LLM-based applications.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要