AI in Focus: Information summarization using an LLM

Introduction

Welcome back to our AI Focus live stream series! I'm Chad Pel, the CEO of Thoughtbot and a developer, and today I'm joined by Jimmy, a developer at Thoughtbot who has spent three years working on various projects including MVPs for small clients and monolith codebases for larger enterprise clients. We’re excited to dive into a fascinating topic: creating an information summarization system using a large language model (LLM).

What is Information Summarization?

Information summarization is the ability to take long pieces of text or extensive information and condense it into shorter, more digestible formats. For instance, it can help create paragraph summaries, a few-sentence abstracts, or even one-sentence summaries. While LLMs are typically quite good at this task, they do present some challenges, particularly with information loss and selecting the correct details to summarize.

In our project today, which is part of our CRM system at Thoughtbot, we aim to help users quickly understand the status of projects without wading through lengthy comment threads. Each entry in the CRM contains not only the original inquiry but also a collection of conversations that provide vital context. We are focusing on two key components – the opportunity descriptions and the associated comments – to build our summary system.

Addressing AI Ethics

Before we begin implementing our feature, we consider the ethical implications of using AI. We follow a series of guiding questions related to AI ethics:

• Transparency: Is it clear what data the LLM uses to generate summaries?
• Fairness: How do we ensure that our AI systems are unbiased and treat all groups fairly?
• Privacy: How do we protect private information while using LLMs?
• Accountability: Can we ensure clear accountability for the AI-generated summaries?
• Security: Are we preventing our AI systems from causing harm and ensuring they remain under human control?

We believe that displaying AI-generated content transparently, incorporating clear notes on data sources, and allowing users to edit summaries will contribute to ethical practices in our project.

Implementation Process

To implement the summarization system, we defined a new GeneratedSummary model associated with the Opportunity model in our application. This also led us to create a generate method that interacts with the existing prompt class we developed in prior sessions.

As we moved through the implementation, we focused on ensuring our application remains adaptable to accommodate various types of content that require summarization. Rather than linking our summary functionality directly to one specific model, we aimed for a more generic solution that could work with multiple models.

We constructed a framework allowing the Opportunity model to inform how it should be summarized by including relevant descriptions and comments. As we created a series of tests, we could iteratively refine the prompts that were sent to the LLM API, gradually improving the quality of the output summaries.

Prompt Engineering

Prompt engineering became crucial as the quality of the summary often depended on how well the prompts were crafted. We tried different formulations like "summarize this project in one sentence" or requests to extract key points from comments. Each iteration taught us more about how to effectively communicate with the AI model.

Ultimately, we recognized the importance of providing context in our prompts – specifying which parts of the input were descriptions versus comments to guide the LLM in generating better summaries.

Conclusion

As we approached the completion of our project, we recognized the potential for expanding this summarization functionality to other areas within our application. Whether you're developing internal tools or customer-facing applications, understanding and implementing AI responsibly is becoming more relevant than ever.

If you are looking to incorporate AI features into your applications or need help improving your existing Rails apps, don’t hesitate to reach out at Thoughtbot for more information on our services and expertise in integrating AI features into applications.