Development of an AI tool for processing documents and forming a legal opinion

Our client, a law firm, needs to process a massive amount of documents to write a legal opinion for insurance company compensation. Previously, to do this, attorneys had to manually review hundreds of pages of insurance claims, police reports, medical records and other handwritten notes (about 250 pages of PDF files per case on average), which was time-consuming and often resulted in errors.
In this case study, we will describe the development of an artificial intelligence (AI)-based tool that made it faster and easier to process information in insurance claims into a consistent and concise legal opinion.

Project Overview:

• Location: USA
• Product: Website, Internal tool
• Technologies: Vue.js, Langchain, FastAPI, Docker Compose
• Team: Back-end Developer, Front-end Developer
• Timeline: since 2020

The first idea of implementing the tool was actually a straightforward solution to the problem: asking ChatGPT for the entire result at once. However, this approach turned out to be impossible because of the huge amount of mixed information (we recall that it is about 250 PDF pages of handwritten medical and police reports, conditions of the incident and testimony). Because of this, ChatGPT can't distinguish groups of topics and be specific about each of them. Therefore, we decided to divide the task and generate the text summary for each topic separately.
Following this, we decided to divide the task into several threads and assigned it to five specialized assistants who were tailored to their needs. These assistants were given the following context:

• Examples of work that attorneys have done by hand, with the widest possible range of outcomes and combinations in the results;
• A decision template clearly showing the order of the conclusions in the file;
• Additional information about special cases that the assistant needs to pay attention to.
  

It is important to note that the assistant should repeatedly point out the importance of one or another aspect in order to improve the quality of the results.
At the initial stage, we created the assistants themselves that allowed us to take some general prompts. Then we started to refine the prompts for the assistants we made so that they give more accurate results. That’s why we decided to ask about a particular aspect in each query to get the desired result. However, this approach was not quite ideal and sometimes there were problems where the result was not obtained and the assistant was difficult to convince that it was wrong. In such cases, we had to reload the chat through the API to initialize a new session.
It is also worth mentioning the iterative approach to creating such AI assistants. If the assistant lacks some knowledge, it is necessary to manually find some data from a set of documents and other systems and add it to the task description, thus improving the context and prioritizing. Sometimes it is even necessary to insert the same information into the prompt, thus showing the importance of some knowledge that should not be missed.
The accuracy of the results also depends on the format of the answer to be asked to the assistant. This seems to be a standard practice already, but it is quite useful for this type of problem.

As a result of five assistants' work, we got a structured text with a list of conclusions and important results highlighted in bold.
Processing one set of documents for one case through OpenAPI costs about $3-4 USD and significantly saves the attorney's time to read 250 pages of text. Our solution highlights important aspects of the case, and the attorney can check the accuracy of the conclusion made by the AI tools and easily supplement it if necessary. Our solution has accelerated the work of attorneys by 1.5 times.