langchain/docs/extras/use_cases/code_understanding.ipynb

Code Understanding

Use case

Source code analysis is one of the most popular LLM applications (e.g., GitHub Co-Pilot, Code Interpreter, Codium, and Codeium) for use-cases such as:

• Q&A over the code base to understand how it works
• Using LLMs for suggesting refactors or improvements
• Using LLMs for documenting the code

Overview

The pipeline for QA over code follows the steps we do for document question answering, with some differences:

In particular, we can employ a splitting strategy that does a few things:

• Keeps each top-level function and class in the code is loaded into separate documents.
• Puts remaining into a seperate document.
• Retains metadata about where each split comes from

Quickstart

In [29]:

!pip install openai tiktoken chromadb langchain

# Set env var OPENAI_API_KEY or load from a .env file
# import dotenv

# dotenv.load_env()

We'lll follow the structure of this notebook and employ context aware code splitting.

Loading

We will upload all python project files using the langchain.document_loaders.TextLoader.

The following script iterates over the files in the LangChain repository and loads every .py file (a.k.a. documents):

In [ ]:

from git import Repo
from langchain.text_splitter import Language
from langchain.document_loaders.generic import GenericLoader
from langchain.document_loaders.parsers import LanguageParser

In [12]:

# Clone
repo_path = "/Users/rlm/Desktop/test_repo"
repo = Repo.clone_from("https://github.com/hwchase17/langchain", to_path=repo_path)

We load the py code using LanguageParser, which will:

• Keep top-level functions and classes together (into a single document)
• Put remaining code into a seperate document
• Retains metadata about where each split comes from

In [14]:

# Load
loader = GenericLoader.from_filesystem(
    repo_path+"/libs/langchain/langchain",
    glob="**/*",
    suffixes=[".py"],
    parser=LanguageParser(language=Language.PYTHON, parser_threshold=500)
)
documents = loader.load()
len(documents)

Out [14]:

1293

Splitting

Split the Document into chunks for embedding and vector storage.

We can use RecursiveCharacterTextSplitter w/ language specified.

In [17]:

from langchain.text_splitter import RecursiveCharacterTextSplitter
python_splitter = RecursiveCharacterTextSplitter.from_language(language=Language.PYTHON, 
                                                               chunk_size=2000, 
                                                               chunk_overlap=200)
texts = python_splitter.split_documents(documents)
len(texts)

Out [17]:

3748

RetrievalQA

We need to store the documents in a way we can semantically search for their content.

The most common approach is to embed the contents of each document then store the embedding and document in a vector store.

When setting up the vectorstore retriever:

• We test max marginal relevance for retrieval
• And 8 documents returned

Go deeper

• Browse the > 40 vectorstores integrations here.
• See further documentation on vectorstores here.
• Browse the > 30 text embedding integrations here.
• See further documentation on embedding models here.

In [19]:

from langchain.vectorstores import Chroma
from langchain.embeddings.openai import OpenAIEmbeddings
db = Chroma.from_documents(texts, OpenAIEmbeddings(disallowed_special=()))
retriever = db.as_retriever(
    search_type="mmr",  # Also test "similarity"
    search_kwargs={"k": 8},
)

Chat

Test chat, just as we do for chatbots.

Go deeper

• Browse the > 55 LLM and chat model integrations here.
• See further documentation on LLMs and chat models here.
• Use local LLMS: The popularity of PrivateGPT and GPT4All underscore the importance of running LLMs locally.

In [32]:

from langchain.chat_models import ChatOpenAI
from langchain.memory import ConversationSummaryMemory
from langchain.chains import ConversationalRetrievalChain
llm = ChatOpenAI(model_name="gpt-4") 
memory = ConversationSummaryMemory(llm=llm,memory_key="chat_history",return_messages=True)
qa = ConversationalRetrievalChain.from_llm(llm, retriever=retriever, memory=memory)

In [30]:

question = "How can I load a source code as documents, for a QA over code, spliting the code in classes and functions?"
result = qa(question)
result['answer']

Out [30]:

'To load a source code as documents for a QA over code, you can use the `CodeLoader` class. This class allows you to load source code files and split them into classes and functions.\n\nHere is an example of how to use the `CodeLoader` class:\n\n```python\nfrom langchain.document_loaders.code import CodeLoader\n\n# Specify the path to the source code file\ncode_file_path = "path/to/code/file.py"\n\n# Create an instance of the CodeLoader class\ncode_loader = CodeLoader(code_file_path)\n\n# Load the code as documents\ndocuments = code_loader.load()\n\n# Iterate over the documents\nfor document in documents:\n    # Access the class or function name\n    name = document.metadata["name"]\n    \n    # Access the code content\n    code = document.page_content\n    \n    # Process the code as needed\n    # ...\n```\n\nIn the example above, `code_file_path` should be replaced with the actual path to your source code file. The `load()` method of the `CodeLoader` class will return a list of `Document` objects, where each document represents a class or function in the source code. You can access the class or function name using the `metadata["name"]` attribute, and the code content using the `page_content` attribute of each `Document` object.\n\nYou can then process the code as needed for your QA task.'

In [33]:

questions = [
    "What is the class hierarchy?",
    "What classes are derived from the Chain class?",
    "What one improvement do you propose in code in relation to the class herarchy for the Chain class?",
]

for question in questions:
    result = qa(question)
    print(f"-> **Question**: {question} \n")
    print(f"**Answer**: {result['answer']} \n")

-> **Question**: What is the class hierarchy? 

**Answer**: The class hierarchy in object-oriented programming is the structure that forms when classes are derived from other classes. The derived class is a subclass of the base class also known as the superclass. This hierarchy is formed based on the concept of inheritance in object-oriented programming where a subclass inherits the properties and functionalities of the superclass. 

In the given context, we have the following examples of class hierarchies:

1. `BaseCallbackHandler --> <name>CallbackHandler` means `BaseCallbackHandler` is a base class and `<name>CallbackHandler` (like `AimCallbackHandler`, `ArgillaCallbackHandler` etc.) are derived classes that inherit from `BaseCallbackHandler`.

2. `BaseLoader --> <name>Loader` means `BaseLoader` is a base class and `<name>Loader` (like `TextLoader`, `UnstructuredFileLoader` etc.) are derived classes that inherit from `BaseLoader`.

3. `ToolMetaclass --> BaseTool --> <name>Tool` means `ToolMetaclass` is a base class, `BaseTool` is a derived class that inherits from `ToolMetaclass`, and `<name>Tool` (like `AIPluginTool`, `BaseGraphQLTool` etc.) are further derived classes that inherit from `BaseTool`. 

-> **Question**: What classes are derived from the Chain class? 

**Answer**: The classes that are derived from the Chain class are:

1. LLMSummarizationCheckerChain
2. MapReduceChain
3. OpenAIModerationChain
4. NatBotChain
5. QAGenerationChain
6. QAWithSourcesChain
7. RetrievalQAWithSourcesChain
8. VectorDBQAWithSourcesChain
9. RetrievalQA
10. VectorDBQA
11. LLMRouterChain
12. MultiPromptChain
13. MultiRetrievalQAChain
14. MultiRouteChain
15. RouterChain
16. SequentialChain
17. SimpleSequentialChain
18. TransformChain
19. BaseConversationalRetrievalChain
20. ConstitutionalChain 

-> **Question**: What one improvement do you propose in code in relation to the class herarchy for the Chain class? 

**Answer**: As an AI model, I don't have personal opinions. However, one suggestion could be to improve the documentation of the Chain class hierarchy. The current comments and docstrings provide some details but it could be helpful to include more explicit explanations about the hierarchy, roles of each subclass, and their relationships with one another. Also, incorporating UML diagrams or other visuals could help developers better understand the structure and interactions of the classes. 

The can look at the LangSmith trace to see what is happening under the hood:

• In particular, the code well structured and kept together in the retrival output
• The retrieved code and chat history are passed to the LLM for answer distillation