OpenAI Embeddings and Vector Databases Crash Course

Introduction

In the realm of AI product development, embeddings and vector databases play vital roles. This article will explore what embeddings and vector databases are, how to create and use them with OpenAI’s APIs, and how to integrate them into your applications. By the end, you will understand how to establish long-term memory for ChatGPT and conduct semantic searches across extensive databases of PDFs linked directly to an AI.

Understanding Embeddings

To begin, let's define what embeddings are. Simply put, an embedding is data that has been transformed into an array of numbers called a vector. This vector captures patterns and relationships, essentially forming a multi-dimensional map to measure similarity.

For an illustrative example, consider a 2D graph where two words, dog and puppy, would be represented as vectors clustered closely together due to their similar usage in context. In practice, these vectors exist in hundreds of dimensions, reflecting the intricate relationships between words.

Even images can be converted into vectors, a technology utilized by Google for similar image searches. Each image is broken down into numeric arrays, enabling the identification of similarity patterns.

After creating an embedding, it can be stored in a database designed for this purpose. A collection of embeddings is often referred to as a vector database, which can be utilized in various ways, including:

• Searching: Results are ranked based on relevance to a query.
• Clustering: Grouping of similar text strings.
• Recommendations: Suggesting related items based on text similarity.
• Classification: Categorizing text strings by their most similar labels.

For the purposes of this tutorial, we will focus on the search functionality.

OpenAI provides models designed specifically for creating embeddings, although it does not offer storage solutions. In this guide, we'll utilize a cloud database for storage.

Creating Embeddings

To create an embedding, you begin by accessing OpenAI's API. This process involves logging into your OpenAI account and heading to the API documentation page dedicated to embeddings.

Here are the steps to follow:

1. API Key Generation: Generate a new secret key for authorizing your API requests under the OpenAI API Keys page.
2. Using Postman: It'll be easiest to interact with the OpenAI API using Postman, a free API platform that allows for easy GET, POST, and other requests.
3. Workspace Creation: Create a new workspace for managing your API queries.
4. Crafting the Request:
   • Select POST as your request type.
   • Enter the embeddings URL: https://api.openai/v1/embeddings.
   • Set up your headers for authorization using the generated API key.
   • Define the body of your request in JSON format with your chosen model and input text.

After sending the POST request, you'll receive a response containing your first embedding represented as an array of numbers.

You can create embeddings for both single words and multi-word phrases. The strength of embeddings lies in chunking large pieces of information (like paragraphs or whole documents) into a single embedding.

Example of an Embedding Request

For instance, a more extensive request may involve embedding an entire page of a contract, which could consist of up to 8,000 tokens (approximately 30,000 characters).

Once embeddings are created, the next step is storing them in a vector database, as OpenAI does not provide database services.

Setting Up a Vector Database

To store embeddings, we will use a cloud provider that supports real-time distributed SQL databases. SingleStore is one such provider that allows you to create a vector database.

1. Account Creation: Sign up for an account, which is initially free and includes credits for usage.
2. Workspace Creation: Set up a workspace for managing your databases.
3. Database Creation: Create a new database under the workspace.
4. SQL Table Setup: Use SQL commands to create a table tailored to store text and vector data.
5. Inserting Data:
   • Use the SQL editor to input your embedding data alongside the original text.

Example SQL Commands

To create a table, you could use the following command:

CREATE TABLE IF NOT EXISTS my_vector_table (
    text TEXT,
    vector BLOB
);

After creating the table, you can insert your previously generated embeddings as new rows, ensuring the structure aligns with your table's design.

Performing Searches in the Vector Database

Searching a vector database involves a few steps:

1. Determine your search query (e.g., "OpenAI").
2. Create an embedding for your search term.
3. Set up an SQL query to search through the database:

SELECT text, 
       DOT_PRODUCT(vector, @search_embedding) AS score 
FROM my_vector_table 
ORDER BY score DESC 
LIMIT 5;

In this query, you will replace @search_embedding with the embedding created from your search term. The results returned will indicate the closest matches based on similarity scores.

Automating with JavaScript

You can automate the embedding creation and database interaction using Node.js. This includes fetching embeddings and inserting them into your vector database programmatically, enhancing efficiency and usability in your applications.

Conclusion

Embeddings and vector databases together facilitate powerful AI applications by providing systems for deep semantic understanding and efficient data retrieval.

Keywords

• Embeddings
• Vector Databases
• OpenAI API
• Search Functionality
• Semantic Search

FAQ

What are embeddings?
Embeddings are numerical representations of data (such as words or images) that capture their meanings and relationships in vector format.

What is a vector database?
A vector database is a type of database explicitly designed to store and manage vector embeddings and offers functionalities like searching, clustering, and more.

How can I create embeddings using OpenAI?
You can create embeddings by making POST requests to the OpenAI API with your desired text and specified model.

What database provider can I use for storing embeddings?
SingleStore is one recommended cloud provider that allows you to create a vector database to store embeddings.

What are some applications of embedding and vector databases?
They can be used for semantic searches, recommendations, classifications, and other AI-enhanced functionalities, such as long-term memory for chatbots.