RAG for Microsoft AI Agent through TextSearchProvider and Redis

Earlier this year I had published an article on implemention of InMemory Vector Embeddings in Semantic kernel.

At that time my focus was to understand how vector embeddings can be used in Semantic Kernel (SK) along with search embeddings without an external vector database.

But there are some inherent drawbacks with that approach, the major one is that all the embeddings are kept in memory. This works well when dealing with small amount of data but as the data increases coupled with user concurrency, your application will hit a major bottleneck. Hence it becomes prudent to store the embeddings and the underlying data in a persistent storage and query only the necessary data based on the user prompt.

In this article, I will reference the InMemory Vector Embeddings approach and explore how it can be integrated with Microsoft Agent Framework and Redis for persistent storage. The idea is not only to store embeddings in Redis but also to make them useful for an agent so that it can retrieve the relevant information based on the user prompt.

Custom Embedding vs Azure AI Search

Custom Embedding approach is more viable if you require complete control over storage, embedding generation, retrieval process and more importantly if most of the underlying data is centralized. But for a more enterprise applications, Azure AI Search is more relevant when the data is spread across different data sources for instance if your data is spread across say Fabric and Cosmos DB.

A custom embedding approach is preferred when the need is for a simple retrieval requirement. Azure AI Search is preferable when you need a no code/low code enterprise-grade search solution with advanced retrieval capabilities.

This article will look into the custom embedding process. For Azure AI search approach please refer to my article on the topic here and here.

Redis

To get started we will spin up an Redis image on Docker.

docker run -d --name redis-stack -p 6379:6379 -p 8001:8001 redis/redis-stack:latest

You will have to spin up a redis-stack and not local-redis else you will get the following error : RedisVectorStore Fails to Create Index and Upsert Records with "no such index"

More details : https://github.com/microsoft/semantic-kernel/issues/12734 and the solution is in the same thread here .

Though the bug report says that its fixed for SK(Semantic Kernel) this issue crops up with MAF(Microsoft Agent Framework) but honestly I don't believe that issue is related to specific agent framework (SK or MAF) but more with SK embedding library.

The bug report highlighted that Redis integration does work with the Microsoft.SemanticKernel.Connectors.AzureAISearch but this library is deprecated so I couldn't use it.

So I used the Microsoft.SemanticKernel.Connectors.AzureOpenAI library but still ended up with the same error.

Thankfully the solution of using redis-stack worked well and the AzureOpenAI library also provides the necessary DI extension method for registering the embedding services.

Redis on docker is running on port number : 6379

TextSearchProvider

This class was introduced in the Microsoft.Agents.AI namespace. It was not available with Semantic Kernel. With TextSearchProvider we can directly inject the results to the AI agents AI invocation context to enable RAG in the agent.

Now you might ask what advantages does it bring ?

With TextSearchProvider, you don't have to manually implement RAG for the agent.

It also maintains the conversation context through TextSearchProviderState class which exposes a property called RecentMessagesText which is handy if you wish to store the conversational messages to an external storage skipping the complexity of implementing your own custom ChatChistoryProvider.

I have an article on ChatChistoryProvider for MAF that you can refer here.

TextSearchProvider also exposes another class called TextSearchResult that can be used to retrieve the raw representation of the search result through the RawRepresentation property along with the Text property which returns the textual context.

Project SetUp

Create a new console application and add the following packages

dotnet add package Azure;
dotnet add package Azure.AI.OpenAI;
dotnet add package Microsoft.Agents.AI;
dotnet add package Microsoft.Extensions.AI;
dotnet add package StackExchange.Redis;
dotnet add package System.Text.Json.Serialization;
dotnet add package Microsoft.Extensions.Configuration;
dotnet add package Microsoft.Extensions.DependencyInjection;
dotnet add package Microsoft.Extensions.Logging;
dotnet add package CommunityToolkit.VectorData.Redis;

Of the above CommunityToolkit.VectorData.Redis is a .NET package that uses Redis as a vector store for semantic search and StackExchange.Redis is the Redis client that provides API for Redis commands.

Microsoft.Extensions.AI exposes an interface IEmbeddingGenerator to create the necessary embeddings .

Add appsetting.json to the project

"AppSettings": { 
    "Chat_DeploymentName": "Deployment Name",,
    "Embed_DeploymentName": "Embedding model",
    "EndPoint": "Azure OpenAI endpoint",
    "ApiKey": "Azure OpenAI API key"
}

Note : Apart from the chat model, reference to the embedding model is required.

Code

Create a DI container

 ServiceCollection servicecollection = new ServiceCollection();

Read credentials and register Chatclient

var credential = new AzureKeyCredential(configuration["AppSettings:ApiKey"]);

servicecollection.AddKeyedChatClient(
    "ChatClient",
    (
        sp =>
            new AzureOpenAIClient(new Uri(configuration["AppSettings:EndPoint"]), credential)                .GetChatClient(configuration["AppSettings:Chat_DeploymentName"])
   .AsIChatClient()
    )
);

Read the embedding generator and add it to the DI container

servicecollection.AddAzureOpenAIEmbeddingGenerator(
deploymentName: configuration["AppSettings:Embed_DeploymentName"],
endpoint: configuration["AppSettings:EndPoint"],
apiKey: configuration["AppSettings:ApiKey"]);

Register ChatClientAgent in the DI container and build the Service Provider

servicecollection.AddSingleton<ChatClientAgent>();
ServiceProvider serviceprovider = servicecollection.BuildServiceProvider();

Let's create some sample data. But before that, we have to define the object structure.

using Microsoft.Extensions.VectorData;

public class Hotel
{
      
[VectorStoreKey]
public string HotelId { get; set; }

[VectorStoreData(IsIndexed = true)]
public string HotelName { get; set; }

[VectorStoreData]
public string Description { get; set; }

[VectorStoreData]
public string Source { get; set; }

[VectorStoreVector(1536)]
public ReadOnlyMemory<float>? DescriptionEmbedding { get; set; }  

[VectorStoreData]
public string[] Tags { get; set; }

[VectorStoreVector(1536)]
public ReadOnlyMemory<float>? TagListEmbedding { get; set; }

}

What we have above is

• VectorStoreKey → This acts as a unique record identifier (primary key)

• VectorStoreData → This is used to store metadata field and can be optionally indexed

• VectorStoreVector(1536) → This is an embedding vector used for similarity search with specified dimension size. In our case we have the dimension size of 1536 for TagListEmbedding and DescriptionEmbedding

Data

Create some sample data that matches the object structure defined above.

  private static List<Hotel> CreateHotelRecords()
  {
      var hotel = new List<Hotel>
  {
      new Hotel {
          HotelId = "1",
          HotelName = "Sea Breeze Resort",
          Description = "Beachfront resort with stunning ocean-view rooms and a seafood restaurant serving fresh, locally sourced cuisine. Guests can enjoy direct access to the beach, relaxing sunsets, spacious accommodations, and a range of recreational activities. The resort also features a swimming pool, spa services, water sports, and comfortable lounge areas, making it an ideal destination for a relaxing seaside vacation.",
          Source = "www.beachfronthotel.com",
          Tags = new[] { "beach", "resort", "seafood", "luxury" }
      },

      new Hotel {
          HotelId = "2",
          HotelName = "City Central Hotel",
          Description = "Modern hotel located in the heart of the downtown area, offering stylish rooms, contemporary amenities, and convenient access to shopping malls, restaurants, cafes, and vibrant nightlife. Guests can enjoy comfortable accommodations, high-speed Wi-Fi, a fitness center, an on-site restaurant, and easy access to public transportation. The hotel is ideal for both business and leisure travelers looking to explore the city's attractions, entertainment venues, and cultural landmarks.",
            Source = "www.partytimehotel.com",
          Tags = new[] { "city", "business", "shopping", "nightlife" }
      },

      new Hotel {
          HotelId = "3",
          HotelName = "Lakeview Retreat",
          Description = "Peaceful retreat near the lake with spa and yoga facilities. Enjoy breathtaking lake views, serene surroundings, and comfortable accommodations designed for relaxation. The property offers rejuvenating spa treatments, daily yoga sessions, wellness activities, and tranquil spaces to unwind. Guests can also explore scenic walking trails, enjoy healthy meals, and experience a peaceful escape from the hustle and bustle of everyday life.",
          Source = "www.lakeviewhotel.com",
          Tags = new[] { "lake", "spa", "relaxation", "yoga" }
      },

      new Hotel {
          HotelId = "4",
          HotelName = "Desert Mirage Inn",
          Description = "Boutique desert hotel offering a unique blend of traditional charm and modern comfort in a peaceful desert setting. Guests can enjoy guided camel tours across the dunes, breathtaking desert sunsets, and an unforgettable outdoor dining experience featuring local cuisine under the open sky. The property also offers comfortable rooms, relaxing lounge areas, cultural activities, and opportunities to experience the natural beauty and tranquility of the desert.",
           Source = "www.desertadventurehotek.com",
          Tags = new[] { "desert", "boutique", "sunset", "adventure" }
      }
  };
      return hotel;
  }

There are two distinct pipeline processes involved: Insertion and Retrieval.

Insertion

Define an Embedding generator

var embeddingGenerator = kernel.Services.GetRequiredService<IEmbeddingGenerator<string, Embedding<float>>>();

Earlier, we registered an embedding model inside the Dependency Injection (DI) container. In the above code, we are requesting the same service to be returned back through the IEmbeddingGenerator interface so that the data embeddings could be generated.

JSON Collection

RedisJsonCollection<string, Hotel> redisCollection = new RedisJsonCollection<string, Hotel>(
ConnectionMultiplexer.Connect("localhost:6379").GetDatabase(), "hotel_index", new() { EmbeddingGenerator = embeddingGenerator });

Embeddings will be stored as type **RedisJsonCollection.**There is a second option called RedisHashSetCollection.

The vabriable redisCollection declared above is essentially a client-side object used to interact with Redis. The actual records and their embeddings are stored in Redis, not inside the redisCollection variable.

With RedisJsonCollection type, the objects are stored in the JSON format while fore RedisHashSetCollection, they are stored in the hash key format.

Our collection is named as hotel_index .The value of the EmbeddingGenerator property is assigned through the variable embeddingGenerator that was declared earlier.

Note : With RedisHashSetCollection, datatype string[] is not supported. Our structure has string[] data for Tags so we cant use the RedisHashSetCollection type.

Data Creation

 await redisCollection.EnsureCollectionExistsAsync();
 var hotelRecords = CreateHotelRecords().ToList();

We first ensure that the Redis collection exists if not then create it and then store the hotel data to the variable hotelRecords.

Note : At this stage we haven't updated the collection with the vector embedding values.

Create Embeddings

To create the embeddings we will have to traverse the collection and create the embeddings for each record in the collection.

async void InsertCollection(RedisJsonCollection < string, Hotel > redisCollection) 
{
  await redisCollection.EnsureCollectionExistsAsync();

  var hotelRecords = CreateHotelRecords().ToList();

  foreach(var hotel in hotelRecords) {
    var descriptionEmbeddingTask = embeddingGenerator.GenerateAsync(hotel.Description);
    var featureListEmbeddingTask = embeddingGenerator.GenerateAsync(string.Join("\n", hotel.Tags));
    hotel.DescriptionEmbedding = (await descriptionEmbeddingTask).Vector;
    hotel.TagListEmbedding = (await featureListEmbeddingTask).Vector;
  }
  await redisCollection.UpsertAsync(hotelRecords);
}

and finally update the redisCollection(declared earlier)with the UpsertAsync method. This concludes our ingestion pipeline.

Verify Redis Collection

Post execution of the ingestion pipeline, we can verify the collection in Redis. For that we will use have to enable redis-cli .

Execute the following command in the Docker terminal to enable it.

docker exec -it redis-stack redis-cli

Get a list of collection

FT._LIST

Return the details for the collection

FT.INFO hotel_index

It returns a list of different attributes of the collection that we just created.

Retrieval

Now that we have verified the collection in Redis, the next step is the retrieval process.

Retrieval will be through the AI Agent AIContextProvider property.

Before that, we have to set the SearchAdapater which acts as an input to AIContextProvider.

async Task <IEnumerable<TextSearchProvider.TextSearchResult>> SearchAdapter(string query, CancellationToken cancellationToken) {

  List <TextSearchProvider.TextSearchResult> results = new();

  var searchVector = await embeddingGenerator.GenerateAsync(query);

  var resultRecords = await redisCollection.SearchAsync(
    searchVector, top: 1, new() {
      VectorProperty = r => r.DescriptionEmbedding
    }).ToListAsync();

  var finalresults = resultRecords.Select(result =>
    new TextSearchProvider.TextSearchResult {

      Text = result.Record.Description,
        SourceName = result.Record.Source,
        RawRepresentation = result.Record.HotelName,

    });

  return await Task.FromResult <IEnumerable< TextSearchProvider.TextSearchResult>> (finalresults);
}

What we are doing above is that, we are embedding the user input with the embeddinggenerator which acts as an input to the redisCollection that was declared earlier and through SearchAsync we the return the TOP 1 matching rows by comparing the VectorProperty of the user input with the DescriptionEmbedding property of the Hotel object.

var searchVector = await embeddingGenerator.GenerateAsync(query);

var resultRecords = await redisCollection.SearchAsync(
  searchVector, top: 1, new() {
    VectorProperty = r => r.DescriptionEmbedding
  }).ToListAsync();

And finally we return the results in form of TextSearchResult property of the TextSearchProvider object.

 var finalresults = resultRecords.Select(result =>
   new TextSearchProvider.TextSearchResult {
     Text = result.Record.Description,
       SourceName = result.Record.Source,
       RawRepresentation = result.Record.HotelName
   });

return await Task.FromResult <IEnumerable< TextSearchProvider.TextSearchResult>> (finalresults);

This adapter now acts as an input of the AIContextProvider of the agent.

We will set it in the ChatClientAgentOptions of the Agent through TextSearchProvider class**.**

var options = new ChatClientAgentOptions
        {
            ChatOptions = new ChatOptions
            {
                Instructions = "You are a helpful chat assistant .Answer briefly and succinctly. Be personable.Recommend the best matching hotel.Explain briefly why it matches the request.Use only provided hotels.If the user query matches any of the hotel's tags, recommend it.If no hotel context is provided, say\"I'm sorry, but there is no matching hotel available for your request. If you have any specific preferences or criteria, please let me know, and I'll do my best to assist you!\".Otherwise assume the provided hotel is the correct match.Use the provided hotel data to answer\r\n- Answer based ONLY on given hotel.Use hotels only from the list provided to you.Add more details about the location and activities that the user can enjoy as bulleted point."
            },           
            AIContextProviders = [new TextSearchProvider(SearchAdapter, textSearchOptions)]

        };

Note the Instructions set for the agent. The instructions are very extensive that provides the agent information on how to handle the user request and how to interpret prompts and what do do incase the underlying data does not have the necessary information for user requests.

Fetch keyed IChatclient from the DI container

var _chatclient = new ChatClientAgent(serviceprovider.GetKeyedService<IChatClient>("ChatClient")).ChatClient;

Set the ChatClientAgentOptions to the above ChatClientAgent

var agent = _chatclient.AsAIAgent(options);

Pass the prompt to the Agent

Console.WriteLine(await agent.RunAsync(new ChatMessage(ChatRole.User, "I am looking for a hotel close to the beach ") 
{
   CreatedAt = DateTimeOffset.UtcNow, AuthorName = "Sachin"
 }, session: agentsession, new AgentRunOptions 
{
   AllowBackgroundResponses = true
 }) + "\n");

Output

Execution

Conclusion

This was an introductory article on how vector embeddings can be created and stored in an external storage that overcomes the shortcomings of InMemoryVector embedding . This approach is more prudent when you need more control over storage ,embeddings and the retrieval process.

Thanks for reading !!!