SummarizingChatReducer in Microsoft Agent Framework

My previous article on Chat History in Microsoft Agent Framework focused on storing and retrieving chat history using a file-based approach.The intention of that article was to provide a simple introduction about implementing a basic chat history mechanism in Microsoft Agent Framework (MAF).

While a file-based approach is useful for understanding the underlying concepts it is definitely not an ideal choice for most real-world production set ups. As the number of users and conversations grows, storing chat history in file systems can be very challenging with added complexities.

In a production environment, chat history needs to be stored on a persistent data store so that it can be accessed across multiple instances of an application. This is where a distributed data stores such as Redis becomes useful. It provides a fast and scalable mechanism for storing messages and the maintain chat history.

But just simply storing the entire conversation indefinitely introduces another important challenge: the size of the conversation context. As a conversation becomes longer, sending the complete chat history to the underlying agents on every request increases token consumption, latency and cost. It can also eventually exceed the model's context window.

Though MAF provides ChatReducer to address this problem, one particularly useful implementation is SummarizingChatReducer that reduces growing conversation by summarizing older messages unlike ChatReducer that simply discards them. This helps to preserve important conversation context in place.

This article will demonstrate storing conversation to Redis and how to leverage SummarizingChatReducer to summarize and reduce long conversations while at the same time maintain the conversation context.

Custom Chat History Provider

In MAF, the most crucial component is the ProviderSessionState object wrt custom chat history implementation. It acts as the abstraction layer through which MAF manages and retrieves chat history through session StateBag.

In my previous article , I used IChatReducer to reduce chat size stored in ProviderSessionState . But drawback with this approach is that you lose the conversation context as the reduction is based on a fixed number (N) of messages. Because of this, important aspects of the conversation is discarded. The agent therefore has access only to the most recent N messages.

Given the drawback of the above approach a more feasible and practical approach is to leverage SummarizingChatReducer where values for targetCount and threshold properties define the number of messages that are summarized and stored so that correct conversational context is maintained.

Property targetCount is the used to specify the number of recent messages that should be retained post reduction while threshold is the number of messages allowed beyond targetCount before summarization.

When the conversation grows beyond targetCount + threshold values , older messages are summarized into a single message that persists the entire conversational context. This message replaces the old messages which are discarded and the most recent messages defined by targetCount stays.

For example : Lets assume that SummarizingChatReducer has targetCount of 1 and threshold of 5 and the no of prompts sent are 6.

Once the total number of chat messages recorded in ProviderSessionState is equal or greater than 6 (1+5), summarization kicks in. 1 recent message is retained post summarization in its original form and all messages are summarized.

There are two major approaches for storing and retrieving summarized conversational context.

In the first approach , you store the entire conversation along with summarized messages in a persistent storage like Redis and retrieve only the summarized messages in the ProvideChatHistoryAsync method. The major advantage with this approach is that you have the true conversation history at your disposal but disadvantage being that the size of the persistent storage would grow and managing it can be challenging.
In the second approach ,only the summarized messages are stored in a persistent storage like Redis along with all the messages above the limit set by the targetCount value. Major advantage with this approach is that you have less data to manage but disadvantage is that you lose all historical trails of the conversation as only the summarized conversational context is available and stored.

Redis

You can either use Redis on Azure or use a Redis image on Docker. I am using Docker. Spin up Redis on Docker through following command in the Docker terminal.

docker run -d --name local-redis -p 6379:6379 redis:latest

Redis is running on port number : 6379

Project SetUp

Create a new console application and add the following packages

dotnet add package Azure;
dotnet add package Azure.AI.OpenAI;
dotnet add package Microsoft.Agents.AI;
dotnet add package Microsoft.Extensions.AI;
dotnet add package StackExchange.Redis;
dotnet add package System.Text.Json.Serialization;
dotnet add package Microsoft.Extensions.Configuration;
dotnet add package Microsoft.Extensions.DependencyInjection;
dotnet add package Microsoft.Extensions.Logging;

Add appsetting.json to the project

"AppSettings": { 
    "Chat_DeploymentName": "Deployment Name",
    "EndPoint": "Azure OpenAI endpoint",
    "ApiKey": "Azure OpenAI API key"
}

Code

Once all the artifacts in place, add the following code to Program.cs

Program.cs >>

var configuration = new ConfigurationBuilder()
.SetBasePath(Directory.GetCurrentDirectory())
.AddJsonFile("appsettings.json", optional: false)
.Build();

Create a DI container

 ServiceCollection servicecollection = new ServiceCollection();

Read credentials and register Chatclient

var credential = new AzureKeyCredential(configuration["AppSettings:ApiKey"]);

servicecollection.AddKeyedChatClient(
    "ChatClient",
    (
        sp =>
            new AzureOpenAIClient(new Uri(configuration["AppSettings:EndPoint"]), credential)                .GetChatClient(configuration["AppSettings:Chat_DeploymentName"])
   .AsIChatClient()
    )
);

Register ChatClientAgent in the DI container and build the Service Provider

servicecollection.AddSingleton<ChatClientAgent>();
ServiceProvider serviceprovider = servicecollection.BuildServiceProvider();

Fetch keyed IChatclient from the DI container

var _chatclient = new ChatClientAgent(serviceprovider.GetKeyedService<IChatClient>("ChatClient")).ChatClient;

Set SummarizingChatReducer properties

SummarizingChatReducer summaryReducer = new SummarizingChatReducer(
chatClient: _chatclient,
targetCount: 2,
threshold: 3);

Here, the targetCount is set to 2 and the threshold is set to 3. This means that the conversation can grow to 5 (2+3) messages before summarization is triggered. Once threshold reaches 5 (2+3), keep 2 (targetCount) most recent messages in its raw form and summarize all the messages.

ChatClientAgentOptions and ChatHistoryProvider

 var options = new ChatClientAgentOptions
 {
     ChatOptions = new ChatOptions
     {
         Instructions = "You are a helpful chat assistant."
     },
     ChatHistoryProvider = new RedisChatHistoryProvider(summarizingChatReducer: summaryReducer, targetCount: 2, threshold: 3)

 };

Why am I passing targetCount and threshold values to the constructor of CustomChatReducer(RedisChatHistoryProvider) when these values have been already configured for SummarizingChatReducer ?

This is because, SummarizingChatReducer does not expose targetCount and threshold as publicly accessible properties. Also I am passing summarizingChatReducer instance as constructor argument.

Agent and Session

 var agent = _chatclient.AsAIAgent(options);
 var agentsession = await agent!.CreateSessionAsync();

Define CustomChatHistoryProvider (RedisChatHistoryProvider)

RedisChatHistoryProvider.cs >>

Define properties of session state in a class called SessionState.

We have defined _targetCount , _threshold and _summarizingChatReducer because we want these properties to be part of InvokingContext through ProviderSessionState (explained later).These properties can be accessed through the ProvideChatHistoryAsync and StoreChatHistoryAsync methods.

   internal class SessionState
 {
     public int _targetCount { get; set; } = 0;

     public int _threshold { get; set; } = 0;

     public SummarizingChatReducer _summarizingChatReducer { get; set; } = null;

     [JsonPropertyName("Messages")]
     public List<ChatMessage> lstChatMessages { get; set; } = [];

     [JsonPropertyName("UserName")]
     public string UserName { get; set; } = "";
   
 }

Redis Connection >>

ConnectionMultiplexer redis = ConnectionMultiplexer.Connect("localhost:6379");
private IDatabase db;

RedisChatHistoryProvider Constructor >>

We create an instance of ProviderSessionState for storing and handling the SessionState.

private readonly Microsoft.Agents.AI.ProviderSessionState<SessionState> _sessionstate;

 public RedisChatHistoryProvider(Func<AgentSession, SessionState> initializer = null, string statekey = "", SummarizingChatReducer summarizingChatReducer = null, int targetCount = 0, int threshold = 0)
 {
     _sessionstate = new(initializer =>
     {
         db = redis.GetDatabase();        
         string messages = db.StringGet(statekey);

         return new SessionState
         {
             lstChatMessages = String.IsNullOrEmpty(messages) ? Enumerable.Empty<ChatMessage>().ToList() : JsonSerializer.Deserialize<List<ChatMessage>>(messages),
             _targetCount = targetCount,
             _threshold = threshold,
             _summarizingChatReducer = summarizingChatReducer,
             UserName = statekey

         };
     }
      , statekey = "Sachin"

    );
 }

I have hardcoded statekey value to "Sachin".You can use any other uniquely identifiable value for statekey.

In the above code, the constructor parameters targetCount, threshold ,summarizingChatReducer are respectively assigned to the Session state properties of _targetCount, _threshold and _summarizingChatReducer. This way these properties are now part of the InvokingContext of the overridden methods ProvideChatHistoryAsync and StoreChatHistoryAsync.

ProviderSessionState is used for initializing and persisting provider state in the session's StateBag. State(in our case it is SessionState object) is a inbuild session state management that maintains conversation history and is stored in the StateBag using the StateKey property as the key.

As mentioned earlier, we can use either of two approaches: store the entire conversation in Redis and retrieve only the relevant summarized/reduced messages when required or store only the summarized messages in Redis.

We will use the second approach, where we store only the summarized messages along with the number of messages specified by targetCount in Redis.

StoreChatHistoryAsync >>

  protected async override ValueTask StoreChatHistoryAsync(InvokedContext context, CancellationToken cancellationToken = default)
  {
      var cntx = _sessionstate.GetOrInitializeState(context.Session);

      IEnumerable<ChatMessage> reducedMessages = new List<ChatMessage>();

      var allNewMessages = context.RequestMessages.Concat(context.ResponseMessages ?? []);
      cntx.lstChatMessages.AddRange(allNewMessages);
      
      var jsonSerializerOptions = new JsonSerializerOptions
      {
          WriteIndented = true,
          Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping
      };

      if (cntx.lstChatMessages.Count >= cntx._targetCount + cntx._threshold)
      {
          
          reducedMessages = await cntx._summarizingChatReducer.ReduceAsync(cntx.lstChatMessages, cancellationToken);
          await db.StringSetAsync(cntx.UserName, System.Text.Json.JsonSerializer.Serialize(reducedMessages, jsonSerializerOptions));
      }
      else
      {
          await db.StringSetAsync(cntx.UserName, System.Text.Json.JsonSerializer.Serialize(cntx.lstChatMessages, jsonSerializerOptions));

      }

  }

In above code there are two lists : allNewMessages and reducedMessages

allNewMessages holds response and request messages which is then added to lstChatMessages of InvokedContext context declared as cntx.

var cntx = _sessionstate.GetOrInitializeState(context.Session);

IEnumerable<ChatMessage> reducedMessages = new List<ChatMessage>();

var allNewMessages = context.RequestMessages.Concat(context.ResponseMessages ?? []);

cntx.lstChatMessages.AddRange(allNewMessages);

Recall that lstChatMessages is a property in the SessionState class

SerializationOption

   var jsonSerializerOptions = new JsonSerializerOptions
   {
       WriteIndented = true,
       Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping
   };

Check if the conversation count is greatest than targetCount and threshold combined together, trigger the ReduceAsync method and save the reduced messages in Redis if not then save all the conversation messages to Redis

  if (cntx.lstChatMessages.Count >= cntx._targetCount + cntx._threshold)
  {
      reducedMessages = await cntx._summarizingChatReducer.ReduceAsync(cntx.lstChatMessages, cancellationToken);

      await db.StringSetAsync(cntx.UserName, System.Text.Json.JsonSerializer.Serialize(reducedMessages, jsonSerializerOptions));
  }
  else
  {
      await db.StringSetAsync(cntx.UserName, System.Text.Json.JsonSerializer.Serialize(cntx.lstChatMessages, jsonSerializerOptions));

  }

Prompts

I will first send the following five prompts together in a single session

Console.WriteLine(await agent.RunAsync(new ChatMessage(ChatRole.User, "My name is Sachin") { CreatedAt = DateTimeOffset.UtcNow, AuthorName = "Sachin" }, session: agentsession, new AgentRunOptions { AllowBackgroundResponses = true }) + "\n");//Prompt 1 

Console.WriteLine(await agent.RunAsync(new ChatMessage(ChatRole.User, "I live in India") { CreatedAt = DateTimeOffset.UtcNow, AuthorName = "Sachin" }, session: agentsession, new AgentRunOptions { AllowBackgroundResponses = true }) + "\n");//Prompt 2

Console.WriteLine(await agent.RunAsync(new ChatMessage(ChatRole.User, "I work in the IT industry") { CreatedAt = DateTimeOffset.UtcNow, AuthorName = "Sachin" }, session: agentsession, new AgentRunOptions { AllowBackgroundResponses = true }) + "\n"); //Prompt 3

Console.WriteLine(await agent.RunAsync(new ChatMessage(ChatRole.User, "I am a software developer") { CreatedAt = DateTimeOffset.UtcNow, AuthorName = "Sachin" }, session: agentsession, new AgentRunOptions { AllowBackgroundResponses = true }) + "\n");//Prompt 4 

Console.WriteLine(await agent.RunAsync(new ChatMessage(ChatRole.User, "I work mainly with C# and .NET") { CreatedAt = DateTimeOffset.UtcNow, AuthorName = "Sachin" }, session: agentsession, new AgentRunOptions { AllowBackgroundResponses = true }) + "\n"); //Prompt 5

Recall that we have set the values of targetCount and threshold to 2 and 3 respectively.

In the above case Prompt 5 and the agent response will be stored to Redis in its raw form while the rest of prompts are summarized.

In the image above 1 in red is the summarized text and 2 and 3 in red are the stored messages that are based on the targetCount value.

Execution

Conclusion

ChatSummarizer provides an effective way to manage conversation history by reducing a large number of messages into a concise summary while preserving the important and required context. This helps prevent conversation history from growing indefinitely and reduces the amount of data that needs to be stored and processed.This gives us a balance between retaining recent conversation details and maintaining a conversational context of older messages.

This helps improve memory efficiency, reduce token usage and there by maintain efficient and relevant context throughout longer conversations.

Thanks for reading !!!

Chat History with Redis, ChatReducer and SummarizingChatReducer in Microsoft Agent Framework

Comments

Microsoft Agent Framework

Microsoft Agent Framework through Dependency Injection

More from this blog

Secure AI Functions in Microsoft Agent Framework with Microsoft Entra OAuth Role-Based Access Control

Build AI Agents with Microsoft Agent Framework to Access Azure Services Using Entra OAuth

Federated Credentials with User-Managed Identity to Access Azure Services through GitHub actions

Understanding Delegated Tokens vs Application Tokens through Claims-Based Authorization in Microsoft Entra ID

Custom Chat History Provider

Redis

Project SetUp

Code

Execution

Conclusion

Command Palette

Comments

Microsoft Agent Framework

Microsoft Agent Framework through Dependency Injection

More from this blog

Custom Chat History Provider

Redis

Project SetUp

Code

Execution

Conclusion