Table of Contents

Introduction: Why AIs are stateless

Large Language Models (LLMs) are stateless. Ask, “How many levels are in Super Mario 64?” and you’ll get an answer. Ask, “How many stars are there?” right after, and the AI often won’t recognize you mean the game. It may return an unrelated number.

Each LLM request is isolated. For AI to understand context, you must send the entire conversation history each time.

With every additional chat question, the number of input tokens rises. You pay for the entire historical text sent back and forth.

The Basic Approach: Agent Sessions

In-Memory Storage:

To solve this, the Agent Framework provides the concept of Agent Sessions. Instead of just calling agent.runAsync("Question"), you create a session and include it with each call. The framework then automatically appends the new messages to a list in the background and sends them with the next call.

// Creating an Agent Session to store short-term context
var session = await agent.GetNewSessionAsync(); 

// Passing the session with each request
var response1 = await agent.RunAsync("How many levels are in Super Mario 64?", session);
var response2 = await agent.RunAsync("How many stars are there?", session); 
// The AI now understands you are still talking about the game!

By default, storage is in-memory only. If the app closes or the server restarts, the AI’s memory is wiped.

The Solution for Long-Term Memory: The ChatHistoryProvider

To offer features like ChatGPT’s left sidebar, where past chats resume, persistence is needed. This is where ChatHistoryProvider helps.

The StateBag Concept

Each session has a StateBag, a flexible key-value store. Store a unique session ID (e.g., a GUID) as a reference for your database or file system. By keeping the ID separate from the chat history, you can securely reference and restore sessions.

Practical Implementation: Saving and Restoring

To build a provider, inherit from the ChatHistoryProvider class and override two main methods:

public class MyDatabaseChatHistoryProvider : ChatHistoryProvider
{
    // Step 1 - Saving
    public override async Task StoreChatHistoryAsync(ChatHistoryContext context)
    {
        // Retrieve our Session ID from the StateBag
        var sessionId = context.Session.StateBag["SessionId"].ToString();
        
        // Grab the newest messages from the context
        var newRequest = context.RequestMessages;
        var newResponse = context.ResponseMessages;

        // Serialize and save the context to disk or a database record
        await SaveMessagesToDatabaseAsync(sessionId, newRequest, newResponse);
    } 

    // Step 2 - Restoring
    public override async Task<IReadOnlyList<ChatMessage>> ProvideChatHistoryAsync(ChatHistoryContext context)
    {
        // Check if the StateBag already has a Session ID
        if (!context.Session.StateBag.TryGetValue("SessionId", out var sessionIdObj))
        {
            // It's a new session, create a unique ID and store it in the StateBag
            context.Session.StateBag["SessionId"] = Guid.NewGuid().ToString();
            return Array.Empty<ChatMessage>(); // No history to load yet
        }

        // If the ID exists, read the previous chat messages from your database
        var sessionId = sessionIdObj.ToString();
        var historicalMessages = await LoadMessagesFromDatabaseAsync(sessionId);
        
        return historicalMessages; 
    }
}

Step 1 - Saving (StoreChatHistoryAsync):

The framework calls this method after the AI responds, but before the user sees it. Here, you can serialize the context and store it. Like writing JSON to disk or a database record.

Step 2 - Restoring (ProvideChatHistoryAsync):

When a user returns and you pass a session with an existing StateBag ID, this method runs. It reads the saved file or database, deserializes the text into chat messages, and hands them to the agent. Crucially, it returns the deserialized messages to the agent so the AI has the context loaded before it processes the user’s new prompt. The AI is caught up and ready to continue.

Conclusion

With ChatHistoryProvider, you control chat storage. The AI remembers the user, even after long breaks.

Now our AI remembers whole conversations. But if the history grows too large, hitting token limits and increasing costs, what then? Next, we’ll explore Chat Reducers: tools for summarizing or trimming old messages to save tokens.

Further Reading