# Observability in Microsoft Agent Framework through Open Telemetry in Azure Application Insights, KQL and Aspire on Docker

In a typical Agent AI and Workflow setups when something goes wrong major questions start piling up:

*   Which agent handled the request?
    
*   Which tool was invoked?
    
*   How long did each step take?
    
*   Where did the failure occur?
    
*   Why is the workflow slower than expected?
    

Without proper observability, answering these questions is very difficult.

This is where Open Telemetry, Azure Application Insights and .NET Aspire come together. They provide a powerful way to gain visibility into what's happening inside your Microsoft Agent Framework applications. Instead of relying on console logs and guesswork, you can trace requests end-to-end, monitor agent interactions, measure performance and quickly identify bottlenecks.

In this article, we'll build a Microsoft Agent Framework application running in Docker, integrate it with Azure Application Insights and use Aspires dashboard to visualize telemetry in real time.

### UseCase

We will use an use case from this [post](https://www.azureguru.net/microsoft-agent-framework-with-background-service-and-azure-service-bus) where an agent runs as a background service and the input to the agent is pushed from Azure Service Bus.

### **SetUp**

Before we move to the **OpenTelemetry** implementation, we will have to set up Aspire service in Docker and the Log Analytics workspace on Azure.

Pull up the Docker desktop and run the following command in the Docker terminal.

```yaml
docker run --rm -it  -p 18888:18888  -p 4317:18889  -p 4318:18890  -d --name aspire-dashboard  mcr.microsoft.com/dotnet/aspire-dashboard:latest
```

![](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/6b10579e-2cc8-40f1-b8b7-665186f1a22e.png align="center")

Once installed , navigate to http://localhost:18888/login . The login page would prompt for a token value

![Microsoft Agent Framework and Agent Workflows](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/4aaabe0b-5dc6-4cc7-a381-23cd78a1341f.png align="center")

Navigate to the Containers tab and under Logs you will find Aspire Dashboard details. We get Dashboard URL, Login URL with the embedded token and the corresponding gRPC and HTTP URLs.

![Microsoft Agent Framework and Agent Workflows](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/40461489-e493-46fc-aca0-74bdff88ee0e.png align="center")

or run the following command in the Docker terminal.

```yaml
docker ps
```

Enter the token value and you will land up on the Aspire Dashboard

![Microsoft Agent Framework and Agent Workflows](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/8e57f52e-d374-4d4f-8411-1f6c405c34d9.png align="center")

Set up **Log Analytics workspace** and **Application Insights** resource, search for Log Analytics workspace and Application Insights in Azure Marketplace.

![Microsoft Agent Framework and Agent Workflows](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/ae46c208-3721-4860-8f83-f404d0aff49e.png align="center")

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/90780ab7-3555-4b3c-b776-2d8797830ec0.png align="center")

While setting up Application Insights you will have to select an option to select the Log Analytics workspace.

The overall setup for these two resources is pretty straightforward.

If required, you can also change the Log Analytics Workspace for the Application Insights.

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/4d9d5d1e-953d-4518-8ece-3896c9c73a01.png align="center")

In the next step, add the following resources to the project

```csharp
dotnet add package Azure.Monitor.OpenTelemetry.Exporter
dotnet add package OpenTelemetry.Logs
dotnet add package OpenTelemetry.Metrics
dotnet add package OpenTelemetry.Resources
dotnet add package OpenTelemetry.Trace
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package System.Diagnostics
```

Then add the following telemetry services

```csharp
 builder.Services.AddOpenTelemetry()
.WithTracing(tracing =>
{
    tracing
        .SetSampler(new AlwaysOnSampler())
        .SetResourceBuilder(
        ResourceBuilder.CreateDefault()
        .AddService("MyApp"))   
        .AddSource("MyApp.Source")                      
        .AddOtlpExporter(options => options.Endpoint = new Uri("http://localhost:4317"))
        .AddHttpClientInstrumentation()
        .AddConsoleExporter()
        .AddAzureMonitorTraceExporter(options =>
        {
         options.ConnectionString = "AppLicationInsights Connection String"
                ;
        });
});

 builder.Logging.ClearProviders();
 builder.Logging.AddConsole();
 builder.Logging.SetMinimumLevel(LogLevel.Trace);
```

Lets break down the major aspects of the above code :

The following code sets the **service name** that appears in the telemetry. Without a service name the app traces appears under a generic or auto-generated name.

```csharp
.AddService("MyApp"))
```

Define a source **MyApp.Source** for the service **MyApp** that listens to the activities created in it.

```csharp
.AddSource("MyApp.Source")
```

Now configure OpenTelemetry to to export traces, metrics, and logs to an OTLP collector. In our case the OTLP collector is an Aspire service running on Docker.

```csharp
.AddOtlpExporter(options => options.Endpoint = new Uri("http://localhost:4317"))
```

You might wonder where did port 4317 come from.

Recall that we had configured 4317 while setting up the Aspire Docker container. So 4317 is the host port and 18889/18890 is the container port.

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/1144404a-7991-492f-a1e0-efaa6b04952c.png align="center")

The next part of the code adds enrichment to logs when one service is calling another service using HTTP. This basically means that OpenTelemetry automatically adds extra context (metadata) to the logs, traces and spans about the outgoing HTTP request.

```csharp
.AddHttpClientInstrumentation()
```

The next piece of code displays the trace to the console window during execution. This part can be optional as I don't feel it does provides much of an value addition. It simply makes the traces visible during execution.

```plaintext
 .AddConsoleExporter()
```

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/109e8930-9bca-4a04-a43b-9f3147e0379b.png align="center")

The following piece of code exports the logs to the Azure Applications Insight resource that we created earlier.

```csharp
.AddAzureMonitorTraceExporter(options =>
{
options.ConnectionString = "AppLicationInsights Connection String";
}     
```

The connection string is available through the Applications Insight dashboard.

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/ada90491-1d0b-4187-9fc6-d8e6c34f4541.png align="center")

In the following code , we first clear all the default loggers that ASP.NET adds. If required we add the logging info to the console and then set the minimum logging level to **Trace** which implies that all the details **Debug**, **Info**, **Warning** etc should be logged.

```csharp
builder.Logging.ClearProviders();
builder.Logging.AddConsole();
builder.Logging.SetMinimumLevel(LogLevel.Trace);
```

In the Background Service, which in our case is named **Worker** we set different levels of span/traces.

First, define an **ActivitySource** called **MyApp.Worker** under the **StartAsync** operation.

```csharp
private static readonly ActivitySource Activity = new("MyApp.Worker");
```

Next, define an **Activity** called **Worker.Start** and trace/spans under it

```csharp
using var activity_start = Activity.StartActivity("Worker.Start", ActivityKind.Server);

activity_start.SetTag("worker.name", "Worker_1");
activity_start.SetTag("Activity Name", activity_start.DisplayName);
activity_start.SetTag("Source Name", activity_start.Source.Name);
```

and then under **RunAsync** event which is a method that runs as a Background service.

Note that we have a different **Activity** called **Worker.Run** where different sets of trace/spans are registered.

```csharp
using var activity = Activity.StartActivity("Worker.Run", ActivityKind.Server);

 activity?.SetTag("receiver.name", "promptqueue");           
 activity?.SetTag("receiver.activity", "servicebusdata");
 activity?.SetTag("operation.type", "background-job");
 activity?.SetTag("operationstatus", "start");
 activity?.SetTag("servicebus.ingestion.start", "promptqueue");
```

Trace the token usage by the agent

```csharp
AgentResponse response = await agent.RunAsync(message.Body.ToString(), session);     
Console.WriteLine(response.Text);
activity?.SetTag("InputTokeCount", response.Usage.InputTokenCount);
activity?.SetTag("OutputTokeCount", response.Usage.OutputTokenCount);
activity?.SetTag("TotalTokenCount", response.Usage.TotalTokenCount);
```

### Complete Code

**Program.cs >>**

```csharp
using Azure;
using Azure.AI.OpenAI;
using Azure.Messaging.ServiceBus;
using Azure.Monitor.OpenTelemetry.Exporter;
using Microsoft.Agents.AI;
using Microsoft.Extensions.AI;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using OpenTelemetry.Logs;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using System.Diagnostics;

internal class Program

{
    private async static Task Main(string[] args)

    {
        HostApplicationBuilder builder = Host.CreateApplicationBuilder();
        var configuration = new ConfigurationBuilder()
      .SetBasePath(Directory.GetCurrentDirectory())
      .AddJsonFile("appsettings.json", optional: false)
      .Build();

        var credential = new AzureKeyCredential(configuration["AppSettings:ApiKey"]);

        builder.Services.AddKeyedChatClient("ChatClient", (sp => new AzureOpenAIClient(
                 new Uri(configuration["AppSettings:EndPoint"]), credential)
                     .GetChatClient(configuration["AppSettings:Chat_DeploymentName"])
   .AsIChatClient()));

        builder.Services.AddSingleton<AIAgent>(sp =>
        {
            Func<ChatClientAgentOptions> func = () =>
            {
                return new ChatClientAgentOptions
                {
                    ChatOptions = new ChatOptions
                    {

                        Instructions = "You are a helpful stock market analysis assistant"
                    }
                };

            };

            return new ChatClientAgent(sp.GetKeyedService<IChatClient>("ChatClient"), options: func());
        }

       );

        builder.Services.AddSingleton(sp =>
        {
            return new ServiceBusClient(configuration["AppSettings:AzureQueue"], new ServiceBusClientOptions
            {
                TransportType = ServiceBusTransportType.AmqpTcp
            });
       });
        
        builder.Services.AddOpenTelemetry()
      .WithTracing(tracing =>
      {
          tracing
              .SetSampler(new AlwaysOnSampler())
              .SetResourceBuilder(
              ResourceBuilder.CreateDefault()
              .AddService("MyApp")) 
              .AddSource("MyApp.Worker")                
              .AddOtlpExporter(options => options.Endpoint = new Uri("http://localhost:4317"))                                                                                              
              .AddHttpClientInstrumentation() 
              .AddConsoleExporter()
              .AddAzureMonitorTraceExporter(options =>
              {
                  options.ConnectionString = "Application Insight Connection String";
              });
      });
        //docker ps
        builder.Logging.ClearProviders();
        builder.Logging.AddConsole();
        builder.Logging.SetMinimumLevel(LogLevel.Trace);
        builder.Services.AddHostedService<Worker>();
        using IHost host = builder.Build();
        await host.RunAsync().ConfigureAwait(false);
    }

    internal sealed class Worker(AIAgent agent, ServiceBusClient servicebusClient,IHostApplicationLifetime appLifetime, ILogger<Worker> logger, IHost host) : IHostedService
    {
        private AgentSession? session;
        private Task? backgroundTask;
        private static readonly ActivitySource Activity = new("MyApp.Worker");

        public async Task StartAsync(CancellationToken cancellationToken)
        {

            using var activity_start = Activity.StartActivity("Worker.Start", ActivityKind.Server);

           
                activity_start.SetTag("worker.name", "Worker_1");
                activity_start.SetTag("Activity Name", activity_start.DisplayName);
                activity_start.SetTag("Source Name", activity_start.Source.Name);
           

            session = await agent.CreateSessionAsync(cancellationToken);
            backgroundTask = RunAsync(appLifetime.ApplicationStopping);
        }


        public async Task RunAsync(CancellationToken cancellationToken)
        {
            await Task.Delay(1000, cancellationToken);

            var receiver = servicebusClient!.CreateReceiver("promptqueue");

            while (!cancellationToken.IsCancellationRequested)
            {
                using var activity = Activity.StartActivity("Worker.Run", ActivityKind.Server);
                activity?.SetTag("receiver.name", "promptqueue");           
                activity?.SetTag("receiver.activity", "servicebusdata");
                activity?.SetTag("operation.type", "background-job");
                activity?.SetTag("operationstatus", "start");
                activity?.SetTag("servicebus.ingestion.start", "promptqueue");

                ServiceBusReceivedMessage message = await receiver.ReceiveMessageAsync(cancellationToken: cancellationToken);

                activity?.SetTag("servicebus.ingestion.end", "promptqueue");

                if (message == null)
                {
                    continue;
                }
                Console.WriteLine("----------------------------------------------------------------------------------------------------------------------------");
                Console.WriteLine("");
                activity?.SetTag("AgentInput", message.Body.ToString());
                activity?.SetTag("SessionId", session.ToString());
                AgentResponse response = await agent.RunAsync(message.Body.ToString(), session);              
                Console.WriteLine(response.Text);
                activity?.SetTag("InputTokeCount", response.Usage.InputTokenCount);
                activity?.SetTag("OutputTokeCount", response.Usage.OutputTokenCount);
                activity?.SetTag("TotalTokenCount", response.Usage.TotalTokenCount);
                Console.WriteLine("");
                Console.WriteLine("----------------------------------------------------------------------------------------------------------------------------");
                await receiver.CompleteMessageAsync(message);
                activity?.SetTag("operationstatus", "end");
            }

        }

        public async Task StopAsync(CancellationToken cancellationToken)
        {
            if (backgroundTask != null)
            {
                await backgroundTask;

            }
        }
    }

}
```

Run the app and check the Logs options in Application Insights and you will see the traces.

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/598990dc-455d-4506-997d-5d695b749c4c.png align="center")

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/6b37eea3-0504-4800-a427-e852900f25c6.png align="center")

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/3bd63584-316c-4b6e-9af5-b192a89e98b2.png align="center")

### KQL Queries

We can leverage KQL queries to query and fetch detailed insights from these trace logs.

```yaml
union traces, requests, dependencies
| order by timestamp desc
```

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/a940da8d-fbe0-495b-9685-939b755fb278.png align="center")

Expanding **customDimensions** gets the token usage details

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/7e3c07b5-ba0f-4a13-a873-56b6a4d09a68.png align="center")

```yaml
requests
| where name == "Worker.Start"
| order by timestamp desc
```

Gets the the traces for event the **Worker.Start**

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/49328550-ca68-45dd-8ced-3f6b8c70c398.png align="center")

```yaml
requests
| where customDimensions["worker.name"] == "Worker_1"
| order by timestamp desc
```

Trace details for worker **Worker\_1**

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/f034c543-258e-407a-8d5a-e8c8540e66b2.png align="center")

```yaml
requests
| project timestamp,name,duration,success
| order by duration desc
```

Duration and success status

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/43b3dbc2-6dd8-4053-bc95-8c07c09927c8.png align="center")

```yaml
requests
| where customDimensions["TotalTokenCount"] >100
| order by timestamp desc
```

Details of requests that costs more than 100 tokens.

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/622b3268-9e37-4e87-9464-2ddd37e92f82.png align="center")

```yaml
requests
| where duration >60000
| order by duration 
```

Gets the details of requests having a duration >60000

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/55f56957-bb88-4191-89b7-b5f51d3d2769.png align="center")

### Aspire Dashboard

Navigate to the **Apsire** dashboard thorough http://localhost:18888/traces that was configured earlier

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/ea707b80-87a5-4273-8f32-3bf629ac6681.png align="center")

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/e0527936-894c-4eb7-b1ab-b7835ad74c6f.png align="center")

**Execution >>**

![Microsoft Agent Framework and Agent Observability](https://cdn.hashnode.com/uploads/covers/6693c62c166ee9c594cffda0/c1d6754f-61cf-4b3e-81f6-80f6d9daf4c3.gif align="center")

### **Conclusion**

Through this article, I demonstrated the most viable options by which Observability in Microsoft Agent Framework through Open Telemetry in Azure Application Insights and Aspire can be implemented.

I would personally prefer Observability implementation in MAF through Azure Application Insights because of strong KQL support with the flexibility of leveraging KQL queries which Aspire lacks. Added to that, Azure Application Insights also seamlessly integrates with Grafana to implement rich dashboard experience which I would possibly cover in another article.

I hope this article helps you get started with **OpenTelemetry** implementation for Microsoft Agent Framework.

Thanks for reading !!!