Azure AI search REST API's for Fabric One Lake Unstructured Data
My previous article was focused on how to set up the Azure AI search service and the underlying components to query unstructured data stored on a Fabric lakehouse. You can out check that article here.
If you want to skip the read up you can instead watch the video below
Our data source have two pdf's of fictional stories stored on Fabric One lake lakehouse.
Below is the Azure portal query screen that we could use to query the knowledge base after we set the Azure AI search service and all the underlying components.
The approach used in the first part had some limitations , not saying that it was incorrect but you wouldn't want the end user to navigate through Azure portal and understand ways to query the Azure AI search service through Azure portal and all the technical definitions and jargons like knowledgebase or indexes or knowledge sources and also considering the security implications of granting end users access to the Azure portal it would not be feasible to provide such access.
So there should be some mechanism where the end user could easily query Azure AI search through an interface that he is comfortable with (can be desktop app, a web or a mobile app).
This is where the Azure AI knowledge retrieval REST API's come in handy. These APIs enable developers to build flexible and customized interfaces that can connect with the search service. By leveraging REST APIs the end users can submit queries, retrieve results, apply filters and refine searches in real time through the interface of their choice.
The endpoint of the knowledge retrieval API is as follows:
POST {endpoint}/knowledgebases('{knowledgeBaseName}')/retrieve?api-version=2025-11-01-preview
If you look at the endpoint above , it looks more or less like OData-like syntax with parentheses required for the knowledgebase name. The api- version used is 2025-11-01-preview though there are other older versions also available but 2025-11-01-preview is the latest one.
These API's aren't just limited to querying the knowledgebase but can also perform other multiple operations like creating/updating/deleting indexes/knowledge sources/knowledgebases.
For example the following endpoint lets your create an index on the Azure Search service
POST {endpoint}/indexes?api-version=2025-11-01-preview
But this article will only be focused on querying the knowledgebases through the API.
Sample API POST request
The main components of the POST requests are:
messages.role
messages.content
In our code we would use the C# serialization library to serialize the POST requests body and Newtonsoft.Json library to deserialize the responses. This is my personal preference as I feel Newtonsoft to be more flexible to handle dynamic responses.
Also, I am using Spectre.Console to enhance the response output to the console screen. Also used is MSAL library to generate access tokens to authenticate underlying services through a service principal. Optionally you could use Azure search API keys but I am not much big fan of using API keys to access underlying services.
Service Principal permissions >> Grant the following API permissions to the service principal.
Only Machine Learning Service API permission is needed but so that there are no other surprises its better to also grant Cognitive Search permissions. The scope to be used is https://search.azure.com/.default
Next, assign the Search Index Reader role to the user login for which you intend to generate the access token for the created Azure AI search service.
If you recall from the previous article on this topic, I had granted Cognitive Services OpenAI Contributor role access to the User Managed Identity.
Apart from this , we will also have to grant Cognitive Services OpenAI access to the service principal of the Azure OpenAI service. It can either be Contributor or User role . I granted Cognitive Services OpenAI User role access to the service principal.
SetUp
Install the following Nuget Packages in your C#.Net console application
dotnet add package Newtonsoft.Json
dotnet add package Microsoft.Extensions.Configuration.FileExtensions
dotnet add package Microsoft.Extensions.Configuration.Json
dotnet add package Microsoft.Extensions.Configuration
dotnet add package Spectre.Console
dotnet add package Microsoft.Identity.Client
We will read the configuration details from appsettings.json file.
Appsettings.json
{
"Logging":
{ "LogLevel": {
"Default": "Information",
"Microsoft": "Warning",
"Microsoft.Hosting.Lifetime": "Information"
}
},
"AllowedHosts": "*",
"ClientId": "Service Principal Client Id",
"AzureAISearchEndPoint": "Azure AI Search Endpoint",
"KnowledgeBase": "Your Azure AI Search knowledge base",
"KnowledgeSource": "Your Azure AI Search knowledge source"
}
}
The code would read the service principal ClientId,AzureAISearchEndPoint and KnowledgeBase from appsettings.json.
Code
I have four classes in the project
Program.cs being the entry point into the application.
Authentication.cs returns the bearer token
Messages.cs defines the structure needed for the POST request
HttpMethods.cs containing the various Http methods
HttpMethods class inherits Authentication class
Next, we define a Messages.cs class to map the required objects and properties of the request .
Messages.cs
using System;
using System.Collections.Generic;
using System.Text;
namespace QueryKnowledgeBase
{
public class Messages
{
public List<messages> messages { get; set; }
public retrievalReasoningEffort retrievalReasoningEffort { get; set; }
public List<knowledgeSourceParams> knowledgeSourceParams { get; set; }
public bool includeActivity { get; set; }
public string outputMode { get; set; }
public Int64 maxRuntimeInSeconds { get; set; }
public Int64 maxOutputSize { get; set; }
}
public class retrievalReasoningEffort
{
public string kind { get; set; }
}
public class messages
{
public string role { get; set; }
public List<content> content { get; set; }
}
public class content
{
public string type { get; set; }
public string text { get; set; }
}
public class knowledgeSourceParams
{
public string filterAddOn { get; set; }
public string knowledgeSourceName { get; set; }
public string kind { get; set; }
public bool includeReferences { get; set; }
}
}
Authentication.cs
To authenticate the user and generate the access token
using Microsoft.Identity.Client;
using Azure.Identity;
namespace Security
{
internal class Authentication
{
public static bool istokencached = false;
public static string clientId = "";
public static string AzureAISearchEndPoint = "";
public static string KnowledgeBase = "";
public static string KnowledgeSource = "";
private static string[] scopes = { "https://search.azure.com/.default"};
private static string Authority = "https://login.microsoftonline.com/organizations";
private static string RedirectURI = "http://localhost";
public async static Task<AuthenticationResult> ReturnAuthenticationResult()
{
string AccessToken;
PublicClientApplicationBuilder PublicClientAppBuilder =
PublicClientApplicationBuilder.Create(clientId)
.WithAuthority(Authority)
.WithCacheOptions(CacheOptions.EnableSharedCacheOptions)
.WithRedirectUri(RedirectURI);
IPublicClientApplication PublicClientApplication = PublicClientAppBuilder.Build();
var accounts = await PublicClientApplication.GetAccountsAsync();
AuthenticationResult result;
try
{
result = await PublicClientApplication.AcquireTokenSilent(scopes, accounts.First())
.ExecuteAsync()
.ConfigureAwait(false);
}
catch
{
result = await PublicClientApplication.AcquireTokenInteractive(scopes)
.ExecuteAsync()
.ConfigureAwait(false);
}
istokencached = true;
return result;
}
}
}
HttpMethods.cs
Handle Http requests
using System.Net.Http.Headers;
using Azure.Core;
using Microsoft.Identity.Client;
namespace Http
{
internal class HttpMethods : Security.Authentication
{
public static double currentfilelength = 0;
public static readonly HttpClient client = new HttpClient();
protected HttpClient Client => client;
public async static Task<String> SendAsync(HttpRequestMessage httprequestMessage)
{
HttpResponseMessage response = await client.SendAsync(httprequestMessage);
response.EnsureSuccessStatusCode();
try
{
return await response.Content.ReadAsStringAsync();
}
catch
{
return null;
}
}
}
}
Program.cs
Here we are serializing and authenticating the request and deserializing the response. We then parse the response json and the display output in the console window through Spectre.Console
using System.Net.Http.Headers;
using System.Text;
using Microsoft.Extensions.Configuration;
using Microsoft.Identity.Client;
using QueryKnowledgeBase;
using Spectre.Console;
using NJson = Newtonsoft.Json.Linq;
namespace AnaylzeDeltaTables
{
internal class Program
{
public static void ReadConfig()
{
var builder = new ConfigurationBuilder().AddJsonFile($"Appsettings.json", true, true);
var config = builder.Build(); Security.Authentication.clientId = config["ClientId"]; Security.Authentication.AzureAISearchEndPoint = config["AzureAISearchEndPoint"]; Security.Authentication.KnowledgeBase = config["KnowledgeBase"];
Security.Authentication.KnowledgeSource = config["KnowledgeSource"];
}
static async Task Main(string[] args)
{
AnsiConsole.MarkupLine(">> [Red] Hello !!! Please type in your query and press ENTER[/]");
AnsiConsole.MarkupLine("");
ReadConfig();
while (true)
{
Messages request = new Messages
{
messages = new List<messages>
{
new messages
{
role = "assistant",
content = new List<content>
{
new content
{
type = "text",
text = "You can answer questions about the stories in the source" +
"Sources have a JSON format with a ref_id that must not be cited in the answer. " +
"If you do not have the answer, respond with 'I do not know'."
}
}
},
new messages
{
role = "user",
content = new List<content>
{
new content
{
type = "text",
text = Console.ReadLine()
}
},
}
},
retrievalReasoningEffort =
new retrievalReasoningEffort
{
kind = "medium"
},
includeActivity = true,
outputMode = "answerSynthesis",
maxRuntimeInSeconds = 60,
maxOutputSize = 100000,
knowledgeSourceParams = new List<knowledgeSourceParams>
{
new knowledgeSourceParams
{
knowledgeSourceName = Security.Authentication.KnowledgeSource,
kind = "searchIndex",
includeReferences = false
},
}
};
string AIrequest = System.Text.Json.JsonSerializer.Serialize<Messages>(request);
HttpRequestMessage httprequestmessage = new HttpRequestMessage
{
Method = HttpMethod.Post,
RequestUri = new Uri($"{Security.Authentication.AzureAISearchEndPoint}/knowledgebases('{ Security.Authentication.KnowledgeBase }')/retrieve?api-version=2025-11-01-preview"),
Content = new StringContent(AIrequest, Encoding.UTF8, "application/json")
};
AnsiConsole.WriteLine("");
AuthenticationResult result = await Security.Authentication.ReturnAuthenticationResult();
Task<string> response = null;
httprequestmessage.Headers.Authorization = new AuthenticationHeaderValue("bearer", result.AccessToken);
var table = new Spectre.Console.Table();
string AIresponse = "";
AnsiConsole.Status()
.Start("[Yellow]Searching for the answer ...[/]", ctx =>
{
response = Http.HttpMethods.SendAsync(httprequestmessage);
var jobject = NJson.JObject.Parse(response.Result.ToString());
Newtonsoft.Json.Linq.JArray jarray_response = (NJson.JArray)jobject["response"];
table.AddColumn("Operation Type");
table.AddColumn("InputTokens");
table.AddColumn("OutputTokens");
table.AddColumn("ElapsedTime");
foreach (NJson.JObject path in jarray_response)
{
foreach (NJson.JObject content in (NJson.JArray)path["content"])
{
AIresponse = content["text"].ToString();
}
}
Newtonsoft.Json.Linq.JArray jarray_activity = (NJson.JArray)jobject["activity"];
foreach (NJson.JObject path in jarray_activity)
{
if (path["type"].ToString() == "searchIndex" || path["type"].ToString() == "agenticReasoning") { continue; }
table.AddRow(path["type"].ToString(), path["inputTokens"].ToString(), path["outputTokens"].ToString(), path["elapsedMs"].ToString());
}
ctx.Status("[Yellow]Almost done...[/]");
Thread.Sleep(2000);
});
AnsiConsole.MarkupLine($"[Green]{AIresponse}[/]");
AnsiConsole.MarkupLine("");
AnsiConsole.MarkupLine("[Blue]>> Token Costs[/]");
AnsiConsole.Write(table);
AnsiConsole.MarkupLine("");
AnsiConsole.MarkupLine(">> [Red] Hello !!! Please type in your query and press ENTER[/]");
AnsiConsole.MarkupLine("");
}
}
}
}
After executing the above code and completing the authentication successfully, the output for the query “Who is Rostov” would be as follows.
Conclusion
With Knowledge Retrieval APIs, the developer gains the flexibility to manage the knowledge base flow and customize the response and control how results are retrieved and shape the final output according to application needs. So go ahead and give it a try.
Thanks for reading !!!
