Customize ClientSecretCredential class for OneLake authentication in Microsoft Fabric

In this blog post I am going to share why and how to customize the inbuilt ClientSecretCredential class in C# for obtaining OAuth2 access tokens for OneLake authentication in Microsoft Fabric.

In Microsoft Fabric, to access and manage OneLake storage you can either use OneLake APIs or DataLakeServiceClient class.

When I say OneLake API, I mean the ADLS2 APIs. OneLake supports the same APIs as Azure Data Lake Storage (ADLS) Gen2 APIs. It means that, you can use the ADLS2 APIs to authenticate and manage the One Lake storage in Microsoft Fabric.I will pen down a separate article on that topic.

If you would like to know more on how to utilize DataLakeServiceClient for authenticating and managing the OneLake storage in Fabric, feel free to check out the article I wrote a couple of months ago which you can find it here.

The approach used in that article was to get tenantId, clientId, clientSecret of the service principal from the Entra Account. Then in the DataLakeServiceClient class, use the ClientSecretCredential object from the Azure.Identity namespace.

Note : ClientSecretCredential class requires tenantId, clientId, clientSecret as arguments to its constructor.

Disclaimer : With this process I am basically converting Authorization Code Flow to Implicit Flow.

Implicit Flow is NOT recommended for server-side applications. Only use this process if you are unable to securely store the client secret and have no other option but to hard code the client secret in the code.

What is the problem with ClientSecretCredential ?

DataLakeServiceClientclass requires TokenCredential as a constructor parameter, along with ServiceUri.

public DataLakeServiceClient (Uri serviceUri, Azure.Core.TokenCredential credential);

Problem :

When you use DataLakeServiceClientclass to fetch and manage the underlying One Lake storage in Microsoft Fabric, you require clientSecret defined and its value set/hardcoded in the code along with clientId and tenantId and pass them as arguments to the constructor of class ClientSecretCredential.

Example :

private static ClientSecretCredential credential;
private static string clientId = "Client Id of the Registered App";
private static string tenantId = "Tenant Id of the Registered App";
private static string clientSecret = "Client Secret of the Registered App";

static async Task ReturnCredentials(string baseUrl)
   {
       credential = new ClientSecretCredential(tenantId, clientId, clientSecret);
   }

As seen above, the ClientSecret value is exposed in the code (which, in my opinion, is not a good idea) in order to set the value for the ClientSecretCredential object and pass it as a TokenCredential interface to the constructor of the DataLakeServiceClient class, as shown below.

  private static DataLakeServiceClient datalake_Service_Client;

  datalake_Service_Client = new DataLakeServiceClient(new Uri(endpoint), credential);

Its always advisable to avoid using clientSecret in the code as it exposes the obvious security vulnerabilities. But unfortunately, with DataLakeServiceClient you require the clientSecret value to set the ClientSecretCredential object. You can check the full code implementation here from my article.

Implementing Azure Key vault to encrypt the clientSecret value is an option(with an increased overhead of maintaining the Key vault) but again the focus of that article was to highlight the usage of DataLakeServiceClient and not to expose any other shortcomings and solutions to overcome them.

Please Note : ClientSecretCredentialclass implements the TokenCredential interface.

Solution

As we all know, that bearer token is a string.Bearer tokens are incompatible with ClientSecretCredential class used for authenticating OneLake through the DataLakeServiceClient class.

To over come this, in our approach we will use the bearer token to return AccessToken through JwtSecurityToken object through a new custom class that inherits from the base ClientSecretCredential class with a constructor parameter AccessToken(which is the bearer token generated through MSAL authentication).

JWTs(RFC 7519) is an open standard while Bearer Token(RFC 6750) is used in OAuth 2.0 authentication.

Note : The Custom class can inherit from the ClientSecretCredential base class or the TokenCredential interface. Either of them is fine. In this example, my custom class AccessTokenCredential is inherited from the ClientSecretCredential class.

To get started, we use MSAL for authentication to create the bearer token. Remember, bearer token is string and cannot be passed as constructor parameter to the DataLakeServiceClientclass in form of type ClientSecretCredential.

Custom class AccessTokenCredential

  public class AccessTokenCredential : ClientSecretCredential
  {
     public AccessTokenCredential(string accessToken)
     {
         AccessToken = accessToken;
     }
     private string AccessToken;
  }

Our custom class AccessTokenCredential inherits from ClientSecretCredential class and provides a constructor that accepts an accessToken. The value passed to the constructor is then assigned to the AccessToken property/variable.

Note : You can define AccessToken as a property or as a variable. I have declared it as a variable.

Next, create a new method FetchAccessToken() that uses the JwtSecurityToken object. The method returns AccessToken from the bearer token.

   private AccessToken FetchAccessToken()
   {
       JwtSecurityToken token = new JwtSecurityToken(AccessToken);
       return new AccessToken(AccessToken, token.ValidTo);
   }

The token generated has a validity equal to the validity of the original bearer token.

Next, we need to override two methods from the original ClientSecretCredentialclass: GetTokenAsync method and the GetToken method.

Both the methods requires the same parameters but the execution of GetToken is asynchronous while GetTokenAsync is asynchronous. GetTokenAsync method being asynchronous, is implemented though ValueTask<T>class and not the usual Task<T> from System.Threading.Tasks namespace that is usually associated in asynchronous operations.

ValueTask<T> is supposedly more efficient and faster than Task<T>.

I’ll resist myself from delving more into ValueTask<T> vs Task<T> comparison to prevent this article from turning into a discussion about C# data structures instead of focusing on Fabric One Lake storage. If you are interested to know more on the subject , you can find the details here.

So, following is the method to override GetTokenAsyncmethod of the ClientSecretCredential class.

  public override ValueTask<AccessToken> GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
  {
      return new ValueTask<AccessToken>(FetchAccessToken());
  }

The above method executes asynchronously and returns the AccessToken.

Method to override the GetToken method from the ClientSecretCredential class.

    public override AccessToken GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
  {
      JwtSecurityToken token = new JwtSecurityToken(AccessToken);
      return new AccessToken(AccessToken, token.ValidTo);
  }

The above method executes synchronously.

Now you might ask as to why do we require GetToken and GetTokenAsync methods in the first place ?

This is because the DataLakeFileSystemClient class has two methods, CreateDirectory and CreateDirectoryAsync.

When you create a new directory in OneLake storage through DataLakeFileSystemClient and use the CreateDirectory method, the GetToken method (which is synchronous) is used to fetch the token and creating a new directory through CreateDirectoryAsync method uses the GetTokenAsync (which is asynchronous)method to fetch the tokens.

For example using the CreateDirectory method

  DataLakeDirectoryClient dataLake_DirClient = dataLake_FileSystem_Client.CreateDirectory("LakeHouse_name.Lakehouse/Files/New_Folder");

will use the GetToken method synchronously while the CreateDirectoryAsync method

  DataLakeDirectoryClient dataLake_DirClient = await dataLake_FileSystem_Client.CreateDirectoryAsync("LakeHouse_name.Lakehouse/Files/New_Folder");

will use the GetTokeAsync method to fetch the token asynchronously.

Here is the complete code for the custom class AccessTokenCredential that inherits from the ClientSecretCredential class

    public class AccessTokenCredential : ClientSecretCredential
    {

        public AccessTokenCredential(string accessToken)
        {
            AccessToken = accessToken;
        }

        private string AccessToken;

        public AccessToken FetchAccessToken()
        {
            JwtSecurityToken token = new JwtSecurityToken(AccessToken);
            return new AccessToken(AccessToken, token.ValidTo);
        }

        public override ValueTask<AccessToken> GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
        {
            return new ValueTask<AccessToken>(FetchAccessToken());
        }

         public override AccessToken GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
         {
             JwtSecurityToken token = new JwtSecurityToken(AccessToken);
             return new AccessToken(AccessToken, token.ValidTo);
         }    

      }

To implement our custom class and send access token to the DataLakeServiceClient class through our custom class ,we simply have to instantiate it and pass it as tokenCredential parameter to the DataLakeServiceClient class as seen below:

AccessTokenCredential tokenCredential = new AccessTokenCredential(access_token);

datalake_Service_Client = new DataLakeServiceClient(new Uri("https://onelake.dfs.fabric.microsoft.com"), tokenCredential);

In the video below, I walk through the entire process step by step.

Source Code

using Azure.Identity;
using Azure.Storage.Files.DataLake;
using Azure.Core;
using Microsoft.Identity.Client;
using System.IdentityModel.Tokens.Jwt;
using System.Net.Http.Headers;
using Microsoft.Identity.Client.Platforms.Features.DesktopOs.Kerberos;
using System.Net;

namespace DataLakeService
{
    internal class Program
    {
        private static string clientId = "Client ID ";
        private static string tenantId = "Tenant ID ";
        private static string clientSecret = "Client Secret";


        private static string workspace = "Workhouse Name";
        private static string lakeHouse = "Lakehouse Name";
        private static string access_token = "";

        private static DataLakeServiceClient datalake_Service_Client;
        private static DataLakeFileSystemClient dataLake_FileSystem_Client;

        private static string RedirectURI = "http://localhost";
        private static string[] scopes = new string[] { "https://storage.azure.com/.default" };
        private static string Authority = "https://login.microsoftonline.com/organizations";
        private static readonly HttpClient client = new HttpClient();
        public HttpClient Client => client;
        static async Task Main(string[] args)
        {
            try
            {

                Console.WriteLine("Creating Folder_WithSecretKey");
                CreateDirectoryWithSecret().GetAwaiter().GetResult();
                Console.WriteLine("Finished creating Folder_WithSecretKey");

                Console.WriteLine("Creating Folder_WithOutSecretKey");
                CreateDirectoryWithOutSecret().GetAwaiter().GetResult();
                Console.WriteLine("Finished creating Folder_WithOutSecretKey");
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
            }

        }
        public static async Task CreateDirectoryWithSecret()
        {
            ClientSecretCredential credential;
            credential = new ClientSecretCredential(tenantId, clientId, clientSecret);
            datalake_Service_Client = new DataLakeServiceClient(new Uri("https://onelake.dfs.fabric.microsoft.com"), credential);
            dataLake_FileSystem_Client = datalake_Service_Client.GetFileSystemClient(workspace);
            await dataLake_FileSystem_Client.CreateDirectoryAsync(lakeHouse + ".Lakehouse/Files/Folder_WithSecretKey");

        }

        public static async Task CreateDirectoryWithOutSecret()
        {
            string responsename = "";
            string dfsendpoint = $"https://onelake.dfs.fabric.microsoft.com/{workspace}?resource=filesystem&recursive=false";
            responsename = await GetAsync(dfsendpoint);

            AccessTokenCredential tokenCredential = new AccessTokenCredential(access_token);

            datalake_Service_Client = new DataLakeServiceClient(new Uri("https://onelake.dfs.fabric.microsoft.com"), tokenCredential);
            dataLake_FileSystem_Client = datalake_Service_Client.GetFileSystemClient(workspace);
            await dataLake_FileSystem_Client.CreateDirectoryAsync(lakeHouse + ".Lakehouse/Files/Folder_WithOutSecretKey");
        }

        public async static Task<string> GetAsync(string url)
        {

            PublicClientApplicationBuilder PublicClientAppBuilder =
                    PublicClientApplicationBuilder.Create(clientId)
                    .WithAuthority(Authority)
                    .WithCacheOptions(CacheOptions.EnableSharedCacheOptions)
                    .WithRedirectUri(RedirectURI);

            IPublicClientApplication PublicClientApplication = PublicClientAppBuilder.Build();
            var accounts = await PublicClientApplication.GetAccountsAsync();
            AuthenticationResult result;
            try
            {

                result = await PublicClientApplication.AcquireTokenSilent(scopes, accounts.First())
                                                   .ExecuteAsync()
                                                   .ConfigureAwait(false);
            }
            catch
            {
                result = await PublicClientApplication.AcquireTokenInteractive(scopes)
                                                  .ExecuteAsync()
                                                  .ConfigureAwait(false);
            }

            client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", result.AccessToken);
            access_token = result.AccessToken;
            HttpResponseMessage response = await client.GetAsync(url);
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadAsStringAsync();
        }

        public class AccessTokenCredential : ClientSecretCredential
        {

            public AccessTokenCredential(string accessToken)
            {
                AccessToken = accessToken;
            }
            private string AccessToken;

            public AccessToken FetchAccessToken()
            {
                JwtSecurityToken token = new JwtSecurityToken(AccessToken);
                return new(AccessToken, token.ValidTo);
            }

            public override ValueTask<AccessToken> GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
            {

                return new ValueTask<AccessToken>(FetchAccessToken());
            }

            public override AccessToken GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken)
            {
                JwtSecurityToken token = new JwtSecurityToken(AccessToken);
                return new AccessToken(AccessToken, token.ValidTo);
            }


        }
    }
}

Thank You !!!