# Using Azure Data Lake Service to manage Fabric Lakehouse

In the previous [article](https://www.azureguru.net/service-principal-in-microsoft-fabric) on Service Principal in Fabric, we explored how service principal improved ways to authenticate and authorize Fabric resources.

In this article we would look how to leverage `Azure DataLake Service` to interact and manage Fabric artifacts outside of the Fabric environment to create and manage folders, upload and download files to and from `lakehouse` .Though there are many actions that possibly could be done, the basic steps in this article would provide a good starting point for exploring further possibilities.

In this article I would use `onelake.dfs` endpoints as we are dealing with Farbric storage. The `onelake.dfs` endpoint provides access to `OneLake` storage APIs allowing operations like data access and management.

### The Setup

You first need to register the application at [**https://entra.microsoft.com**](https://entra.microsoft.com/#view/Microsoft_AAD_RegisteredApps/ApplicationMenuBlade/~/Overview/quickStartType~/null/sourceType/Microsoft_AAD_IAM/appId/ad53ea30-aa6b-4232-a596-b00f31920145/objectId/1d59f9c6-36a8-43f3-a862-32e7d087ddde/isMSAApp~/false/defaultBlade/Overview/appSignInAudience/AzureADMyOrg/servicePrincipalCreated~/true)

Once logged in, navigate to **Applications >>App registrations >> New registration**

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1724690897503/35730069-9cfa-438d-b3b5-d15cd01735f7.png?auto=compress,format&format=webp align="left")

and register a new application

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1728338902210/02e52a2c-dedf-49e4-ac52-4e8c15bd1627.png align="center")

I have registered the application under the name `Azure Data Service Fabric`. We would required `Client ID` and `Tenant ID` values to reference in the code from the registered app.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1728339301451/463363d8-fa8e-4db6-9ae0-70c3a6e332ad.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1728339210038/a5db642e-a673-4709-aef1-d08db31a1eed.png align="center")

Also `Client Secrets` will be required to be referenced in the code.

### The Code

Create a new Console application and declare a bunch of variables.

We would use Azure `DataLakeServiceClient` and `DataLakeFileSystemClient` class references to manipulate `lakehouse` resources and file system.

```csharp
 private static string clientId = "Client Id of the Registered App";
 private static string tenantId = "Tenant Id of the Registered App";
 private static string clientSecret = "Client Secret of the Registered App";
 private static string workspaceName = "Your Workspace";
 private static string lakeHouse = "Your LakeHouse";
 private static ClientSecretCredential credential;
 private static string endpoint = $"https://onelake.dfs.fabric.microsoft.com";
 private static DataLakeServiceClient datalake_Service_Client;
 private static DataLakeFileSystemClient dataLake_FileSystem_Client;
```

Method to return a `Credential` object for the service principal

```csharp
static async Task ReturnCredentials(string baseUrl)
   {
       credential = new ClientSecretCredential(tenantId, clientId, clientSecret);
   }
```

Lets create a folder in a `lakehouse` through Azure Data Lake services

```csharp
  public static async Task CreateFolder(string endpoint)
  {
      DataLakeDirectoryClient dataLake_DirClient = await dataLake_FileSystem_Client.CreateDirectoryAsync($"{lakeHouse}.Lakehouse/Files/New_Folder");
      System.Console.WriteLine($"Directory: {dataLake_DirClient.Name} created");
  }
```

Call to the above method

```csharp
 await CreateFolder(endpoint);
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1728066521022/c01451fd-84f2-4fa0-a586-559eb7fa35d4.png align="left")

Now rename the created folder:

```csharp
  public static async Task RenameFolder()
  {
      DataLakeDirectoryClient dataLake_DirClient_1 = dataLake_FileSystem_Client.GetDirectoryClient($"{lakeHouse}.Lakehouse/Files/New_Folder");
      DataLakeDirectoryClient dataLake_DirClient_2 = await dataLake_DirClient_1.RenameAsync($"{lakeHouse}.Lakehouse/Files/Old_Folder");
      System.Console.WriteLine($"Directory {dataLake_DirClient_1.Name} has been renamed. New name: {dataLake_DirClient_2.Name}");
  }
```

Call to the above method:

```csharp
 await RenameFolder();
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1728066713992/379c78ce-2b61-461d-b596-af1089ef0ade.png align="left")

Now lets upload all files from a given local directory to a folder on the Lakehouse that we had created earlier

```csharp
 public static async Task UploadFiles(string uploadfrom)
 {
     DirectoryInfo d = new DirectoryInfo(uploadfrom);
     DataLakeDirectoryClient dataLake_DirClient = dataLake_FileSystem_Client.GetDirectoryClient($"{lakeHouse}.Lakehouse/Files/Old_Folder");
     foreach (FileInfo file in d.GetFiles())
     {
         DataLakeFileClient fileToUploadClient = dataLake_DirClient.CreateFile(file.Name);
         FileStream fileStream = System.IO.File.OpenRead(file.FullName);
         await fileToUploadClient.AppendAsync(fileStream, offset: 0);
         await fileToUploadClient.FlushAsync(position: fileStream.Length);
     }
     await foreach (PathItem pathItem in dataLake_DirClient.GetPathsAsync(recursive: true))
     {
         System.Console.WriteLine($"Uploaded file: {pathItem.Name}");
     }
 }
```

Call to the above method

```csharp
 await UploadFiles("Your Upload Path");
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1728067046097/09ab103d-bff0-4cbd-9901-e876ce284d28.png align="left")

Now lets download the same files from the `lakehouse` folder to a local directory

```csharp
  public static async Task DownloadFiles(string downloadto)
  {
      DataLakeDirectoryClient dataLakeDirectoryClient = dataLake_FileSystem_Client.GetDirectoryClient($"{lakeHouse}.Lakehouse/Files/Old_Folder");

      await foreach (PathItem pathItem in dataLakeDirectoryClient.GetPathsAsync(recursive: true))
      {

          DataLakeFileClient fileToDownload = dataLakeDirectoryClient.GetFileClient(Path.GetFileName(pathItem.Name));
          Response<FileDownloadInfo> downloadResponse = await fileToDownload.ReadAsync();

          StreamReader reader = new StreamReader(downloadResponse.Value.Content);

          FileStream fileStream = System.IO.File.Open(downloadto + "\\" + Path.GetFileName(pathItem.Name), FileMode.OpenOrCreate);

          using (var reader_1 = new StreamReader(fileStream))
          {
            using (var writer = new StreamWriter(fileStream))
              {
               while (!reader.EndOfStream)
                  {
                      string line = await reader.ReadLineAsync(); // Asynchronously read the line
                      await writer.WriteLineAsync(line); // Asynchronously write the line                           
                  }
                  writer.Close();
              }
              System.Console.WriteLine($"Downloaded file: {(pathItem.Name)}");
              reader_1.Close();
          }
          reader.Close();
         
      }
  }
```

Call to the above method

```csharp
await DownloadFiles("Your Download Path");
```

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1728074480820/fc6e474b-325a-456f-a412-52f318d302fe.gif align="left")

### Complete code

```csharp
using Azure;
using Azure.Identity;
using Azure.Storage.Files.DataLake;
using Azure.Storage.Files.DataLake.Models;

namespace Azure_DataLake_Fabric
{
    internal class Program
    {
        private static string clientId = "Client Id of the Registered App";
        private static string tenantId = "Tenant Id of the Registered App";
        private static string clientSecret = "Client Secret of the Registered App";
        private static string workspaceName = "Your Workspace";
        private static string lakeHouse = "Your LakeHouse";
        private static ClientSecretCredential credential;
        private static string endpoint = $"https://onelake.dfs.fabric.microsoft.com";
        private static DataLakeServiceClient datalake_Service_Client;
        private static DataLakeFileSystemClient dataLake_FileSystem_Client;


        static async Task Main(string[] args)
        {

            ReturnCredentials(endpoint);
            datalake_Service_Client = new DataLakeServiceClient(new Uri(endpoint), credential);
            dataLake_FileSystem_Client = datalake_Service_Client.GetFileSystemClient(workspaceName);
            
             /* Method Calls */
             await CreateFolder();
             await RenameFolder();
             await UploadFiles("Your Upload Path");
             await DownloadFiles("Your Download Path");
        }

        public static async Task CreateFolder()
        {
            DataLakeDirectoryClient dataLake_DirClient = await dataLake_FileSystem_Client.CreateDirectoryAsync($"{lakeHouse}.Lakehouse/Files/New_Folder");
            System.Console.WriteLine($"Directory: {dataLake_DirClient.Name} created");
        }
        public static async Task RenameFolder()

        {
            DataLakeDirectoryClient dataLake_DirClient_1 = dataLake_FileSystem_Client.GetDirectoryClient($"{lakeHouse}.Lakehouse/Files/New_Folder");
            DataLakeDirectoryClient dataLake_DirClient_2 = await dataLake_DirClient_1.RenameAsync($"{lakeHouse}.Lakehouse/Files/Old_Folder");
            System.Console.WriteLine($"Directory {dataLake_DirClient_1.Name} has been renamed. New name: {dataLake_DirClient_2.Name}");
        }
        public static async Task UploadFiles(string uploadfrom)
        {

            DirectoryInfo d = new DirectoryInfo(uploadfrom);
            DataLakeDirectoryClient dataLake_DirClient = dataLake_FileSystem_Client.GetDirectoryClient($"{lakeHouse}.Lakehouse/Files/Old_Folder");

            foreach (FileInfo file in d.GetFiles())
            {

                DataLakeFileClient fileToUploadClient = dataLake_DirClient.CreateFile(file.Name);
                FileStream fileStream = System.IO.File.OpenRead(file.FullName);
                await fileToUploadClient.AppendAsync(fileStream, offset: 0);
                await fileToUploadClient.FlushAsync(position: fileStream.Length);

            }

            await foreach (PathItem pathItem in dataLake_DirClient.GetPathsAsync(recursive: true))
            {
                System.Console.WriteLine($"Uploaded file: {pathItem.Name}");
            }
        }

        public static async Task DownloadFiles(string downloadto)
        {
            DataLakeDirectoryClient dataLakeDirectoryClient = dataLake_FileSystem_Client.GetDirectoryClient($"{lakeHouse}.Lakehouse/Files/Old_Folder");

            await foreach (PathItem pathItem in dataLakeDirectoryClient.GetPathsAsync(recursive: true))
            {

                DataLakeFileClient fileToDownload = dataLakeDirectoryClient.GetFileClient(Path.GetFileName(pathItem.Name));
                Response<FileDownloadInfo> downloadResponse = await fileToDownload.ReadAsync();
                StreamReader reader = new StreamReader(downloadResponse.Value.Content);
                FileStream fileStream = System.IO.File.Open(downloadto + "\\" + Path.GetFileName(pathItem.Name), FileMode.OpenOrCreate);

                using (var reader_1 = new StreamReader(fileStream))
                {
                    using (var writer = new StreamWriter(fileStream))
                    {

                        while (!reader.EndOfStream)
                        {
                            string line = await reader.ReadLineAsync(); // Asynchronously read the line
                            await writer.WriteLineAsync(line); // Asynchronously write the line                           
                        }
                        writer.Close();
                    }
                    System.Console.WriteLine($"Downloaded file: {(pathItem.Name)}");
                    reader_1.Close();
                }
                reader.Close();

            }
        }
        static async Task ReturnCredentials(string baseUrl)
        {
            credential = new ClientSecretCredential(tenantId, clientId, clientSecret);
        }

    }
}
```

### Conclusion

In conclusion, we can leverage `Azure DataLake Service` to manage a Fabric Lakehouse efficiently . By leveraging the capabilities of `Azure DataLake Service`, we automate storage and retrieval processes outside of the Fabric ecosystem. Whether uploading, downloading, or managing files, the seamless interaction with the Fabric Lakehouse enhances overall data management and operational efficiency.

Thanks for reading !!!