Azure Databricks Integration

To integrate with Azure Databricks in Compyl, you’ll need to configure API access using Azure Databricks account/workspace details and/or Microsoft Entra ID service principal. These credentials allow secure access to the Azure Databricks REST APIs and are required to configure the Azure Databricks integration within Compyl.

Generate Your API Credentials

1. Log into the Azure portal.

2. Create a Microsoft Entra application (service principal) by going to Microsoft Entra ID.

   

3. Next, click + Add and choose App registration.

   

   If you're using the newer Entra ID Admin Center, the path is slightly different. Expand below to see the screenshot.

   New Entra ID Admin Center

   

4. Name the app and click Register.

   

5. From the Overview page, copy the Application (client) ID and Directory (tenant) ID.

   

6. Expand Manage in the sidebar and select Certificates & secrets.

   

7. Click on New client secret.

   

8. Enter in a description, an expiration, and then click Add.

   

9. Copy the secret Value.

   

Add the service principal to Azure Databricks (account & workspace)

1. Open the Azure Databricks Accounts console.

2. Go to User management > Service principals > Add service principal.

3. Paste the Application (client) ID from Step 5 above and save.

4. Assign appropriate account-level roles for your use case.

5. In your Databricks workspace, click on your user icon in the upper right and select Settings.

   

6. Then click Identity and access and choose Manage under Service principals.

   

7. Select Microsoft Entra ID managed and then paste in your Application (client) ID from above and name the principal.

   

8. Click on your newly created Service Principal and select Permissions. Then click on Grant access to grant the required workspace permissions.

   

Find your Azure Databricks Account ID

1. In the Accounts console, open Settings or Account details and copy the Account ID.

2. To find your Workspace Instance URL, launch your Databricks workspace and copy the base URL from the browser address bar (for example, https://adb-1234567890123456.7.azuredatabricks.net. If your environment shows the dbc-*.cloud.databricks.com format, use that.

   

Add Configuration

1. Navigate to the Integrations tab using the left-hand menu, find Azure Databricks, and select +Add Configuration.

   

2. Next, fill in the details below and select Continue.

   

   Name: Enter the name of the integration.

   is Enabled: Toggle button to ensure data is populated into Compyl.

   Account ID: Your Azure Databricks Account ID used for account-level requests.

   Workspace Instance: Your Azure Databricks workspace instance URL for workspace-level requests. Format: dbc-a1b2345c-d6e7.cloud.databricks.com

   Client ID: Paste the Application (client) ID from Step 5.

   Client Secret: Paste the Client Secret from Step 9.

   Microsoft Entra Tenant ID: Provide your Microsoft Entra Tenant ID from Step 5.
   
   

   🚧

   Note

   This is only if you want to use your Microsoft Entra ID for OAuth.

3. Choose your Endpoints and Begin Validation to validate your connection.

   

4. Select any Blueprints you wish to associate with this integration.

   

5. After you hit Continue, you'll see a summary screen. Click Return to Integrations to complete the configuration.

   

Once you have completed filling out the form click Return to Integrations.

You'll know the integration is successfully added when the status updates to Complete.

Data is synced on a recurring schedule configured by your Compyl administrator. Each sync pulls the latest data from all enabled endpoints.

To query the data from the integration, refer to the Blueprints article.