Setting Up an Azure Databricks Administration Connection Manager
The REST Connection Manager is an SSIS connection manager that can be used to establish connections and facilitate data integration with REST based APIs and services such as Azure Databricks Administration, among many more.
Integration for Azure Databricks Administration is enabled by the following two components:
- Azure Databricks Administration REST Source Component
- Azure Databricks Administration REST Destination Component
To add a new connection, right-click the Connection Manager area in your Visual Studio project and choose "New Connection..." from the context menu. You will be prompted to the "Add SSIS Connection Manager" window. Select the "REST (KingswaySoft)" item to add a new REST Connection Manager.
The REST Connection Manager contains 3 pages of configuration.
- General
- Advanced Settings
- Custom Headers
General Page
The General page allows you to specify connection properties and login credentials for Azure Databricks Administration REST service:
- Service Name
-
The REST service that will be connected to.
- Subscription ID
-
The ID for your Azure Subscription. This can be found in your Azure portal under the subscription tab.
- API Throttle Rate
-
The API Throttling Rate will limit the number of requests that can be sent per second. Setting the Throttle Rate to 0 will disable throttling
- Authentication Mode
- The Authentication Mode option allows you to specify the authentication mode when working with Azure Maps service. Available options are:
-
-
- OAuth Authorization Code
- Client Credentials
- Certificate
-
- OAuth Authorization Code
-
- Generate Token File
- This button allows you to log in to the service endpoint and authorize your app to generate a token.
- Here you can enter the Tenant ID, Client ID, Client Secret, and Redirect URL The Redirect URL would be the one that you had specified in the App settings. The ellipsis button next to the scope will provide a list to help in finding the correct scopes.
-
- Use Default Browser to Sign In: When this option is checked the Sign In and Authorize button will open your default web browser to complete the OAuth authentication. When this option is unchecked, the Sign In and Authorize button will complete the entire OAuth authentication process inside the toolkit.
- Sign In and Authorize: This button allows you to log in to the service endpoint and authorize your app to generate a token.
- Path To Token File
- The path to the token file on the file system. Please note that this field supports both file system paths as well as Azure Blob Storage Shared Access Signature (SAS) URL path. Note: The component supports Azure Blob Shared Access Signature (SAS) URL in the token file path.
- Token File Password
- The password to the token file.
- Client Credentials
-
- Tenant ID
- The ID for your Azure Tenant
- Client ID
- The Client Id option allows you to specify the unique ID which identifies the application making the request.
- Client Secret
- The Client Secret option allows you to specify the client secret belonging to your app.
- Certificate
-
- Tenant ID
- The ID for your Azure Tenant
- Client ID
- The Client Id option allows you to specify the unique ID which identifies the application making the request.
- Certificate Location
- The Certificate location can be either a Store or a File system.
- Certificate Thumbprint
- The Certificate Thumbprint option allows you to specify the thumbprint of the certificate created for the application you have created.
- Test Connection
- After all the connection information has been provided, you may click the "Test Connection" button to test if the user credentials entered can connect to the selected REST service.
Advanced Settings
- Proxy Mode
- The Proxy
Mode option allows you to specify how you want to configure the proxy server setting. There are three options available.
- No Proxy
- Auto-detect (Using system-configured proxy)
- Manual
- Proxy Server
-
Using the Proxy Server option allows you to specify the name of the proxy server for the connection.
- Port
-
The Port option allows you to specify the port number of the proxy server for the connection.
- Username (Proxy Server Authentication)
-
The Username option (under Proxy Server Authentication) allows you to specify the proxy user account.
- Password (Proxy Server Authentication)
-
The Password option (under Proxy Server Authentication) allows you to specify the proxy user's password.
Note: The Proxy Password is not included in the connection manager's ConnectionString property by default. This is by design for security reasons. However, you can include it in your ConnectionString if you want to parameterize your connection manager. The format would be ProxyPassword=myProxyPassword;(make sure you have a semicolon as the last character). It can be anywhere in the ConnectionString.
- Timeout
- The Timeout (secs) option allows you to specify the maximum number of seconds that the Connection Manager will wait while trying to establish a connection with the Salesforce service.
- Retry on Intermittent Errors
- This is an option designed to help recover from possible intermittent outages or disruption of service so the integration does not have to be stopped because of such temporary issues. Enabling this option will allow service calls to be retried upon certain types of failure.
- Ignore Certificate Error
- This option can be used to ignore those SSL certificate errors when connecting to Dynamics 365 Customer Insights servers.
- Warning: Enabling the "Ignore Certificate Errors" option is generally NOT recommended, particularly for production instances. Unless there is a strong reason to believe the connection is secure - such as the network communication is only happening in an internal infrastructure, this option should be unchecked for best security.
-
Custom Headers Page
The Custom Headers page will allow you to set the Header Name and Header Value. You may select the green icon to add another row, and the red icon to remove an existing header you have configured.