Working with Dynamics 365 online, your integration performance is subject to what is allowed by the server infrastructure managed by Microsoft. In particular, Microsoft has some throttling strategy in place which is to prevent exhaustive resource usage on the server side from each individual tenant. this is for the good reason of making sure that the servers operate in a healthy environment so that it does not degrade the overall performance of all other tenants in the same server group while one tenant is performing a high-paced data load. However, this can create challenges when your data load has to be completed in a particular time window, in which case you might have exhausted all of your options to speed up the data load performance.
The following is the typical error message you would receive when your service calls are throttled. Note that this error does not typically show up as an error when using our software since we gracefully handle such throttling errors by making retries. When such errors happens in our data load process, it is typically surfaced in the SSIS log as a warning message.
CRM service call returned an error: Combined execution time of incoming requests exceeded limit of 1200000 milliseconds over time window of 300 seconds. Decrease number of concurrent requests or reduce the duration of requests and try again later. (Error Code: -2147015903, Detail Message: Combined execution time of incoming requests exceeded limit of 1200000 milliseconds over time window of 300 seconds. Decrease number of concurrent requests or reduce the duration of requests and try again later.)
In respecting the service availability governance model created by Microsoft while recognizing the needs that we all want to load data as quickly as possible in our large-scale data load processes, we have shipped a new User Multiplexing feature in our CDS/CRM connection manager in our 2022 release wave 1, which offers the option to specify the connection manager to use multiple application users instead of one to achieve a better data load performance.
Setting up the User Multiplexing configuration
To access the feature, you need to make sure that you are using one of the following two OAuth options in the connection manager.
- Client Credentials
- Certificate
Once you have selected the right OAuth option, the User Multiplexing feature becomes available as a button, as shown in the following screenshot.
Click the User Multiplexing button to launch the User Multiplexing configuration window.
From this window, you can add new application users using the + button. The idea is that Dynamics 365 online service throttling is enforced per connection user, so this way you can add multiple connection users in order to achieve a higher data load performance.
Once you have added all necessary application users, you can complete the configuration by clicking the OK button. In doing so, you will return to the main UI of the connection manager. From there, you can click the Test Connection button to make sure that all entered application user credentials are correct.
Leveraging the User Multiplexing feature
Once you have done the user multiplexing configuration as we have shown above, you can now put the feature in action and take full advantage. Although the feature is implemented for all service calls, you would typically get most from the CRM/CDS destination components, since they are generally the bottleneck of your data flow tasks.
To do so, you would head to your CRM/CDS destination component to increase the thread number specified there. In theory, if you have two application users configured in the connection manager, you can very much double the thread number, in which case, you should technically see a double performance improvement. If you have provided 3 application users, then you should be able to triple the thread number in the destination component without encountering any server throttling errors (provided that your previous configuration did not give you any throttling errors) while entertaining the 3-fold performance improvement. The list keeps going on.
However, you should keep in mind, there is an organization-level throttling which might kick in if you have specified too many connection users. We would say that using 3 or 4 application users should be relatively safe. In summary, you want to push the limit using a moderate configuration instead of being too aggressive, in which case, it might impact other tenants in the same server group, or otherwise even the user experience of your own tenant. With that being said, with great power comes great responsibility, and you want to be mindful of the impact when you leverage such an advanced feature. At the end of the day, we want to be respectful of the resource governance needs on Microsoft platform in order for them to provide us with an overall healthy operational environment.
We hope you find this feature useful.