If you haven't been made aware yet, Microsoft has recently announced their new xMultiple framework, which provides a new way to interact with Dataverse, Common Data Service (CDS), and Dynamics 365 online platforms using some newly introduced SDK messages. These messages facilitate more optimized data writing that can help achieve some greater write performance when used. In this blog post, we will cover some technical characteristics of the new API, with the hope that you can use it to make justified decisions about whether and how to utilize the features in our Dynamics 365 toolkit. For your reference, xMultiple API support was added in our v23.2 release of the Dynamics 365 toolkit.
In a nutshell, the newly introduced xMultiple framework consists of support of the following write operations:
- Create (via CreateMultiple message)
- Update (via UpdateMultiple message)
- Upsert (via UpsertMultiple message)
- Delete (via DeleteMultiple message)
How does the new API work and how does it differentiate from the traditional ExecuteMultiple message?
Before we get much deeper, let's have a quick look at the API evolution over time in terms of batch processing.
- When the Dynamics CRM API was first made available (as early as Microsoft Dynamics CRM 3.0), it only supported row-by-row writing, in which case each SDK service call can contain up to one record each time. This of course limits the throughput of SDK service calls due to the overhead associated with each request. It is worth noting that such row-by-row writing is still available in the most current platform.
- ExecuteMultiple SDK message was first introduced at the time when Microsoft Dynamics CRM 2011 Update Rollup 12 was made available in January 2013, which marks a new API milestone since it is the first time the Dynamics SDK provides support for batch write operations. This enhancement offers some substantial performance improvement when compared to row-by-row writing. I wrote an article describing some of the details at the time for MS Dynamics World. In one of the preliminary tests that I have done, I was able to see a performance improvement of 11 times.
- Starting from Microsoft 2023 Release Wave 1, the Dynamics 365/Dataverse team started introducing the new xMultiple framework, which utilizes those Homogeneous batch operation messages (such as CreateMultiple, UpdateMultiple, UpsertMultiple, and DeleteMultiple) to facilitate further improved batch writing and processing.
Now, with such background in mind, the key is to understand how the new xMultiple framework works differently from the ExecuteMultiple message that has been around for over 10 years. The following is my understanding.
- xMultiple leverages a new batch framework which is designed to be more efficient for writing operations through the use of those homogeneous batch operation messages, including CreateMultiple, UpdateMultiple, UpsertMultiple, and DeleteMultiple. With the new API design, each batch request will only contain write operations of the same SDK message (Create, Update, Upsert, or Delete), so that the platform can apply some batch processing strategies that would be otherwise not possible when using the ExecuteMultiple message.
- xMultiple messages should have a relatively smaller payload than their equivalent version of the ExecuteMultiple messages which can make it more efficient as well.
How do you make use of it if you are a hardcore developer?
If you are a hardcore developer, you should head to the Microsoft documentation page to find code examples there. The documentation page is fairly thorough, it has covered all scenarios including how you would check whether each xMultiple message is actually supported by the platform. You can also find the sample code that shows you how to make the service calls.
How can you achieve it if you are running an ETL process?
If you happen to be an ETL developer, we are happy to share that we have added a new option in our most recent v23.2.2 release, called "Homogeneous Batch Operation Messages", which can be used to enable the xMultiple messages to speed up your writing.
We have done some preliminary testing, we could see a performance improvement of nearly 100% when working with record creation. The following are some numbers we have collected in our tests. In our tests, we mainly work with the account entity. If you happen to be working with an elastic table entity, you should see some much greater improvement.
It should be noted that our test was very preliminary, we have not put much effort in its optimization. The test was done in a temporary trial instance, which resides in a resource-restricted tenant. You should be able to see some better improvement if you are working with a higher environment. Again, this test was run against a standard entity (account), not an elastic table entity.
How do you decide whether to use the feature or not?
While the feature provides some incredible performance improvement, it also comes with some constraints that should be taken into consideration when deciding whether and how to use it.
- The most important issue with the new batch operation is, that the new batch message has a different error handling behavior. To be more specific, there is no partial batch process available as we do in the traditional ExecuteMultiple message. In other words, when you submit a batch (CreateMultiple, UpdateMultiple, or DeleteMultiple), if there is one record failing in the batch due to any reason (such as data type or data length/size overflow errors), then the entire batch will fail. There isn't a ContinueOnError option like we do in the ExecuteMultiple message. If you have the trust that your input data is always valid for writing to the target entity, then you are encouraged to leverage the new feature.
- The new message does not actually provide any throttling improvement. You will still be throttled by the same rates, in other words, there is a ceiling of how fast your write can be. The ceiling should be very much the same regardless of whether you are using the new batch message or the traditional ExecuteMultiple message. However, if you couple the new option with the user multiplexing feature available in our software, you could technically raise the ceiling to a certain extent with a relatively minimal effort.
Having said that, we generally believe it is beneficial to use the new batch messages due to the vast performance improvement they provide. The partial exception situation mentioned in the above first bullet point would not be an issue if your source data is clean and in good shape. In case you don't have full control of your source data, you can always apply some transformation logic in your data flow to correct your data before writing, which would make the concern less relevant.
We hope this is helpful.