Announcing Migration Starter Pack for CRM Online and CRM On-Premise

Update December 17, 2020: We have released v4 of our Migration Starter Pack, find more details on the changes we made.

CRM migration can be a daunting task as there are many things that can go wrong during the migration development process. For this reason, we are very excited to announce the general availability of Migration Starter Pack for CRM Online and CRM On-Premise, a sample CRM migration package that is built on top of our SSIS Integration Toolkit software, which you can use, or customize to fit your specific migration needs.

In almost every CRM migration project, you have to have a starting point, and you have to deal with CRM entity dependency in almost every single project. Our hope is, using the sample package, you can save a significant amount of time when developing the migration tasks for those commonly used system entities that you have to deal with again and again, as we have sorted out their dependencies including handling many other details. Among them, there are a few other things that I would like to highlight which you might benefit from using the sample package. 

• We have demonstrated some design patterns that you can utilize if you need to work with your own custom entities, including how to work with the dependencies, how to retain createdby, createdon field values, etc.
• We have showed the techniques that you can utilize migrating CRM activity records without losing the display sequence in the activity wall (or social wall). More details about the approach can be found in our other blog post.
  
• We have demonstrated the approaches that can be utilized to migrate CRM product and product family records in the sample package.There are three main challenges:
  
  • There are more than a few entities involves, and the dependencies among those entities are not obvious.
  • When creating a product or product family record within a product family tree, you really only get one chance to get it in the right spot in the tree, as CRM does not offer a way to move a product around with the CRM product hierarchy.
  • Products have to be in certain states in order for them to be used by associated entities.
    
• We have demonstrated the way of migrating CRM quote records, which we have covered in our other blog post.
• Migrating recurring appointment is a tricky business, which is covered by the sample package and was further discussed in our other blog post as well.

Furthermore, we have learned some interesting lessons while creating the package.

• The ownerid field of opportunityproduct entity does not have a corresponding owneridname field, so we are trying to perform two joins in the source component to bring the owning user name or team name, which is used for the text lookup setting in the destination component. The same applies to quotedetail, salesorderdetail and activitymimeattachment entity.
  
• There are a number of entities in CRM that have sort of an entity-level circular references. When this is the case, it requires multi-pass in order to migrate the involved entities properly.
• For some entities that have a parent-child relationship, the parent has to remain open until the migration of the child entity is done. So what we have to do is to first migrate the parent entity without having the statecode/statuscode fields mapped, essentially having the parent entity records migrated in an open or active status, then migrate the child entity records, finally we use another data flow task to specifically update the parent entity's status using an Update or Upsert action. Such examples include opportunity / opportunityproduct, salesorder / salesorderdetails, etc.

The migration package can be used for different migration scenarios, such as:

• Migrating from CRM On-Premise to CRM On Online.
• Migrating from CRM Online to CRM On-Premise.
• Migrating from one CRM instance to another one of the same deployment type.
• Migrating from an old version of CRM installation to a newer one.

There might be other scenarios that you could possibly use the packages as well, we leave that to your imagination and decision.

The migration package is available at our product download page. The package is created using SSIS 2008 R2, and it should automatically upgrade if you are using a newer version of SSIS. The whole package looks like something below.

To use the package, you would first need to:

• Update the connection manager to point to your CRM instances, including changing the login credentials, discovery service URL, authentication type, etc.
• For any entities in this package that you might have new metadata, you need to click the "Refresh CRM Metadata" button in the CRM source component and CRM destination component UI.
• The package was written for CRM 2016. If you use an older version of CRM server (such as CRM 2015, 2013, or 2011), you need to click the "Refresh CRM Metadata" button for each data flow component just in case there are some CRM 2016 specific fields that may not be available in the version that you are using(You can do that for only the data flow tasks that fail). If it is the case that you are using a newer version of Dynamics 365 Customer Engagement software, you would be doing the same in order to surface any new fields that might be made available to new versions. In case you are targeting CRM 4.0 or 3.0, you would have to manually change the service endpoint in CRM connection managers using their Properties window, and re-select the CRM organization (CRM 4), then click the "Refresh CRM Metadata" button for each data flow component that uses the connection manager. All components will need to be updated using the "Refresh CRM Metadata" button if it is changed to target CRM 4.0 or 3.0 due to the metadata discrepancies in those versions.
  
  

While developing the package, we have made the following assumptions:

• We assume that the base currency is the same across the two CRM instances (we use text lookup feature for any lookup fields that refer to transactioncurrency entity)
• We assume that you want to retain the createdby values.
• To retain createdby values, we assume that all system users have been created in the target CRM system using the same name with the same name sequence format. (we use text lookup feature for any lookup fields that refer to systemuser entity. Those fields include createdby, modifiedby, ownerid, etc.)
• We assume that you want to retain the createdon values, so we have mapped createdon field from the source to the overriddencreatedon field in the destination component in all data flow tasks.

Some additional notes about the starter package:

• We have tried to organize the data flow tasks based on the underlying relationship dependencies. Adding new data flow tasks to the package should take such dependencies into consideration.
• All data flow tasks are using a single-destination component. If you are using a more recent version of our software (v9.0 or later), we would recommend you turn on the "Enable Multithreaded Writing" option in all destination components and choose a right thread number. Doing so, you will be writing to CRM simultaneously which can help you improve the write performance. It is important to choose the right combination of thread number and batch size to achieve the best possible performance. Keep in mind, CRM online throttles ExecuteMultiple requests. You can raise a ticket with Microsoft Dynamics online team to have the throttling lifted or otherwise you will have to always use single thread in the destination components.
  
• We have mapped createdon field from the source to the overriddencreatedon field in the destination component in all data flow tasks in order to retain the createdon value from the source system.
• Note that the migration of some entities involves multiple data flow tasks due to the dependencies of certain fields.
• We use Upsert action in most of our data flow tasks, so technically you can re-run the package multiple times without any harm. This can be helpful in the case that your source system is still in use while the migration is in progress in parallel. You can do multiple runs before the final push, which can minimize the time window required for the final cut over.
• Note that we have used "Ignore Unchanged Fields" option in the data flow tasks whenever possible, as this option can help improve performance in the case you have to do multiple runs of the migration package.

Please feel free to give the sample package a try and let us know if you have any further comments. We would like to receive your feedback to make it better, and we certainly hope that you find the package useful.