Data Migration in Dynamics 365 – Key considerations

I was recently watching the FastTrack TechTalks on Data Migration.

https://community.dynamics.com/365/b/techtalks/posts/data-migration-january-16-2018

Below are some of the key points from the session à

Customer 360 View is more often than not, the starting point of the data migration discussions. Storing all the data in Dynamics 365 might not be a good idea as it could impact the performance and more importantly, the storage has a cost associated with it.

Learn more about the storage in Power Platform here

https://docs.microsoft.com/en-us/power-platform/admin/whats-new-storage

Important Considerations as part of the Data Migration Strategy

Data Source

What is the source of the migration? Is it a single system or multiple systems from which the data needs to be consolidated.
The appropriate connectors or technology needed to fetch or extract the data from the source system(s). (SQL, File-based, Web Service, etc.)
Need for a staging database for transformation and cleansing of data before migrating it to Dynamics 365.
Having a well-defined Dynamics 365 data dictionary and model.

Data Cleansing

Removal of the old data.
De-duplicate of the data.
Checking for null, missing required fields and references.
Identify and define the uniqueness of the record.
Need for defining Alternate keys

Data Transformation (mapping with Dynamics 365 entities and fields)

Transform the data to adapt to the target Dynamics 365 Data model.
Format and Range of the fields.
Calculate or combination of different fields.
Filtering and Enriching of the fields

Mapping OptionSet in KingswaySoft SSIS Integration Toolkit for Dynamics 365

Mapping OptionSet in Scribe Online

Lookup and Cache Transformation component in KingswaySoft SSIS Integration Toolkitfor Dynamics 365

Data Loading

Is it one-time migration (plus delta load daily) or n times migration
Based on the time it takes, the batch can run over the weekend or after office hours daily.
Order of data load – Master \ Configuration Data à Parent records àChild records.
Use Create or Update whenever possible instead of Upsert.
For Inactive records, first, we need to create those records as active, associate any child records to it if any, before updating the status of the record.
Use multithreading and execute multiple requests.
Figure out the optimum batch size and thread.

Optimum batch size

For Upsert in KingswaySoft SSIS Integration Toolkit for Dynamics 365

For Delete in KingswaySoft SSIS Integration Toolkit for Dynamics 365

Volume

How many records are to be migrated and how much time will it take.

Storage / Scalability

Do we need all the data to be stored in Dynamics 365, or for Customer 365, can we leverage Power BI reports which could pull data from different data sources?

Do we need all the data for all the entities or last x months?

Compliance and regulations

We need to consider what information is allowed to be stored/accessed by the users in Dynamics 365.

Other key points

Use WebAPI for migration.
Use Execute Multiple for batch operations.
Leverage Application User that provides Server to Server authentication.
To update CreatedOn field use OverriddenCreatedOn field.
Use Create OnBehalfOf / CallerID for Created By field.
Map ModifiedBy and ModifiedOn to custom fields for the historical purpose.
Check for the sequence of the auto-number fields in the entities.
If possible for improved performance – during migration disable duplicate detection, workflows, plugins, audit, activity feeds, etc.
Leverage the same location/region for running the package as the Dynamics 365.
For faster date-time conversions set regional settings to UTC for the service account being used.