I was recently watching the FastTrack TechTalks on Data Migration.
Below are some of the key points from the session à
Customer 360 View is more often than not, the starting point of the data migration discussions. Storing all the data in Dynamics 365 might not be a good idea as it could impact the performance and more importantly, the storage has a cost associated with it.
Learn more about the storage in Power Platform here
Important Considerations as part of the Data Migration Strategy
- What is the source of the migration? Is it a single system or multiple systems from which the data needs to be consolidated.
- The appropriate connectors or technology needed to fetch or extract the data from the source system(s). (SQL, File-based, Web Service, etc.)
- Need for a staging database for transformation and cleansing of data before migrating it to Dynamics 365.
- Having a well-defined Dynamics 365 data dictionary and model.
- Removal of the old data.
- De-duplicate of the data.
- Checking for null, missing required fields and references.
- Identify and define the uniqueness of the record.
- Need for defining Alternate keys
Data Transformation (mapping with Dynamics 365 entities and fields)
- Transform the data to adapt to the target Dynamics 365 Data model.
- Format and Range of the fields.
- Calculate or combination of different fields.
- Filtering and Enriching of the fields
Mapping OptionSet in KingswaySoft SSIS Integration Toolkit for Dynamics 365
Mapping OptionSet in Scribe Online
Lookup and Cache Transformation component in KingswaySoft SSIS Integration Toolkitfor Dynamics 365
- Is it one-time migration (plus delta load daily) or n times migration
- Based on the time it takes, the batch can run over the weekend or after office hours daily.
- Order of data load – Master \ Configuration Data à Parent records àChild records.
- Use Create or Update whenever possible instead of Upsert.
- For Inactive records, first, we need to create those records as active, associate any child records to it if any, before updating the status of the record.
- Use multithreading and execute multiple requests.
- Figure out the optimum batch size and thread.
Optimum batch size
For Upsert in KingswaySoft SSIS Integration Toolkit for Dynamics 365
For Delete in KingswaySoft SSIS Integration Toolkit for Dynamics 365
How many records are to be migrated and how much time will it take.
Storage / Scalability
Do we need all the data to be stored in Dynamics 365, or for Customer 365, can we leverage Power BI reports which could pull data from different data sources?
Do we need all the data for all the entities or last x months?
Compliance and regulations
We need to consider what information is allowed to be stored/accessed by the users in Dynamics 365.
Other key points
- Use WebAPI for migration.
- Use Execute Multiple for batch operations.
- Leverage Application User that provides Server to Server authentication.
- To update CreatedOn field use OverriddenCreatedOn field.
- Use Create OnBehalfOf / CallerID for Created By field.
- Map ModifiedBy and ModifiedOn to custom fields for the historical purpose.
- Check for the sequence of the auto-number fields in the entities.
- If possible for improved performance – during migration disable duplicate detection, workflows, plugins, audit, activity feeds, etc.
- Leverage the same location/region for running the package as the Dynamics 365.
- For faster date-time conversions set regional settings to UTC for the service account being used.
Check out the below articles
Hope it helps..