Easily set up Dynamics 365 CE \ CRM replication (incremental) to Azure SQL / SQL On-Premise using Skyvia’s Data Integration services

In the earlier post, we saw how we can use Skyvia’s query feature to execute SQL command against our Dynamics 365 CE

https://nishantrana.me/2019/10/25/execute-sql-select-insert-update-and-delete-command-on-dynamics-365-customer-engagement-data-using-skyvia-query/

In this post, we will cover how we can use Skyvia’s replication feature to create a copy of our cloud data i.e. Dynamics 365 Online data in a relational database like SQL Server and most interestingly keep it up to date automatically with basic configuration steps.

Creating replication of cloud data can be useful in different data integration scenarios and most importantly for reporting and data analysis need as with online we are limited with the use of Fetch XML.

The Skyvia’s replication service takes care of syncing Create, Update and Delete operation performed on Dynamics 365 Entity records with the destination SQL Database.

For this we need to Skyvia expects an object’s key as well as modifiedon, createdon fields to be included for replication.

For delete to work it expects, audit history to be enabled on that Entity.

In this post, we will take a simple example of replicating Contact entity records from Dynamics 365 Online to SQL Azure.

Login to Skyvia

https://app.skyvia.com/#/packages

Create the connections

Source Dynamics CRM Connection:

Destination SQL Azure Connection:

We have a database named SampleDB already created in SQL Azure.

With connections created, now let us create a new package of type replication

For the replication, we have specified source connection as the CRM connection and destination as the SQL Azure.

We have selected the option of Incremental Updates along with Create Tables and Drop Tables option which will delete if there is an existing table of the same name as the entity and will create a new one.

In the Select Objects section, we have a specified Contact entity. We can also select multiple entities for replication.

Edit task option in the selected entity let us specify the fields and filter criteria on the source.

We also have the option of scheduling that we will look at in the end of the post.

Save the package and click on Run to see it in action.

While it is running, the Run History section shows the details

It auto refreshes with all the details.

We can see the 228 records of contact records replicated in SQL Azure.

Now let us create a new contact and run the package again.

Here we have created a contact record as shown below in CRM.

On running the package, we can see that it has added a new record in the Run History

Back in SQL Azure also our count has increased to 229.

Now let us update one of the records in the source and run the package. The run history shows one of the records being updated.

Finally, let us delete a couple of records (9) from the source and run the package. (We need to make sure we have enabled Audit on Contact entity for delete to work)

Back in SQL Azure, we can see total record count decreased

Here we were running the package manually, however, in real-world scenarios we would be scheduling it.

For this, we can edit our package and navigate to the Schedule section and click on Set Schedule.

For e.g. we have scheduled here it to run every hour starting from 6 p.m. till 10 a.m.

On save, we can check the run history

Thus, we saw how easy it is to replicate the data from Dynamics 365 CE to our SQL Database using Skyvia’s Integration service.

To achieve the same using Scribe Online Replication Service

https://nishantrana.me/2019/01/09/using-scribe-online-replication-services-rs-to-replicate-dynamics-ce-data/

and KingswaySoft 

http://www.kingswaysoft.com/blog/2017/04/04/How-to-Retrieve-Incremental-Changes-from-CRM-in-your-ETL-Process

Hope it helps..

Single Instance / Organization Considerations in Dynamics 365 Customer Engagement / Dynamics CRM

Let us start from the scenario that the enterprise has decided to use Microsoft Dynamics CRM /365 CE as a preferred customer relationship solution for their organization.

For a large enterprise and/or having multiple different departments, the first question that comes up is, do they need to have a single instance or organization or multiple instance or organization. Well, that decision is not that easy but one thing is for sure the decision should be made as early as possible in the project.

Well you could refer to these posts that could help us to make the decision

https://medium.com/capgemini-dynamics-365-team/does-enterprise-scale-dynamics-require-multiple-instances-f1931a443930

https://nishantrana.me/2019/07/23/single-organization-vs-multiple-organization-single-instance-vs-multiple-instance-key-points-to-consider-in-dynamics-365-ce-on-premise/

Now suppose we have made the decision to go with the single instance \ organization, so what are some of the key points we need to consider.

There could be multiple ways of implementing and defining the segregation layer as well as what needs to be shared from data and customization perspective among the different business units, the most important being the security model and the business unit structure being one of the key components of it.

Check out the below Webinar –

http://www.xrmcoaches.com/2017/07/free-webinar-friday-creating-your-dynamics-365-security-model/

also listen to the below podcast to get some guidance around the setting up the security model

http://crmmvppodcast.com/episode-19-setting-up-your-security-architecture-like-a-crm-mvp

In one of our projects, the way we segregated different departments, after defining the appropriate business unit structure, was by having the Business Unit lookup (custom field) in all the common entities that were being shared by the departments and populating that field accordingly, every time record is created either through User Interface or through code and to refer the same field in the customizations (plugin, workflow etc.) as and when applicable.

In the other project, we were referring to the Business Unit lookup field in the System User / Team Entity.

Well there could be multiple ways of doing the same with its own pros and cons.

But what if you have already implemented for one of the units and didn’t consider that the same instance would need to be extended to accommodate the different business unit(s).

Well, then it is time to modify a couple of things to make the Business Unit structure scalable (if it is not).

Let us go through the each of the main components one by one and how they can be segregated among different business units.

Wrong root business unit name?

Set Parent Business Unit lookup field as optional, open the Business Unit record and rename it.

Need to rename the organization?

For Online à

Navigate to Admin Portal, select the instance and click on edit and update the organization name and if required the URL as well as shown below.


For On-Premise à

  • Follow the below steps to update the Display Name

https://blogs.msdn.microsoft.com/darrenliu/2013/09/21/change-crm-organization-display-name/

  • Updating Organization Database Name

https://blogs.technet.microsoft.com/crmbipl/2008/01/14/how-to-change-crm-4-0-organizations-database-name/

Forms

Create separate new Main forms for different business units.

Enable appropriate security roles for the Form.

Views

Fields

Reuse the fields that are common instead of ending up creating a large number of fields.

Field security profiles can also be used if required when certain entity’s records are shared across the business unit and we want to hide the sensitive information.

Business Rules

Business rule can be configured to run on the scope of Form instead of the Entity.

Dashboards

For Dashboards, we can enable the security roles.

Workflows

  • Use scope fields to make sure the workflows fire appropriately.
  • Or put the first step to check for which business unit it is firing (depends on the segregation logic implemented)

Plugins

Have a field in the records to identify to which Business Unit it belongs to and check for this field as the first thing in the plugin or check for the Business Unit of the user under whose context the plugin is getting fired or use the secure \ unsecure configuration etc. it purely depends upon the way the segregation logic has been implemented.

SLAs

Use SLA lookup field to apply the SLA dynamically through plugin or workflow.

https://nishantrana.me/2019/07/24/apply-sla-choosing-the-sla-in-the-sla-lookup-field-for-the-sla-enabled-entities-in-dynamics-365-ce/

Business Process Flow

If there are multiple business process flow, make sure appropriate one is applied

Apply business process flow while creating an entity record.

https://www.powerobjects.com/blog/2019/04/24/business-process-flows-switch-bpf-dynamics-365-javascript/

There would be multiple other key points to be considered apart from the one listed above specific to each implementation; the above points are some of the things that can be taken as guidelines.

Lastly we must consider  creating different apps for different department \ teams\ users and add required solution components to it.

Design custom business apps by using the app designer

Please feel free to add comments and share your thoughts.

Hope it helps..

The workflow cannot be published or unpublished by someone who is not its owner error while importing solution in Dynamics 365 CE

Recently after upgrade of our Dynamics 365 CE from 8.2 to 9.1, when the System Administrator tried importing the solution (managed), he got the below error

The workflow cannot be published or unpublished by someone who is not its owner.

Error while importing workflow {ec05f40d-fd14-46ab-96b3-be1f5b108ef3} type Workflow name Marketing List – Update CT Relationship: The workflow cannot be published or unpublished by someone who is not its owner.

What the error says is that the user who is importing the solution needs to be the owner of the workflow that is there in that solution.

http://gonzaloruizcrm.blogspot.com/2011/05/processesworkflow-ownership-faqs.html

https://stackoverflow.com/questions/17159920/crm-2011-ru13-the-workflow-cannot-be-published-or-unpublished-by-someone-who-is

The thing was that we have never got this issue earlier for that System Admin user and we would have imported the solution at least 10 times earlier as well. This time the only thing different was that it was in the newer version 9.1

Changing the owner of the workflow and assigning it to the one who is importing will fix this issue. However interestingly and as expected when we tried importing the same solution using a different system administrator (again not the owner of the workflow in the target system), the import was successful with warning.

“The original workflow definition has been deactivated and replaced”

Read the below post to understand how workflows are managed internally

https://www.inogic.com/blog/2010/11/workflows-behind-the-scene/

Please share your thoughts, if someone has faced this issue recently and how they managed to fix this?

And what caused the issue to the first system admin and worked for the other.

Requesting the backup of Microsoft Dynamics 365 Online Database

In theory, requesting the backup of the online database seems very easy and straightforward. We basically need to raise a support ticket with Microsoft and provide the following details.

  • SQL Version – Standard or Enterprise
  • SQL Year of the version. In our case it was (Microsoft SQL Server 2016 (SP1) (KB3182545) – 13.0.4001.0 (X64) Oct 28 2016 18:17:30 Copyright (c) Microsoft Corporation Standard Edition (64-bit) on Windows Server 2016 Datacenter 6.3 <X64> (Build 14393)
  • Global Admin E-mail Address – to who the backup will be shared.

We had requested the same, some 3 weeks back. In our case, the Database size of our production was around 600 GB out of which 200 GB were occupied by Audit Data. For some business-specific reason, we wanted to have the backup of Audit Data with us before we delete it. Now, this Audit has a separate story with it, when we tried deleting it from UI we got the SQL Time Out issue and on raising the support ticket when Support Team tried the same through back end it, resulted in our Production Server being down briefly for few minutes. We also tried to take the back up through 3rd party tools using SSIS, however again we ran into some issue where the tool was not able to pull the Audit Data for few of the entities.

Well to cut the story short, it is around 21 days after we raised the request, and we are yet to receive the backup of our production database, we have been told that support team is actively working on it and it is because of the large size of the organization.

Will keep posting the update on the backup and the Audit deletion.

The purpose of this blog post was to share that certain things might seem straightforward and would have taken maximum few hours, could still run into issues and challenges causing delays. So, when we plan for it we need to aware of these factors as well.

Hope it helps..

HideCustomAction and Display \ Enable rule in Dynamics 365

Imagine a scenario where we have both the disable rule and HideCustomAction implemented for a ribbon button. Let us see with an example what will happen in this scenario.

First, let us implement the display rule to hide the Delete button from Contact form when it is in a disabled state.

For active record à

We can see the delete button in the command bar.

For disabled record à The delete button is not visible.

Now let us hide the button using HideCustomAction

Interestingly the Delete button is not available in case of the active contact record this time.

The reason for this is because the HideCustomAction button removes the specified node from the ribbon so that it is not rendered instead of hiding it. Therefore any other enable or display rules are not applied to that button.

https://docs.microsoft.com/en-us/dynamics365/customer-engagement/developer/customize-dev/define-custom-actions-modify-ribbon#hide-custom-actions

Hope it helps..

Plugin on Pre-Validation Stage in Dynamics 365 CE

Recently we had a requirement to delete the Account record without deleting the associated Contact records.

If we try deleting the account record we’d get the following message box

The relationship definition can’t be updated as well to achieve this

So, we wrote a plugin on the pre-validation stage of pre-delete event of Account, which will retrieve and loop through all the child contact records and set its parent customer field as null.

In case of pre-operation the child records were not available.


public void Execute(IServiceProvider serviceProvider)
{
ITracingService tracingService = (ITracingService)serviceProvider.GetService(typeof(ITracingService));
IPluginExecutionContext pluginContext = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
IOrganizationServiceFactory serviceFactory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
IOrganizationService organizationService = serviceFactory.CreateOrganizationService(pluginContext.UserId);
EntityReference targetEntity = (EntityReference)pluginContext.InputParameters["Target"];
QueryExpression getContacts = new QueryExpression("contact");
getContacts.Criteria.AddCondition(new ConditionExpression("parentcustomerid", ConditionOperator.Equal, targetEntity.Id));EntityCollection allContacts = organizationService.RetrieveMultiple(getContacts);foreach (Entity contact in allContacts.Entities)
{
Entity contactToBeUpdated = new Entity("contact");
contactToBeUpdated.Id = contact.Id;
contactToBeUpdated.Attributes["parentcustomerid"] = null;
organizationService.Update(contactToBeUpdated);
}}

 

Another practical scenario

https://www.inogic.com/blog/2017/03/plugin-pre-validation-operation-to-show-an-error-message-as-well-as-log-the-error/

Hope it helps..

User Security Manager – My first XrmToolBox Plugin

Being one of the biggest fan and the most frequent daily user of XrmToolBox, always had this wish to write a plugin for it.

Thanks to Prashant Maurya  (a dear friend and ex-Microsoft Employee) for making this wish come true. (Who played the major role in developing it).

The tool is called – User Security Manager.

The Plugin make it easy for the administrators to manage all the security related aspects of the system users. The tool also gives 360-degree information of the user in correspondence to business unit, security roles and teams.

In our current project we had around 180 users using the system divided into more than 50 business units, which involved frequent update of their business unit, changes in the security roles assigned, update in team assignment and also during testing we had to assign the same BU and roles to the test users in the org, which wasn’t that easy to do from the user interface.

So, the idea was to come up with a tool that will let an administrator to

  • View all the security roles that assigned to the user either directly or through the team that the user is member of.
  • View all the Teams the selected user is part of.
  • With just a single click, remove all the security roles assigned to the user.
  • With just a single click, remove user from all the teams that he is part of.
  • Easily assign and unassign single or multiple security role to the user.
  • Easily add or remove user from a single or multiple team.
  • Change the Business Unit of the user.

User Guide: –

Load all the users

After connecting to the organization, we can click on Load Users to load all the system users of the organization.

List details of the user selected

  • For the selected user, in User Detail Tab’s General section, we can get the general information of the user like First Name, Last Name, Title, Primary Email and Access Mode.
  • Next to the General section, we can see all the security roles assigned to the user either directly or through the teams the user is member of.
  • The Roles tab shows all the Roles assigned to the user and also the roles that are available in the business unit to which user belongs in Assigned and Available section along with their counts.
  • The Teams tab shows all the teams user is member of and also the teams that are available in the organization in Assigned and Available section along with their counts.

Remove all security roles of the user.

To remove all the security roles of the user, click on Remove All Roles button.

Remove user from all teams.

To remove the user from all the teams, click on Remove All Teams button.

Assign Security role to the user.

To assign security role to the user, check the security roles from the available security roles section and click on “<<” button.

To remove security role from the user.

To remove security roles from the user, select the roles from the Assigned section and click on “>>” button.

Assign User to Team(s).

To assign user to team(s), select the teams from the available section and click on “<<” button.

Remove user from Team(s)

To remove user from Team(s), select the team from the Assigned Section and click on “>>” button.

Change the business unit of the user.

To change the business unit of the user, select the Business Unit from the Business Unit drop down in General Section of User Details.

Please install it, use it and share your feedback which would help us to improve it

https://www.xrmtoolbox.com/releases/

Hope it helps..


	

PLAN PROPERLY before using Back and restore and Copy instances in Dynamics 365

Update – 25 June 2018 – Took nearly 20 days to get both the instances up.

Update – 12 June 2018Support Engineer has been not been able to resolve either of the support ticket which were raised last Thursday 7th June. One instance is not opening and the other instance is not restored with the backup we had requested. They are unable to find the backup till now.

Recently we tried copying our DEV to one of the TEST instances. The idea was we were upgrading the click dimensions solution in our DEV so we wanted to make sure everything works fine after the upgrade and in case it didn’t we could then again copy it back from the TEST.

We had faced back and restore issues (instance failed) in the past so decided to go the Copy instance way this time.

However, we got the Instance Failed error on the TEST instance to which we were copying DEV to.

Interestingly the Edit, Reset, Delete etc option were still available and we could open the TEST instance (the original, as the copy of DEV had failed) without any issues.

Then we thought of giving a try again and started copy from DEV to TEST again. This time again it failed. And with no option of Edit, Reset, delete etc. and the Instance was also not opening.

The error à

Along with that we also tried Restoring the Backup in one of the other TEST instances, that also expectedly failed.

So basically, we ended up with 2 unusable instances.

A few weeks back also we faced the same with our Dev environment and the only way out was to raise the support ticket with Microsoft. That time it took 5 days for DEV to be restored. Imagine the development activity that came to halt.

Last year also we had faced this issue in Production and had to work with the support team for 16 – 20 hours continuously to get it restored.

So, if we have some critical activity planned like Production Deployment and we are counting on Back and Restore or Copy an instance to rescue us in case anything goes wrong after production deployment, we need to plan for this Instance Failure as well while restoring or copying.

Basically, be ready to raise a support ticket with the highest severity to restore things back, which could take hours or in worst case days and do all the production deployment over the weekend or have enough downtime allocated.

Hope it helps..

Insufficient Permissions error while updating records using Excel Online in Dynamics 365

Recently one of our users reported the below while trying to bulk update the records using Excel Online feature.

The user had the Update rights for those records.

Basically, the Excel Online update features creates a data import job in the background.

https://debajmecrm.com/2015/11/14/dynamics-crm-excel-online-field-changes-save/

So the user had to be given the import data rights.

We resolved the issue by giving the following rights to the user at organization level.

  • Data Import
  • Data Map
  • Import Source File
  • Web Wizard
  • Web Wizard Access Privilege
  • Wizard Page

https://community.dynamics.com/crm/b/crmpowerobjects/archive/2015/08/18/data-import-privileges

Hope it helps..

The unit schedule id is missing error while trying to migrating product in Dynamics 365

Recently while trying import products from our Dev to Test environment using the Configuration Migration tool we got the below error

The reason being we were using default unit as a unit group in out DEV and the same wasn’t there in the TEST environment. Interestingly if we set the correct GUID in the data.xml generated by Configuration Migration tool and try importing, still the products do not get imported.

So basically, if we try importing the products entity alone we might run into issues. To migrate it properly we need to add the related entities along with it as mentioned in the Microsoft documentation.

https://docs.microsoft.com/en-us/dynamics365/customer-engagement/admin/manage-product-catalog-configuration

We created a new schema file with Product and its related entities, and this time migration worked properly.

Hope this helps.