Category: Microsoft Dynamics CRM

Easily set up Dynamics 365 CE \ CRM replication (incremental) to Azure SQL / SQL On-Premise using Skyvia’s Data Integration services

In the earlier post, we saw how we can use Skyvia’s query feature to execute SQL command against our Dynamics 365 CE

https://nishantrana.me/2019/10/25/execute-sql-select-insert-update-and-delete-command-on-dynamics-365-customer-engagement-data-using-skyvia-query/

In this post, we will cover how we can use Skyvia’s replication feature to create a copy of our cloud data i.e. Dynamics 365 Online data in a relational database like SQL Server and most interestingly keep it up to date automatically with basic configuration steps.

Creating replication of cloud data can be useful in different data integration scenarios and most importantly for reporting and data analysis need as with online we are limited with the use of Fetch XML.

The Skyvia’s replication service takes care of syncing Create, Update and Delete operation performed on Dynamics 365 Entity records with the destination SQL Database.

For this we need to Skyvia expects an object’s key as well as modifiedon, createdon fields to be included for replication.

For delete to work it expects, audit history to be enabled on that Entity.

In this post, we will take a simple example of replicating Contact entity records from Dynamics 365 Online to SQL Azure.

Login to Skyvia

https://app.skyvia.com/#/packages

Create the connections

Source Dynamics CRM Connection:

Destination SQL Azure Connection:

We have a database named SampleDB already created in SQL Azure.

With connections created, now let us create a new package of type replication

For the replication, we have specified source connection as the CRM connection and destination as the SQL Azure.

We have selected the option of Incremental Updates along with Create Tables and Drop Tables option which will delete if there is an existing table of the same name as the entity and will create a new one.

In the Select Objects section, we have a specified Contact entity. We can also select multiple entities for replication.

Edit task option in the selected entity let us specify the fields and filter criteria on the source.

We also have the option of scheduling that we will look at in the end of the post.

Save the package and click on Run to see it in action.

While it is running, the Run History section shows the details

It auto refreshes with all the details.

We can see the 228 records of contact records replicated in SQL Azure.

Now let us create a new contact and run the package again.

Here we have created a contact record as shown below in CRM.

On running the package, we can see that it has added a new record in the Run History

Back in SQL Azure also our count has increased to 229.

Now let us update one of the records in the source and run the package. The run history shows one of the records being updated.

Finally, let us delete a couple of records (9) from the source and run the package. (We need to make sure we have enabled Audit on Contact entity for delete to work)

Back in SQL Azure, we can see total record count decreased

Here we were running the package manually, however, in real-world scenarios we would be scheduling it.

For this, we can edit our package and navigate to the Schedule section and click on Set Schedule.

For e.g. we have scheduled here it to run every hour starting from 6 p.m. till 10 a.m.

On save, we can check the run history

Thus, we saw how easy it is to replicate the data from Dynamics 365 CE to our SQL Database using Skyvia’s Integration service.

To achieve the same using Scribe Online Replication Service

https://nishantrana.me/2019/01/09/using-scribe-online-replication-services-rs-to-replicate-dynamics-ce-data/

and KingswaySoft 

http://www.kingswaysoft.com/blog/2017/04/04/How-to-Retrieve-Incremental-Changes-from-CRM-in-your-ETL-Process

Hope it helps..

Single Instance / Organization Considerations in Dynamics 365 Customer Engagement / Dynamics CRM

Let us start from the scenario that the enterprise has decided to use Microsoft Dynamics CRM /365 CE as a preferred customer relationship solution for their organization.

For a large enterprise and/or having multiple different departments, the first question that comes up is, do they need to have a single instance or organization or multiple instance or organization. Well, that decision is not that easy but one thing is for sure the decision should be made as early as possible in the project.

Well you could refer to these posts that could help us to make the decision

https://medium.com/capgemini-dynamics-365-team/does-enterprise-scale-dynamics-require-multiple-instances-f1931a443930

https://nishantrana.me/2019/07/23/single-organization-vs-multiple-organization-single-instance-vs-multiple-instance-key-points-to-consider-in-dynamics-365-ce-on-premise/

Now suppose we have made the decision to go with the single instance \ organization, so what are some of the key points we need to consider.

There could be multiple ways of implementing and defining the segregation layer as well as what needs to be shared from data and customization perspective among the different business units, the most important being the security model and the business unit structure being one of the key components of it.

Check out the below Webinar –

http://www.xrmcoaches.com/2017/07/free-webinar-friday-creating-your-dynamics-365-security-model/

also listen to the below podcast to get some guidance around the setting up the security model

http://crmmvppodcast.com/episode-19-setting-up-your-security-architecture-like-a-crm-mvp

In one of our projects, the way we segregated different departments, after defining the appropriate business unit structure, was by having the Business Unit lookup (custom field) in all the common entities that were being shared by the departments and populating that field accordingly, every time record is created either through User Interface or through code and to refer the same field in the customizations (plugin, workflow etc.) as and when applicable.

In the other project, we were referring to the Business Unit lookup field in the System User / Team Entity.

Well there could be multiple ways of doing the same with its own pros and cons.

But what if you have already implemented for one of the units and didn’t consider that the same instance would need to be extended to accommodate the different business unit(s).

Well, then it is time to modify a couple of things to make the Business Unit structure scalable (if it is not).

Let us go through the each of the main components one by one and how they can be segregated among different business units.

Wrong root business unit name?

Set Parent Business Unit lookup field as optional, open the Business Unit record and rename it.

Need to rename the organization?

For Online à

Navigate to Admin Portal, select the instance and click on edit and update the organization name and if required the URL as well as shown below.


For On-Premise à

  • Follow the below steps to update the Display Name

https://blogs.msdn.microsoft.com/darrenliu/2013/09/21/change-crm-organization-display-name/

  • Updating Organization Database Name

https://blogs.technet.microsoft.com/crmbipl/2008/01/14/how-to-change-crm-4-0-organizations-database-name/

Forms

Create separate new Main forms for different business units.

Enable appropriate security roles for the Form.

Views

Fields

Reuse the fields that are common instead of ending up creating a large number of fields.

Field security profiles can also be used if required when certain entity’s records are shared across the business unit and we want to hide the sensitive information.

Business Rules

Business rule can be configured to run on the scope of Form instead of the Entity.

Dashboards

For Dashboards, we can enable the security roles.

Workflows

  • Use scope fields to make sure the workflows fire appropriately.
  • Or put the first step to check for which business unit it is firing (depends on the segregation logic implemented)

Plugins

Have a field in the records to identify to which Business Unit it belongs to and check for this field as the first thing in the plugin or check for the Business Unit of the user under whose context the plugin is getting fired or use the secure \ unsecure configuration etc. it purely depends upon the way the segregation logic has been implemented.

SLAs

Use SLA lookup field to apply the SLA dynamically through plugin or workflow.

https://nishantrana.me/2019/07/24/apply-sla-choosing-the-sla-in-the-sla-lookup-field-for-the-sla-enabled-entities-in-dynamics-365-ce/

Business Process Flow

If there are multiple business process flow, make sure appropriate one is applied

Apply business process flow while creating an entity record.

https://www.powerobjects.com/blog/2019/04/24/business-process-flows-switch-bpf-dynamics-365-javascript/

There would be multiple other key points to be considered apart from the one listed above specific to each implementation; the above points are some of the things that can be taken as guidelines.

Lastly we must consider  creating different apps for different department \ teams\ users and add required solution components to it.

Design custom business apps by using the app designer

Please feel free to add comments and share your thoughts.

Hope it helps..

The workflow cannot be published or unpublished by someone who is not its owner error while importing solution in Dynamics 365 CE

Recently after upgrade of our Dynamics 365 CE from 8.2 to 9.1, when the System Administrator tried importing the solution (managed), he got the below error

The workflow cannot be published or unpublished by someone who is not its owner.

Error while importing workflow {ec05f40d-fd14-46ab-96b3-be1f5b108ef3} type Workflow name Marketing List – Update CT Relationship: The workflow cannot be published or unpublished by someone who is not its owner.

What the error says is that the user who is importing the solution needs to be the owner of the workflow that is there in that solution.

http://gonzaloruizcrm.blogspot.com/2011/05/processesworkflow-ownership-faqs.html

https://stackoverflow.com/questions/17159920/crm-2011-ru13-the-workflow-cannot-be-published-or-unpublished-by-someone-who-is

The thing was that we have never got this issue earlier for that System Admin user and we would have imported the solution at least 10 times earlier as well. This time the only thing different was that it was in the newer version 9.1

Changing the owner of the workflow and assigning it to the one who is importing will fix this issue. However interestingly and as expected when we tried importing the same solution using a different system administrator (again not the owner of the workflow in the target system), the import was successful with warning.

“The original workflow definition has been deactivated and replaced”

Read the below post to understand how workflows are managed internally

https://www.inogic.com/blog/2010/11/workflows-behind-the-scene/

Please share your thoughts, if someone has faced this issue recently and how they managed to fix this?

And what caused the issue to the first system admin and worked for the other.

Requesting the backup of Microsoft Dynamics 365 Online Database

In theory, requesting the backup of the online database seems very easy and straightforward. We basically need to raise a support ticket with Microsoft and provide the following details.

  • SQL Version – Standard or Enterprise
  • SQL Year of the version. In our case it was (Microsoft SQL Server 2016 (SP1) (KB3182545) – 13.0.4001.0 (X64) Oct 28 2016 18:17:30 Copyright (c) Microsoft Corporation Standard Edition (64-bit) on Windows Server 2016 Datacenter 6.3 <X64> (Build 14393)
  • Global Admin E-mail Address – to who the backup will be shared.

We had requested the same, some 3 weeks back. In our case, the Database size of our production was around 600 GB out of which 200 GB were occupied by Audit Data. For some business-specific reason, we wanted to have the backup of Audit Data with us before we delete it. Now, this Audit has a separate story with it, when we tried deleting it from UI we got the SQL Time Out issue and on raising the support ticket when Support Team tried the same through back end it, resulted in our Production Server being down briefly for few minutes. We also tried to take the back up through 3rd party tools using SSIS, however again we ran into some issue where the tool was not able to pull the Audit Data for few of the entities.

Well to cut the story short, it is around 21 days after we raised the request, and we are yet to receive the backup of our production database, we have been told that support team is actively working on it and it is because of the large size of the organization.

Will keep posting the update on the backup and the Audit deletion.

The purpose of this blog post was to share that certain things might seem straightforward and would have taken maximum few hours, could still run into issues and challenges causing delays. So, when we plan for it we need to aware of these factors as well.

Hope it helps..

Different Updating Instance status during Dynamics 365 Upgrade

The day has come when we had scheduled the upgrade of our Dynamics 365 instance Test Instance(from 8.2 to 9.0).

We can see the below message during the scheduled time à “The update will being within 24 hours. No rescheduling is available”.

The following Reschedule Update option will go missing during the scheduled time.

After waiting for few minutes, we see the Update Status as Queue (Not Started)

Followed by the status as “Database Upgrade”

In between the Status message will be Backup and Restore.

https://docs.microsoft.com/en-in/dynamics365/customer-engagement/admin/manage-updates#update-status

Then it kept interchanging the status from Updating Instance (with no status) to Updating Instance – Database Upgrade for close to 2.5 hours.

Then although the status was Updating Instance the Open link got enabled after 2.5 hours.

Which opened the following notification

And finally, after 10 minutes or so à

We were all set with version 9.1

In total it took around 3 hours approximately.

Time to enjoy the successful upgrade and explore the new features ..

 

Upgrading from Dynamics CRM 2016 On-Premise to Dynamics 365

Make sure the Solution is compatible to Dynamics 365 (8.2), for this check the Version Compatibility

https://docs.microsoft.com/en-us/previous-versions/dynamicscrm-2016/developers-guide/gg328109%28v%3dcrm.8%29#version-compatibility

Upgrade the existing CRM 2016 On-Premise Server to CRM 2016 (8.1) for its solution to be compatible with Dynamics 365 Online using cumulative updates.

https://support.microsoft.com/en-gb/help/3142345/microsoft-dynamics-365-onpremise-cumulative-updates

If upgrading from previous version i.e. prior to CRM 2016, use “Migrate by using a new instance of Microsoft SQL Server” approach

https://technet.microsoft.com/en-us/library/hh699669.aspx

https://technet.microsoft.com/en-us/library/hh699747.aspx

Once updated to compatible CRM 2016 On Premise (8.1), go to settings à customizations and export the existing solutions or create a new solution, add all the required solution components to it as unmanaged which will be imported to the new Dynamics 365 online environment.

Create a new Dynamics 365 Online Environment and synchronize it with your existing Active Directory

https://blogs.msdn.microsoft.com/crm/2013/07/18/how-to-synchronize-crm-online-with-your-active-directory/

With solution ready and imported to new Dynamics 365 Online, next step is migration of data.

Consider using a data migration tool like KingswaySoft.

Check out their Migration Starter Pack.

https://www.kingswaysoft.com/blog/2016/09/16/Announcing-Migration-Starter-Pack-for-CRM-Online-and-CRM-On-Premise

or Scribe

https://www.scribesoft.com/solutions/dynamics-365/

Use following tools to update the JavaScript to Version 9.0 of XrmToolBox.

https://www.xrmtoolbox.com/plugins/XrmToolBox.Dynamics365V9JavascriptValidator/

  • Script Finder (to update the form and fields for passing the executionContext)

https://www.xrmtoolbox.com/plugins/MsCrmTools.ScriptsFinder/

https://community.dynamics.com/crm/b/develop1/archive/2017/11/11/executioncontext-hits-the-big-time

Update JavaScript to use Web API.

https://github.com/jlattimer/CRMRESTBuilder

SQL Based report, if possible, needs to be converted to use Fetch XML. Use the following tool to speed up the process.

http://www.sql2fetchxml.com/

If the reports are too complex, plan to use Power BI reports.

https://technet.microsoft.com/en-us/library/dn708055.aspx

Plugin and Custom workflow needs to be updated to run in an isolated environment i.e. sandbox along with references to the latest SDK assemblies.

https://docs.microsoft.com/en-us/dynamics365/customer-engagement/developer/plugin-isolation-trusts-statistics#web-access

https://www.nuget.org/packages/Microsoft.CrmSdk.CoreAssemblies/

If the existing On-Premise is integrated to other applications, make sure they can still talk with new Dynamics 365 Online.

Recurring jobs if needed, can be deployed to Azure as Azure WebJobs.

https://docs.microsoft.com/en-us/azure/app-service/websites-dotnet-deploy-webjobs

Use Hybrid Connection to talk with On-Prem resources.

https://docs.microsoft.com/en-us/azure/app-service/app-service-hybrid-connections

These were some of the points I could quickly think of. Please share your thoughts and experiences in comments. I’d update this list.

Do check out :

Microsoft Dynamics CRM (on-premises) to Microsoft Dynamics
Online Migration Guide

https://download.microsoft.com/download/6/D/6/6D67BDEA-1D67-42B4-A52A-CF13CD547CB5/OPtoCRMOnlineMigration.pdf

Hope it helps..

Error validating profile registration while configuring Microsoft Dynamics 365 Data Export Service

While configuring the Data Export Service we got the below error while validating the profile

It mostly occurs if there is some issue with the connection string specified in the PowerShell Script used for generating the key vault.

Always copy the connection string from the Azure SQL Connection Strings property and double check the username and password provided there. And make sure the user specified has all the required rights.

https://technet.microsoft.com/en-us/library/mt744592.aspx#Anchor_1

Correcting the connection string resulted in successful validation of the export profile.

Hope it helps..

Power BI with Dynamics 365 CE – Dynamics 365 Content Pack

In previous posts we covered

Creating a Power BI Report using Dynamics 365 Online Service.

Publishing it and Showing it inside Dynamics 365 and scheduling its Refresh.

Apart from creating our own Power BI Report another way of quickly be up and running is to use the Content Packs provided by Third Party Service, for e.g. here we will be using Sales Analytics for Dynamics 365 content pack provided by Microsoft.

Login to the Power BI service, click on Get Data, select Services.

Select Apps and choose the Sales Analytics for Dynamics 365 from the AppSource.

If we search for Dynamics 365, we can see around 15 Apps provided by Microsoft.

Back into our Sales Analytics for Dynamics 365 App, when we click on Get in now we are presented with the option where we need to Specify the URL of our Dynamics CRM Online Service and Fiscal Year End Month Number for that organization.

In the next screen, we need to select OAuth2 as the authentication method and followed by signing in.

This installs the app which would be available in the Apps section.

Clicking on the installed Sales Analytics App opens the Dashboard

Clicking on Ask a question about your data inside Dashboard allows us to analyze the data by allowing us to choose the criteria

Here we have filtered the accounts by created on by month along with the owner

Also, we can see around 10 different reports created inside that app.

  • Sales Performance
  • Sales Leaderboard
  • Win/Loss Analysis
  • Top Won/Lost Details
  • Sales Pipeline
  • Sales Pipeline Dashboard
  • Sales Activity
  • Open Activities
  • Lead Analysis
  • Account Analysis

Similarly, we can create our own Content Packs to be shared with either specific group or with the entire organization.

Click on settings gear inside Power BI Service and select Create Content Pack

Here we are creating a content pack choosing the Dashboard, Reports and the Dataset that we created earlier in the previous posts.

Once published, the user 2 can click on Get Data and search for and can see the content pack.

Clicking on Connect adds the Dashboard, Report, and the corresponding Dataset to the My Workspace of the user. Only the user who had created the content pack can edit the artifacts that are part of Content Pack, however, the other user of the organization can save a copy of it and can work on it.

More on it

https://docs.microsoft.com/en-us/power-bi/service-organizational-content-pack-manage-update-delete

Hope it helps..

Power BI with Dynamics 365 CE – Refreshing the Dataset

In the previous post, we used Dynamics 365 Dashboard and Power BI Tile to show Power BI Dashboard and report inside Dynamics 365.

https://nishantrana.me/2018/08/20/power-bi-with-dynamics-365-ce-showing-power-bi-dashboard-tile-inside-dynamics-365-ce/

Now as a next step, we want to make sure that the users are looking at the latest data possible inside the Dashboard\Reports.

In case of Online Services like Dynamics 365 (Online), we have the option to manually Refresh the data.

To see it in action, let us create a lead record in Dynamics 365.

This is how our Dashboard looks like for now à

Now back in Power BI Service right, click on ellipsis for the Dataset and select Refresh Now.

We can see the newly created lead in our Power BI Report inside Power BI Service and Dynamics 365 CE.

The other option we have is to Schedule the refresh.

Navigate to the Dataset and select Schedule Refresh

We can select the Refresh Frequency as Daily or Weekly.

And we can set it to refresh up to 8 times per day as shown below.

With Power BI Pro License, we can complete the scheduled refresh 8 times a day whereas the Power BI Premium allows the refresh up to 48 times per day.

https://powerbi.microsoft.com/en-us/power-bi-premium/

Some helpful links

https://docs.microsoft.com/en-us/power-bi/refresh-scheduled-refresh

https://www.powerobjects.com/2018/01/11/power-bi-data-refresh-performance/

https://crmtipoftheday.com/category/power-bi/

Hope it helps..

Power BI with Dynamics 365 CE – Publishing Power BI Report to the web

In the previous post, we learned how to create a simple Power BI Report using Power BI Desktop

https://nishantrana.me/2018/08/20/power-bi-with-dynamics-365-ce-creating-report/

Now let us publish the report to Power BI Service from within the Power BI Desktop.

To do so, select the Publish button

Publish window will ask us to save the report file (if not saved) before we could publish it.

Here we are publishing it to the default “My Workspace” app workspace within Power BI Service.

Publishing the report from Power BI Desktop publishes both the report and dataset associated to it in the Power BI Service. We can think of DataSet as the Database.

After successful publish we can see the Report and Dataset being added inside the Power BI Service (https://app.powerbi.com)

From with Power BI Service, we have the option of Publishing the report on Web so that it could be accessed by anyone having that URL.

To do so,

Select File
à
Publish to Web to publish it.

The Embed dialog box pops up as shown below with all the information.

Clicking on Create Embed Code will open the confirmation dialog box, that shows the potential risk associated with sharing the information over the internet.

Here, clicking on Publish generates the link for the report and also the HTML code which can be used for embedding it within an IFrame.

Within Dynamics 365 CE, we can use this public URL to show it inside the IFrame component of the Dashboard.

The report within Dynamics 365 CE’s Dashboard.

Next, we will see how to create Dashboard from the reports created and how to share them with other users and how to use the Power BI Dashboard and Power BI Tiles feature of Dynamics 365.

Hope it helps.