Using Conditional Split Transformation in SSIS

Recently we had a requirement to Upsert Customer’s Spend related information from SQL DB to our CRM Online.

We had the spend captured per brand wise in the DB as shown below.

And in our CRM, we had one custom entity that had fields corresponding to each of these Brand as shown below

i.e. one section for each Brand and inside that section 5 fields corresponding to the columns in the SQL Table.

The GCR column of the table was the Alternate Key column in our custom entity which we used for Upsert.

Now to make sure we update only the corresponding Brand fields in CRM, we can make use of Conditional Split Transformation component.

Just after OLE DB source, add a Conditional Split component

Create one output per brand i.e. (select distinct Brand from the source table records) as shown below

Now add a CRM Destination Component and bind it to the corresponding output

And inside Columns section, map only the corresponding field of the selected Brand.

The finished package à

Hope it helps..

Optimum Batch Size while using SSIS Integration toolkit for Microsoft Dynamics 365


Recently while working in a data integration task, we were evaluating the performance of using KingswaySoft’s Dynamics 365 Toolkit from within Visual Studio by trying out different possible combinations of batch size and threads.

The environment was CRM Online and had concurrent ExecuteMultiple max limit set to 20.

By default, it is set as 2 for online, but we can raise support ticket with Microsoft to get this limit increased. However, we need be very careful while using them as large number of ExecuteMultiple running in parallel might degrade the performance.

Below are the 2 settings that we played around in the CRM Destination Component

These are the results we got with different batch size and thread.

  • Message Used– Upsert.
  • Total Number of Records – 20000.

It clearly shows that having batch size as 100 gives the best result.

And this is what is recommended by KingswaySoft as well.

Hope it helps..

CRM service call returned an error: The server is busy and the request was not completed. Try again later while inserting records using CRM Destination Component of KingswaySoft’s SSIS Integration Toolkit for Dynamics 365

While running one of our package we got the below error in few of the records

“CRM service call returned an error: CRM service call returned an error: The server is busy and the request was not completed. Try again later. (Error Code: -2147180543, Detail Message: The server is busy and the request was not completed. Try again later.)”

“CRM service call returned an error: CRM service call returned an error: The request channel timed out while waiting for a reply after 00:01:59.7991956. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. (Error Type / Reason: Timeout)”

One way to get around this error is to increase the Timeout (secs) in the Service Timeout Setting of CRM Connection Manager.

We updated it to 600 seconds and the number of records that failed got drastically reduced.

Also make sure that following option is checked

Hope it helps..

Lookup not getting set while using CRM Destination Component of KingswaySoft’s SSIS Integration Toolkit for Dynamics 365

Recently while inserting a custom entity (Transaction) records using the CRM Destination Component we realized that Contact’s lookup was not being set.

We were using Alternate Key to set the lookup as shown below

To get it working we had to uncheck the Remove Unresolvable References option from the Optional Settings of CRM Destination Component. It seemed that it ignored it, even though values were there for the GCR (alternate key field) in the input source.

I think it might not be working as expected in case if lookup is done using Alternate Key.

BTW, I would strongly recommend using KingswaySoft’s SSIS Integration Toolkit for any or all data integration tasks. It is definitely the best tool out there.

Give it a try à

Hope it helps..

View Designer not showing SSIS Package in Design mode in Visual Studio

Time to time we’d see our Package not opening in Designer mode in Visual Studio.

To fix this,

Add the package again in the solution or restart the Visual Studio.

Hope it helps..

Using Lookup Transformation and Cache Transformation in SSIS for Dynamics 365 (using Kingswaysoft)

Recently while working in data migration project, we had a requirement to insert a record in CRM from SQL Database and assign it to the team insert.

For e.g.

We had a Transaction Record that had a Brand lookup in it i.e. n – 1 relationship.

And the Brand record had a lookup of Team in it again n – 1. So basically, while inserting the transaction record we wanted to assign the record to the team of the corresponding brand in it.

So, for this, we had to set the owner id and owner id type field while inserting the record.

And for owner id i.e. team GUID we had to do a lookup on the Brand entity.

To achieve so, we did the following step

Added 2 Data flow task in the Control Flow, one for Caching the Brand entity’s detail and other for using the fetching data from SQL DB, Lookup Transformation task and inserting the data into CRM.

For the first data flow task,

We added the Dynamics CRM Source and fetched the Brand Name and Team Lookup’ GUID field.

name field which will be used for mapping and teamid field whose value we need from it.

Then we added a Cache Transform task with following details,

Now for our second data task flow,

First, we added the OLE DB Source to fetch data from the source SQL View, having Brand (string) column in it.

Then we added the Lookup Transformation task with following properties,

Setting Connection to Cache Connection Manager defined earlier.

Now the most important part, the columns properties. Here we defined the mapping between the Brand field of the SQL View and name field of the Brand Entity of CRM and selected the teamid field as the lookup column whose value we would like to fetch as shown below.

Then we added Derived Column task to add one more column for specifying ownerid type field to be used in CRM Destination Component while mapping. We defined the field type as string and hardcoded the value as “team”.

And as the final step mapped the above fields to the corresponding ownerid and owneridtype field in CRM Destination Component for Create operation.

For ownerid, we selected the below option as we were setting the GUID of the team.

On running the package, we can see the transaction records getting created and properly assigned to the corresponding team of the brand.

The transaction record with Team set as Owner.

Brand record with Team lookup in it.

Hope it helps..