Transfer files from local drive to Azure Blob using Azure Blog Upload Task – SSIS


Similar to Premium File Transfer Task,

The Azure Blog Upload Task component can be used to easily transfer files from local drive to Azure Blob storage.

https://docs.microsoft.com/en-us/sql/integration-services/control-flow/azure-blob-upload-task

Let us take a simple example to see it in action.

Here we will pick the folder Customer and its subfolders along with the files inside it and will move it or transfer to the Azure Blob Container.

Create a new SSIS Package and drag the Azure Blob Upload Task to the control flow designer.

Double click the task and specify the following values as shown below

AzureStorageConnection – specify the SSIS Connection Manager for Azure Storage.

Blob Container – the name of the existing blob container

Local Directory – the local directory containing the files to be uploaded.

Search Recursively – specify whether to search for files within Sub-directories.

File Name – specify the pattern for the files to be selected.

Time Range from/to – to pick files modified within that range.

Let us execute the package.

We can see the content transferred successfully to Azure Blog storage

Also, check out –

Using Azure Blob Storage component with Dynamics 365

https://nishantrana.me/2020/10/20/using-kingswaysoft-azure-blob-storage-component-with-dynamics-365/

Hope it helps..

Transfer files from local drive to Azure Blob using Premium File Transfer Task – SSIS


The Premium File Transfer Task component of KingswaySoft can be used to easily transfer files from local drive to Azure Blob storage.

https://www.kingswaysoft.com/products/ssis-productivity-pack/help-manual/premium-file-pack/premium-file-transfer-task

Let us take a simple example to see it in action.

Here we will pick the folder Customers and its subfolders along with the files inside it and will transfer it’s content to the Azure Blob Container.

Create a new SSIS Package and drag the Premium File Transfer Task to the control flow designer.

Double click the task and specify the following Source Properties

  • Action – Send Files – the other options are – Delete files, Create Directory, Remove Directory.
  • Check the option – Include Subdirectories
  • Connection Manager – Local File
  • Directory Path – specify the folder

Similarly, for Destination, we can specify the Azure Blog Storage Connection Manager and the directory path as shown below

Note – Make sure we have already added the connection for it to be available inside the connection manager option of Premium File Transfer Task. The other connection types supported are FTPS, SFT, Amazon S3, Azure Data Lake Storage, Box, Dropbox, Hadoop, OneDrive, SharePoint.

Let us run the package.

We can see the content transferred successfully to Azure Blog storage

 

 

Also, check out –

Using Azure Blob Storage component with Dynamics 365

https://nishantrana.me/2020/10/20/using-kingswaysoft-azure-blob-storage-component-with-dynamics-365/

https://nishantrana.me/2020/10/16/ssis-kingswaysoft-and-dynamics-365/

Hope it helps..

Process Optimization – Dynamics 365 CE / CRM / CDS SSIS Integration toolkit – KingswaySoft


Process Optimizations is a new feature added in the CDS / CRM Destination Component of KingswaySoft’s SSIS Integration Toolkit as part of November 2020 Release.

Check other posts on SSIS and Dynamics 365

https://nishantrana.me/2020/10/16/ssis-kingswaysoft-and-dynamics-365/

These options if enabled, will turn off or disable plugin, workflow, entity auditing during pre-execution temporarily, during the writing process, and will revert the changes to the original state after successful execution.

  • Tentatively Disable Relevant Plugins (if any)
  • Tentatively Disable Relevant Workflows (if any)
  • Tentatively Disable Relevant Auditing (if any)

Let us take a simple example to see it in action.

Here we have a simple data flow that creates contact records inside CDS using Data Spawner.

https://nishantrana.me/2020/05/26/using-data-spawner-component-ssis-to-generate-sample-data-in-dynamics-365/

Let us run the package and see the status of the plugin, workflow, and audit with regards to the contact entity.

As expected, we do not see any Audit records created for contact records creation.

We see an event where Audit is disabled and then enabled later for the contact entity, during the duration of package running and after its successful completion.

We can also see our workflow in status – Draft

In the case of Plugin, we didn’t see the step being disabled and it was still being triggered

As per the documentation –

Get the toolkit here

https://www.kingswaysoft.com/products/ssis-integration-toolkit-for-microsoft-dynamics-365

Hope it helps..

Changes in CDS / CRM Destination Component – SSIS Integration Toolkit for Dynamics 365


With version 20.2 November 2020 release, there have few updates added to the KingwaySoft’s CDS/CRM Destination Component, few are changes in the label and others are metadata changes, that one should be aware of before updating.

Check Post on SSIS and Dynamics 365

https://nishantrana.me/2020/10/16/ssis-kingswaysoft-and-dynamics-365/

  • Label changes.

Version 20.1 –

Version 20.2 –

  • CrmRecordId renamed to SavedRecordId

Version 20.1 –

Version 20.2 –

  • CrmErrorMessage renamed to ErrorMessage

Version 20.1 –

Version 20.2 –

Thus, any package saved with the new version will not work with the older version, and it is recommended to take full back up of the package before updating to the new version.

Hope it helps..

Extract attachment from Notes in Dynamics 365 / CE / CDS – SSIS (KingswaySoft)


Recently we were exploring ways of extracting attachment from notes to a local drive for our case entity. We also wanted to organize it based on Case Number.

For this, we used KingswaySoft’s SSIS components.

Other posts on SSIS and Dynamics 365 –

https://nishantrana.me/2018/11/26/ssis-and-microsoft-dynamics-365/

This is how our final package looked like

For CDS Source, we used Source Type as Fetch XML.

And the below Fetch XML to fetch notes details where objecttypecode is incident and isdocument property are true (i.e. has an attachment) and case as the linked entity to get the case ticket number.

Result View –

CDS / CRM Source Component

Make sure to use the alias for the attributes of the linked entity.

Next drag the Premium Derived Column, here we will use the DecodeBase64 encoding function to decode the file content stored in the documentbody field using as base-64 encoded string.

DT_Image output column will have the binary content of the attachment

Next, we have added another Premium Derived Column, this time to write the binary content to the local drive, using the WriteBinaryContent function.

Click on the ellipsis to specify the expression.

Here we are creating a folder based on the case ticket number and storing the corresponding attachments inside it.

Let us run the package

On successful execution, we can see the folders created with case ticket number and attachments added to it.

Get all the details here

http://www.kingswaysoft.com/blog/2017/03/21/Extracting-CRM-Attachments-with-Ease

Hope it helps..

Replicate / Export Dynamics 365 (CDS) data – different options


The most common use case to export Dynamics 365 or Common Data Service data to an external database/datastore is to have more control over the data (without going through API route) with the ability to run analytics such as Power BI Reporting, Machine Learning, Data Warehousing, Integration, etc.

For analytics, we can also use our Microsoft Dynamics 365 AI apps that use Dynamics 365 data as a source

https://docs.microsoft.com/en-us/dynamics365/ai/

Below listing down few of the options –

Data Export Service – Configuration

It is a free Add on-service provided by Microsoft for exporting Dynamics 365 or CDS Data to Microsoft Azure SQL Database or Microsoft Azure SQL Server on Microsoft Azure Virtual Machine.

Microsoft recommends Azure SQL Database Premium P1 or a better plan for Data Export Service

https://docs.microsoft.com/en-us/power-platform/admin/replicate-data-microsoft-azure-sql-database

Notes on Data Export Service –

https://nishantrana.me/2018/10/04/my-notes-on-dynamics-365-data-export-service/ 

Export to Data Lake Service – Configuration

The service enables continuous replication of CDS Data to Azure Data Lake Gen 2.

https://docs.microsoft.com/en-us/powerapps/maker/common-data-service/export-to-data-lake

Few posts on Azure Data Lake

https://nishantrana.me/category/azure-data-lake/

Azure Data Factory – GUI based integration tool

Cloud-based integration service.

https://docs.microsoft.com/en-us/azure/data-factory/connector-dynamics-crm-office-365

Check out the informative posts on Azure Data Factory –https://dynamicscrmgirl.wordpress.com/tag/azure-data-factory/

We can also run SSIS Packages on Azure Data Factory

https://nishantrana.me/2020/10/21/posts-on-azure-data-factory/

Scribe Online Replication Services – GUI based integration tool

Cloud-based integration service.

https://nishantrana.me/2019/01/09/using-scribe-online-replication-services-rs-to-replicate-dynamics-ce-data/

Another option is to use Scribe Insight

https://trellispoint.com/quickly-copy-dynamics-365-online-data-to-sql-server-using-scribe-insight/

 

Skyvia’s Data Integration service

 

Informatica

 

Cloud-based integration service.

https://nishantrana.me/2019/11/13/easily-set-up-dynamics-365-ce-crm-replication-incremental-to-azure-sql-sql-on-premise-using-skyvias-data-integration-services/

Another GUI based integration tool that can be considered is Informatica

https://network.informatica.com/cloud/index.htm#page/cloud-microsoft-dynamics-365-for-sales-connector-guide/Introduction_to_Microsoft_Dynamics_365_for_Sales_Connector.html

KingswaySoft Integration Components – SSIS

Leading data integration software – provide SSIS components to integrate applications/databases.

http://www.kingswaysoft.com/blog/2017/04/04/How-to-Retrieve-Incremental-Changes-from-CRM-in-your-ETL-Process

Another SSIS based option is CozyRoc

https://www.cozyroc.com/ssis/dynamics-crm

Or we can use Script Component instead of using 3rd party components/ tool

https://nishantrana.me/2020/10/19/connect-to-dynamics-365-through-script-component-ssis/

More on SSIS and Dynamics 365

https://nishantrana.me/2020/10/16/ssis-kingswaysoft-and-dynamics-365/

Use SQL to query CDS data (Preview)

The capability to query CDS Data.

https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/cds-sql-query

Check out all the articles on CDS T-SQL

https://markcarrington.dev/category/msdyn365/t-sql/

https://nishantrana.me/2020/05/21/setting-up-using-sql-to-query-data-in-dynamics-365-preview/

Query CDS data using Web API / Organization Service – Development

Write a custom tool using Dynamics 365 APIs

https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/webapi/query-data-web-api

https://docs.microsoft.com/en-us/powerapps/developer/common-data-service/org-service/entity-operations

Hope it helps..