Posts on Azure Data Lake


How to – Access Dataverse environment storage using SAS Token

We can access our environment’s storage using a SAS token.  Download – Azure Storage Explorer – https://azure.microsoft.com/features/storage-explorer/#overview Select Connect to Azure resources option Select ADLS Gen2 container or directory for the Azure storage type. Select the Shared access signature URL for the connection To get the SAS URL – https://%5BContainerURL%5D/CDS?%5BSASToken%5D, we need ContainerURL and SASToken.…

Fixed – Azure Synapse Link for Dataverse not starting

Recently while trying to configure “Azure Synapse Link forDataverse” for one of our environments, we faced an issue, where it got stuck as shown below. Also, there were no files/folders created inside the Storage Account. https://nishantrana.me/2020/12/10/posts-on-azure-data-lake/ The accounts were properly setup – https://docs.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-synapse#prerequisites Here we had only selected contact table for sync initially, and it…

Fixed – Authorization failed. The client with object id does not have authorization to perform action ‘Microsoft.Authorization/roleAssignments/write’ over scope ‘/subcriptions’ while configuring Azure Synapse Link for Dataverse

Recently while configuring Azure Synapse Link for Dataverse, for exporting data to Azure Data Lake we got the below error – {“code”:”AuthorizationFailed”,”message”:”The client ‘abc’ with object id ‘d56d5fbb-0d46-4814-afaa-e429e5f252c8’ does not have authorization to perform action ‘Microsoft.Authorization/roleAssignments/write’ over scope ‘/subscriptions/30ed4d5c-4377-4df1-a341-8f801a7943ad/resourceGroups/RG/providers/Microsoft.Storage/storageAccounts/saazuredatalakecrm/providers/Microsoft.Authorization/roleAssignments/2eb81813-3b38-4b2e-bc14-f649263b5fcf’ or the scope is invalid. If access was recently granted, please refresh your credentials.”} As well…

How to – Export Dataverse (Dynamics 365) data to Azure SQL using Azure Data Factory pipeline template

[Visual Guide to Azure Data Factory – https://acloudguru.com/blog/engineering/a-visual-guide-to-azure-data-factory%5D Using the new Azure Data Factory pipeline template – Copy Dataverse data from Azure Data Lake to Azure SQL – we can now easily export the Dataverse data to Azure SQL Database. https://docs.microsoft.com/en-us/power-platform-release-plan/2021wave1/data-platform/export-dataverse-data-azure-sql-database Check other posts on Azure Data Factory Select Pipeline from template option inside the…

Fixed – AuthorizationFailed. The client with object id does not have authorization to perform action ‘’Microsoft.Authorization/roleAssignments/write’ over scope ‘storageaccount’ – Azure Data Lake

While configuring the Azure Synapse Link/ Export to Data Lake service, we were getting below error for one of the users. {“code”:”AuthorizationFailed”,”message”:”The client ‘nishantr@pmaurua105.onmicrosoft.com’ with object id ‘d56d5fbb-0d46-4814-afaa-e429e5f252c8’ does not have authorization to perform action ‘Microsoft.Authorization/roleAssignments/write’ over scope ‘/subscriptions/30ed4d5c-4377-4df1-a341-8f801a7943ad/resourceGroups/RG/providers/Microsoft.Storage/storageAccounts/saazuredatalakecrm/providers/Microsoft.Authorization/roleAssignments/2eb81813-3b38-4b2e-bc14-f649263b5fcf’ or the scope is invalid. If access was recently granted, please refresh your credentials.”} The current…

How to setup – Azure Synapse Link – Microsoft Dataverse

Azure Synapse Link (earlier known as Export to Data Lake Service) provides seamless integration of DataVerse with Azure Synapse Analytics, thus making it easy for users to do ad-hoc analysis using the familiar T-SQL with Synapse Studio, build Power BI Reports using Azure Synapse Analytics Connector or use Azure Spark in Azure Synapse for analytics.…

Advanced configuration settings – Azure Synapse Link / Export to Data Lake service (Dataverse/ Dynamics 365)

The Export to Data Lake service now has some Advanced configuration settings available. To learn more on Export to Data Lake service https://nishantrana.me/2020/12/10/posts-on-azure-data-lake/ The new settings allow us to configure how the DataVerse / CRM table data is written to Azure Data Lake. In-Place update or upsert (default) Append Only With the in-place update, the…

DSF Error: CRM Organization cannot be found while configuring Azure Synapse Link / Export to Data Lake service in Power Platform

Recently while trying to configure the Export to Data Lake service from the Power Apps maker portal, we got the below error. DSF Error: CRM Organization <Instance ID> cannot be found. More on configuring Export to Data Lake service – https://nishantrana.me/2020/12/10/posts-on-azure-data-lake/ The user through which we were configuring had all the appropriate rights. https://docs.microsoft.com/en-us/powerapps/maker/data-platform/export-to-data-lake#prerequisites The…

How to – Migrate Dataverse environment to a different location within the same Datacentre region – Power Platform

When we create an environment in the Power Platform admin center, we get the option of specifying the datacenter region, but not the location within it. Find the Data Center Region / Location of your Dataverse Environment- https://nishantrana.me/2021/04/27/finding-the-datacenter-region-location-of-the-microsoft-dataverse-environment/ E.g. we have specified Region as the United Arab Emirates. Now within the UAE region, we had…

Azure Data Lake Storage Component in KingswaySoft – SSIS

Download and install the SSIS Productivity Pack https://www.kingswaysoft.com/products/ssis-productivity-pack/download/ Drag the Azure Data Lake Storage Source component in the data flow Double click and click on New to specify the connection Provide the connection details and test the connection It supports both Gen 1 and Gen 2 Supports the below Authentication modes Inside the Azure Data…

Use query acceleration to retrieve data from Azure Data Lake Storage

Few key points about query acceleration – Query acceleration supports ANSI SQL like language, to retrieve only the required subset of the data from the storage account, reducing network latency and compute cost. Query acceleration requests can process only one file, thus joins and group by aggregates aren’t supported. Query acceleration supports both Data Lake…

Snapshot in Azure Data Lake (Dynamics 365 / CDS) – Azure Synapse Link

In the previous post, we saw how to export CDS data to Azure Data Lake Gen 2 https://nishantrana.me/2020/09/07/export-data-from-common-data-service-to-azure-data-lake-storage-gen2/ Here let us have a look how the sync and snapshot work. We have already done the configuration and have synced the Account and Contact entity. As the diagram depicts – there is initial sync followed by…

Error – We don’t support the option ‘HierarchicalNavigation’. Parameter name: HierarchicalNavigation when trying to load table in Power BI Desktop using Azure Data Lake Storage Gen 2 CDM Folder view (beta)

While trying to connect to a table within Azure Data Lake Storage Gen2 through CDS Folder View we got the below error Users have reported this issue with the August 2020 Update of Power BI Desktop. As suggested in the forums, downgrading to June 2020 Update fixed the issue for us. Check out Export CDS data…

How to – Use Azure Synapse Link – Export data from Common Data Service to Azure Data Lake Storage Gen2

Azure Data lake store gen 2 can be described as a large repository of data, structured or unstructured built on top of Azure Blob storage, that is secure (encryption – data at rest), manageable, scalable, cost-effective, easy to integrate with. Also check out – setup Azure Synapse with Dataverse – https://nishantrana.me/2021/06/16/how-to-setup-azure-synapse-link-microsoft-dataverse/ Export to Data Lake…

Comparing the three main ways to get Dynamics 365 data into Power BI


Ulrik (CRM Chart Guy)'s avatarcrm chart guy

There are a few different ways to get your Dynamics 365 data into Power BI. Now that the Common Data Service connector has come out of preview and is generally available, I figured it would be a good time to have a look at the different methods and how they stack up.

The different direct methods are:

  • Dynamics 365 Online connector
  • Common Data Service (CDS) Connector
  • FetchXML (via Power Query Builder for XrmToolBox)

You can also use OData directly, but since this option doesn’t have any benefits over the other three, I am going to leave it out. The Data Export Service does have some benefits, but I will have to leave that for a different blog post.

The Dynamics 365 Online connector is probably the most commonly used option as it is the result you get when searching for “Dynamics 365” in Power BI’s Get Data dialog.

D365 legacy connector

The Common…

View original post 2,494 more words

Use query acceleration to retrieve data from Azure Data Lake Storage


Few key points about query acceleration –

Query acceleration supports ANSI SQL like language, to retrieve only the required subset of the data from the storage account, reducing network latency and compute cost.

Query acceleration requests can process only one file, thus joins and group by aggregates aren’t supported.

Query acceleration supports both Data Lake Storage (with hierarchical namespace enabled) and blobs in the storage account.

Query acceleration supports CSV and JSON formatted data as input.

Let us take a simple example to see it in action.

Within mydatalakegen (StorageV2 (general purpose v2)), we have All Contacts.csv with the mycrmcontainer.

Open the Windows PowerShell command window

Sign in to Azure subscription

  • Connect-AzAccount

Register the query acceleration feature

  • Register-AzProviderFeature -ProviderNamespace Microsoft.Storage -FeatureName BlobQuery

Register the resource provider

  • Register-AzResourceProvider -ProviderNamespace ‘Microsoft.Storage’

Create a console application project in Visual Studio and add the following NuGet Packages

Sample Code –

&lt;/pre&gt;

using System;
using System.Globalization;
using System.IO;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using Azure.Storage.Blobs.Specialized;
using CsvHelper;
using CsvHelper.Configuration;

namespace MyQuery
{
class Program
{
static void Main(string[] args)
{

// Initialize the BlockBlobClient
BlockBlobClient myBlobClient = new BlockBlobClient(connectionString: "DefaultEndpointsProtocol=https;AccountName=mydatalakegen;AccountKey=orc8e1Dpclu5P3Ox9PIlsLG2/x8KZLcmgyhOEgz6yFTmzFJty+EpHQ==;EndpointSuffix=core.windows.net",
containerName: "mycrmcontainer", blobName: "All Contacts.csv");

// Define the query
// First Name - space in the column header
// _4 - referring the 4th column in the csv file
// LIMIT - limit to first 10 records
string query = @"SELECT ""First Name"", _4, email FROM BlobStorage LIMIT 10";

var blobQueryOptions = new BlobQueryOptions();
blobQueryOptions.InputTextConfiguration = new BlobQueryCsvTextOptions() { HasHeaders = true };

var result = myBlobClient.Query(query, blobQueryOptions);
var reader = new StreamReader(result.Value.Content);

var parser = new CsvReader(reader, new CsvConfiguration(CultureInfo.CurrentCulture) { HasHeaderRecord = true });

while(parser.Read())
{
Console.Out.WriteLine(String.Join(" ", parser.Context.Record));
}

Console.ReadLine();
}
}
}
&lt;pre&gt;

Output –

Get all the details here –

https://docs.microsoft.com/en-us/azure/storage/blobs/query-acceleration-sql-reference

https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-query-acceleration

https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-query-acceleration-how-to?tabs=azure-powershell%2Cpowershell

 

Posts on Azure Data Lake

Hope it helps..

Advertisements

Fixed – AADSTS7000218: The request body must contain the following parameter: ‘client_assertion’ or ‘client_secret


While trying to access Azure resources using UsernamePasswordCredential credential we were getting the below error

https://docs.microsoft.com/en-us/dotnet/api/azure.identity.usernamepasswordcredential?view=azure-dotnet

Azure.Identity.AuthenticationFailedException: ‘UsernamePasswordCredential authentication failed: A configuration issue is preventing authentication – check the error message from the server for details.You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details. Original exception: AADSTS7000218: The request body must contain the following parameter: ‘client_assertion’ or ‘client_secret’.

Trace ID: ef6c9e2b-862a-4a8b-9519-9a9072d23301

Correlation ID: 5f9bae95-e45a-4da5-b27c-ad9704e7334e

Timestamp: 2020-11-28 05:58:05Z’

This was because Allow public client flows was disabled for the application registered in Azure AD.

Enabling it fixed the issues for us.

https://docs.microsoft.com/en-gb/azure/active-directory/develop/scenario-desktop-acquire-token?tabs=dotnet#username-and-password

More on ROPC

https://nishantrana.me/2019/08/23/connect-to-dynamics-365-web-api-using-oauth-2-0-resource-owner-password-credential-ropc/

Hope it helps..

Advertisements

Working with large CDS datasets in Power Automate Flows and Logic Apps


Thanura Wijesiriwardena's avatarEVOLVED365

During the last 12 months I have been busy working on projects and lost my way a little bit on writing and sharing my experiences with the community. However on a positive note I learned a lot of new things during that period and now I’ve got some great content planned and waiting to be shared with you all.

Some of those learnings were around working with large CDS datasets in Microsoft Power Automate Flows and Azure Logic Apps. Here are some of those easy to learn tips, tricks, guidelines and limitations that will help you for sure.

Connecting to a Common Data Service (CDS) environment

There are two main connectors you can use. “Common Data Service” connector and “Common Data Service (current environment)” connector. Let’s look at few points to consider when selecting a connector for your workflow.

Common Data Service connectorCommon Data Service (current…

View original post 1,303 more words

Fixed – AADSTS65001: The user or administrator has not consented to use the application with ID


The below error occurs for the application registered with Azure AD (Delegated Permissions), which requires either user or an administrator’s consent for the permissions it needs.

“Azure.Identity.AuthenticationFailedException: ‘UsernamePasswordCredential authentication failed: AADSTS65001: The user or administrator has not consented to use the application with ID ‘9ea6c0e6-5ab5-4816-b787-5391cd41fd7b’ named ‘MyKVApp’. Send an interactive authorization request for this user and resource.”

The below setting specifies that all users can allow applications to access the organization’s data on their behalf.


Here the admin can grant the consent through the portal as shown below from Home > App > API Permissions



or can also use Consent URL

https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/grant-admin-consent#construct-the-url-for-granting-tenant-wide-admin-consent

https://login.microsoftonline.com/{tenant-id}/adminconsent?client_id={client-id}


When trying to access the consent URL using another non-admin user, we might get the below message, which means that only the admin can provide the required consent.


Signing in with the Admin account presents the below message for granting the app the required permissions.


Admin can also revoke the admin consent (along with the permission as shown below) from the portal as shown below through Remove admin consent option.


Get all the details here –

https://docs.microsoft.com/en-us/azure/active-directory/develop/application-consent-experience#consent-and-permissions

Hope it helps..


Advertisements

Nishant Rana's Weblog

Everything related to Microsoft .NET Technology

Skip to content ↓