Microsoft Dynamics 365 – Nishant Rana's Weblog

Why SQL4CDS Record Counts May Not Match Advanced Find for Date Filters (Dataverse / Dynamics 365)

While validating some Dynamics 365 Field Service data recently, we came across an interesting scenario where SQL4CDS and Advanced Find returned different record counts even though the date filters appeared to be identical.

At first glance it was surprising to see different record counts being returned despite using what appeared to be the same date range. After investigating further, we found that the difference was related to time zone handling and the behavior of User Local date fields.

In this post, we’ll walk through the issue, explain why it happens, and show how to get matching results between Advanced Find and SQL4CDS.

The Scenario

We had a user in Auckland, New Zealand running the following Advanced Find query against Work Orders.

Date Window Start

On or After 01/01/2026
On or Before 02/01/2026

Advanced Find returned:

5,755 records

The generated FetchXML looked like this:

To validate the result, we ran the following query in SQL4CDS:

SELECT COUNT(*)
FROM msdyn_workorder
WHERE msdyn_datewindowstart >= ‘2026-01-01 00:00:00’
AND msdyn_datewindowstart <= ‘2026-01-02 00:00:00’;

The results were unexpected.

Query Method	Time Zone Used	Result
Advanced Find (Auckland User)	Auckland (NZDT)	5,755
SQL4CDS	UTC Mode	3,027
SQL4CDS	Local Mode (India)	3,026

At this point, it was clear that Advanced Find and SQL4CDS were evaluating different date boundaries, even though the filters appeared very similar. The next step was to understand why.

Understanding the Date Window Start Field

The key detail was the configuration of the Date Window Start field.

The Date Window Start field is configured as a Date Only field with User Local behavior.

Although users only see a date value, Dataverse stores an underlying UTC datetime value and performs time zone conversion based on the user’s personal settings.

To better understand what was happening, we queried some of the underlying values directly.

SELECT msdyn_workorderid,
msdyn_datewindowstart
FROM msdyn_workorder
WHERE msdyn_datewindowstart >= ‘2026-01-01 00:00:00’
AND msdyn_datewindowstart <= ‘2026-01-02 00:00:00’;

When running SQL4CDS in UTC mode, many records had values such as:

2026-01-01 11:00:00

This initially looked unusual because users only see a date value in the application.

However, the explanation becomes clear when we consider the Auckland user’s time zone.

In January, Auckland operates on New Zealand Daylight Time (NZDT), which is UTC+13.

For a User Local Date Only field, Dataverse converts the user’s local date into UTC before storing it.

Date Seen by Auckland User	Stored UTC Value
01-Jan-2026	31-Dec-2025 11:00 UTC
02-Jan-2026	01-Jan-2026 11:00 UTC
03-Jan-2026	02-Jan-2026 11:00 UTC

This explains why so many records appear with a value of 11:00 UTC when viewed in SQL4CDS running in UTC mode.

Why Advanced Find Returned More Records

When the Auckland user enters:

01/01/2026
to
02/01/2026

Advanced Find interprets those dates using the user’s personal time zone.

The actual UTC boundaries become:

>= 2025-12-31 11:00:00 UTC
< 2026-01-02 11:00:00 UTC

This represents two complete calendar days for the Auckland user.

Our original SQL4CDS query was searching a different range entirely:

>= 2026-01-01 00:00:00 UTC
<= 2026-01-02 00:00:00 UTC

Although the dates appear similar, the actual UTC boundaries are very different.

Finding the Correct SQL4CDS Query in UTC Mode

To reproduce the Advanced Find results, we converted the Auckland user’s date range into UTC and updated the SQL4CDS query accordingly.

SELECT COUNT(*)
FROM msdyn_workorder
WHERE msdyn_datewindowstart >= ‘2025-12-31 11:00:00’
AND msdyn_datewindowstart < ‘2026-01-02 11:00:00’;

This returned:

5,755 records

which matched Advanced Find exactly.

What If SQL4CDS Is Running in Local Mode?

The example above used SQL4CDS running in UTC mode. However, SQL4CDS can also be configured to use Local Time mode.

In our scenario, SQL4CDS was running on a machine configured for India Standard Time (IST), which is UTC+5:30.

To match the Advanced Find results in Local Mode, we need to convert the Auckland UTC boundaries into the local time zone used by SQL4CDS.

Earlier we determined that the Auckland user’s date range:

01-Jan-2026 to 02-Jan-2026

corresponds to the following UTC boundaries:

31-Dec-2025 11:00 UTC
to
02-Jan-2026 11:00 UTC

When SQL4CDS is running in Local Mode on an India machine, those UTC values need to be converted to IST.

UTC Boundary	IST Boundary
31-Dec-2025 11:00 UTC	31-Dec-2025 16:30 IST
02-Jan-2026 11:00 UTC	02-Jan-2026 16:30 IST

The equivalent SQL4CDS query becomes:

SELECT COUNT(*)
FROM msdyn_workorder
WHERE msdyn_datewindowstart >= ‘2025-12-31 16:30:00’
AND msdyn_datewindowstart < ‘2026-01-02 16:30:00’;

This query also returned:

5,755 records

matching Advanced Find exactly.

The results can now be summarized as follows:

Validation Method	Query Boundary	Result
Advanced Find (Auckland User)	User Time Zone	5,755
SQL4CDS UTC Mode	31-Dec-2025 11:00 UTC → 02-Jan-2026 11:00 UTC	5,755
SQL4CDS Local Mode (India)	31-Dec-2025 16:30 IST → 02-Jan-2026 16:30 IST	5,755

References

For a deeper understanding of how SQL4CDS handles date and time values, I highly recommend Mark Carrington’s article:

https://markcarrington.dev/2021/05/21/date-time-handling-in-sql-4-cds

This article explains how SQL4CDS interprets date and time values in both UTC and Local Time modes and was a useful reference while investigating this scenario.

Key Takeaways

The investigation highlighted that there may be three different time zones involved when validating results:

The Dataverse user’s personal time zone used by Advanced Find.
The SQL4CDS Local Time setting.
UTC when SQL4CDS is configured to use UTC mode.

Even when the same date values are entered, the actual UTC range being queried may be different.

For the most reliable comparison:

Identify the time zone of the user who ran Advanced Find.
Convert the date boundaries to UTC.
Run SQL4CDS in UTC mode.
Use explicit UTC values in your query.

We also recommend using an exclusive upper boundary:

WHERE Field >= StartBoundaryUTC
AND Field < EndBoundaryUTC

instead of:

WHERE Field <= EndOfDay

This avoids potential issues with milliseconds and provides more predictable results.

SQL4CDS can match Advanced Find in either UTC Mode or Local Mode. The important requirement is that the date boundaries represent the same moment in time. We generally prefer UTC Mode because the query behaves consistently regardless of the machine or user executing it.

Hope it helps..

Advertisements

Dataverse Web API: Returning Record Data During Create Using Prefer: return=representation

When creating records using the Dataverse Web API, the default behavior is to return a successful response without including the record data in the response body.

A standard create request does provide access to the newly created record’s GUID through the OData-EntityId response header. However, if we need additional column values from the newly created record, we would typically need to perform a separate retrieve request.

Dataverse provides a convenient way to return record data as part of the create operation itself by using the Prefer: return=representation request header.

Let’s see how it works.

Creating a Record and Returning Data

Suppose we want to create a Contact record and immediately retrieve some of its values.

POST https://.crm.dynamics.com/api/data/v9.2/contacts?$select=contactid,fullname,emailaddress1

Content-Type: application/json

Prefer: return=representation

Request Body:

{
    "firstname": "Nishant",
    "lastname": "Rana",
    "emailaddress1": "nishant@example.com"
}

The key part of the request is the following header:

Prefer: return=representation

This tells Dataverse to return the created record in the response.

Using $select to Limit Returned Columns

In the example above, the request URL includes a $select clause:

?$select=contactid,fullname,emailaddress1

This allows us to control which columns are returned in the response.

Returning only the required columns helps reduce the response size and makes the response easier to process.

Response

When the request succeeds, Dataverse returns an HTTP 201 Created response along with the requested column values.

Example response:

{
    "contactid": "4e7d5c6a-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "fullname": "Nishant Rana",
    "emailaddress1": "nishant@example.com"
}

Because the record data is already available in the response, there is no need to perform an additional retrieve request to obtain these values.

Standard Create vs Return Representation

Let’s compare the behavior with a standard create request.

A normal create request returns an HTTP 204 No Content response.

HTTP/1.1 204 No Content

The response also includes an OData-EntityId header containing the URI of the newly created record.

OData-EntityId: https://.crm.dynamics.com/api/data/v9.2/contacts(4e7d5c6a-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

This means we can still obtain the GUID of the created record without using Prefer: return=representation.

However, if we need additional values such as calculated fields, default values applied by Dataverse, or other column data, a separate retrieve request would typically be required.

Feature	Basic Create	Create with Data Returned
Status Code	204	201
Response Body	No	Yes
Extra Retrieve Needed	Usually	No
Performance	Faster	Slightly More Expensive
Best For	Bulk Operations	Integrations & UI Scenarios

When Should We Use It?

Using Prefer: return=representation can be useful when:

An integration needs values from the newly created record immediately.
We want to retrieve calculated or system-generated values.
A client application needs to display record information immediately after creation.
We want to avoid an additional retrieve request and simplify the integration logic.

If only the GUID of the newly created record is required, the standard create operation is usually sufficient.

Get more details – Create with data returned

Hope it helps..

Advertisements

Testing the New RunJobForSandbox Option in Bulk Delete Jobs (Preview) – Dataverse / Dynamics 365

Microsoft recently introduced a preview feature for Dataverse Bulk Delete Jobs that provides additional control over bulk delete processing. One of the new options available when creating a bulk delete job through the API is RunJobForSandbox.

According to the documentation, this option is intended to control sandbox processing during bulk delete operations, which could be particularly useful in environments where delete plugins or custom workflows impact large-scale data cleanup activities.

For our testing, we created a Bulk Delete Job using Postman and included the following option in the request payload:

{
  "QuerySet": [
    {
      "EntityName": "contact",
      "Criteria": {
        "FilterOperator": "And",
        "Conditions": [
                   {
            "AttributeName": "createdon",
            "Operator": "OnOrBefore",
            "Values": [
              {
                "Value": "2026-09-07T23:59:59Z",
                "Type": "System.DateTime"
              }              
            ]
          }
        ]
      }
    }
  ],
  "JobName": "Sample Bulk Delete Job with Run Job For Sandbox True",
  "SendEmailNotification": false,
  "RecurrencePattern": "",
  "StartDateTime": "2026-05-18T00:00:00Z",
  "ToRecipients": [],
  "CCRecipients": [],
  "Options": {
    "CanRecoverDeletedRecords": false,
    "RunJobForSandbox": true
  }
}

The complete job targeted Contact records based on their Created On date and was created successfully.

To understand how this option behaves, we registered a simple plugin on the Delete message of the Contact table. The plugin was intentionally designed to throw an InvalidPluginExecutionException whenever a record deletion was attempted.

Our expectation was that enabling RunJobForSandbox would prevent the sandbox plugin from executing during the bulk delete process, allowing the records to be deleted successfully.

However, the results were different from what we anticipated.

When the bulk delete job was executed, the delete plugin was still triggered. Because the plugin threw an exception, all targeted records failed to delete. The Bulk Delete Job completed with failures and reported errors indicating that the deletion operation had been aborted by a plugin or custom workflow.

Since this capability is currently in Preview, it is possible that the feature is still evolving, has limitations that are not yet documented, or requires additional configuration. To better understand the observed behavior, we have raised a Microsoft Support ticket and are awaiting clarification from the team.

Even though our initial test did not produce the expected result, this is still a very promising feature. Once fully implemented and generally available, the ability to control sandbox processing during bulk delete operations could make large-scale data cleanup significantly easier, especially in environments where plugins and custom workflows frequently interfere with bulk deletion activities.

We’d update this post once we receive additional information from Microsoft regarding the current behavior and intended functionality of RunJobForSandbox.

The feature is documented here:

Control Bulk Delete Processing (Preview)

Hope it helps..

Advertisements

Troubleshooting Hidden Dependency Errors in Dynamics 365 and Dataverse

Recently, while cleaning up some old customizations in Dynamics 365, we came across an interesting dependency issue that was not immediately obvious from the user interface. What initially looked like a simple Business Process Flow deletion turned into a deeper investigation into how Dataverse manages published dependencies internally.

While trying to delete a BPF, Dataverse returned the following error: – Failed to delete (). Object dependencies exist, please review before deleting.

Normally, the first step in this situation is to use the built-in dependency viewer. However, in this case, the dependency screen itself was not particularly helpful. Even though the platform clearly stated that dependencies existed, the dependency dialog was not showing any actual records / components blocking the deletion.

To investigate further, we opened the browser developer tools and inspected the network response generated during the delete operation. The response payload contained much more detail than what was shown in the UI, including the GUID of the component and information indicating that dependencies still existed behind the scenes.

The Entity(99980477-1249-447b-8514-6d11fe6f1b1e) component cannot be deleted because it is referenced by 7 other components.

{“error”:{“code”:”0x8004f01f”,”message”:”The Entity(99980477-1249-447b-8514-6d11fe6f1b1e) component cannot be deleted because it is referenced by 7 other components. For a list of referenced components, use the RetrieveDependenciesForDeleteRequest.”}}

–changesetresponse_aa58f311-830f-4c7f-af82-f21732502a1e–

–batchresponse_26fe11bd-bb7c-4cef-ad4b-91121ac828cf—

With the component GUID available, we switched to SQL 4 CDS to query the dependency table directly. The following query was used:

SELECT *

FROM dependency

WHERE requiredcomponentobjectid = ‘99980477-1249-447b-8514-6d11fe6f1b1e’

The results immediately started revealing useful information. One of the most important columns in the results was:

dependencytype = 2, dependencytypename = Published

This turned out to be the key detail. A dependency type of Published means the reference is actively present in the environment and is currently preventing deletion of the component.

The dependency records also showed the following values:

dependentcomponenttype = 29, dependentcomponenttypename = Workflow

That told us the blocking component was actually a workflow. To identify the workflow names, we queried the workflow table directly using the workflow IDs returned by the dependency query, which took us to the below Workflow. When we opened this workflow, we found that it still contained a step creating or referencing the process we were attempting to remove.

We removed the workflow referencing the process, published the changes, and then retried the deletion. Once the workflow dependency was removed, the process was deleted successfully without any further errors.

This was a useful reminder that the dependency viewer in Dynamics 365 and Dataverse does not always surface every dependency clearly, especially with older workflows and process-related customizations. In these situations, SQL 4 CDS and direct dependency table queries can be extremely valuable for identifying hidden references.

Hope this helps..

Advertisements

Capture UTM Parameters in Dynamics 365 Marketing Forms Using JavaScript (Dynamics 365 Customer Insights)

When running marketing campaigns, UTM parameters help us understand where our leads are coming from. They help us track whether a lead came from Google Ads, Facebook campaigns, email campaigns, or some other source.

Recently while working with a Dynamics 365 Marketing form, we used a small JavaScript snippet to automatically capture UTM parameters from the URL and store them directly into marketing form fields.

 <script>
document.addEventListener("d365mkt-afterformload", function () {
    const params = new URLSearchParams(window.location.search);
      const mappings = [
        { param: "utm_source", name: "custom_utm_source" },
        { param: "utm_medium", name: "custom_utm_medium" },
        { param: "utm_campaign", name: "custom_utm_campaign" },
        { param: "utm_term", name: "custom_utm_term" },
        { param: "utm_content", name: "custom_utm_content" },
        { param: "gclid", name: "custom_gclid" },
        { param: "gclsrc", name: "custom_gclsrc" },
        { param: "fbclid", name: "custom_fbclid" }
    ];
    mappings.forEach(function (m) {
        const field = document.querySelector(`[name="${m.name}"]`);
        const value = params.get(m.param);

        if (field && value) {
            field.value = value;        
        }
    });
});
</script>

Suppose the marketing form URL is opened like this:

https://contoso.com/form?utm_source=google&utm_medium=cpc

In the example below, the values from the URL are automatically populated into the form fields.

The script first waits for the Dynamics 365 Marketing form to fully load using the d365mkt-afterformload event. This is important because the fields may not yet exist when the page initially loads.

After that, the script reads the query string from the URL using URLSearchParams. So if the URL contains values like utm_source=google or utm_medium=cpc, those values become available to the script.

The mappings array is used to map URL parameters to marketing form fields. For example, utm_source maps to custom_utm_source and utm_medium maps to custom_utm_medium.

The script then loops through each mapping, finds the matching field inside the marketing form, and sets the value automatically.

Using this approach helps us capture campaign attribution data directly inside Dataverse during form submission

References –

https://paulinekolde.info/javascript-library-for-real-time-marketing-form-in-customer-insights

Extend Customer Insights – Journeys marketing forms using code

Hope it helps..

Advertisements

Hidden Required Fields Causing “Please ensure all required fields are filled out” Error While Disqualifying a Lead in Dynamics 365 / Dataverse

While working on a Lead Disqualification scenario in Dynamics 365, we ran into a strange issue.

When trying to Disqualify a Lead, Dynamics 365 was showing the generic error message:

“Please ensure all required fields are filled out and have valid info.”

Even more confusing, the form itself was not showing any missing required fields.

This is one of those classic “ghost validation” problems in Dynamics 365 where fields are being marked as required dynamically through JavaScript or Business Rules, even though they are not visible on the form. The message was generic and did not indicate which field was causing the validation failure.

Since the form was not highlighting any required fields, we suspected that some fields were being set as required dynamically in the background. To identify them, we executed the following JavaScript in the browser console.

(function () {
    var missingFields = [];
    var attributes = Xrm.Page.data.entity.attributes.get();
    attributes.forEach(function (attribute) {
        var requiredLevel = attribute.getRequiredLevel();
        var value = attribute.getValue();
        var isEmpty =
            value === null ||
            value === "" ||
            (Array.isArray(value) && value.length === 0);
        if (requiredLevel === "required" && isEmpty) {
            missingFields.push(attribute.getName());
        }
    });
    if (missingFields.length) {
        alert("Missing required fields: " + missingFields.join(", "));
    } else {
        alert("No missing required fields found.");
    }

})();

The script immediately showed the fields that were marked as required internally:

Even though these fields were not visible on the form, they were still configured as required dynamically somewhere in the background.

To fix it – we found and updated the JavaScript method that was setting those fields as required. This is the cleaner and recommended approach.

The other quick fix though not recommended is to reset the fields to non-required on OnLoad/ OnSave/ OnChange.

formContext.getAttribute("custom_enquirytype")
    .setRequiredLevel("none");
formContext.getAttribute("custom_decisionmaker")
    .setRequiredLevel("none");

Sometimes these generic validation popups in Dynamics 365 can be a bit tricky because the actual field causing the issue is not even visible on the form.

Running a quick console script like the one above helped us immediately identify which hidden fields were still marked as required and blocking the Disqualify action

Hope it helps..

Advertisements