Recently we wrote a flow that will run daily once and will pick all the tasks due in the last 24 hours i.e. schedule end date less than equal to utcNow() and greater than equal to addDays(UTCNow(),-1)
Interestingly we observed one of the task records not picked.
The scheduled end date on the task record was – 2024-07-22T20:00:00Z
And for the flow the filter condition was –
scheduledend le 2024-07-23T20:00:35.5173871Z and scheduledend ge 2024-07-22T20:00:35.5173943Z
If we look at the date for the greater than equal condition, we can see that the seconds part is 35, the exact time when the List rows step would have run, and in case of that particular task record is 00, so it was not picked.
Then we applied the below formatDateTime function, excluding the time part.
(scheduledend le ‘@{formatDateTime(utcNow(),’yyyy-MM-dd’)}’ and scheduledend ge ‘@{formatDateTime(addDays(utcNow(),-1),’yyyy-MM-dd’)}’ and _regardingobjectid_value ne null and statecode eq 0)
Again we saw few tasks not picked,
The records that were not picked had scheduledenddate as
2024-07-25 18:00:00.000
2024-07-25 19:00:00.000
And as per new condition
scheduledend = ‘2024-07-24’ which essentially was
scheduledend = ‘2024-07-24 00:00:00.0000’
Eventually we updated the flow’s Filter Rows condition to include only the hour and minutes, ignoring the seconds/milliseconds because of which we got the issue in the first place.
(scheduledend le ‘@{formatDateTime(utcNow(),’yyyy-MM-dd HH:mm’)}’ and scheduledend ge ‘@{formatDateTime(addDays(utcNow(),-1),’yyyy-MM-dd HH:mm’)}’ and _regardingobjectid_value ne null and statecode eq 0)
One more example for more clarity –
Below we are creating a contact record and setting values for 3 date time fields, UTC1, UTC2, UTC3.
Let us first use the data spawner component to generate sample data for a custom table for which we just have 2 new custom first name and last name fields created as well as mapped with 100 K records.
Let us first run the Package with batch size = 1000, threads as 20, multiplexing user = 5, and homogeneous batch operation disabled.
Below is the User Multiplexing option in the Connection Manger.
Here we have defined 5 different application users.
Now let us run the same with the Homogenous Batch Operation option checked.
Below are the findings with different variations of Batch Size, Threads, Multiplexing Users, Homogenous Batch Operation for the – 100K records – Custom table
Batch Size
Threads
Multiplexing Users
Homogenous Batch Operation
Duration (minutes)
1000
20
5
N
5:48
1000
20
5
Y
1:54
500
20
5
N
4:16
500
20
5
Y
1:29
250
20
5
N
3:58
250
20
5
Y
1:38
100
20
5
N
4:47
100
20
5
Y
1:58
500
50
5
N
4:00
500
50
5
Y
1:24
We can see huge performance improvements while using Bulk Operations (Homogenous batch option) for our custom table, with threads around 20 and multiplexing users as 5. Increasing the number of multiplexing users will provide further performance improvement here.
Now let us run it against the Contact table and this time we take 10K as a sample instead of 100K as a sample.
10K records Contact table
Batch Size
Threads
Multiplexing Users
Homogenous Batch Operation
Duration (minutes)
500
1
1
N
25:26
500
1
1
Y
42:14
100
1
1
N
24:34
100
1
1
Y
36:06
100
5
1
N
21:56
100
5
1
Y
16:45
100
10
1
N
6:59
100
10
1
Y
12:54
100
10
2
N
6:14
100
10
2
Y
11:28
100
10
5
N
3:26
100
10
5
Y
9:36
100
15
5
N
2:56
100
15
5
Y
9:57
100
20
5
N
2:34
100
20
5
Y
10:17(Ran into a server-side throttling error.)
1000
20
5
N
5:30 (Ran into a server-side throttling error.)
1000
20
5
Y
5:02 (Ran into a server-side throttling error.)
500
20
5
N
4:20(Ran into a server-side throttling error.)
500
20
5
Y
2:36 (Ran into a server-side throttling error.)
100
20
1
N
18:00
(Ran into a server-side throttling error.)
100
20
1
Y
11:20
(Ran into a server-side throttling error.)
With the higher Batch size along Threads + Multiplexing users + Homogenous Batch Operation message option, we could get a good performance improvement, however, we can see that we ran into server-side throttling errors on increasing the batch size. So with tables having a higher number of fields/relationships, we need to be more careful than a custom/table with fewer relationships and fields
[CDS Destination] Warning: An exception has occurred while processing the service request, the same request will be attempted again immediately. KingswaySoft.IntegrationToolkit.DynamicsCrm.WebAPI.WebApiServiceException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. (Error Type / Reason: KeepAliveFailure, Detailed Message: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.)
[CDS Destination] Warning: A server side throttling is encountered, the same request will be retried after 5 minutes (as instructed by the returned throttling error message from the server). KingswaySoft.IntegrationToolkit.DynamicsCrm.WebAPI.WebApiServiceException: The remote server returned an error: (429) . (Error Type / Reason: 429, Detailed Message: {“error”:{“code”:”0x80072321″,”message”:”Combined execution time of incoming requests exceeded limit of 1200000 milliseconds over time window of 300 seconds. Decrease number of concurrent requests or reduce the duration of requests and try again later.”}})
Below we can see the resource cell template (or view) applied that defines the images, values, and fields displayed for the resource in the Schedule Board.
Now suppose we want to show the Account (custom field) value also, that would make it easy for the Dispatcher to schedule them from within the Schedule board.
For this, we need to select Board Settings for the Schedule Board.
Navigate to the Other section within the Board Settings.
We’d first start by adding/defining a new Resource Cell Template and a new Retrieve resource query template.
We have added below Div tag below to show the account name.
Save this new template
Next, add the below attribute tag for the account field in the Fetch XML for it to retrieve the value of the account, here name property holds the schema name of the field.
Select Save as new to add the new template.
On refreshing the Schedule Board we can see the Account value added to the view, however, it shows the Guid of the account record.
To get the label /name, edit the Resource Cell Template and add the below UFX Bag (UFX directives for querying the data) to fetch the name of the account.
Update the Sample Resource Query, Save the changes, and refresh the schedule board.
We can see the Guid replaced by the Account name there.
Nishant Rana's Weblog 