Dataset request submit failing for longer tagId's

[kiranharidas187]

@manjudr We are trying to create a dataset submit request using the following curl and its giving an error

Curl from portal

curl --location --request POST 'https://staging.sunbirded.org/report/request/submit' \ --header 'Accept: application/json, text/plain, */*' \ --header 'Accept-Language: en-GB,en;q=0.9' \ --header 'Connection: keep-alive' \ --header 'Content-Type: application/json' \ --header 'Cookie: connect.sid=s%3AwYHoVlLa6kUx4wRe8kQJ_j01zmifoQvH.6fFwEM5QYzuqJI6m9%2B6XI%2FCTnFE8DSbGgUjRcZPqNBY' \ --header 'DNT: 1' \ --header 'Origin: https://staging.sunbirded.org/' \ --header 'Referer: https://staging.sunbirded.org/program' \ --header 'Sec-Fetch-Dest: empty' \ --header 'Sec-Fetch-Mode: cors' \ --header 'Sec-Fetch-Site: same-origin' \ --header 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36' \ --header 'sec-ch-ua: "Google Chrome";v="105", "Not)A;Brand";v="8", "Chromium";v="105"' \ --header 'sec-ch-ua-mobile: ?0' \ --header 'sec-ch-ua-platform: "Linux"' \ --data-raw '{ "request": { "dataset": "druid-dataset", "tag": "60a245158eee6c5040d16a4a_4c4e7a7a-d44e-45cc-9319-d22d84f749bd_2f76dcf5-e43b-4f71-a3f2-c8f19e1fce03", "requestedBy": "4c4e7a7a-d44e-45cc-9319-d22d84f749bd", "datasetConfig": { "type": "ml-observation-with-rubric-status-detail-exhaust", "params": { "filters": [ { "type": "in", "dimension": "status", "values": [ "started", "completed", "inprogress" ] }, { "type": "equals", "dimension": "private_program", "value": "false" }, { "type": "equals", "dimension": "program_id", "value": "607d320de9cce45e22ce90c0" }, { "type": "equals", "dimension": "solution_id", "value": "60a245158eee6c5040d16a4a" }, { "type": "equals", "dimension": "district_externalId", "value": "2f76dcf5-e43b-4f71-a3f2-c8f19e1fce03" } ] }, "title": "Status Report" }, "output_format": "csv" } }'

Response

{ "id": "ekstep.analytics.dataset.request.submit", "ver": "1.0", "ts": "2023-03-29T11:01:11.212+00:00", "params": { "resmsgid": "62d973fe-484f-47bd-9c85-b4f573b7f0ed", "err": "ERROR", "status": "failed", "errmsg": "Ask timed out on [Actor[akka://application/user/job-service-actor#-1186517923]] after [20000 ms]. Message of type [org.ekstep.analytics.api.service.DataRequest]. A typical reason for AskTimeoutException is that the recipient actor didn't send a reply.", "client_key": null }, "responseCode": "ERROR", "result": null }

On further investigation, we found that the API is failing when the tagId param has a length of more than 78 characters. Please let us know why and where is this restriction and how can we remove this restriction. This issue is happening in production and some of our reports are getting failed.

@aks30 @5Amogh @vijiurs @SanthoshVasabhaktula

[SanthoshVasabhaktula]

@kiranharidas187 - As per your understanding, what should be the tagId limit?
Let us know why 78 characters are not sufficient for a tagId and what would be the right length.

[SanthoshVasabhaktula]

@kiranharidas187
ok - Then what should be the length? If more and more fields are being added as tag, then no limit will be sufficient. In fact, 78 itself is too high, no id shouldn't be more than 20 characters.

Why don't you do a md5Hash of all the fields to generate unique tag, which would be sufficient?

[kiranharidas187]

@SanthoshVasabhaktula This should work. But what will happen to the existing tags which are created using the old combinations? All the requests created using the old combination will be hidden from the UI after the implementation of the new logic right??

[SanthoshVasabhaktula]

@kiranharidas187 - You can figure out two approaches.

1. Run a migration script to update the DB with the new tag
2. Add logic in UI to read the dataset requests with both tags and merge on the fly.

You can choose the approach the approach that is quicker to implement

[kiranharidas187]

ya, I feel 1st approach is the permanent solution .@Shakthieshwari do you find any challenge in taking up approach 1?

[Shakthieshwari]

@kiranharidas187 where will the md5Hash logic will takes place ? In Front-end or Backend ?

@SanthoshVasabhaktula where do we write a migration script ?

Thanks

[kiranharidas187]

@Shakthieshwari For new tags, the frontend can handle the md5 hash logic but for old ones, migration should be written. But how will you be able to identify ML-specific tags from DB?

[Shakthieshwari]

@kiranharidas187 with the help of job id = druid-dataset we can figure out ML specific tags from DB. We need to implement the md5Hash logic in the backend for migration purpose

@SanthoshVasabhaktula please let us know, where do we write a migration script ?

[SanthoshVasabhaktula]

@Shakthieshwari - SB doesn't contain any data migration scripts. It is always with the instance, i.e DIK in your case. Reach out to DIK team for any info about the data migration.

Please close this thread here and do not tag any DIK folks on this thread.

@kiranharidas187 @vijiurs

[Shakthieshwari]

Sure @SanthoshVasabhaktula