Using rollup_ml_datasource we have a issue creating charts in Hawkeye in Staging environment

[Ashwiniev95]

Hi Team,

As part of 5.0.0 release we have a story https://project-sunbird.atlassian.net/browse/SB-23100 created.

So we need to create reports on newly created 3 rollup data sources correspondingly specs have been pushed to git and it is merged.
PR: https://github.com/project-sunbird/sunbird-data-pipeline/pull/1578. These 3 supervisor specs are submitted in staging environment and the data sources are created as an when the data was pushed to kafka.

But we are facing the issue while creating the charts to be rendered on HawKeye Portal. Here are the steps that we followed to create charts.

1. Created the chart using submit api, sharing the curl here
   curl --location --request POST 'https://staging.sunbirded.org//api/data/v1/report/jobs/submit' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJQbEhoYXhnNFR5bEU0cW53WUp0dUs5TUxOYjBhSHFERSJ9.ZIk-J6IrTaEBSRVHhH6u0rRSkB1DXZK3LHLlmVHguUY' \ --data-raw '{ "request": { "reportId": "ml_no_of_users_test3", "config": { "streamQuery": true, "reportConfig": { "id": "ml_no_of_users_test3", "mergeConfig": { "postContainer": "reports", "container": "reports", "basePath": "/mount/data/analytics/tmp", "reportPath": "ml_no_of_users_test3.csv", "rollup": 0 }, "labels": { "COUNT_DISTINCT(user_id)": "total_content_plays_on_portal", "legend": "No of users", "total_content_plays_on_portal": "No of users", "parent_channel": "parent_channel", "district_name": "District Name", "state_name": "State Name" }, "dateRange": { "interval": { "startDate": "2020-01-01", "endDate": "2022-12-31" }, "granularity": "all", "intervalSlider": 0 }, "metrics": [ { "metric": "total_content_plays_on_portal", "label": "total_content_plays_on_portal", "druidQuery": { "dataSource": "ml-survey-status-rollup", "limitSpec": { "type": "default", "limit": 50000, "columns": [ { "dimension": "COUNT_DISTINCT(user_id)", "direction": "descending" } ] }, "filters": [], "granularity": "all", "dimensions": [ { "fieldName": "state_name", "aliasName": "State Name" }, { "fieldName": "parent_channel", "aliasName": "parent_channel" }, { "fieldName": "district_name", "aliasName": "District Name" } ], "aggregations": [ { "fieldName": "user_id", "type": "cardinality", "name": "total_content_plays_on_portal" } ], "queryType": "groupBy" } } ], "output": [ { "type": "csv", "metrics": [ "total_content_plays_on_portal" ], "dims": [ "date" ], "fileParameters": [ "id", "dims" ] } ], "queryType": "groupBy" }, "container": "reports", "store": "azure", "key": "hawk-eye/" }, "createdBy": "fca2925f-1eee-4654-9177-fece3fd6afc9", "description": "Big Number", "reportSchedule": "Daily" } }'
   response was 200 success

2. executed this job in jenkins
   https://10.20.0.14/jenkins/job/Deploy/job/staging/job/DataPipeline/job/RunReportjob/

3. But data was not generated in Azure, Container Name : reports and here are the paths

   hawk-eye/ml_no_of_users_test3/ -> folder
   apekx/ml_no_of_users_test3,json & csv -> files

4. When checked in Logs using this job : https://10.20.0.14/jenkins/job/Deploy/job/staging/job/DataPipeline/job/AnalyticsFetchLogs/215/execution/node/3/ws/ansible/data-products/logs/data-products/2022-07-27-job-execution.log. We found this error

22/07/27 08:16:07 INFO KafkaProducer: [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
com.ing.wbaa.druid.client.HttpStatusException: Received response with HTTP status code 500 Internal Server Error
[ERROR] [07/27/2022 04:55:34.652] [scruid-actor-system-akka.actor.default-dispatcher-5] [akka://scruid-actor-system/system/pool-master] connection pool for Pool(shared->http://11.3.2.25:8082/) has shut down unexpectedly
java.lang.IllegalStateException: Pool shutdown unexpectedly

@sowmya-dixit @anandp504 Can you please help us out, Its a blocker for us in testing staging environment. We are waiting since a week on this for your response in slack.

-cc @vijiurs @Shakthieshwari @aks30 @kiranharidas187 @snehangsude

Thanks

[sowmya-dixit]

@Ashwiniev95
We see there is empty filter "filters": [] part added in the query which was causing issue when Druid was trying to construct the query, below is the error from Druid service when report was executed -

"2022-08-01T11:24:46,476 ERROR [qtp114012533-141] org.apache.druid.server.QueryResource - Exception handling request: {class=org.apache.druid.server.QueryResource, exceptionType=class com.fasterxml.jackson.databind.exc.ValueInstantiationException, exceptionMessage=Cannot construct instance of org.apache.druid.query.filter.AndDimFilter, problem: AND operator requires at least one field at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 685] (through reference chain: org.apache.druid.query.groupby.GroupByQuery["filter"]), query=unparseable query, peer=11.3.3.7} (com.fasterxml.jackson.databind.exc.ValueInstantiationException: Cannot construct instance of org.apache.druid.query.filter.AndDimFilter, problem: AND operator requires at least one field at [Source: (org.eclipse.jetty.server.HttpInputOverHTTP); line: 1, column: 685] (through reference chain: org.apache.druid.query.groupby.GroupByQuery["filter"]))

We have updated your report config by removing that filters part and re-ran the report. It has ran successfully now. Please check for report files in Azure.

[Shakthieshwari]

Thanks @sowmya-dixit .

Where did you get this error ?

we didn't find any such error in both execution and job log ?

From where do we get this error ?

Please help us out

Thanks

[Ashwiniev95]

Hi @sowmya-dixit ,

yes it worked. Thanks for the help.

But we didn't find this error anywhere, If you could help in finding these logs it would be more helpful, Since we didn't know the exact error message, we were not able to figure out what was going wrong and we was depending on you and waiting.

Thanks

[sowmya-dixit]

@Ashwiniev95 The above logs are from Druid Broker service. All the query logs can be found under broker service logs but Druid sends back only 500 Internal Server Error to user hence we were not able to figure out exact error from just Druid response.

[Ashwiniev95]

@sowmya-dixit Do we have any jenkin jobs to get the logs from Druid Broker service? or Did you get it with the help of devops.
Because it would be helpful for us in future to check if any issues in similar way.

Thanks