Cost and usage information is missing

[JoBruns]

I am missing the cost information both in the UI of Workload Discovery and in the diagrams of Graph Explorer. I made sure that the cost feature is working in my account; I set it up myself last December. The S3 buckets carrying the exported cost and usage reports as well as the crawler results are all in place and contain data. So I checked the Athena queries processing the cost information in the crawler results and found this error message:

COLUMN_NOT_FOUND: line 1:31: Column 'product_servicename' cannot be resolved or requester is not authorized to access requested resources
This query ran against the "athenacurcfn_aws_perspective_cost_and_usage" database, unless qualified by the query. Please post the error message on our forum  or contact customer support  with Query Id: 2c31540f-b860-41e1-a7ee-d3077c69c29

In fact, the column 'product_servicename' is not being exported with the AWS cost report. You can see this from the attached manifest describing the export schema.

How can I fix this issue? I set up Workload Discovery last December. Has it been fixed meanwhile in the current software version?

workload-discovery-cost-and-usage-511133000301-Manifest.json

[svozza]

This is just a guess but if you set up the cost feature in December, there was a bug in our documentation at the time that has been fixed. The fix for step ten on the cost setup page (https://docs.aws.amazon.com/solutions/latest/workload-discovery-on-aws/set-up-the-cost-feature.html) was this

Before: 10. Set the Report prefix path to workload-discovery.
After: 10. Set the Report prefix path to aws-perspective.

Can you verify that the report prefix you used says aws-perspective?

[JoBruns]

The situation has changed a bit but the error still remains. I repeated the steps in the above mentioned page to create the AWS Cost and Usage Report in the deployment account. I made sure that the report prefix reads "aws-perspective". Still, Workload Discovery shows no accumulated cost for service SageMaker in July 2024 although the Cost and Billing report does.

You will also see that the Athena query reports an error saying COLUMN_NOT_FOUND. It appears that the query does not match the schema of the queried Athena database table as it reads:

SELECT product_servicename, line_item_usage_account_id, Array_join(Array_distinct(Array_agg(product_region)), '|') AS region, pricing_term, sum(line_item_unblended_cost) AS cost, line_item_currency_code FROM perspective_glue_crawler WHERE line_item_usage_account_id IN ('511133000301') AND product_region IN ('eu-west-1','eu-west-2','eu-west-3','eu-central-1','eu-north-1','us-east-1','us-east-2','us-west-1','us-west-2','ap-south-1','ca-central-1','sa-east-1') AND product_servicename LIKE '%QuickSight%' AND line_item_usage_start_date >= TIMESTAMP '2024-06-01' AND line_item_usage_end_date <= TIMESTAMP '2024-06-30' GROUP BY  product_servicename, line_item_usage_account_id, pricing_term, line_item_currency_code HAVING sum(line_item_unblended_cost) > 0 ORDER BY cost DESC

COLUMN_NOT_FOUND: line 1:291: Column 'product_region' cannot be resolved or requester is not authorized to access requested resources
This query ran against the "athenacurcfn_aws_perspective_cost_and_usage" database, unless qualified by the query. Please post the error message on our forum  or contact customer support  with Query Id: 566aa2bf-e8f1-470c-8796-417016db3c55

Please find attached the screenshots and and Excel workbook with the S3 bucket content used by Workload Discovery.

Issue with Cost Reporting_02.xlsx

[svozza]

I'm sorry but I can't reproduce this. I copy and pasted that query you supplied into a test system and ran it in Athena and got no errors. The product_region field has been part of this query since we released the cost feature 3 years ago (you can see when it was committed here: https://github.com/aws-solutions/workload-discovery-on-aws/blame/090b6675869959de34d64b819f0380be2451499c/source/backend/functions/cost-parser/src/athenaQueryBuilder.js#L82).

On my system it exists in the Athena DB:

In the error message it says Column 'product_region' cannot be resolved or requester is not authorized to access requested resources, is there any way this could be a permission issue. Or is there some configuration in the CUR setup that would mean that the product_region field is not present in the report?

[JoBruns]

I exported the CUR as a "zipped" CSV file. It does contain the the missing columns and many more. So I have to figure out why the CUR exported for Workload Discovery is different.