fix(thumbnail cache): Enabling force parameter on screenshot/thumbnail cache

[fisjac]

SUMMARY

Both the /api/v1/chart/{{id}}/cache_screenshot/ and /api/v1/dashboard/{{id}}/cache_dashboard_screenshot/ endpoints enable generating and retrieve a screenshot from the respective resource.

The endpoints accept sending force=true in the query parameters, but currently this parameter does nothing. It should instead invalidate the previous image and generate a new one.

This PR introduces a fix as well as some improvements to the functioning of the thumbnail cache, such that the status of the screenshot can be tracked by the relevant api endpoints as well.

Rather than storing plain bytes objects in the cache, this PR introduces a new class ScreenshotCachePayoad which can store the image bytes as well as timestamp and status values. Status values include
Pending: screenshot is awaiting being processed
Computing: screenshot job has been picked up by the worker and is processing
Updated: screenshot has been updated
Error: an error occured while the webdriver was processing the screenshot task.

While in a Computing or Updated state, requests to regenerate the screenshot will be blocked unless the force parameter is set to True

Additional update_status and updated_at fields have been added to 202 response payload for endpoints as follows:

{
  "cache_key": "ef13dd170eda0581d74a5d1c84983778",
  "dashboard_url": "http://superset_app:8088/superset/dashboard/15/",
  "image_url": "http://superset_app:8088/api/v1/dashboard/15/thumbnail/ef13dd170eda0581d74a5d1c84983778/",
  "update_status": "Pending",
  "updated_at": "2025/01/16-14:58:34"
}

How to reproduce the bug
Create a chart.
Perform a modification in the data.
Send a GET request to /api/v1/chart/{{id}}/cache_screenshot/?q=(force:!t).
Access the URL returned in the image_url value.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

tests have been added in the tests/unit_tests/utils/screenshot_tests.py

For manual testing

• ping the /api/v1/chart/<pk>/cache_screenshot/ endpoint. Ensure that a 202 response is received.
• refresh periodically until status changes to "Computing." Then again to "Updated"
• once updated, ensure the screenshot url returns the proper screenshot.
• refresh again, and ensure the status has not changed from "Updated"
• ping the endpoint again with force parameter /api/v1/chart/cache_screenshot/<pk>/?q=(force:!t) to ensure that the status changes to "Pending"

Repeat the steps for the POST /api/v1/dashboard/<pk>/cache_dashboard_screenshot/

• access the GET /api/v1/chart/<pk>/thumbnail/<digest> endpoint to ensure that a 202 response is received.
• access again once the status changes to "Updated" ensure that the endpoint results in a 200 response with the screenshot.
• repeat with GET /api/v1/dashboard/<pk>/thumbnail/<digest>

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags: THUMBNAILS, ENABLE_DASHBOARD_SCREENSHOT_ENDPOINTS
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[korbit-ai]

Based on your review schedule, I'll hold off on reviewing this PR until it's marked as ready for review. If you'd like me to take a look now, comment /korbit-review.

Your admin can change your review schedule in the Korbit Console

[codecov]

Codecov Report

Attention: Patch coverage is 84.71338% with 24 lines in your changes missing coverage. Please review.

Project coverage is 83.55%. Comparing base (76d897e) to head (a63c5ec).
Report is 1331 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/charts/api.py	50.00%	12 Missing ⚠️
superset/utils/webdriver.py	12.50%	7 Missing ⚠️
superset/utils/screenshots.py	93.58%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #31757       +/-   ##
===========================================
+ Coverage   60.48%   83.55%   +23.06%     
===========================================
  Files        1931      540     -1391     
  Lines       76236    39205    -37031     
  Branches     8568        0     -8568     
===========================================
- Hits        46114    32756    -13358     
+ Misses      28017     6449    -21568     
+ Partials     2105        0     -2105     

Flag	Coverage Δ
hive	48.76% <28.66%> (-0.41%)	⬇️
javascript	?
mysql	75.99% <61.14%> (?)
postgres	76.05% <61.14%> (?)
presto	53.26% <28.66%> (-0.54%)	⬇️
python	83.55% <84.71%> (+20.06%)	⬆️
sqlite	75.53% <61.14%> (?)
unit	61.15% <56.68%> (+3.52%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[korbit-ai]

Korbit doesn't automatically review large (500+ lines changed) pull requests such as this one. If you want me to review anyway, use /korbit-review.