Expose new plugin quality metrics to the hub

[aaronsteers]

As discussed in office hours, 2022-06-22

Replacing:

[Meltano Stats:](https://hub.meltano.com/singer/docs#meltano-usage-metrics)
Executions (Last 3 Months)
Projects (Last 3 Months)

Should we expose metrics?

• we think 'yes'

How to surface information without metrics?

• Aggregated into statuses: 'Running', 'Unknown', 'Unstable'
• Even with metrics, the 'words' are helpful for context. "Is 500 executions a lot or a little?"

Can we try to distinguish 'dev' and 'prod' environments?

Can we bias metrics for final execution being more relevant?

• It failed 5 times and then succeeded on the 6th.
• If succeeded 100 times and then has been failing for the latest 10.

Questions to answer per plugin:

• Is this stable?
• Is this maintained?
• Did this recently break?
• Are 'enough' people using this?
• How many people are using this, in relation to other plugins or other variants of this plugin?
• How much of the API is covered? (stream names and/or num streams)

Metrics

• Monthly active projects using this plugin
• Bronze/silver gold?
• Details:
• Total number of executions, failures
  • last 30 days
• "Warning: Recent instability"
• "Unique projects" is in some cases more helpful than "total executions"

Available time periods:

• Start with 7 days, then add 30
• 90?
• 365?

Should we do a "gold/silver/bronze" rating?

• What would this be based on?

Have a Meltano badge that maintainers can put on their own repos.

TODO:

• After squared metrics are built, this issue or another issue would tackle the frontend rendering of the stats.

[tayloramurphy]

@aaronsteers converted this to a discussion. We do have:

• Rebuilt plugin-specific usage metrics #570

to track updating the current metrics to be accurate and variant specific with the Snowplow data.

[rwfeather]

(pasted from slack)

I think there might be more of a conversation to be had around fetching the latest data dynamically rather than baking it into the static site. Depends on what our goals are for hub metrics going forward. Forcing this data into the static site paradigm will be a limiting factor if we want to expand on it or get handed tighter requirements for freshness.

Some possible approaches with better long-term prospects:

• The metrics.yml and audit.yml are in s3, and so it might be trivial to host those files publicly. Users visiting the site would be fetching the data files directly and they'd be getting fresh numbers as soon as the files are updated in s3
• AFAIK, these data files are being extracted from our data warehouse on an interval. It might make sense to rethink the extraction and find some way to expose the view/table as a small "hub data api". That way the metrics come directly from the source and will always reflect our current warehouse picture.

[tayloramurphy]

@rwfeather thanks for sharing. I'm generally in favor of making the Hub API more robust and useful. Right now I don't think the current API returns data about usage, just the definitions for the plugins. We'll get to the place where we want to serve more useful data but the static site solution works for now.

I'm actually okay with the delay in numbers since we're doing it daily. The increased complexity of getting fresher data quicker while a fun challenge likely wouldn't be a huge value add, so it'd be hard to prioritize it right now.