Metrics Definitions

[jessepasswaters]

For BI Tools that aren't currently supported, or for general business rules that wouldn't be captured somewhere, it would be great to have an area to document how specific fields are used to provide a definition of another field. This could be used somewhere in data lineage or just as another area to view how a specific metric is defined.

Could be a simple CSV upload or a dashboard that allows for linking of already existing tables/columns that exist.

[harshach]

Hi @jessepasswaters thanks for starting the discussion around Metrics. Yes, they are on the roadmap. Here is the definition of the Metrics entity https://github.com/open-metadata/OpenMetadata/blob/main/catalog-rest-service/src/main/resources/json/schema/entity/data/metrics.json . This of course is just a starting point, we would like to see what are some of the interesting fields you would like to see we capture. For example, in the case of Superset, we can gather the queries and dimensions out of the charts and populate the Metrics and link them to the dashboard->chart as the lineage.

[sureshms]

@jessepasswaters few clarifying questions

For BI Tools that aren't currently supported, or for general business rules that wouldn't be captured somewhere, it would be great to have an area to document how specific fields are used to provide a definition of another field.

Can you add more details on this? Can you please give examples of BI Tools? There are entities from BI Tools (say reports) that become available in OpenMetadata as an entity. So not having integration with a BI Tool means we will not have a corresponding entity. This makes establishing relationships with those entities hard.

If you are talking about capturing information on how BI tools use certain fields of entities in OpenMetadata (like a table or column of a table) and being able to manually document it, (I think this is what you are talking about), what should be added in addition to markdown for description for table and column we have. Are you suggesting that we should add a mechanism to document rich usage notes that can describe how a field is used and possibly connect it to the entity (in this case a BI metrics, as an example) where such usage occurs? Such connections can then also serve as lineage. Am I understanding this right?

Could be a simple CSV upload or a dashboard that allows for linking of already existing tables/columns that exist.
Instead of editing each table/column, you are proposing a convenient CSV upload. Right?

[jessepasswaters]

Can you add more details on this? Can you please give examples of BI Tools? There are entities from BI Tools (say reports) that become available in OpenMetadata as an entity. So not having integration with a BI Tool means we will not have a corresponding entity. This makes establishing relationships with those entities hard.

MicroStrategy is a BI tool that is currently used that has around 4,000 or so metric definitions.

If you are talking about capturing information on how BI tools use certain fields of entities in OpenMetadata (like a table or column of a table) and being able to manually document it, (I think this is what you are talking about), what should be added in addition to markdown for description for table and column we have. Are you suggesting that we should add a mechanism to document rich usage notes that can describe how a field is used and possibly connect it to the entity (in this case a BI metrics, as an example) where such usage occurs? Such connections can then also serve as lineage. Am I understanding this right?

Short answer, Yes. Thinking of Business approved metrics/derivation logic, whether they are in the BI tool or not, that are utilized across multiple reports/defacto business rules.

It would be nice to have a central repo of definitions with links to those sources (to also view lineage) to describe how that specific value/column is derived or what is used to define that metric.