Replies: 13 comments 42 replies
-
One small point on needing to allow custom aggregations for metrics and definitions for dimensions, as people do all kinds of strange stuff here which can’t even be covered by composing metrics from others. |
Beta Was this translation helpful? Give feedback.
-
THIS IS INCREDIBLE!!! Thank you all for putting the spec together, I can't wait to get my hands on live product. For the spec:
|
Beta Was this translation helpful? Give feedback.
-
are filters being left out in this new spec? I find them to be incredibly useful in |
Beta Was this translation helpful? Give feedback.
-
Looks great! Two comments:
Several ways of doing something can generate problems when several people collaborate in the same project. The first is simpler, but I don't have a strong preference for any of the two.
|
Beta Was this translation helpful? Give feedback.
-
👋 @nhandel Currently there are 2 libraries, From my understanding, Will I saw that Finally, what will be missing from the |
Beta Was this translation helpful? Give feedback.
-
Considering the extensive use of S3 as data storage and partition keys being the key to performant queries, do you see a way that partition keys could also be part of semantic model definitions? I imagine these could then be passed from model to measure to metric to allow users to limit their metric queries by the appropriate partition keys. I guess there are 2 distinct examples: time based partition keys and categorical partition keys. Time based partition key example:
Categorical partition key example:
Not sure if these are perfect examples, but I hope you understand the direction of thought. Have you considered any of these questions? Would explicitly defining and showing partition keys help users call the metrics in a better way? |
Beta Was this translation helpful? Give feedback.
-
dimensions This time around, do you really mean time or is it still going to be date? |
Beta Was this translation helpful? Give feedback.
-
I feel like we could do with some kind of diagram showing the possible objects in metricflow, semantic-interfaces and dbt-core showing which things are in where and how they relate to each other. |
Beta Was this translation helpful? Give feedback.
-
Cool! Overall, this feels like a worthwhile transition that will be short term painful for a much better long term implementation! Excited to see it come together! A few questions...
|
Beta Was this translation helpful? Give feedback.
-
Nice! Very happy to see this. At first glance this addresses the limitations/issues we saw in the previous implementation, namely:
Some notes/questions on things I'd like to see:
|
Beta Was this translation helpful? Give feedback.
-
As an update to this issue, we now have a very early version of our third repo with the semantic interfaces here. Let us know what you think! Through building this, we’ve updated a few elements of the spec. Specifically:
This is also now reflected in the examples in the top post! Nick |
Beta Was this translation helpful? Give feedback.
-
How will it be possible to use that with PowerBI? dbt_metrics could just end up being a VIEW that would be easy to query. |
Beta Was this translation helpful? Give feedback.
-
Hi @nhandel . Thanks. |
Beta Was this translation helpful? Give feedback.
-
Summary & Background
It's finally here! Community members have been asking us about the future of the semantic layer and what it will look like since we announced the acquisition two months ago. Since then, we've integrated teams, had an offsite to share context and, and spent a few weeks polishing up what we'll be working on for the foreseeable future. In that vein, this discussion will contain the proposed specs for the new dbt & MetricFlow semantic objects. Before we move any further, these are only proposed specs. They are not finalized - the broad strokes will most likely remain the same, but we may still change them based on community feedback.
Proposal
These specs will not live in
dbt-core
orMetricFlow
- both libraries will depend on these objects, so we plan to create a third repository,dbt-semantic-interfaces
containing these protocols. Both libraries will import that repo.New Objects
semantic_models
Reason: The Semantic Model will be the new foundational building block upon which the Semantic Layer will rest. It contains many object types (such as entities, measures, & dimensions ) that allow MetricFlow to construct the queries for metric definitions.
Location:
third-repo
Description:
See properties below
YML Properties
Property Names
entities
Description: Entities are the join key columns in your semantic model that can be used to join to other semantic models. For example, the
user_id
entity defined in a semantic model could be aprimary
type which means that each row in that object represents the entity of auser
. Or it could beforeign
, which means that it points to an entity of auser
in another semantic model.Callout: This is not a new node type as was previously proposed in but a property that lives inside the
semantic_model
. It corresponds to a key or identifier inside the logical model and represents a single instance of the entity in question. This is the basis of how MetricFlow handles joins.See properties below
YML Properties
Property Names
measures
Description: A measure is an aggregation that can result in either your final metric, and/or serve as the foundational building blocks for a metric. For those familiar with dbt_metrics, this takes the place of some of the information stored in
expression
andcalculation_method
.Callout : The aggregation types that were previously defined in the
calculation_method
are now defined in theagg
property of the measure. The supported aggregation types will be defined inside MetricFlow. This means that the previously supported behavior of overwriting macros to use custom aggregation types will no longer be supported.See properties below
YML Properties
Property Names
dimensions
Description: A dimension is a categorical or time field that can be used to group or filter metric(s). Similar to the dbt metric dimension property except, it now exists as a reusable component of the semantic model with additional metadata.
See properties below
YML Properties
Property Names
Altered Objects
metrics
Description: A metric! Same as before but with new properties.
Callout: This is a breaking change to the metric spec. Metrics will also only exist on top of the new Semantic Models object instead of sql/python models.
Reason: Much of the information currently living inside the metric definition will move to different locations. Some of it will go to the semantic model and some to the component measure(s) it depends on.
See properties below
YML Properties
Property Names
Example
This yaml contains:
customers
andtransactions
)entities
(transaction
andcustomer
)metrics
(revenue
andsales_minus_revenue
)FAQ
semantic_models
andentities
different things? It seems like they are similar.semantic_model
that does not have a primary entity and only contains relationships between foreign entities. By maintaining a difference between semantic models and entities, we can correctly represent semantic_models as containing 1:M connections to entities.Beta Was this translation helpful? Give feedback.
All reactions