Net-new test configs

[jtcohen6]

Describe the feature

Let's add configs for:

• where
• limit
• warn_if, error_if (picking up from Threshold argument for Test Severity #2219 + Expect Failure Mode for Tests #2982)

Following the work in #3257, these will be configurable from config() + dbt_project.yml + .yml instantiation (for generic tests).

Mechanism

1. Add to TestConfig:
   https://github.com/fishtown-analytics/dbt/blob/7203123d56e42007d569a03c76bea0f351fb1611/core/dbt/contracts/graph/model_config.py#L437-L440

2. Add to schema test builder (along with methods lower down):
   https://github.com/fishtown-analytics/dbt/blob/7203123d56e42007d569a03c76bea0f351fb1611/core/dbt/parser/schema_test_builders.py#L187-L188

3. Add to shortcut method for "fast" schema tests:
   https://github.com/fishtown-analytics/dbt/blob/7203123d56e42007d569a03c76bea0f351fb1611/core/dbt/parser/schemas.py#L490-L491

[prratek]

Not sure if this issue is the right place for this but @jtcohen6 what do you think of supporting where at the model level? Use case:

With high-volume date-partitioned tables (especially for web traffic data), we often want to be able to test only the most recent partition each time. Testing the entire table each time would be too expensive and adding a where to each test individually doesn't seem particularly clean. I'd love to be able to just do something like this

models:
  - name: some_table
    test_where: "date_col = date_sub(current_date, interval 1 day)"

columns:
  ...

[jtcohen6]

@prratek Such a good thought. I was also thinking about this in light of whether, for models with the require_partition_filter config set (new in v0.20), dbt should "automatically" set the where config for tests.

I'm thinking about this in a few ways:

1. A workaround that's possible today
2. A better (but still verbose) way possible in v0.20.0
3. How this could be better in the future

Today, it's be possible to define an ephemeral model that wraps ref('some_table') in a where condition, and then define tests on that ephemeral model instead. This doesn't feel like "first-class" support, but it's totally plausible and accomplishes what you're after.

In v0.20.0, it will be possible to define test configs in dbt_project.yml. Unfortunately, test FQNs are pretty silly today, so in practice this looks like:

tests:
  my_project:
    schema_test:
      exact_name_of_schema_test:
        +where: "date_col = date_sub(current_date, interval 1 day)"
      exact_name_of_other_schema_test:
        +where: "date_col = date_sub(current_date, interval 1 day)"

We could improve test FQNs (#3259) so that they include the name and path of the model they're defined on, allowing them to instead work like:

tests:
  my_project:
    subdirectory:
      some_table:
        +where: "date_col = date_sub(current_date, interval 1 day)"

Finally, in a post-#2401 world (reconciliation of node configs and resource properties), we may be able to get quite close to the proposal you make above. I'm interested in tests being able to "inherit" configs from the parent models they're defined on (#3256), so that you could define a where config as a model-level property, and have those be inherited by the tests beneath it. How would this work in practice, given that the implementation in #3336 adds where only as a test config, not as a model config? I'm not so sure.

[prratek]

This all sounds really great, particularly the improvements to test FQNs and what that enables! My only thought re: potentially inheriting this from model config is that the where doesn't intuitively feel like model config. What do you think? Being able to specify a where clause for tests by model name/path on the other hand makes sense to me.