Where in project folder structure to store python dict used for data validation?

[meaningfromdata]

I have a dictionary that makes use of the pandera library to define certain checks (values of dict) for a given column name (keys of dict) for data validation nodes in my pipeline. I'm not sure where to store this dictionary within the kedro folder structure. I was thinking maybe in parameters, but it seems fairly involved compared to other parameters examples I've seen. Here's a sample of what the dict looks like but there are around 50 keys at present:

column_validation_dict = {
    "person_id" Column(int, nullable=False, required=True, coerce=False),
    "birth_date": Column(pd.Timestamp, nullable=False, required= True, coerce=True),
    "last_name": Column(str, nullable=False, required=True, coerce=True),
    "first_name": Column(str, nullable=False, required=True, coerce=True)
}

Any thoughts on where this dict should be defined and stored in the kedro folder structure? Some options:

1. In parameters.yml -- so far I have not had success with getting this to work
2. In the catalog after pickling and storing in some new-non data folder
3. Just define it in the validation_nodes.py file
4. Somewhere else?

[datajoely]

since you can't use the python Column object you have to use the pandera.io.from_yaml method.

I can't quite seem to find many examples on their docs - but this piece gives you the syntax:

https://pandera.readthedocs.io/en/stable/schema_inference.html#write-to-yaml

Perhaps to work it out you could define stuff with the pandera python API and then use the to_yaml method to work out the syntax?

[datajoely]

Beyond maintaining something that mapped keywords to functions in scope using something like a hook, I'm not actually sure this is possible in YAML.

We've been flirting with the idea of giving an option to enable 'unsafe' YAML which would be powerful, but introduce some security risk.
https://pyyaml.org/wiki/PyYAMLDocumentation

[meaningfromdata]

Thanks, was unaware of unsafe yaml. Looks interesting. I think for now I'll just define it inside a function in the validation_nodes.py file, though that feels inconsistent to me. Btw, speaking of hooks, do you know if kedro viz can visualize a hook step (node/input) in a pipeline?

[datajoely]

Today that's not available in Viz but it's a neat idea! Maybe raise a feature request in https://github.com/kedro-org/kedro-viz !

[antonymilne]

FYI I've seen the pattern of validation nodes before and I think it's ok - certainly better than pickling and including the validation specification as a dataset.

Depending on how your pipeline is structured, there's a few ways you could do it:

• a dedicated pipeline that just has validation nodes (can then run with kedro run --pipeline=validation)
• one or more validation nodes per pipeline, with the nodes tagged as validation (can then run with kedro run --tag=validation)

One advantage of the tagging approach is that you can easily add if "validation" in node.tags into a before/after_node_run hook to invoke some sort of special behaviour for these validation nodes.

[meaningfromdata]

Thanks, @AntonyMilneQB. The node tagging approach sounds very useful.