Open Metadata & Snowflake : How can we reduce the cost?

[cyclonstep]

Hello and as always thank you very much for the great tools!

Lately I've been trying to integrate our existing data in Snowflake with this tools. But I found out that this tools potentially can increasing our snowflake usage cost and make it more costly. AFAIK snowflake is charging based on the durations of the activity of the compute warehouse and based on my test before, Open metadata profiling ingestions will run many queries (some queries's execution time will be longer on big tables) for the sake of getting the profile data. So based on that, I would like to ask people's suggestions on how can I tackle this and ultimately reducing the cost as minimum as possible.

Some of the considerations that I'm thinking:

1. Create trigger-only ingestion pipeline which only affecting one database/table.
2. Reduce profile sample (how many percentage?)
3. Not running profiling at all but potentially lost a good amount data that can help us do the analysis.

Thank you very much!

[harshach]

hi @cyclonstep Thanks for starting the discussion thread

1. You can limit the profile to only your most important tables
2. You can define the profiler sample at the workflow level that applies to all the tables or you can also add config at each table to reduce the amount of data profiler queries
3. Does your snowflake tables as a partition key, if so you can set this partition key at the table level and profiler can query only the last partition instead of the overall table. So every time we run the profiler we are only looking at the last partition rather than the entire table
   cc @TeddyCr for any other ideas.

[pmbrull]

An addition to this, some tables can have dozens (and hundreds) of columns, and not all of them are going to be equally important. Including/excluding certain columns from big tables can be also another big winning point to speed up the full process

[TeddyCr]

@cyclonstep we also have some best practices in the documentation that might help you.

[TinaBleu]

The page is not available :(

[Prajwal214]

@TinaBleu You can refer to the Best Practice