Feature Engineering for Machine Learning with DBT

[emekdahl-palmetto]

What is the main problem you are solving? What is your solution? This should help form your core thesis.

Feature engineering is a hard problem in ML that impacts model performance. Using dbt to clean data and engineer features speeds up the data-centric approach to iterative modeling.

Why should the reader care about this problem? Why is your solution the right one? This should help form your specific target audience.

The ability to accurately describe and predict the answers to business important questions (and allow stakeholders to do the same) is a clear indicator that the data team is adding value to the organization.

Can you list the steps of your solution for the reader here? This should help you form the overall narrative arc and sketch out an example use case to illustrate it.

• exploration
• iteration
  • modeling
  • testing
  • documentation
• (maybe) reproducibility
• feeding predictions back into the data warehouse

Are there any resources that helped inspire or inform your idea? (eg slack discussions, articles, external product docs, etc. -- if so please link)

Data science use cases

Standard examples of feature engineering

Are there other existing solutions that solve the problem, and if so, how is this solution better or different? If so please share any links here.

• discuss existing feature stores
• discuss what types of teams / data / company size this approach might be suited for (e.g. small to midsize teams / data with few low latency requirements)

[gwenwindflower]

hi, emily! this is a fantastic topic -- the emergence of growing excitement in the past 6 months particularly about using dbt for ML has been really amazing and motivating to see.

and thank you for doing such a perfect job of sparking this Discussion and covering all the sections thoroughly - we truly appreciate that!

my initial sense is that this is a big and important topic -- so i want to drill into it a bit further so we can get a handhold on where to start digging and crafting some shared knowledge around this together. to do that, i'm curious if you could answer these two (somewhat interconnected) questions:

• is there a narrative from your direct experience that you feel like illustrates some or all aspects of this really well? one foundational principle of our blog is to ensure that practitioners stories and experiences are getting highlighted vs more abstract overviews.
• alternatively, is there a direct technical aspect from the above that you'd like to highlight? do you feel like you've landed on a recommendable process for, say, iterating on models that serve a feature store, and have examples and code you'd like to share for that? we can also start breaking this down from a more technical rather than narrative standpoint and build this up that way!

let me know which of these resonates more with what you want to build!

[emekdahl-palmetto]

@gwenwindflower I was hoping to do a post on how feature engineering works in practice using dbt, dbt_ml_preprocessing, and fal with some code snippets. How does that sound to you?

[gwenwindflower]

heck yea - that sounds awesome!

i think that would be super valuable either way, in terms of output -- whether being what we call a Guide (an iteratively updated, evergreen set of technical steps and best practices) or a Blog post (a more narrative exploration of a topic with a more distinct voice, but still technical with illustrative code examples etc) i think this one really just depends on your preference.

if you want to take it in a blog post direction the first step would be to spike out a basic outline -- just headings and any points you know you want to touch on within those -- within Notion, Google Docs, or FigJam (if you prefer a more visual style). we can then upgrade to an issue and iterate and expand from there, and once we have a collaboratively built up outline together you'll be good to write a draft!

a Guide would be a little less upfront work: that would just involve spiking out the steps and code snippets in the process here, and when we feel good about it we can upgrade it to an issue and get it live. from there, you and others would be able to continue pushing updates and expansions. 'iterative collective knowledge' is the triple buzzword of the Guides :)

if you're not sure which output resonates more that's totally fine, as either way getting this outlined together is the best next step. we definitely are very psyched to get this created and live regardless of which direction feels better for what you're envisioning!

[emekdahl-palmetto]

Thanks, @gwenwindflower! I'll start drafting the outline. I will target early May for delivery. I want to put some thoughts down and then get feedback from a few community members.

[gwenwindflower]

fantastic! please feel free to share the outline here (with limited permissions) whenever you feel comfortable and loop any of us or other community members in for feedback whenever!

[callum-mcdata]

👋 hey @emekdahl-palmetto ! Hope you're doing well!

I figured I'd check back in on the status of the outline that you were working on. I was going through the discussion posts and when I saw this one I got pretty excited about the possibility. As someone that has not yet delved into the world of ML, doing so through the lens of dbt feels like the entry point that many people like myself are looking for.

If you'd like any help drafting it up or walking through it, more than happy to pair on it!

[lostmygithubaccount]

also interested 👀