I want to write about adopting a modern data stack in advocacy/political orgs and nonprofits

[thebbennett]

Overview

I work at a progressive political organization, and chose to adopt the modern data stack on to my team fairly early in our life cycle. In particular, we adopted dbt in the first year of running the data team. The modern data stack has paid off for my team and for my organization, but I am in community with other Data Directors of advocacy orgs and few of them use dbt. I want to share my success story of implementing dbt and training my junior analysts to be analytics engineers with the broader progressive community.

The problem

I want to write about why other organizations should adopt the modern data stack, but I also want to be mindful of the real challenges the progressive tech ecosystem faces. Data teams in progressive politics are often underfunded and overworked. Many data folk in this field are self taught, and may lack experience in git/version control, advanced SQL, test driven development, and another software engineering best practices that fuel analytics engineering. We also tend to prefer tools that are built by progressives, for progressives. Finally, much of our infrastructure is usually built in Civis, and breaking out of Civis can be a herculean efforts.

How you can help

For those of you that have made the decision to adopt dbt/a modern data stack:

• What was the "green light" that helped you say yes to this decision?
• What concerns did your data team have?
• What concerns did non-technical staff or stakeholders have?
• How long did it take you to migrate?
• Did you have to teach your data staff new skills? How did you do this?

[zaneselvans]

Since it seems quiet here, and I'd really like to see this get written... We've decided to make this transition or something like it, but it is still a work in progress.

• The 💡 for us was the unmaintainability of our existing codebase. As a small team we just can't keep accumulating new data products and processes, and integrating additional datasets without standardizing how all the different inputs and outputs and dependencies are managed, and using a uniform system for persisting the data -- and all of that seems like challenging problems that have been solved well by others, that we shouldn't try or need to solve ourselves.
• A big (technical) concern / holdup for us has been uncertainty about whether we are choosing the right tools for the jobs that we're trying to do. It'll take a lot of person-hours to learn the new tools and refactor our existing code to work with them. If we get it wrong (or at least, not good enough) it'll be a huge waste that we might not be able to recover from. Honestly this has been kind of paralyzing, and has led to continuing (and frustrating) investment in our existing homebrew systems. Should we keep using our own data extractors, since our sources are arcane and useless to most people? Or should we reorganize them within a framework like Airbyte so we can benefit from the design thinking others have done for similar tasks? Which of our transformations could we / should we do with pure SQL/dbt? Which should stay in Python? How do we marry those systems to each other? Is this unclear to us because we don't understand SQL well enough? Or because our source data is very messy and poorly integrated and needs grungy cleaning and entity resolution that benefit from Python? Will we have to move all development onto cloud resources? Or can we still do local runs and testing? Will everybody on our team need to know how to use all of these tools? Or should we split into folks that work on the backend, and folks that use the "finished" data products?

[krevitt]

@amosbudde2 curious if you'd have thoughts here / know any orgs that have gone through this transition?

[amosbudde2]

I used to work at a company that sold data technology to progressive and advocacy organizations, and saw the situation that @thebbennett mentions. I left that world a little over 2 years ago, and the ecosystem has changed a lot since then, so I never got to see a political or advocacy organization go through the transition to newer solutions.

I could be wrong about this, but I think the biggest barrier is probably having a good data replication solution for the primary sources of advocacy and campaign data - just like @zaneselvans is mentioning. If you could reliably get data into your cloud data warehouse of choice, it unlocks all the other tooling.