Open Sourcing Polaris Viz

[nickpresta]

Overview

This discussion is about open sourcing Polaris Viz in Q4 2021, based on our initial Polaris Viz Engineering Roadmap.

I have never run a large-scale open source project before but I have contributed to many, and I'm a user of many more. This discussion contains my thoughts, concerns, and proposals for moving Polaris Viz into an open source model.

Context

In our roadmap, we've outlined the following tasks that need to be complete before we can open source:

• Ensuring our repository (this one!) and license meet Shopify's Open source requirements.
• Having a permanent place for documentation and code samples. This could be Storybook hosted somewhere, or a microsite, or adding onto Alpha Polaris/Polaris.
• Implementing some additional features, such as print support in Safari and Firefox.
• Upgrading our foundational dependencies (e.g. d3)
• Having a better build system in place, including features like tree-shaking, and reducing our overall package size.

We have also identified some ongoing work that is important we do regularly and is very important for an open source project:

• Fixing security issues and updating dependencies with security issues.
• Responding to issues and questions opened by other teams.
• Supporting teams as they use Polaris Viz.

Defining Open Source

When we talk about open source, what do we really mean? For example, do we want to open Polaris Viz to outside contributors, who can open issues, write code, influence the direction of the project and the work that gets done?

What are our objectives in open sourcing Polaris Viz? Is it primarily a marketing play? Supporting our Partners? Having others write code to save us time and effort? I think it's important we articulate why we want to open source the project and for what benefit to Shopify, to our team, to the tech industry, etc.

Concerns

The burden of being an open source maintainer is pretty well documented. For example:

• https://nolanlawson.com/2017/03/05/what-it-feels-like-to-be-an-open-source-maintainer/
• http://antirez.com/news/129
• https://caddy.community/t/the-realities-of-being-a-foss-maintainer/2728

In concrete terms, we have to be ready to deal with:

• Many people using and looking at our code and wanting to contribute changes that don't always align with our vision.
• Providing a short and long-term roadmap for our users so they understand how Polaris Viz will evolve over time.
• Spending time triaging issues, reviewing code, and releasing changes in a timely manner. For example, Polaris has 200 issues and over 70 pull requests active. Someone has to triage those issues, review that code, communicate with people and work through issues that are specific to their environment, request reproducible code examples, etc.
• Changes that are specific to Shopify may not always mesh well with everyone who isn't at Shopify. Do we prioritize our needs over the community? What about Partners? How do we make it clear how we decide?
• Code of conduct and other community-management tasks. Do we have open discussions? A tag on Stack Overflow? How do we ensure interacts are free from harassment? For example, Polaris' Code of Conduct.
• We're competing in a crowded space. There are lots projects that offer high-quality visualizations (e.g. Highcharts, Chart.js, Chartist.js, Plotly) and document comparisons between them and Polaris Viz, migration guides, etc.

There are best practices we can follow, and we can lean on the experience of those at Shopify who maintain popular projects (e.g. Polaris) to guide us.

We also have the benefit of working at Shopify and all the support that comes with it -- we're paid to work on Polaris Viz, we have teams who use Polaris Viz and offer feedback and help, and our network of Engineers, Designers, Content Writers, etc to help.

Proposal

I firmly believe we can make Polaris Viz a great open source library that the entire world can benefit from -- especially Shopify and our Partners (and ultimately our Merchants). Given our current team size and experience with running an open source project, I propose that we open up Polaris Viz in a staged manner:

• Tackle the "pre-work" that needs to happen before we open source.
• Continue publishing our package to npm.
  • Technically, anyone can use Polaris Viz right now if they know where to look and want to read the code for how to use it.
• Open up the Polaris Viz project to a select few Partners who are designing visualization apps. Help them migrate the existing visualizations they have, let them ask questions, file GitHub issues, use our Storybook, etc. We can control access to this project via GitHub permissions.
  • The idea here is to work with a manageable number of Partners for a few weeks and learn from their experiences. For example, were they able to get started with our current documentation? Figure out how to migrate from their existing library to Polaris Viz? Do they understand how they could contribute code back to Polaris Viz if they found an issue?
• In parallel, build out the "marketing" site for the project, ensure our code samples are accurate and run on something like Code Sandbox. This ensures we're putting our best foot forward when we open up to the world.
• Once we're satisfied that we have a high-quality library for non-Shopifolk, open up Polaris Viz and start marketing it.
  • Posting on Hacker News, Twitter, Reddit, JavaScript/Data Viz newsletters, etc.

At this point, we are a fully open sourced project that has gone through a round of integration with some Partners before we've opened up to the entire world.

What Next?

If any of these points resonated with you -- either because you agree, or disagree, please leave a comment below. In the words of Shigeru Miyamoto:

A delayed game open source project is eventually good, but a rushed game open source project is forever bad

I think that taking a staged approach to open sourcing Polaris Viz will ultimately result in a better project overall, and a much better Partner/community reception.

Resources

I'm trying to dig up as many resources on how we open sourced Polaris, since it will help us understand some of the things we will need to do for Polaris Viz. Feel free to edit and add to this section.

• https://www.shopify.ca/partners/blog/how-to-get-the-most-out-of-polaris-shopify-s-new-design-system

[carysmills]

Thanks for covering all these points with so much detail, @nickpresta!

All the points resonate with me, so just here to ➕1.

What this really suggests to me is that we need to create a thorough plan around what open sourcing looks like and exactly what the steps are to get there and make sure we are aligned as a team, in terms of the steps and resourcing required.

I think the idea of reaching out to some select partners first is a great idea. Perhaps we could start with those who have asked Polaris about chart components in the past. Some of us reached out to a few of them a couple of years ago for a Hack Days project.

[krystalcampioni]

Thanks for getting this conversation going, @nickpresta!

Personally, I do have experience in both contributing to and maintaining popular open source projects, so I can 100% relate to the pain points described.

That being said, I'd like to raise a couple things that are important for us to consider:

• Open sourcing PV is for sure marketing play: It will assist our ambitious efforts in attracting talent, and showcasing the type of work we do at Shopify to the engineering community.
• I agree we're competing in a crowded space, but we've also seen that each of these available libraries always had some major pain point that would make us develop our own solutions, thus the need to create Polaris Viz in the first place. The same way that there are many UI components libraries out there ( I'd imagine even more than data viz libraries ) and Shopify decided Polaris was worth the investment as it represents what we consider to be the ideal approach for UI components, Polaris Viz should be the same. We're defining together what Data Viz components should look like in a Shopify context, and open sourcing these decisions could:
  • Assist our partners in creating experiences based on what we consider best practices with low effort and are consistent with Shopify's context
  • Be a learning opportunities for us, as having external people using our project is a great way to see blindspots and make our components even better

➕ 1 to sharing it with some selected partners first. It would be a great opportunity to test drive the library!