Set up Scarf to collect download metrics

[GeorgLink]

I propose to set up Scarf to collect download metrics on GrimoireLab. This kind of metrics are not yet part of CHAOSS but are becoming more accepted across open source. It is done in a privacy-preserving way. Apache and the Linux Foundation have partnered with Scarf, so we are in good company.

For transparency, this has been discussed within the CHAOSS Board. We want to learn more about Scarf and how it can be used.

[sduenas]

I think this is something we need to discuss in the open. I would like to know the opinion of other members of GrimoireLab community. Pro and cons of using Scarf.

In my opinion, Scarf sounds like a good opportunity to know more about the users and the community of GrimoireLab. Plus it could be a good option to integrate in the future with our toolset.

The problems that I see using Scarf are to change the way people will have to download the docker images and packages. Of course, this will be optional. People will still being able to download everything from the main services - scarf is only a bridge/proxy - but, to be accurate with the stats, we should ask for the collaboration of users to download from Scarf and not somewhere else.

If users start downloading using Scarf, we should also provide ways that guarantee that what they are downloading are the right artifacts and not something malicious. We already sign the docker images but we have to provide ways to check if the signature is correct. We don't do this with Python packages or tarballs, so it we'll have to work on that too.

Besides this extra work, I think it will be good to add it but I would like to know other people opinion.

[GeorgLink]

The extra work sounds worthwhile, regardless of whether we adopt Scarf. How much time do you think this will require?

[GeorgLink]

I'd like to provide the reasons for collecting data on the use of GrimoireLab.

Currently, we have no measure for the usage. It is always awesome when we hear from someone that they tried GrimoireLab or successfully started using it. We started the ADOPTERS.md to capture voluntary disclosure and get a better understanding of how people use GrimoireLab. This is not giving us an understanding of how popular GrimoireLab really is or which versions are being downloaded.

I proposed to start using Scarf because they partnered with the Linux Foundation to provide this kind of insights to LF projects. Scarf is also used by Apache projects.

Scarf does more than download metrics that tell us which versions are in use. Scarf also has tracking pixels that we can embed in the documentation to learn more about how the documentation is used. Scarf also allows collecting telemetry from running GrimoireLab instances; I wouldn't start with this, but that could be an option in the future to get a more accurate picture of usage. Ultimately, this would help prioritize efforts in the project as we get immediate feedback on improvements we make, for example as we simplify installing and getting started.

I propose to start using Scarf in two ways:

1. Scarf Gateway [1], which is a proxy in front of PyPi and Dockerhub. It will not be mandatory to go through this but we want to encourage people to use it.
2. Tracking pixel [2] on documentation and website. This is similar to how Google Analytics works.

While Scarf does not store any personally identifying information, it does send various bits of IP-derived information as outlined here [1, 2]. This data should be made as transparent as possible by granting dashboard access to the GrimoireLab Maintainers, CHAOSS Board, and any other relevant means of sharing/surfacing it that we encounter (Community Meetings, Slack, etc).

[1] https://docs.scarf.sh/gateway/
[2] https://docs.scarf.sh/web-traffic/

[oleg-nenashev]

Sounds like a good plan. I evaluated Scarf for one of the projects a year ago. It looks pretty solid. Unless there are plans to have something with similar functionality in CHAOSS, definitely+1

[GeorgLink]

There are no plans to replicate the functionality of scarf in any CHAOSS tool.

[arjundevarajan]

Appreciate the appreciation, @oleg-nenashev !

[jjmerchante]

Scarf could be integrated into various parts of GrimoireLab to provide a comprehensive overview of the community.

From a technical perspective, for users to leverage the images or packages that utilize Scarf, they would need to trust and use it in a specific way. For example, instead of using the command:

docker pull grimoirelab/grimoirelab

They would need to switch to something like:

docker pull scarf-admin.docker.scarf.sh/grimoirelab/grimoirelab

The domain is something that can be configured. Normally, users work with a Docker Compose file, so this change shouldn't be too demanding in this case.

For package management, the process is similar to Docker's case. With pip install, users would need to add the following option:

--extra-index-url https://my-python-project-domain.com

For tracking in README files and documentation, it's fairly straightforward to add a tracking pixel that renders with the text. However, this cannot be done in GitHub README files because GitHub replaces links with their own (more information: https://docs.scarf.sh/web-traffic/),
but can be done in the GrimoireLab tutorial or other places.

I believe Scarf can be useful, but it’s essential that when installing packages or running the images, the end user "collaborates"; otherwise, the results may be biased.

[sduenas]

So, I don't see much concern or opposition about it, so I think we can go ahead with this.

@GeorgLink @arjundevarajan how can we sign up for the service? I'd like the domain to be grimoirelab.org.

[GeorgLink]

I've signed up to Scarf. We need to configure the domain and start using the proxy URLs for downloads. Let's sync up directly and have a working session to do this.