modrinth
Community verifiers programme #249
Replies: 3 comments 4 replies
-
This feels like a bit of conflicting interest. If someone is attempting to upload malware, obviously they don't want verifiers to run. With that said though, I think monetization of verifiers would be hard. Sure, the community could donate a few percent to verifiers, but what is then stopping your neighbor Joe from creating a verifier that always skips, and then takes a share of that money? Additionally, I'm not too sure if a few percent here and a few percent there would make enough revenue to cover even very basic hosting for a verifier. While I focus on malware detection in this comment, you also mention,
This would need some kind of opt-in system where authors can opt-in to always have a specific verifier run the checks, not everyone would want to e.g sign their mods.
Wouldn't we want to send the request to every known verifier that i.e scans for malware, not just one? Also need to counter in what I mentioned above about opt-in.
If "danger" reports it to moderators for manual review, what does "caution" do? Might also be able to combine "skip" and "fail"? Would there be a separate status for warning the project creator? Would the project creator be able to push a new version anyways? In general, it all feels a bit over engineered to me. Wouldn't it be easier for developers wishing to build something like this, to simply pull versions pushed in the last x hours, scan them, wait x hours, repeat? Then if it found something, alert the developer, so it can be manually reviewed by the developer and reported to moderators if not a false-positive? At what point would it benefit Modrinth to build their own malware detection system, rather then build this? If it is all automated by verifiers, I also see false-positives becoming a problem. It would probably be smart to make the program be applied for / invite-only, and ask that the developers dry test their verifiers before letting it automatically send i.e "danger". With all that said, I think it's an interesting idea, and I thank you for taking the initiative. |
Beta Was this translation helpful? Give feedback.
-
I was thinking of an additional “unpaid” call-out status (since statuses are displayed publicly), which is suitable for exactly the case of disabled paid verifiers and verifiers that are paid for manually by the creators (the same way software publishers pay for certificates).
The fact that each verifier has to be vetted into the programme by Modrinth and skips won't be paid for, cause that implies that verifier didn't do anything.
It's somewhat similar to the bots on GitHub, where you can open bot's site to add it to your project, so can verifiers have their own sites where you can configure them to check your files and how to do that, as well as some other options.
Yeah, this is not exactly set in stone. Sending requests can be overwhelming and will expose Modrinth's servers. Alternative solution could be an API endpoint returning all recently published versions or websocket endpoints to which verifiers connect and get notifications through. The goal of this compared to manually fetching is to avoid overwhelming API (delivering same data easier than have verifiers bombard API with requests to retrieve data) and ensure that notifications are always received (delivery backlog), so verifiers are doing their job reliably.
No status automatically reports file to the moderators, but can serve as a potential signal if verifiers of a certain category all red (“danger”). This is more of a sign to the users and launchers. The distinction between “danger” and “caution” is that “danger” implies immediate danger and generally suggests to avoid the content, whereas caution implies that the content may be unsafe (e.g., missing signature), so you should verify it manually if you can. Reports to moderators are created with the special verifiers reporting endpoint, and show separately to the Modrinth moderators.
This I'm still thinking of, I can totally see say Mallory could upload a mod that's red across the board, and then delete it, work to obfuscate it and then publish to get it to orange or greens, and then claim that her mod was wrongly detected. One solution could be to have GitHub style of activity feed, where you can see that creators has indeed published the version, that verifiers marked it all red, and that the version has since been deleted (would be worth bringing up as a separate idea).
At no point, Modrinth has to become a sustainable company and hire security experts to produce a solid malware detection system, whereas this allows to delegate this task to the community full of people with creative ideas of detecting the malware, and other issues with mods. Some big actors like AVS companies can also join in, as this makes for a great promotion of their AV software. Of course, before someone is vetted into the programme they have to show the efficiency of their verifier or explain how exactly it works.
If majority of verifiers report a certain version, it should be put up in the moderators' review queue. Whereas single reports are generally okay to postpone. Verifiers submitting a large number of false positives will be disqualified from the programme, so yeah, they are encouraged to test it first before starting submitting reports, e.g., by verifying triggers and submitting reports themselves where it's confirmed.
Exactly how I imagine this programme be. I don't see how gaining everyone access to it will help verifiers community credibility. |
Beta Was this translation helpful? Give feedback.
-
|
If this becomes a thing, I'd like to run a verifier |
Beta Was this translation helpful? Give feedback.
-
|
I see issues with this program as a mod dev. First of all, Reviewing mods will not be sustainable for anyone, as Modrinth really doesn't pay all that much (I have ~$50 USD for almost 220K downloads, I doubt you can make a living off of ad revenue alone in mod dev) For cases like Borderless Windows (the malware, NOT borderless mining), the dev still used their machine to create the malware, so it would still appear as a "Trusted" environment to reviewers, not to mention someone who's had their machine backdoored via a RAT (Remote Access Trojan) can still compile the mod in their own environment, and upload it while their modrinth login is still valid. Modrinth has tens of thousands of mods, most of which likely get updated semi-frequently, so there would also need to be a large number of verifiers actively working in order to verify every new mod. If a mod has a large codebase and is closed-source, that means painstakingly going through each class file and looking for suspicious indicators, which can take a while. So for me, this proposal is: 1) Not financially feasible for reviewers due to them getting a small cut of an already small amount of revenue. 2) Isn't focused enough for the most likely case of malware suddenly being mass-distributed (rouge/hacked developer uploading malware from their own machine). And 3) is almost impossible to do in a decent amount of time. |
Beta Was this translation helpful? Give feedback.
-
Verifiers are meant to be automated and community-ran. I definitely don't see anyone verifying the mods manually, but if anyone wants to do so, they are free to create their dashboard that will allow them to review content and submit review statuses, but again, this is not feasible.
Your computer is not a trusted environment. This was a broad example of the verifier that checks for artefact attestation. If your computer is compromised, I don't think any malware will automatically run Gradle on your behalf, plus, you definitely should not build, sign and upload mods on your own computer. There's no point in a verifier that just checks your username.
Once again, verifiers are meant to be automated and most of them are likely to check for known patterns in decompiled sources or bytecode. It's not that big of a deal for a machine. |
Beta Was this translation helpful? Give feedback.
-
I'm aware, but if a hacker gets full control of your system, your browser probably has your github and modrinth account still logged in, which can be controlled without root access. It only takes one RCE exploit or a dodgy executable to do basically anything on a compromised machine.
I don't like the mentality of checking for bytecode patterns with a machine, then potentially misleading people looking at a mod by saying it's clear, when it's not, because there are lots of ways to pretty easily sneak malware into a mod while doing things that some minecraft mods do with innocent intent (say, download a changelog from a developer's site, then get its contents via a crappy parser that's written with native code), that could also be done in a malicious way. Also, Modrinth does already have some automated systems to scan projects in place, and it seems that they're improving them further due to the malware. |
Beta Was this translation helpful? Give feedback.
-
I don't think any malware publishes code with Git, as it's quite convoluted process to automate correctly, so like 99.9% of cases this verifier will be useful to have alongside other verifiers. And even if malware possesses a modded account, the build is in public, therefore you'll be able to see that the malicious code was added, if you want to check for that. The goal of verifier should also be publicly stated, a verifier that checks that the code is built in a trusted environment only confirms that the build process was attested, not the safety of the code that was built.
That's the goal of having verifiers. They don't have to be ideal. Nothing is ideal. But having those signals is valuable, because if several verifiers flag a file, it's definitely worth taking a look at it. This is how VirusTotal works, for example. If you see some unnamed antiviruses flag the file, you can skip it, but if the list screams red, avoid the file like a plague. In our case, verifiers can range in purpose, from checking the bytecode to running the mod in sandboxed and trapped environment, and then analyse the behaviour (e.g., if the mod tries to access fake chromium directory, this should immediately raise an alarm).
To my knowledge, Modrinth only has validators that check if the project type is correct, and those are broken right now as well. What makes you believe that Modrinth, a company spread thin and working on many different projects at a time, is able to create verifiers that would be far better than a dozen of verifiers created by the invested community? |
Beta Was this translation helpful? Give feedback.
-
This proposes the creation of necessary interfaces and establishment of a community verifiers programme.
What is a Community verifier?
Community verifier is as an actor within the Modrinth ecosystem whose purpose is to verify the content published on the site and provide reporting to both the users and the moderators of Modrinth according to verifier's purpose.
Verifier's purpose is not necessarily to check content for viruses, but for any other verifiable information, e.g, whether the uploaded mod JAR is signed by the original creator, or if the mod JAR was built in a trusted environment and can be asserted to be safe.
The programme
It was an expressed interest in community to aid Modrinth with moderation on the site. In order for verifiers mission to provide the most efficient result, a programme has to be established to vet verifiers in.
To verifiers work?
When a user is first publishing their project, or a project gets updated with a new version, a known verifier receives a request to which it has to respond within a limited time to claim the job.
Depending on verifier's purpose, it can respond with a status immediately, or report a pending status, in which the status is further deferred.
Once verifier completes it's job, it can report a status such as “skip” (verifier does not apply), “pass” (verifier does not see a problem with the published content), “fail” (verifier was not able to verify the content), “caution” (verifier has found a problem with the content), “danger” (verifier has found content to be dangerous).
Verifiers also have an ability to report projects to moderators with a special reason, either to signal to check the content, or to alert of a possible threat. The latter should be used carefully, at the risk of expulsion from the programme due to false reporting.
Acknowledgement of verifiers
Verifiers, given this programme becomes a reality, will be doing tough work. In order to know what each verifier is doing, and to provide an acknowledgement, each verifier will gets its own page. It can use it to describe verifier's purpose, or to advertise itself as a product.
Monetisation venue
Because verifiers will be costing money to operate, it can be possible to make so that each project can opt in to pay around 2-5% of their revenue on top of existing revenue, to fund the programme. The likely consumers won't be the ones paying, so it's a community effort to make this operation sustainable. Verifiers that are not cheap to operate can then skip the projects that do not share their revenue.
The goal
The goal is to provide the community with the tools to allow efficient additive moderation, to catch stuff that Modrinth can't or won't catch.
Discussion
This is a hasty draft of an idea that I had for a long time inspired by many posts that I've seen where people wanted to implement their own safety mechanisms in Modrinth. However, “say” is one thing, and “do” is another, so I'm genuinely interested to hear whether verifiers is something that people would want to build or see. Share your thoughts in comments to this discussion.
Beta Was this translation helpful? Give feedback.
All reactions