Look for corruption in downloaded files

[fmarier]

https://community.brave.com/t/ads-and-trackers-blockers-breaks-all-websites/524842 reported widespread webcompat problems. This was due to a corrupted cookie list. We should be validating filter lists after downloading them. List of things to check (in priority order):

• If there is a checksum on an adblock list, verify the checksum at crx build time in a place similar to https://github.com/brave/adblock-resources/blob/master/index.js#L7
• File size being too small
• Check the MIME/file type (should be a text file)
• Generate a warning if the file has invalid rules sent out via Sentry
• Check for lines which are too long (the whole file should not be in one line for example)
• Check file diff sizes before publishing (shouldn’t be too large)
• Check against “known-good” sites before publishing (nothing should be blocked): https://github.com/brave/devops/issues/11117

In all these cases, we should not publish a new version of the list/component and send a Sentry alert like how it's done already in the code: https://github.com/brave/brave-core-crx-packager/blob/master/scripts/generateAdBlockRustDataFiles.js#L101

[ryanbr]

https://github.com/ryanbr/fanboy-adblock/blob/master/scripts/validateChecksum.pl also

[fmarier]

Example of data corruption:

singkinderlieder.de##+js(set-cookie, acceptMatomo, true)

became:

singkinderlieder.de-o+js(set-clert, acceptMatomo, true)

[fmarier]

For added robustness, it may be worth verifying the checksum both when downloading the upstream file / building the component, as well as client-side when loading the list.

[ShivanKaul]

From checking S3, it looks like the corrupted file was shipped to clients. To be safe though, we should check the list before loading into brave-core, since the corruption might happen as part of the CRX zipping process.

[ShivanKaul]

I edited the top-level issue description to have list of validation checks we can do.

[bbondy]

I added an item to the top level issue (item 0) "Verify checksums when available at crx build time in a place similar to https://github.com/brave/adblock-resources/blob/master/index.js#L7"