deployment monitoring and bug reporting: TriblerHealth community

[synctext]

Brainstorm
Solve the blindness problem we have with critical bugs. (formulated above as more general: "Health monitoring of client state")

problem1. To crash or not to crash. We have a time consuming manual process of interactive debugging with user involvement using our forum. We have to make a choice to display only a warning to the screen, to the logs or do a full TraceBack crash. Catch Tribler exceptions and bugs, do we kill Tribler or not? (our choice as developers).

TriblerHealth community idea. We could implement a dedicated gossip community where you share errors anonymously. Now everybody can monitor the health and frequency of warning in logs, errors, and TraceBacks. Scrub you logging entries of personal identifying info. Share them only with neighbours. Protect privacy plus transparent method for monitoring users. Always opt-in for stable releases. For experimental and debug release this is opt-out in settings screen. Would this have prevented our two "packet-of-death" incidents?

problem2. We don't understand, for example, how database became corrupt, where did that Unicode error come from, or why the UDP packet was filtered. We need interactivity to identity the root-cause-of-failure. Users usually stop responding on our forum, we also fail to make custom-builds for them.

Debug patching in the field Implement later even two-way communication. Really dangerous idea: Inject maximum 1k of code/config for debugging or variables to monitor/sample. We always need to understand what is going on. In ideal target for the future: we have interactive debug monitoring within Sentry. So TriblerHealth can also gossip signed Python code/debug config script in a more advanced version. For non-invasive deployment monitoring this would be implemented as a config-language. Each module in Tribler would have a configurable logging-level. By sending out signed packets with a certain target audience we can temporarily increase the log-level of modules we want to inspect for debugging. Log-levels are a simple number and do not require invasive and dangerous full backdoor access with Python code injection.

This feature is always opt-in for stable releases. For experimental and debug release this is opt-out in settings screen.