Restructures & Improvements for Milk

[ShindouMihou]

Milk was created back when I wasn't that used to JavaScript and related. As such, it may suffer some design flaws that were pretty bad for maintainability. This PR intends to resolve that and adds additional changes, such as supporting JSON, and documentation for both third-parties and Beemo developers.

TODO

• Restructuring to make things more straightforward and simple.
• Code simplification ¹
• Support JSON output. ²
• Fixes some pretty awful bugs (i.e. trying to serialize bigint)
• Add support for paginating logs. ³
• Document the APIs and Kafka Client (via OpenAPI and a Kafka Markdown file).
• Testing the Kafka Client ⁴

Additional Notes

Following ³, a little bit of testing reveals that this is really a big problem for our current plans as a sample run with 500,000 users resulted in 5.83s computation time (likely faster on the dedicated machine) with over 36.66 MB and that doesn't look good as our largest log to date is Midjourney's 450,000 user log.

A key solution for this, to be discussed, is to limit the amount of users fetched to a specific amount. The key reasoning for this solution is that as we now have a proper JSON route, third-party developers wishing to use the logging service no longer need to be considered in the text route (JSON will use pagination), as such, regular users who simply want to see a quick view of their raid would generally only need to see at most 100-500 users.

Although the above solution is up to further debate internally.

Footnotes

1. Code simplification still needs to be looked into and the documentation may change due to this as it would likely change our Kafka client's implementation, or even the API. ↩

2. Third-party and clients may want to integrate with our logs, therefore, we can simplify this process for them a bit by allowing them to receive JSON format of our log, this will especially help with plans for a web viewer for logs. ↩

3. We've had raid logs that go beyond 200,000 users and as we are no longer serving static files in this version, this becomes a significant issue as we'll have to fetch and cache 200,000 users, although our caching mechanism saves the string content only to possibly save us some pain, but still, this issues a need for pagination and likely better caching with those pagination. ↩ ↩²

4. Testing of the Kafka client is currently needed, but is blocked with the pending merge of the new antiraid refactor, as such, this will be drafted until testing is performed (this Pull Request should not be merged until we've completed tested the Kafka client). ↩

[ShindouMihou]

Pagination of /raid/:id.json is now implemented with each page containing, at maximum, 200 users with the complete details of the Raid also provided. In addition, the /raid/:id route (Text API) is now limited to viewing 2,000 users at a maximum with a note for raids that go beyond that links viewers to a documentation link over how to read beyond that limit by using the JSON API.

Text API Response

JSON API Response

[ayubun]

strawberry milk 😍

your pr descriptions are so lovely to read, thank you for that miu ❤️

TIL you can use footnotes in them