Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat/purge messages upto 24 hrs in current channel #982

Draft
wants to merge 35 commits into
base: develop
Choose a base branch
from
Draft

Feat/purge messages upto 24 hrs in current channel #982

wants to merge 35 commits into from

Conversation

ankitsmt211
Copy link
Member

@ankitsmt211 ankitsmt211 commented Dec 10, 2023 •
edited
Loading

For earlier context related to PR, #945

resolves #912

Workflow

  1. Message Sent: A user sends a message in a Discord channel.
  2. Log Message Metadata: The Message Listener detects the new message and logs its metadata into the Database Table.
  3. Periodic Cleanup: The Cleanup Routine periodically checks the database for messages older than 24 hours and deletes them.
  4. Database Overload Prevention: The Emergency Brake activates if the database approaches its record limit, preventing spamming and overload.
  5. Database Size Management: The Record Management Subroutine deletes the oldest records to make space for new entries when nearing the database limit for this feature(~7.5k records).

Includes

  • Database Table: Tracks message metadata.
  • Cleanup Routine: Removes records older than 24 hours.
  • Message Listener: Logs new messages into the database.
  • New Package: Dedicated to the feature.
  • New Command: /purge-in-channel.
  • Issue Resolution: Resolves Delete last X messages from user for moderation #912.

Additions

  • Emergency Brake: Prevents spamming DB after cetain limit.
  • Record Management Subroutine: Deletes oldest records to free up DB limit for this feature once triggered.

Command Interface

image

when there's no records:
image

if there's any records within selected duration:
image

Note: Duration is optional, if skipped it defaults to last hour.

ankitsmt211 and others added 29 commits November 6, 2023 03:52
@ankitsmt211
new table to hold message history
1c12b9d
@ankitsmt211
add package for purge command feature
d368324
@ankitsmt211
message listener to store message data in db
79277c9
@ankitsmt211
add routine to clear message history and purge command
093aa32
@ankitsmt211
adding to features
bde724c
@ankitsmt211
refactor based on suggestions
f6757a9
@ankitsmt211
refactor variables
357d41d
@ankitsmt211
fix doc
b25ee1e
@ankitsmt211
improve doc
a6627ba
@ankitsmt211
method to ignore messages from bot or webhooks
117c0cc
@ankitsmt211
refactor stream using try-with-resource
650388e
@ankitsmt211
updated catch block
ec443d3
@ankitsmt211
add try-with-resource and logger
5feb1e6
@ankitsmt211
minor improvements
0bdc627
@ankitsmt211
method to handle reasons
672d319
@ankitsmt211
minor improvements
c3f3532
@ankitsmt211
update privacy policy
c6e81af
@ankitsmt211
improved message for user and delete already pulled records
45bda14
@ankitsmt211
adds option for duration, message expiration from db increased
d38d874
@ankitsmt211
add an atomic counter as safety brake
b9f5aed
@ankitsmt211
refactor variable name
0700661
@ankitsmt211
refactor counter methods
8e80ba0
@ankitsmt211
update record limit from 7 to 7.5k
20a2eb3
@ankitsmt211
refactor classes to be final
fedb8c6
@ankitsmt211
log level from warn to debug on record limit
53a5b72
@ankitsmt211
Merge branch 'develop' into feat/delete-last-hour-messages
cdc511f
@ankitsmt211
auto trim routine and minor improvements
5813ae3
@ankitsmt211
add choices for duration, better UX
9a2c3da
@ankitsmt211
add duration enum for constants and fix
10d3664 
* enum will hold duration constants for feature
* tiny fixes in message for user of command
* hrs-> hr for single message and in x hrs -> within x hrs
@ankitsmt211 ankitsmt211 added the enhancement New feature or request label Dec 10, 2023
@ankitsmt211 ankitsmt211 added new command Add a new command or group of commands to the bot priority: low labels Dec 10, 2023
@ankitsmt211 ankitsmt211 self-assigned this Dec 10, 2023
@ankitsmt211 ankitsmt211 requested review from a team as code owners December 10, 2023 18:21
surajkumar
surajkumar suggested changes Dec 15, 2023
View reviewed changes
application/src/main/java/org/togetherjava/tjbot/features/moderation/history/Duration.java Outdated Show resolved Hide resolved
.../src/main/java/org/togetherjava/tjbot/features/moderation/history/MessageHistoryRoutine.java Outdated Show resolved Hide resolved
.../src/main/java/org/togetherjava/tjbot/features/moderation/history/MessageHistoryRoutine.java Outdated Show resolved Hide resolved
.../src/main/java/org/togetherjava/tjbot/features/moderation/history/MessageHistoryRoutine.java Outdated Show resolved Hide resolved
.../src/main/java/org/togetherjava/tjbot/features/moderation/history/MessageHistoryRoutine.java Outdated Show resolved Hide resolved
...n/src/main/java/org/togetherjava/tjbot/features/moderation/history/PurgeMessageListener.java Show resolved Hide resolved
...n/src/main/java/org/togetherjava/tjbot/features/moderation/history/PurgeMessageListener.java Show resolved Hide resolved
...n/src/main/java/org/togetherjava/tjbot/features/moderation/history/PurgeMessageListener.java Outdated Show resolved Hide resolved
...n/src/main/java/org/togetherjava/tjbot/features/moderation/history/PurgeMessageListener.java Show resolved Hide resolved
application/src/main/resources/db/V14__Add_Message_History.sql Show resolved Hide resolved
@ankitsmt211 ankitsmt211 changed the title Feat/delete last hour messages Feat/purge messages upto 24 hrs Dec 16, 2023
@ankitsmt211 ankitsmt211 changed the title Feat/purge messages upto 24 hrs Feat/purge messages upto 24 hrs in current channel Dec 16, 2023
ankitsmt211
ankitsmt211 commented Feb 23, 2024
View reviewed changes
...n/src/main/java/org/togetherjava/tjbot/features/moderation/history/PurgeMessageListener.java
private static final int MESSAGES_RECORDS_MAX_LIMIT = 7500;
private static final int MESSAGES_RECORDS_THRESHOLD = MESSAGES_RECORDS_MAX_LIMIT - 100;
private static final int RECORDS_TO_TRIM = 500;
static AtomicInteger recordsCounter = new AtomicInteger(0);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since the PR is getting too old now and so much stuff to review , flagging the important bit, is using atomic justified here?
the goal is to limit only certain no of db entries(recordsCounter), which is modified across package in PurgeMessageListener (incrementCounterByOne) on each new message and in both PurgeHistoryCommand and PurgeExpiredMessageHistory (decrementCountByOne) on each deletion of record/message.

@ankitsmt211 ankitsmt211 mentioned this pull request Mar 21, 2024
Open
@ankitsmt211
Copy link
Member Author

converting this PR to draft, feel like it needs a bit of refactoring considering moderation usecase per channel basis might not be suited anymore.

@ankitsmt211 ankitsmt211 marked this pull request as draft May 13, 2024 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marko-radosavljevic marko-radosavljevic marko-radosavljevic left review comments

@surajkumar surajkumar surajkumar requested changes

@SquidXTV SquidXTV SquidXTV left review comments

At least 2 approving reviews are required to merge this pull request.

Assignees

@ankitsmt211 ankitsmt211

Labels
enhancement New feature or request new command Add a new command or group of commands to the bot priority: low
Projects
Together Java
Status: No Status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Delete last X messages from user for moderation
4 participants
@ankitsmt211 @marko-radosavljevic @surajkumar @SquidXTV