Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/purge messages upto 24 hrs in current channel #982

Draft
wants to merge 35 commits into
base: develop
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
1c12b9d
new table to hold message history
ankitsmt211 Nov 5, 2023
d368324
add package for purge command feature
ankitsmt211 Nov 5, 2023
79277c9
message listener to store message data in db
ankitsmt211 Nov 5, 2023
093aa32
add routine to clear message history and purge command
ankitsmt211 Nov 5, 2023
bde724c
adding to features
ankitsmt211 Nov 5, 2023
f6757a9
refactor based on suggestions
ankitsmt211 Nov 6, 2023
357d41d
refactor variables
ankitsmt211 Nov 6, 2023
b25ee1e
fix doc
ankitsmt211 Nov 6, 2023
a6627ba
improve doc
ankitsmt211 Nov 6, 2023
117c0cc
method to ignore messages from bot or webhooks
ankitsmt211 Nov 6, 2023
650388e
refactor stream using try-with-resource
ankitsmt211 Nov 6, 2023
ec443d3
updated catch block
ankitsmt211 Nov 6, 2023
5feb1e6
add try-with-resource and logger
ankitsmt211 Nov 6, 2023
0bdc627
minor improvements
ankitsmt211 Nov 6, 2023
672d319
method to handle reasons
ankitsmt211 Nov 6, 2023
c3f3532
minor improvements
ankitsmt211 Nov 7, 2023
c6e81af
update privacy policy
ankitsmt211 Nov 7, 2023
45bda14
improved message for user and delete already pulled records
ankitsmt211 Nov 7, 2023
d38d874
adds option for duration, message expiration from db increased
ankitsmt211 Nov 8, 2023
b9f5aed
add an atomic counter as safety brake
ankitsmt211 Nov 12, 2023
0700661
refactor variable name
ankitsmt211 Nov 12, 2023
8e80ba0
refactor counter methods
ankitsmt211 Nov 12, 2023
20a2eb3
update record limit from 7 to 7.5k
ankitsmt211 Nov 12, 2023
fedb8c6
refactor classes to be final
ankitsmt211 Nov 12, 2023
53a5b72
log level from warn to debug on record limit
ankitsmt211 Nov 12, 2023
cdc511f
Merge branch 'develop' into feat/delete-last-hour-messages
ankitsmt211 Nov 14, 2023
5813ae3
auto trim routine and minor improvements
ankitsmt211 Nov 14, 2023
9a2c3da
add choices for duration, better UX
ankitsmt211 Nov 17, 2023
10d3664
add duration enum for constants and fix
ankitsmt211 Nov 25, 2023
ec27d05
writes should happen below limit
ankitsmt211 Dec 15, 2023
6f7f08d
refactoring to a better name for clarity
ankitsmt211 Dec 16, 2023
abe1e89
Merge branch 'develop' into feat/delete-last-hour-messages
ankitsmt211 Jan 27, 2024
9bd90cd
refactor better names for roles & replacing get(0) with getFirst()
ankitsmt211 Jan 27, 2024
671d92a
refactor enum Duration for clarity
ankitsmt211 Feb 18, 2024
1f6209e
add doc
ankitsmt211 Feb 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion PP.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ For example, **TJ-Bot** may associate your `user_id` with a `message_id` and a `

**TJ-Bot** may further store data that you explicitly provided for **TJ-Bot** to offer its services. For example the reason of a moderative action when using its moderation commands.

Furthermore, upon utilization of our help service, `user_id`s and `channel_id`s are stored to track when/how many questions a user asks. The data may be stored for up to **30** days.
Furthermore, upon utilization of our help service, `user_id`s and `channel_id`s and `timestamp`s are stored to track when/how many questions a user asks. The data may be stored for up to **30** days.

The stored data is not linked to any information that is personally identifiable.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
import org.togetherjava.tjbot.features.moderation.audit.AuditCommand;
import org.togetherjava.tjbot.features.moderation.audit.ModAuditLogRoutine;
import org.togetherjava.tjbot.features.moderation.audit.ModAuditLogWriter;
import org.togetherjava.tjbot.features.moderation.history.MessageHistoryRoutine;
import org.togetherjava.tjbot.features.moderation.history.PurgeHistoryCommand;
import org.togetherjava.tjbot.features.moderation.history.PurgeMessageListener;
import org.togetherjava.tjbot.features.moderation.modmail.ModMailCommand;
import org.togetherjava.tjbot.features.moderation.scam.ScamBlocker;
import org.togetherjava.tjbot.features.moderation.scam.ScamHistoryPurgeRoutine;
Expand Down Expand Up @@ -106,6 +109,7 @@ public static Collection<Feature> createFeatures(JDA jda, Database database, Con
.add(new AutoPruneHelperRoutine(config, helpSystemHelper, modAuditLogWriter, database));
features.add(new HelpThreadAutoArchiver(helpSystemHelper));
features.add(new LeftoverBookmarksCleanupRoutine(bookmarksSystem));
features.add(new MessageHistoryRoutine(database));

// Message receivers
features.add(new TopHelpersMessageListener(database, config));
Expand All @@ -119,6 +123,7 @@ public static Collection<Feature> createFeatures(JDA jda, Database database, Con
features.add(new CodeMessageManualDetection(codeMessageHandler));
features.add(new SlashCommandEducator());
features.add(new PinnedNotificationRemover(config));
features.add(new PurgeMessageListener(database));

// Event receivers
features.add(new RejoinModerationRoleListener(actionsStore, config));
Expand Down Expand Up @@ -159,6 +164,7 @@ public static Collection<Feature> createFeatures(JDA jda, Database database, Con
features.add(new BookmarksCommand(bookmarksSystem));
features.add(new ChatGptCommand(chatGptService));
features.add(new JShellCommand(jshellEval));
features.add(new PurgeHistoryCommand(database));

FeatureBlacklist<Class<?>> blacklist = blacklistConfig.normal();
return features.stream().filter(f -> blacklist.isEnabled(f.getClass())).toList();
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package org.togetherjava.tjbot.features.moderation.history;

/**
* Holds duration constants used for both purge history command and message history routine.
*/
public enum Duration {
tj-wazei marked this conversation as resolved.
Show resolved Hide resolved
// All new/existing options for duration in PurgeHistoryCommand should be within max and min
// duration
PURGE_HISTORY_MAX_DURATION(24),
PURGE_HISTORY_MIN_DURATION(1),
PURGE_HISTORY_THREE_HOURS(3),
PURGE_HISTORY_SIX_HOURS(6),
PURGE_HISTORY_TWELVE_HOURS(12),
PURGE_HISTORY_TWENTY_FOUR_HOURS(24);
ankitsmt211 marked this conversation as resolved.
Show resolved Hide resolved

private final int hours;

Duration(int hours) {
this.hours = hours;
}

public int getHours() {
return hours;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
package org.togetherjava.tjbot.features.moderation.history;

import net.dv8tion.jda.api.JDA;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.togetherjava.tjbot.db.Database;
import org.togetherjava.tjbot.db.generated.tables.records.MessageHistoryRecord;
import org.togetherjava.tjbot.features.Routine;

import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.concurrent.TimeUnit;
import java.util.stream.Stream;

import static org.togetherjava.tjbot.db.generated.Tables.MESSAGE_HISTORY;
ankitsmt211 marked this conversation as resolved.
Show resolved Hide resolved

/**
* Routine that deletes records from message_history post expiration hours.
*/
public final class MessageHistoryRoutine implements Routine {
ankitsmt211 marked this conversation as resolved.
Show resolved Hide resolved
private static final Logger logger = LoggerFactory.getLogger(MessageHistoryRoutine.class);
private static final int SCHEDULE_INTERVAL_SECONDS = 30;
tj-wazei marked this conversation as resolved.
Show resolved Hide resolved
private static final int EXPIRATION_HOURS = Duration.PURGE_HISTORY_MAX_DURATION.getHours();
private final Database database;

/**
* Creates a new instance.
*
* @param database the database that contains records of messages to be purged.
*/
public MessageHistoryRoutine(Database database) {
this.database = database;
}


@Override
public Schedule createSchedule() {
return new Schedule(ScheduleMode.FIXED_RATE, 0, SCHEDULE_INTERVAL_SECONDS,
TimeUnit.SECONDS);
}

@Override
public void runRoutine(JDA jda) {
Instant now = Instant.now();
Instant preExpirationHours = now.minus(EXPIRATION_HOURS, ChronoUnit.HOURS);

Stream<MessageHistoryRecord> messageRecords =
database.writeAndProvide(context -> context.selectFrom(MESSAGE_HISTORY)
.where(MESSAGE_HISTORY.SENT_AT.lessThan(preExpirationHours))
.stream());

try (messageRecords) {
messageRecords.forEach(messageRecord -> {
messageRecord.delete();
PurgeMessageListener.decrementRecordsCounterByOne();
});
} catch (Exception exception) {
tj-wazei marked this conversation as resolved.
Show resolved Hide resolved
logger.error(
"Unknown error happened during delete operation during message history routine",
exception);
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
package org.togetherjava.tjbot.features.moderation.history;

import net.dv8tion.jda.api.entities.Member;
import net.dv8tion.jda.api.entities.Role;
import net.dv8tion.jda.api.entities.User;
import net.dv8tion.jda.api.entities.channel.unions.MessageChannelUnion;
import net.dv8tion.jda.api.events.interaction.command.SlashCommandInteractionEvent;
import net.dv8tion.jda.api.interactions.InteractionHook;
import net.dv8tion.jda.api.interactions.commands.OptionMapping;
import net.dv8tion.jda.api.interactions.commands.OptionType;
import net.dv8tion.jda.api.interactions.commands.build.OptionData;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.togetherjava.tjbot.db.Database;
import org.togetherjava.tjbot.db.DatabaseException;
import org.togetherjava.tjbot.db.generated.tables.records.MessageHistoryRecord;
import org.togetherjava.tjbot.features.CommandVisibility;
import org.togetherjava.tjbot.features.SlashCommandAdapter;

import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.stream.Stream;

import static org.togetherjava.tjbot.db.generated.Tables.MESSAGE_HISTORY;

public final class PurgeHistoryCommand extends SlashCommandAdapter {

private static final Logger logger = LoggerFactory.getLogger(PurgeHistoryCommand.class);
ankitsmt211 marked this conversation as resolved.
Show resolved Hide resolved
private static final String USER_OPTION = "user";
private static final String REASON_OPTION = "reason";
private static final String COMMAND_NAME = "purge-in-channel";
private static final int PURGE_MESSAGES_DEFAULT_DURATION =
Duration.PURGE_HISTORY_MIN_DURATION.getHours();
private static final String DURATION = "duration";
private final Database database;

/**
* Creates an instance of the command.
*
* @param database the database to retrieve message records of a target user
*/

public PurgeHistoryCommand(Database database) {
super(COMMAND_NAME, "purge message history of user in same channel",
CommandVisibility.GUILD);

String optionUserDescription = "user who's message history you want to purge within %s hr"
.formatted(PURGE_MESSAGES_DEFAULT_DURATION);

String optionDurationDescription =
"duration in hours (default: %s)".formatted(PURGE_MESSAGES_DEFAULT_DURATION);

OptionData durationData =
new OptionData(OptionType.INTEGER, DURATION, optionDurationDescription, false);
durationData.addChoice("last 3 hrs", Duration.PURGE_HISTORY_THREE_HOURS.getHours())
.addChoice("last 6 hrs", Duration.PURGE_HISTORY_SIX_HOURS.getHours())
.addChoice("last 12 hrs", Duration.PURGE_HISTORY_TWELVE_HOURS.getHours())
.addChoice("last 24 hrs", Duration.PURGE_HISTORY_TWENTY_FOUR_HOURS.getHours());

getData().addOption(OptionType.USER, USER_OPTION, optionUserDescription, true)
.addOption(OptionType.STRING, REASON_OPTION,
"reason for purging user's message history", true)
.addOptions(durationData);
this.database = database;
}

@Override
public void onSlashCommand(SlashCommandInteractionEvent event) {
event.deferReply(true).queue();

OptionMapping targetOption =
tj-wazei marked this conversation as resolved.
Show resolved Hide resolved
Objects.requireNonNull(event.getOption(USER_OPTION), "target is null");
Member author = Objects.requireNonNull(event.getMember(), "author is null");
String reason = Objects.requireNonNull(event.getOption(REASON_OPTION).getAsString(),
"reason is null");

OptionMapping durationMapping = event.getOption(DURATION);
int duration = durationMapping == null ? PURGE_MESSAGES_DEFAULT_DURATION
tj-wazei marked this conversation as resolved.
Show resolved Hide resolved
: durationMapping.getAsInt();

handleHistory(event, author, reason, targetOption, event.getHook(), duration);
}

private void handleHistory(SlashCommandInteractionEvent event, Member author, String reason,
OptionMapping targetOption, InteractionHook hook, int duration) {
Instant now = Instant.now();
Instant purgeMessagesAfter = now.minus(duration, ChronoUnit.HOURS);

User targetUser = targetOption.getAsUser();
String sourceChannelId = event.getChannel().getId();

if (!validateHierarchy(author, targetOption)) {
hook.sendMessage("Cannot purge history of user with a higher role than you").queue();
return;
}

List<String> messageIdsForDeletion = new ArrayList<>();

Stream<MessageHistoryRecord> fetchedMessageHistory =
database.writeAndProvide(context -> context.selectFrom(MESSAGE_HISTORY)
.where(MESSAGE_HISTORY.AUTHOR_ID.equal(targetUser.getIdLong())
.and(MESSAGE_HISTORY.CHANNEL_ID.equal(Long.valueOf(sourceChannelId)))
.and(MESSAGE_HISTORY.SENT_AT.greaterOrEqual(purgeMessagesAfter)))
.stream());

try (fetchedMessageHistory) {
fetchedMessageHistory.forEach(messageHistoryRecord -> {
String messageId = String.valueOf(messageHistoryRecord.getMessageId());
messageIdsForDeletion.add(messageId);
messageHistoryRecord.delete();

PurgeMessageListener.decrementRecordsCounterByOne();
});
} catch (DatabaseException exception) {
logger.error("unknown error during fetching message history records for {} command",
COMMAND_NAME, exception);
}

handleDelete(messageIdsForDeletion, event.getChannel(), event.getHook(), targetUser, reason,
author, duration);
}

private void handleDelete(List<String> messageIdsForDeletion, MessageChannelUnion channel,
InteractionHook hook, User targetUser, String reason, Member author, int duration) {

if (messageIdsForDeletion.isEmpty()) {
handleEmptyMessageHistory(hook, targetUser, duration);
return;
}

if (hasSingleElement(messageIdsForDeletion)) {
ankitsmt211 marked this conversation as resolved.
Show resolved Hide resolved
String messageId = messageIdsForDeletion.get(0);
String messageForMod = "message purged from user %s in this channel within last %s hr."
.formatted(targetUser.getName(), duration);
channel.deleteMessageById(messageId).queue();
hook.sendMessage(messageForMod).queue();
return;
}

int noOfMessagePurged = messageIdsForDeletion.size();
channel.purgeMessagesById(messageIdsForDeletion);

String messageForMod = "%s messages purged from user %s in this channel within last %s hrs."
.formatted(noOfMessagePurged, targetUser.getName(), duration);
hook.sendMessage(messageForMod)
.queue(onSuccess -> logger.info("{} purged messages from {} in {} because: {}",
author.getUser(), targetUser, channel.getName(), reason));
}

private boolean hasSingleElement(List<String> messageIdsForDeletion) {
return messageIdsForDeletion.size() == 1;
}

private void handleEmptyMessageHistory(InteractionHook hook, User targetUser, int duration) {
String messageForMod = "%s has no message history in this channel within last %s hrs."
.formatted(targetUser.getName(), duration);

hook.sendMessage(messageForMod).queue();
}

private boolean validateHierarchy(Member author, OptionMapping target) {
int highestRole = 0;
ankitsmt211 marked this conversation as resolved.
Show resolved Hide resolved
Role targetUserRole = Objects
.requireNonNull(target.getAsMember(), "target user for purge command is not a member")
.getRoles()
.get(highestRole);
Role authorRole = author.getRoles().get(highestRole);

return targetUserRole.getPosition() >= authorRole.getPosition();
}
}
Loading
Loading