The bot bans all new members.

[winhex]

Суть проблемы - бот с какого-то момента выдает бан всем подряд новым участникам за любой пост.
конфиг
--telegram.token=************** \ --telegram.group=************* \ --files.samples=/etc/tg-spam/ \ --logger.enabled \ --logger.file=/etc/tg-spam/tg-spam.log \ --logger.max-size=10M \ --message.spam="Этот пользователь заблокирован из-за подозрения на спам" \ --max-emoji=5 \ --admin.group=************* \

stop-words.txt
Ищу людей Ищу партнеров доп.доход доп.доходе доп.доходa yдаленной pаботы соtрудничеству финансового благополучия Нaбop в кoмaндy Требуется человек
кусок лога:

{"ts":"2024-07-23T00:17:45Z","display_name":"********","user_name":"********","user_id":********,"text":"Доброй ночи,проблема с медитацией Анапанасати,как только начинаю концентрироваться на точке не ловлю своё дыхание начинаю задыхаться"} {"ts":"2024-07-23T06:38:33Z","display_name":"********","user_name":"********","user_id":********,"text":"Доброго времени суток вопрос такой,начал практиковать анапанасати проблемы с дыханием,когда концентрируясь на носе пересиаю дышать полностью и понял,что когда засыпаю ночью дышу сам не автоматически из за чего это может быть,пробывал задерживать дыхания и тогда пару минут слежц за ним и странное чувство"} {"ts":"2024-07-23T06:55:14Z","display_name":"********","user_name":"********","user_id":********,"text":"Доброе утро вопрос такой,начал практиковать анапанасати, как только концентрируясь на точке носа перестают дышать,как будто задерживается дыхание"} {"ts":"2024-07-23T06:55:20Z","display_name":"********","user_name":"********","user_id":********,"text":"Не могу наблюдать своё дыхание. Так как оно само останавливается. Помогает задержать дыхание и тогда дышу и вижу дыхание,но не надолго"} {"ts":"2024-07-23T06:55:30Z","display_name":"********","user_name":"********","user_id":********,"text":"Конкретно концентрация на объект любой даёт такую реакцию"}

Пользователя вручную разбанил, сейчас бот отключен. Если бота запустить, он удалит ВСЕ посты и забанит юзера снова.
Как видно в stop-words.txt нет тех слов, которые есть в сообщении.

---

UPD: by @umputun - translation

The essence of the problem is that the bot, at some point, starts issuing bans to all new members indiscriminately for any post.
The user was manually unbanned, and now the bot is disabled. If the bot is restarted, it will delete ALL posts and ban the user again. As seen in stop-words.txt, the words in the message are not present.

[umputun]

what exactly are you running? I mean, dockerized or binary and what version? Are you sure this is even tg-spam? The logs your provided look odd, as tg-spam doesn't have structural logs.

This is what valid logs look like (with --dbg for more details):

2024/07/23 12:13:33.206 [DEBUG] {events/listener.go:232 events.(*TelegramListener).procEvents} ban initiated for {Text:this is spam: "Алина" (1860693084) Send:true BanInterval:9600h0m0s User:{ID:1860693084 Username: DisplayName:Алина} ChannelID:0 ReplyTo:546207 DeleteReplyTo:true CheckResults:[{Name:stopword Spam:false Details:not found} {Name:emoji Spam:true Details:4/2} {Name:images Spam:false Details:no images without text} {Name:cas Spam:true Details:spam detected} {Name:multi-lingual Spam:false Details:0/5} {Name:similarity Spam:false Details:0.15/0.50} {Name:classifier Spam:true Details:probability of spam: 100.00%}]}
2024/07/23 12:13:33.210 [DEBUG] {app/main.go:520 main.makeSpamLogger.func1} spam detected from {1860693084  Алина}, text: Мы в поиске партнеров для долгосрочного сотрудничества.   Предлагаем:  🔹Удаленная деятельность 🔹Высокий доход 🔹Бесплатное обучение 🔹Круглосуточная поддержка специалистов  В отличии от большинства компаний, наши партнеры достигают хороших результатов уже в первый месяц.  Не упустите свой шанс поменять свою жизнь.  Количество мест - ограничено.   Подробности здесь: PetrovDanil0
2024/07/23 12:13:33.235 [INFO]  {storage/detected_spam.go:75 storage.(*DetectedSpam).Write} detected spam entry added for user_id:1860693084, name:
2024/07/23 12:13:33.545 [INFO]  {events/events.go:195 events.banUserOrChannel} user {1860693084  Алина} banned by bot for 9600h0m0s
2024/07/23 12:13:33.545 [DEBUG] {events/admin.go:42 events.(*admin).ReportBan} report to admin chat, ban msgsData for {1860693084  Алина}, group: -4037717836
2024/07/23 12:13:33.546 [DEBUG] {events/admin.go:595 events.(*admin).sendWithUnbanMarkup} action response "change ban": user {ID:1860693084 Username: DisplayName:Алина}, msgID:546207, text: "**permanently banned [{1860693084  Алина}](tg://user?id=1860693084)**\\n\\n\u2060\u2060\u2060\u2060\u2060\u2060\u2060\u2060\u2060\u2060Мы в поиске партнеров для долгосрочного сотрудничества.   Предлагаем:  🔹Удаленная деятельность 🔹Высокий доход 🔹Бесплатное обучение 🔹Круглосуточная поддержка специалистов  В отличии от большинства компаний, наши партнеры достигают хороших результатов уже в первый месяц.  Не упустите свой шанс поменять свою жизнь.  Количество мест - ограничено.   Подробности здесь: PetrovDanil0\\n\\n"
2024/07/23 12:17:27.222 [DEBUG] {events/listener.go:203 events.(*TelegramListener).procEvents} {"message_id":546208,"from":{"id":692696840,"first_name":"July L","username":"uulng"},"date":1721755047,"chat":{"id":-1001358715993,"type":"supergroup","title":"radio-t chat","username":"radio_t_chat","photo":null,"location":null},"new_chat_members":[{"id":692696840,"first_name":"July L","username":"uulng"}],"message_auto_delete_timer_changed":null,"proximity_alert_triggered":null,"voice_chat_scheduled":null,"voice_chat_started":null,"voice_chat_ended":null,"voice_chat_participants_invited":null}

Если бота запустить, он удалит ВСЕ посты и забанит юзера снова.

This is not how tg-spam bot is working, it doesn't remove messages retroactively, but only reacts on new messages. So, let me repeat my question - are you sure you are running tg-spam bot?

[umputun]

if the log you gave is from the log file, this is not what we need to investigate the issue. I would like to see logs from the docker (or the binary) and better with --dbg set.

pls note: stop-words.txt is not the only reason to ban, you also may have samples with the spam/ham dataset.

[winhex]

what exactly are you running? I mean, dockerized or binary and what version?

I'm running binary, version 1.13.2
And bot work on linux, x86_64

Are you sure this is even tg-spam?

I don't have any other bot, so yes )

This is not how tg-spam bot is working, it doesn't remove messages retroactively, but only reacts on new messages. So, let me repeat my question - are you sure you are running tg-spam bot?

I checked twice, downloaded the binary again and verified it, so yes
There is an idea that the problems started after I answered to the spamer’s post with the /spam command. Is this possible?

if the log you gave is from the log file, this is not what we need to investigate the issue. I would like to see logs from the docker (or the binary) and better with --dbg set.

ok, but I wouldn't like to publish the log here - there's a lot of private information. Can I send it by mail or telegram?

[umputun]

Can I send it by mail or telegram?

sure, feel free to send it to my tg (same user name as here)

There is an idea that the problems started after I answered to the spamer’s post with the /spam command. Is this possible?

hard to say, but in theory if you have no ham/spam samples (they are pre-populated in docker version, but you may have not copied them or made them manually) and started to make spam samples manually, the classifier without any ham samples could detect messages that remotely resemble the spam sample as new spam. By the way, you can see in the logs what exactly triggered spam detection, check for "ban initiated for" messages and see "check results" in this line, i.e.

CheckResults:[{Name:stopword Spam:false Details:not found} {Name:emoji Spam:true Details:4/2}
{Name:images Spam:false Details:no images without text} {Name:cas Spam:true Details:spam detected} 
{Name:multi-lingual Spam:false Details:0/5} {Name:similarity Spam:false Details:0.15/0.50} 
{Name:classifier Spam:true Details:probability of spam: 100.00%}]}

You will see one or more checks with Spam:true, and this will give you a clue. If you see the classifier triggering all of those, just make a ham-samples.txt file in your /etc/tg-spam/. The content should be a set of your "non-spam messages" you can just get from your tg chat, or copy the default one from here https://github.com/umputun/tg-spam/blob/master/data/ham-samples.txt

As of now, I don't really see any bug to fix, but suspecting configuration issue of some sort. Will convert this issue to a discussion in FAQ section

[winhex]

Okay, I created ham-samples.txt based on some typical messages in my chat (it was empty before). And now the bot works great!
@umputun, thanks for your help!

[winhex]

I spent some time testing my new ham-samples.txt file with samples of regular posts in my chat (it now contains about 5000 words) and using a Message Checker I discovered that there are still many "good" posts from my chat being detected as spam. ((
This is a buddhist chat that contains a lot of buddhist terminology. How many samples should I add so that these words are not detected as spam?

[umputun]

First of all, check if those false-positives are from the classifier. You can do this by either examining the output logs (with --dbg) or using the web UI (either the detected_spam tab or the checker tab). If the classifier is indeed responsible for these false positives, you can add those messages to the ham samples either directly or via the web UI (refer to the manage samples tab). This will help train the classifier to better recognize what ham is and ultimately reduce the number of false positives.

You may also want to consider setting up the "Admin group" (refer to https://github.com/umputun/tg-spam?tab=readme-ov-file#admin-chatgroup). With this group/chat, you will be able to receive all spam detection reports directly in Telegram and take immediate action from this group (such as reverse banning, confirming, and checking details).

Another important point to consider is running the tg-spam bot in training mode for a while until you are satisfied with the classifier results. Refer to https://github.com/umputun/tg-spam?tab=readme-ov-file#training-the-bot-on-a-live-system-safely