New mess detection algorithm

[nickspring]

In progress

[chris-ha458]

I am looking into other bitflag alternatives or ways to collapse the if chains
(while maintaining correctness or performance)

But I don't see why it shouldn't be merged as is. we can always fix later

[chris-ha458]

Do you think it would be a good idea to add Cargo.lock into .gitignore?

According to best practices, it is up to the project.
Ignoring it helps with development churn. But divergent crates could harm reproducibility.
Keeping it helps with reproducible builds, but not alone. You require proper reproducibility toolchain as well.

[nickspring]

I read about it and didn't decided what's better. But I guess Cargo.lock was there but at the moment it was incompatible with my system (I think it can be incompatible if we will build lib for different system and push Cargo.lock to the repo. However, I can try to add it again.

[chris-ha458]

can be rewritten like this?

foreign_long_watch |= (!character.is(MessDetectorCharFlags::LATIN) || (character.is(MessDetectorCharFlags::ACCENTUATED))
                    && !character.is(MessDetectorCharFlags::CJK)
                    && !character.is(MessDetectorCharFlags::HANGUL)
                    && !character.is(MessDetectorCharFlags::KATAKANA)
                    && !character.is(MessDetectorCharFlags::HIRAGANA)
                    && !character.is(MessDetectorCharFlags::THAI)

[nickspring]

Yes, sure. I didn't think about idiomatic code at all - my plan was to do it and see new performance and after that moment rewrite in more idiomatic way with performance controlling :)

[chris-ha458]

As for idiomatic vs speed,

I think the biggest change here in the code is many of the chars and flags are owned by the MessDetectorChar rather than moved around.

If this gets merged I will

• fold and invert several if/elses (I think i can identify a few redundancies, but i might be wrong)
• look into using enumflags2 because it seems safer while keeping all speed benefits.
• check speed and correctness with a more match based code
• import the newer unicode crates. (they are used very similar to the new md code so you might not even notice a difference)