Skip to content

Commit

Permalink
Properly support Unicode Strings (#3)
Browse files Browse the repository at this point in the history
* Properly support unicode strings

```
iex(47)> "✔︎" |> String.replace(~r/[\p{P}\p{S}]/, " ") |> String.valid?
false
iex(48)> "✔︎" |> String.replace(~r/[\p{P}\p{S}]/u, " ") |> String.valid?
true
```

* add a test case for unicode strings

Co-authored-by: Ryan Johnson <ryan@grain.com>
  • Loading branch information
patrickdet and bismark authored Nov 8, 2022
1 parent 5bba136 commit 107d544
Show file tree
Hide file tree
Showing 2 changed files with 11 additions and 1 deletion.
2 changes: 1 addition & 1 deletion lib/fuzzy_compare/preprocessor.ex
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ defmodule FuzzyCompare.Preprocessor do
alias FuzzyCompare.Preprocessed

# Replaces all punctuation
@regex ~r/[\p{P}\p{S}]/
@regex ~r/[\p{P}\p{S}]/u

@spec process(binary(), binary()) :: {Preprocessed.t(), Preprocessed.t()}
def process(left, right) when is_binary(left) and is_binary(right) do
Expand Down
10 changes: 10 additions & 0 deletions test/preprocessor_test.exs
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
defmodule PreprocessorTest do
use ExUnit.Case

@subject FuzzyCompare.Preprocessor

test "when replacing whitespace and punctuation in unicode strings the string remains a valid unicode string" do
result = @subject.process("✔︎")
assert String.valid?(result.string) == true
end
end

0 comments on commit 107d544

Please sign in to comment.