Can we add SPARK NLP support for Presidio ?

[ashishmgofficial]

Just thinking wouldnt support for Spark NLP instead of Spacy provide more efficiency for spark based workloads ? Now Im new to this whole NLP concept, so kindly pardon if this is a stupid thought.

[omri374]

Hey @ashishmgofficial,
I'm not familiar enough with Spark NLP, but based on my understanding it shouldn't be an issue extending Presidio to support it, for example using a Remote Recognizer: https://microsoft.github.io/presidio/analyzer/adding_recognizers/#creating-a-remote-recognizer

While Spark NLP could also potentially be replacing spaCy, this might require more work and code changes. One way around it is to use a small spacy model (like en_core_web_sm) and use Spark NLP as an additional recognizer of PII entities. the small spacy model would be used for things like tokenization and lemmatization, and the Spark NLP model would be used for entity recognition.

If you're interested in creating a sample of using Spark NLP with Presidio, this could be super useful!