execution via AWS Lambda

[ashwallace]

Would really love to see piicatcher execute against event data from a lambda invocation so can check for PII in event driven/streaming workflows!

[vrajat]

Definitely. PIICatcher has an API. However these scan databases or files. What should an API for streaming look like? Do you have examples of other python libraries that process streaming data?

Also can you give more details about your architecture and use case?

[ashwallace]

Lambda is handling receipt of the stream, so if the piicatcher API was able to scan variables in my code that would be great.

Getting a reference to the specifically detected keys (for JSON) or character location ranges (for a text blob) as an API response would be even more amazing, that way i can replace the PII too.

In my case i am receiving a variety of semistructured data sources into Amazon Kinesis streams, and i use lambda to do some processing before it continues on to long term storage. There is a risk of some PII not being filtered before being received, hence the ask for piicatcher API support for data inside variables.

[vrajat]

Thanks for the info on your use case.

A few thoughts:

• My assumption is that at the basic level a text blob has to be scanned.
• The blob can be part of a larger data structure like lists, tuples, dicts or dict of dicts etc.
• PIICatcher detects PII by scanning metadata and contents. Is there metadata available or only content can be scanned?
• Scanning content can be slow. PIICatcher gets around the problem by scanning a small sample of rows in a table. Can sampling be used for streaming as well?
• Keys or character ranges can be easily returned in the results.

[ashwallace]

• PIICatcher detects PII by scanning metadata and contents. Is there metadata available or only content can be scanned?

not really

• Scanning content can be slow. PIICatcher gets around the problem by scanning a small sample of rows in a table. Can sampling be used for streaming as well?

you could think of stream consumers processing from a source row by row, or window by window (many rows in a window). So since it wouldnt be a lot of data at each piicatcher invocation, then i imagine it should be pretty quick? Most streaming sources are stateful for a period of time, so the stream consumer could get around any slowness of piicatcher by scaling the consumer horizontally to get performance via parallelism, then the risk of data expiring in the source stream becomes a mitigated risk..

[vrajat]

Per invocation time can get add up. However the streaming source can choose to sample in a way that makes sense in that context. What are the typical data types you see?

• int
• float
• str
• list
• dict
• complex combinations of list, dicts and basic types

Anything else I am missing?

[dikshapandey-ml]

hi , i am trying to create lambda function and trying to use piicatcher module.I've created the layer but upon running simple import piicatcher command, i am getting error nable to import module 'lambda_function': No module named 'piicatcher'.Can someone pls help.I've tried on all python version runtime 3.8 and above. any help @vrajat