Pipeline doesn't work as expected when using dynamo db streams with NEW_AND_OLD_IMAGES

[an-khoros]

Let's say I have a dynamodb record as below :
{empId : 1, empName: John Doe, Department: Marketing}

The dynamo db streams is turned on for the table with type as NEW_AND_OLD_IMAGES. I have a open search index with name as emp-department-${Department}. Now when the above record is created the data prepper would have created an index as emp-department-Marketing and added a document under it. If I go ahead and delete the record the dynamo db stream would pass new data as null and old data with whatever was there before deletion. But data prepper is ignoring the old data in this case and the event doesn't have any data. When the sink processes this event of type delete it fails because it is unable to determine/parse the index name. Due to this the document will not get deleted from the open search index. This looks like a bug?

[graytaylor0]

Please see (#4261)

[an-khoros]

So an issue has been reported. Thank you @graytaylor0