Send events to kafka in plain text format

[JavadHosseini]

I have configured a Kafka destination in self-hosted environment. Currently events will be send to Kafka in JSON. But I need events to be stored in Kafka destination in plain text (e.g. id: x, name: y) that later can be converted to parquet.
Do I need to change the source code or there is an easier way?

[arajguha]

Hi @JavadHosseini. Can you provide a bit more context or maybe an example of the payload mapping?
Currently, JSON payload is being sent to Kafka as message value (JSON string) and the message key in our Kafka integration is the userId. W.R.T your example, do you mean that you need to parse the JSON body into comma-separated key-value pair string and then send it as the value?

[JavadHosseini]

Thanks for reply,
Yes, this is actually somehow is happening with Postgres (+Minio as a temp storage) destination which events has been stored in csv format, I wonder can it be done with kafka in key-value string?
I looked into the source code but I couldn't find anything related to that.

[SumanthPuram]

@JavadHosseini We are adding support for parquet files as a destination. That should be out in 2 weeks.

May I know why would you want to send it to Kafka to generate parquet? Some of our users do the following - write JSON files to object storage (S3/Minio). Then trigger a lambda function which converts them to parquet.

CC: @sivashanmukh

[JavadHosseini]

@SumanthPuram Wow, great news! thanks.
Actually I'm considering my options right now. If I want to be more specific, I have 2 branches (destination in rudder) in my pipeline, one is for my relational data warehouse (postgres + minio as a temp storage) for some separated functionality, and the other for data analysis and ML purpose which it has a Kafka followed by hadoop hdfs.
I can write a spark code to consume from Kafka and convert it to parquet and then push it into hdfs. I didn't know about your solution, can it be done with transferring parquets file from minio to hdfs?