JSON returned to kafka topic is more than expected.

[QuidneIT]

When I perform a JSON curl request to the specific REST API (post request). I get this back:
{"PM2.5":"9","Temperature":"22.6","Relative Humidity":"32","Air Pressure":"1021","TVOC":"990","CO2":"466","CO":"0","Ozone":"7.5","NO2":"17.5","virusIndex":3,"Timestamp":1616417426,"DateTime":"2021-03-22 12:50"}
Which is what I expect.

But when I perform the request through this source connector, I get all this back:
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"doc":"HTTP Record Value","field":"value"},{"type":"string","optional":true,"doc":"HTTP Record Key","field":"key"},{"type":"int64","optional":true,"doc":"HTTP Record Timestamp","field":"timestamp"}],"optional":false,"name":"com.github.castorm.kafka.connect.http.Value","doc":"Message Value"},"payload":{"value":"{"PM2.5":"9","Temperature":"22.6","Relative Humidity":"31.9","Air Pressure":"1020.8","TVOC":"927","CO2":"468","CO":"0","Ozone":"7.5","NO2":"18.9","virusIndex":3,"Timestamp":1616418626,"DateTime":"2021-03-22 13:10"}","key":"045915bc-a2a0-3ea0-9e1d-800a035d0c8d","timestamp":1616418624013}}

Am I missing some properties in the properties file?

Thank you for the great Source Connector.
Raoul.

[castorm]

Hi @QuidneIT,

From what I can see you expect the body of the Kafka message to contain just the payload of the HTTP response, whilst the connector wraps it around some custom schema.

Kafka Connect offers you some transformations out of the box, and there is one in particular than can be of help here: ExtractField (https://docs.confluent.io/platform/current/connect/transforms/extractfield.html), although I'm afraid it won't completely remove the schema.

The alternative would be to implement a custom KvSourceRecordMapper that doesn't wrap the response around this schema.

I suggest you play around with the ExtractField SMT, maybe that's enough for you. Anyway, I'll make available a new KvSourceRecordMapper that does exactly that, for whoever might be interested.

Best regards.

[castorm]

I just realized, this already existed, you can take a look at #19 for reference. The two options offered above are further described there.

[QuidneIT]

Castor, Thank you very much for this. This has help to resolve the problem. 😊 And sorry, to have asked the question as I should have looked there first. Raoul From: Cástor Rodríguez ***@***.***> Sent: 26 March 2021 07:14 To: castorm/kafka-connect-http ***@***.***> Cc: Raoul van Eijndhoven ***@***.***>; Mention ***@***.***> Subject: Re: [castorm/kafka-connect-http] JSON returned to kafka topic is more than expected. (#113) I just realized, this already existed, you can take a look at #19<#19> for reference. The two options offered above are further described there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#113 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AF2MOWHWHFFL7NA3H5QV7ODTFQX4FANCNFSM4ZTC4RXA>.