Support for avro union types

[kvedes]

Hi

I'm trying to use Arroyo on a Kafka topic from Confluent. I can successfully set up the connection and get the Avro schema which is stored in the Schema Registry. However, when I try to create a pipeline my source shows as JSON instead of Avro.
I suspect this is due to my Avro schema containing Union types. I cannot find any documentation related to union types, so I suppose it might not be supported.
Will this be coming the with the Protobuf support? - Protobuf has similar semantics just called oneof. Otherwise I would like to request it.

P.S. I'm brand new to Arroyo

[timonviola]

Hey, did you check this part of the docs: https://doc.arroyo.dev/connectors/confluent#avro

Just to understand you are seeing the expected behavior (quoting the docs):

For Avro, there are some features that cannot be converted to SQL types:

• Unions (aside from unions that are a single type and null, which are converted to nullable columns)
  ...
  Fields with these types will be re-encoded as JSON and stored in a JSON column, in which form they can be processed via SQL JSON functions.

Is this what you are referring to "source shows as JSON"?

[kvedes]

Hi @timonviola

Thanks, yes this is exactly what I'm seeing. Was hoping to see support for general union types, but doesn't sound like that would be on the roadmap.

[kvedes]

@mwylde Just saw that arroyo 0.12 has full support for protobuf, does this include the "oneof" type in protobuf? - If so supporting Avro Union types should require a very similar implementation.