Athena result object should have type information, and a way to make it a dataframe

[nick-megaport]

If i connect to athena and query it, i get back a result object like so;

> result |> Map.keys()
[:columns, :rows, :__struct__, :output_location, :query_execution_id, :statement_name]

livebook will display this result with typed fields

to convert it to a dataframe i do

df = result |> Explorer.DataFrame.new()

but now it's untyped. I think we need a common way to get data from these external data sources into typed dataframes. I'd love to see elasticsearch support for instance. I'm not sure the best way to go about it.

I should also note that it would be great if datetime columns were also converted (another pain point!)

[josevalim]

Does Athena return the type information? If so, we can add it to the struct and then use it on Table.Reader. Check out ReqAthena. :)

[josevalim]

Maybe we should revert it. We should store :aws_athena_types, with no conversion, and whoever is capable of converting does it.

[jonatanklosko]

Still with namespace right? {:athena, :types}?

[josevalim]

I think :aws_athena_types are {:athena, :types} are both namespaced, no? Just in different ways?

[jonatanklosko]

Yeah, it's just that Table.Reader allows tuples, not arbitrary atoms :)

[josevalim]

Oh, so {:athena, :types} it is!

[nick-megaport]

Yep looks like it!

https://github.com/livebook-dev/req_athena/blob/main/lib/req_athena.ex#L297-L314
https://docs.aws.amazon.com/athena/latest/APIReference/API_GetQueryResults.html

[nickkaltner]

req_athena now exports the type information, but the version hasn't been yet released. See livebook-dev/req_athena#34 for some discussion on it. It's the first step for this feature.