-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Better timezone functionalities #10368
Comments
See the result in Postgres
I think we should not return 9:00:00, because the local time for timezone +1 is 8:00:00. so |
btw, I think there might be some bug in datafusion, the behavior is unlike Postgres
|
Thanks for the response!
I wonder whether it has to do with the
This is the result I would expect See: https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-ZONECONVERT Now that I think about it, it might be that what I am proposing is actually for the |
I haven't read this ticket in detail yet, but there is a collection of other issues on #8282 |
In postgresql the In datafusion this expression casts from one |
I have filed #10602 with a summary of how I understand the usecase of "how do we bin timesamps in timezones with daylight savings time correctly" and some ideas on how to proceed |
Is your feature request related to a problem or challenge?
It would be nice if Datafusion has support for better timezone functionalities. Though painful, local time is often used in energy analytics and data reporting (especially when the reporting is meant for official authorities).
Specifically, I would like to have the following functionalities:
date_part
) for example, might be a nice to havechrono
seems to prefer failing/returningNone
when there is ambigous results regarding daylight savings, which I assume is an inherited behavior for Arrow. If we can have some kind of flag to not fail, but either do something else , it would be great.Describe the solution you'd like
I am not sure how this can be implemented in a generalized way. Perhaps by adding a function
to_local_time
that takes a timestamp/column as an argument, and a timezone as another argument. For exampleThe difference between this and the
at time zone
operator is:The
at time zone
operator seems to be only adding timezone/offset information to the timestamp if I understood correctly. the proposedto_local_time
function should ideally apply that offset to the provided timestamp. The same example above would be:Describe alternatives you've considered
I have currently a hacky alternative to aggregate using local time. What I do basically is the following:
time
columntime
using:((time AT TIME ZONE 'UTC' at time zone 'Europe/Brussels') - (time AT TIME ZONE 'Europe/Brussels' at time zone 'UTC'))::interval
(time AT TIME ZONE 'Europe/Brussels' at time zone 'UTC')
fails during daylight savings hour. This then becomes for example:The daylight savings predicates are dynamically generated in an application layer
time
This all becomes something like:
Additional context
No response
The text was updated successfully, but these errors were encountered: