feat(python): handle PyCapsule interface objects in write_deltalake #2534
Conversation
kylebarron
requested review from
wjones127,
fvaleye,
roeap and
ion-elgreco
as code owners
|
ACTION NEEDED delta-rs follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. |
ion-elgreco
force-pushed
the
kyle/write-pycapsule
branch
from
77d03f0
to
1c805e1
Compare
|
@kylebarron can you fix the linting issues? Then we can merge it Also wondering, how we should typehint this now, since an input can have the c_stream attribute or not |
ion-elgreco
changed the title
I'm pretty packed but I can try to find some time soon.
You can use these type hints: https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html#protocol-typehints |
|
@kylebarron ah nice, do you mind adding those typehints when you find the time |
|
I believe I fixed the lint and fixed the type hinting. In the future, a more involved PR could remove pyarrow as a required dependency entirely by passing the C stream pycapsule directly to Rust (arrow-rs has an example of how to do that here) |
|
@kylebarron that would be nice, we could potentially make it opt-in since it would then only be needed for reading |
ion-elgreco
force-pushed
the
kyle/write-pycapsule
branch
from
387b819
to
dfe6e4a
Compare
Description
Adds support for the Arrow PyCapsule interface.
Since pyarrow is already a required dependency, this takes the minimal route of converting pycapsule interface objects into pyarrow objects. This requires pyarrow 15 or higher for the stream conversion (apache/arrow#39217).
This doesn't modify the existing hard-coded support for pyarrow and pandas
Related Issue(s)
Documentation