Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [WIP] Barrage Refactor Read/Write Chunk Factories and Default Type Mappings #6065

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

nbauernfeind
Copy link
Member

@nbauernfeind nbauernfeind commented Sep 13, 2024

Fixes #58 (custom type serialization / deserialization).
Fixes #936 (ColumnConversionModes is being replaced with easy to integrate codecs).
Fixes #2984 (refactoring has good interface documentation).
Fixes #3403 (by supporting a variety of mappings, these now must match client wiring).
Fixes #4533 (java barrage client do put support).
Fixes #5258 (snapshot/subscribe methods with default w2w options).
Fixes #5453 (support other Timestamp arrow wire encodings).
Fixes #5864 (support for uint64_t).
Fixes #5969 (supports custom wire mappings).
Fixes #6114 (supports ObjectVector properly).

This PR adds support to (some to be implemented as this is a WIP):

  • enables users to target wire types and configurations
  • enables users to target deephaven types, including custom codecs
  • improves snapshot/subscription APIs for default options
  • adds a do put style API from dh table for worker to worker

Copy link
Member

@rcaudy rcaudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A little nervous about the scope, and the willingness to truncate inputs or infer units.

engine/chunk/src/main/java/io/deephaven/chunk/Chunk.java Outdated Show resolved Hide resolved
@@ -18,7 +20,56 @@
/**
* Consumes Flight/Barrage streams and transforms them into WritableChunks.
*/
public interface ChunkReader {
public interface ChunkReader<ReadChunkType extends WritableChunk<Values>> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We wondered if you could make the bound looser, e.g. PoolableChunk or SafeCloseable. Looks like maybe not? Should PoolableChunk be a Chunk (it's not right now)?

}
}

private static long factorForTimeUnit(final TimeUnit unit) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ryan's placeholder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment