Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ref(api): Move chunk data types to submodule #2086

Merged
merged 1 commit into from
Jun 10, 2024

Conversation

szokeasaurusrex
Copy link
Member

@szokeasaurusrex szokeasaurusrex commented Jun 7, 2024

  • Create a new data_types submodule under the api module.
  • Crate a chunking submodule under the data_types module.
  • Move all data types related to chunking from src/api/mod.rs to the chunking submodule, splitting their definitions further into submodules under chunking.

This PR only moves existing code, adding mod and use statements and adjusting visibility as needed. No other modifications were made.

Ref #2077

use serde::{Deserialize, Deserializer};

#[derive(Debug, Clone, Copy, Ord, PartialOrd, Eq, PartialEq, Default)]
pub enum ChunkCompression {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

driveby question: how much effort would it be to add zstd compression to the whole workflow?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure – have not worked too extensively with compression in the CLI yet.

What would be the benefit of adding zstd? Do we have any docs written about it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, we don’t have any docs on that.
and zstd is pretty much the best choice if you want to trade compression ratio for resource usage.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, we should probably investigate how much effort this would be, will open an issue

 - Create a new `data_types` submodule under the `api` module.
 - Crate a `chunking` submodule under the `data_types` module.
 - Move all data types related to chunking from `src/api/mod.rs` to the `chunking` submodule, splitting their definitions further into submodules under `chunking`.

Ref #2077
Copy link
Contributor

@loewenheim loewenheim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These modules are a lot smaller than I would make them, but it's up to you :)

@szokeasaurusrex szokeasaurusrex merged commit b37ff70 into master Jun 10, 2024
12 checks passed
@szokeasaurusrex szokeasaurusrex deleted the szokeasaurusrex/data-types branch June 10, 2024 15:44
@szokeasaurusrex
Copy link
Member Author

@loewenheim Fair enough, I guess it comes down to personal preference a bit.

My reasoning is that each of these new modules has now a distinct responsibility; all code in a single module is closely related. When files get long, I have difficulty keeping track of everything in the file. It is much easier for me to work with many small files where I can clearly wrap my head around each one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants