Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a dynamic chunkSize service on the Transfer-Service #70

Open
jgoldverg opened this issue Apr 15, 2023 · 0 comments
Open

Create a dynamic chunkSize service on the Transfer-Service #70

jgoldverg opened this issue Apr 15, 2023 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@jgoldverg
Copy link
Contributor

jgoldverg commented Apr 15, 2023

Problem: Currently the chunksize found in the infoList of EntityInfo.java objects results in the FilePartitioner splitting the file into symmetric chunks. The issue being that we sometimes want to resize the chunksize to reduce or increase strain on the network/service in general.

Solution: Make a class that manages the chunksize found in the FilePartitioner that will enable re-sizing of all the chunks in the ConcurrentQueue found in the FIlePartitioner. Create a controller that will enable a user to re-size the chunks on the fly of an ongoing transfer. An interesting idea is that the filePartioiner will produce a chunk of a size that is the current chunkSize in the class. This way we dont need to pre-compute all the chunks, we can simply query the next chunk based on the chunkSize found in this class.

What we want to do is create a single service that is responsible for sizing the next chunk that the reader will produce.

@jgoldverg jgoldverg added the enhancement New feature or request label Apr 15, 2023
@lokesh-1144 lokesh-1144 self-assigned this Apr 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants