vdk-oracle: Pass ingestion payload rows in uniform batches #3194
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DeltaMichael
force-pushed
the
person/mdilyan/oracle-uniform-batches
branch
from
091b618 to
679b21d
Compare
DeltaMichael
commented
Mar 12, 2024
projects/vdk-plugins/vdk-oracle/src/vdk/plugin/oracle/ingest_to_oracle.py
Outdated
Show resolved
Hide resolved
duyguHsnHsn
reviewed
Mar 12, 2024
projects/vdk-plugins/vdk-oracle/src/vdk/plugin/oracle/ingest_to_oracle.py
Outdated
Show resolved
Hide resolved
duyguHsnHsn
approved these changes
Mar 13, 2024
DeltaMichael
force-pushed
the
person/mdilyan/oracle-uniform-batches
branch
2 times, most recently
from
b0a36bd to
da46bea
Compare
DeltaMichael
commented
Mar 15, 2024
projects/vdk-plugins/vdk-oracle/src/vdk/plugin/oracle/ingest_to_oracle.py
Show resolved
Hide resolved
DeltaMichael
force-pushed
the
person/mdilyan/oracle-uniform-batches
branch
3 times, most recently
from
bf4139a to
b24c0c0
Compare
duyguHsnHsn
approved these changes
Mar 22, 2024
yonitoo
approved these changes
Mar 22, 2024
projects/vdk-plugins/vdk-oracle/src/vdk/plugin/oracle/oracle_configuration.py
Outdated
Show resolved
Hide resolved
projects/vdk-plugins/vdk-oracle/src/vdk/plugin/oracle/ingest_to_oracle.py
Outdated
Show resolved
Hide resolved
DeltaMichael
force-pushed
the
person/mdilyan/oracle-uniform-batches
branch
2 times, most recently
from
b0405a9 to
ec3a18f
Compare
gabrielgeorgiev1
approved these changes
Mar 22, 2024
DeltaMichael
force-pushed
the
person/mdilyan/oracle-uniform-batches
branch
from
ff0934c to
eb1bf5e
Compare
Why? The current ingestion implementation batches payload rows by column keyset. Payloads with the same keyset are batched together and passed to an executemany() call. This is not ideal becauase it can result in a large number of executemany() calls What? Make each payload row uniform by identifying missing columns and filling them out with null values. Pass data rows to executemany() in uniform batches. How was this tested? Functional tests What kind of change is this? Feature/non-breaking Signed-off-by: Dilyan Marinov <mdilyan@vmware.com>
DeltaMichael
force-pushed
the
person/mdilyan/oracle-uniform-batches
branch
from
eb1bf5e to
667e4c6
Compare
yonitoo
pushed a commit
that referenced
this pull request
Mar 26, 2024
## Why? The current ingestion implementation batches payload rows by column keyset. Payloads with the same keyset are batched together and passed to an executemany() call. This is not ideal becauase it can result in a large number of executemany() calls ## What? Make each payload row uniform by identifying missing columns and filling them out with null values. Pass data rows to executemany() in uniform batches. ## How was this tested? Functional tests ## What kind of change is this? Feature/non-breaking --------- Signed-off-by: Dilyan Marinov <mdilyan@vmware.com> Co-authored-by: Dilyan Marinov <mdilyan@vmware.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why?
The current ingestion implementation batches payload rows by column keyset. Payloads with the same keyset are batched together and passed to an executemany() call. This is not ideal becauase it can result in a large number of executemany() calls
What?
Make each payload row uniform by identifying missing columns and filling them out with null values.
Pass data rows to executemany() in uniform batches.
How was this tested?
Functional tests
What kind of change is this?
Feature/non-breaking