Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adapter batching #19

Merged
merged 10 commits into from
Jul 20, 2023
Merged

Adapter batching #19

merged 10 commits into from
Jul 20, 2023

Conversation

aMahanna
Copy link
Member

@aMahanna aMahanna commented Jul 7, 2023

pip install torch
pip install git+https://github.com/arangoml/pyg-adapter.git@feature/batching

As we work towards cleaning up the ArangoDB to PyG & PyG to ArangoDB transformations, reworking the batching process is crucial to optimizing the adapter's memory consumption.

Branch has been based off of feature/exceptions in order to allow for its review & merge after #17

New changes:

  1. ArangoDB to PyG batching via the batch_size parameter of the db.aql.execute() method (python-arango)
  2. PyG to ArangoDB batching via a new batch_size parameter in the pyg_to_arangodb method
  3. Reworking the fetch_adb_docs method to only fetch the document attributes specified by the metagraph (get_aql_return_value)
adbpyg_adapter.arangodb_to_pyg("imdb", metagraph, batch_size=100) # Fetches & processes up to 100 ArangoDB Documents/Edges at a time

adbpyg_adapter.pyg_to_arangodb("karate", data, metagraph, batch_size=100) # Processes & ingests up to 100 PyG Nodes/Edges at a time

@aMahanna aMahanna marked this pull request as draft July 7, 2023 18:47
@aMahanna aMahanna marked this pull request as ready for review July 17, 2023 20:37
@aMahanna aMahanna changed the title WIP: Batching Adapter batching Jul 17, 2023
@aMahanna aMahanna changed the base branch from feature/exceptions to master July 18, 2023 16:09
Copy link
Contributor

@geenen124 geenen124 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - nice work!

@geenen124 geenen124 merged commit 0f14366 into master Jul 20, 2023
6 checks passed
@aMahanna aMahanna self-assigned this Jul 20, 2023
@aMahanna aMahanna deleted the feature/batching branch October 25, 2023 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants