-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Streamline upload process on the backend #75
Labels
Comments
@juliamullen : Thinking about this further, it'd be good to decouple and expose the jsonvalidation process to check the validation before trying the POST command (either in the command-line and/or GUI on the website). ideally, the validation process should be checked first, and then make the POST request. might also make it easier to take advantage of the elasticsearch python library capabilities. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uploading large chunks of data is a pain, since there's not a good way to queue the data to be uploaded, and due to the complexity of the .json validation before ES-insertion, 300 records takes ~ 5 min to upload.
There are at least a few limits to queuing large amounts of data:
Ideally, we could queue a buncha records and let it do its thing overnight. This may involve moving away from the front-end interface, but we'll still have problems with the multiprocessing inserting duplicates.
The text was updated successfully, but these errors were encountered: