You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Top: current system for retrieval. Bottleneck is present when saving PubMed articles
Bottom: proposed design for making both retrieval and storage of PubMed articles using a bunch of "workers" (mostly likely EC2 instances).
Also, the "retrieval" call from ReCiter to PubMed service will be async. ReCiter will be notified of the success/failure of retrieval (list of pmids retrieved successfully, list of pmids retrieved unsucessfully) via a callback.
PubMed service will orchestrate the retrieval "workers" (retrieve PubMed articles) and storage "workers" (storing PubMed articles into DynamoDB). PubMed Retrieval Service still does what it does now: accept PubMed queries and returns results of the queries.
Q: How do you divvy up work among the workers? 100 each?
A: # pubmed_articles / # workers
The text was updated successfully, but these errors were encountered:
Top: current system for retrieval. Bottleneck is present when saving PubMed articles
Bottom: proposed design for making both retrieval and storage of PubMed articles using a bunch of "workers" (mostly likely EC2 instances).
Also, the "retrieval" call from ReCiter to PubMed service will be async. ReCiter will be notified of the success/failure of retrieval (list of pmids retrieved successfully, list of pmids retrieved unsucessfully) via a callback.
PubMed service will orchestrate the retrieval "workers" (retrieve PubMed articles) and storage "workers" (storing PubMed articles into DynamoDB). PubMed Retrieval Service still does what it does now: accept PubMed queries and returns results of the queries.
Q: How do you divvy up work among the workers? 100 each?
A: # pubmed_articles / # workers
The text was updated successfully, but these errors were encountered: