[Feat.] build ChekPoint system for the scrapper. #9
Labels
easy
GSSOC label for beginner tag
enhancement
New feature or request
good first issue
Good for newcomers
gssoc20
GSSOC label for gscco20 tag
help wanted
Extra attention is needed
In case of a sudden failure and crash, we don't want the process to restart and start scrapping from the beggining.
** REQUIREMENT **
1 . Add method of Database class inside db.py to store check point information
2. Let user implementing plugin decide how to use the checkpointing system.
Note: Database class uses pymongo and contains no data that is modified after creation of object. So it's safe to say it is thread safe.
The text was updated successfully, but these errors were encountered: