Source: background file downloads for FileSource #670
Conversation
tim-quix
changed the title
| def stop(self): | ||
| logger.info("Stopping file download thread...") | ||
| self._stopped = True | ||
| self._executor.shutdown(wait=False) |
There was a problem hiding this comment.
we should cancel the in-progress future before shutting down.
We also should wait=True the shutdown to make it clean
There was a problem hiding this comment.
So the original reason I did this was to hopefully kill the file download immediately rather than waiting to complete it, as it seems you cant in actuality cancel already running futures (you can still call cancel on it, but from what I can tell, it won't do anything).
However, I think for now I'll do a wait, and later on if we really want to we can add a mid-download stop (where you read chunks of data of the file at a time and look for stopping). I can't imagine people will be stopping in the middle very often anyway.
There was a problem hiding this comment.
In any case we have the source shutdown timeout
tim-quix
force-pushed
the
source/file-download-queue
branch
from
09eb826 to
8d7cbcf
Compare
This adds a simple queue to the FileSource connector to download the next file while the current one is being processed.