-
Notifications
You must be signed in to change notification settings - Fork 76
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
earthaccess.download()
ignores errors
#581
Comments
@mfisher87 This seems interesting, what should we proceed |
@Sherwin-14, we want to proceed with both, meaning that we want both options available to the user by doing the following:
|
@Sherwin-14 don't forget to take a look at #36 for more context! Do you want to be assigned this issue? |
def get(
self,
granules: Union[List[DataGranule], List[str]],
local_path: Union[Path, str, None] = None,
provider: Optional[str] = None,
threads: int = 8,
) -> List[str]:
if local_path is None:
today = datetime.datetime.today().strftime("%Y-%m-%d")
uuid = uuid4().hex[:6]
local_path = Path.cwd() / "data" / f"{today}-{uuid}"
elif isinstance(local_path, str):
local_path = Path(local_path)
if len(granules):
files = self._get(granules, local_path, provider, threads)
return files
else:
raise ValueError("List of URLs or DataGranule instances expected")
@singledispatchmethod
def _get(
self,
granules: Union[List[DataGranule], List[str]],
local_path: Path,
provider: Optional[str] = None,
threads: int = 8,
) -> List[str]:
raise NotImplementedError(f"Cannot _get {granules}") @mfisher87 @chuckwondo I had been looking into this and it seems like |
@Sherwin-14, just search the code base for Whichever functions you find it in, do the following:
Then, you have to find the places where those functions are called (i.e., where the functions that call
Also, in each modified function, update the docstring appropriately to describe the added Writing tests for these changes will likely be tricky, so you can ignore this for the moment and focus first on the steps above and make sure that existing tests pass (i.e., make sure you didn't break existing stuff before worrying about writing tests for the new stuff). |
We found the
pqdm
library is the thing swallowing all the error messages.pqdm
has anexception_behavior
option, which defaults to"ignore"
, surprisingly. The other two settings,"deferred"
and"immediate"
, are both good options for earthaccess.@chuckwondo and I agree
"immediate"
is the most intuitive default, and we can establish a new flagfail_fast: bool = True
to override it to"deferred"
by passingdownload(..., fail_fast=False)
. Prior discussion in #36. Please continue discussion here :)I feel exposing the
"ignore"
behavior isn't necessary because the user can do their own exception handling withtry
.Thoughts?
The text was updated successfully, but these errors were encountered: