-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Corrupt record handling #53
Comments
Thanks for your very clear bug report! I'll look into the details. I see that I'll follow both of your suggestions: trap such errors better and also report the problem to arxiv, if there's a problem either with the record or with their API. |
Okay, I get it. For this search, you get proper results if limit <= 77, but if limit >= 78, it returns NULL. If batchsize < limit and you're in this latter case, you get the error about assigning attributes to NULL.
|
Certain records seem to cause a crash. We have narrowed it down to this query, which should retrieve all records submitted in a one-minute period of 22:16 to 22:17 on January 24, 2018.
dfy<-arxiv_search(query = "submittedDate:[201801242216 TO 201801242217]", limit = 15000, batchsize=2000)
which returns an error of:
We can isolate the record, which appears to be this one:
https://arxiv.org/abs/1610.04266
If we were to search using title, the same error appears:
dfy<-arxiv_search(query = "ti:Fourfolds", limit = 1200, batchsize=300)
We therefore think that either the record is corrupt (e.g., hidden unintentional column delimiter, etc.)
A similar error occurs on this single-date range, though we have not isolated the individual record causing the error:
dfy<-arxiv_search(query = "submittedDate:[201612030000 TO 201612040000]", limit = 15000, batchsize=2000)
Does the query need to be modified? Can the query auto-skip corrupt records? Should arxiv be notified?
The text was updated successfully, but these errors were encountered: