Request for tips on code 429 and 500 .. btac in docker and behind nordvpn

[edzob]

I am running btac with nordvpn in a docker container.

See readme.md and dockerfile below.
I needed to upload Dockerfile as .md due to upload restrictions.

README.md
Dockerfile.md

I recieved some 429 and 500 warnings in the outout
I think these warnings are no issue, since they are warnings.

For example:

[unpaywall] ahmad_investigation_2015: WARNING: response: 500 INTERNAL SERVER ERROR in 0.446s
[s2] ciborra_platform_1996: WARNING: response: 429 in 0.535s
[dblp] barreto_cyber-argus_2019: WARNING: response: 429 Too Many Requests in 1.19s

Question:

Can I tweak something in the timeout settings in btac or in the vpn server country, that would minimize the amount of warnings?

[dlesbre]

These warnings indicate a request to a database couldn't complete correctly:

• 429 is "too many requests", the server complains that it is overloaded and can't handle btac's query. I try to respect API rate limits when querying with btac but these still crop up from time to time.
• 500 is "internal server error", i.e. something went wrong on the server.

As you mentioned, getting a few of them isn't much of an issue (you will just lack completion data for some entries from the erroneous source), however getting a lot can be a problem (if all sources fail on a single entry, then no data will be added...). I agree that spamming output is bad practice, but so is hiding warnings about how well our request are going.

[dlesbre]

So to answer your question: not really I'm afraid. These warnings depend on the dbs btac is querying, not btac itself. I could modify the script so it aggregates warnings and displays a summary of failed requests at the end instead of a warning for each failed request though.

[edzob]

For me the warnings are perfect. It is a level of verbose that helps me know something is going on, and if I want less verbose I can pipe the output into a file.

It does worry me that a not so big a .bib file is now running for several hours (8+) without completion, and I do not know how to determine if it is due to my setup (docker/ nordvpn) or just bad luck, or something else.

Perhaps I should use the verbose option to see per record what is going on.

[dlesbre]

8h without completion sounds REALLY excessive. Yes the verbose option would help figure out what is going on, as it prints one line per completed entry and shows how many requests have been performed for each source.

There are two source of delays in btac:

• Waiting for response after sending a request. By default the timeout is set to 20s on these, and they are multi-threaded (querying every db in parallel). So in theory worse case time is 40 seconds per entry from the timeouts, but in practice most queries complete in 1-2s.
• Rate limits: most API require you stick to a reasonable number of queries per unit of time, by default this is one query every 0.02s, but it can be changed dynamically by the response headers. Usually this sticks to reasonable delays allowing a few queries per minute but its possible you got hit by a message with a very high delay request (especially given rate limits are often set based on IP address, and if you use a VPN you share your IP with all other VPN users).

[dlesbre]

Unfortunately btac can't be safely interrupted without losing all data... I should probably add a safety dump in case of interruptions, but I never imagined it would take this long to run.

(EDIT: I've implemented a dump to temporary file in case of interruption in v1.3.3)

[edzob]

and I found out that I am an idiot. Somehow it was not processing a sub-collection of zotero (export as bib), but the whole library. That explains that it was running for so long.

So I basically did an accidental performance test.
and BTAC did wonders in that regard :)