Replace master_scrape with a single api call

[jojo2357]

There exists a place in the API to get a list of all current offerings.

Pros:

• Single call, great for slower connections
• Removes need for step 4 since we get the newest version as well as the size in one go. This will then tell us if there is an update or not

Cons:

• Makes it harder to find the previous version. If we just cache the sha's then this doesnt really matter
• Request is big, 5_100_257 bytes. A lot more information is transmitted than is strictly required

I figure that if you plan to download an archive, then 5MB isnt too bad and that having to wait for multiple requests can be a bigger issue.

Thoughts? @Jaifroid @DocDrydenn

[DocDrydenn]

Interesting.... I like the idea of a single call, but what's the time difference we're looking at? I mainly ask because most users will be running the script twice. Once for the "dry-run" to see what will happen, then a "real" run to commit.

Could the initial "dry-run" API call be cached, then utilized by any follow-up script runs? Maybe utilize a timestamp to determine if the cache has "expired". Fresh caches get reused, but an expired cache causes a new fetch. Not sure... just thinking out loud...

[jojo2357]

Yes I was thinking about caching, probably only re-fetch once daily

The speed difference is fairly pronounced, replace all of the time to build an index (aka step 2) with this command:
curl "https://library.kiwix.org/catalog/v2/entries?count=-1" > /dev/null
and tell me roughly what you see

Despite resulting in a large response, it is still faster because it only uses one connection instead of 20.

[jojo2357]

I ran wget -q -O - "https://library.kiwix.org/catalog/v2/entries?count=-1" | grep -i 'application/x-zim' | grep -ioP '(?<=href=")[\w:\/\-.]+(?=\.meta4")' | grep -ioP "$BaseURL[^/]+/\K.*" which prints line by line each zim (only the newest version)

I am currently downloading zims because i may or may not have accidentally deleted my entire zim library, and it still took only 3.5 seconds.

It is fast. if you adapt the processing, you can get all the file sizes, all the url's, and all of the available categories (so if you want to go back to search for older versions, you now know the links)

Right now we run 21 wgets (1 to build the /zims index, 20 for the 20 results) and this one command replaces all of them. I think I will proceed with reworking this and hopefully reducing the overall number of arrays we have because it is getting unwieldy

[jojo2357]

I pushed my changes on my master branch. Please test it, but be warned it has not been stress-tested. I only got my first successful run just now. Notice anything different?