OrgBook BC API returns a 400 Error

[amanji]

It was reported that the (v3/v4)/search/topic endpoint returns a 400 Error when paging past a certain number of results.

For example: https://orgbook.gov.bc.ca/api/v4/search/topic?ordering=-score&q=abc&inactive=false&latest=true&revoked=false

Returns 541 results

The first 10 pages of results can be accessed:

https://orgbook.gov.bc.ca/api/v4/search/topic?inactive=false&latest=true&ordering=-score&page=10&q=abc&revoked=false

The result of the 10th page shows a URL to the next page available:

Accessing the 11th page onwards returns a 400 Error. This is consistent regardless of the query used.

[swcurran]

That’s a “feature” that @WadeBarnes added to stop bots from scraping the database.

Who reported the issue?

[WadeBarnes]

@swcurran, Correct

[esune]

Closing as this is a known behaviour.

For context, the limit to query paging was introduced to avoid issues deriving from performing queries that fetch large data sets (unbounded) that would cause out-of-memory issues in the search engine.

References:

• Don't allow unbounded search queries  #762
• Investigate how Aries-VCR is performing paging when querying Solr #757
• Update search-engine resource allocations orgbook-configurations#137 (comment)