Customizable Pagination with Limit and Offset in Kinto

[luis-doudeau]

Hello,

I'm currently working on a project that utilizes Kinto, and I'm looking to implement a more personalized pagination system that allows for specifying a limit and an offset. At present, the pagination for querying a collection is standardized: it returns 100 rows at a time along with a token for fetching the next 100 records. I'm interested in modifying this behavior to enable setting an offset (for example, 100) and a limit (for example, 20), allowing access to records in the range from 100 to 120.

This customization would greatly enhance our ability to navigate through data in a more efficient and tailored manner. I'm reaching out to the Kinto maintainers or anyone with in-depth knowledge of Kinto for guidance. Could you please advise on how I might develop a plugin or modify the SQL code to introduce these parameters (limit and offset) for achieving this modified pagination system?

Thank you very much for your assistance and time.

Best regards,
Luis

[leplatrem]

We would welcome any contribution that would introduce the necessary settings or code change.

For example the code that controls the default limit/paginate_by is this:

kinto/kinto/core/resource/__init__.py

Lines 1044 to 1057 in cb9cbf7

	def _extract_limit(self):
	"""Extract limit value from QueryString parameters."""
	paginate_by = self.request.registry.settings["paginate_by"]
	max_fetch_size = self.request.registry.settings["storage_max_fetch_size"]
	limit = self.request.validated["querystring"].get("_limit", paginate_by)
	# If limit is higher than paginate_by setting, ignore it.
	if limit and paginate_by:
	limit = min(limit, paginate_by)
	# If limit is higher than what storage can retrieve, ignore it.
	limit = min(limit, max_fetch_size) if limit else max_fetch_size
	return limit

By setting this value, the pages will be smaller:

kinto.paginate_by = 20

As for the offset, if you sort by a specific field, and you add gt_field={offset} you may be able to obtain what you're looking for.

I would suggest you to make some rough code changes in a draft pull-request, see if you obtain what you want, and then we can guide you from there to turn it into a real contribution, with tests etc.

Hope that helps :)

[luis-doudeau]

First and foremost, thank you for your prompt and detailed response. However, I believe there might be some misunderstanding regarding the proposed solution for implementing an offset mechanism using gt_field={offset}. My goal is to establish a pagination system that operates independently of query result order and can be applied across various collections without necessitating the addition and sorting by a specific field in every collection. Essentially, the suggested method seems less feasible for my use case, as it requires sorting query results by a specific field, which does not align with the flexibility I'm aiming for in pagination. Specifically, I'm seeking to implement a SQL-like pagination feature, where 'limit' and 'offset' parameters allow for fetching a precise range of records, independent of the field-based sorting or any additional modifications to the dataset.

Given this context, I'd like to further inquire whether addressing this pagination issue could be effectively achieved through plugin development, or if it necessitates direct modifications to the Kinto source code. What would be your recommendation for proceeding in a manner that aligns with Kinto's architecture and practices?

Thank you again for your time and assistance :)