For the moment we blocked the pagination of any API to be under the max result window of ElasticSearch (pageSize * page < 1000)
@raphael.flores @celia.michotey @cyril.pommier @jeremy.destin
In this branch, I propose a solution to implement deep pagination on BrAPI calls. We've had problems with BrAPI client trying to harvest our big datasets, and we'd like to propose a reasonable solution without having to change all of the BrAPI.
The data discovery calls are for now still restricted on pagination in this branch (see the @MaxResultWindowPagination annotation on DataDiscoveryCriteriaImpl.java)
In order to fetch data over the max result window we can either use the scroll API or the search after API. The scroll API cannot be used to fetch pages in random access whereas the search after API can so I chose to implement deep pagination with the search after API.
The search after API requires the result to be sorted (here we use the sort on _id by default) and the previous page last document sort values.
The ESGenericFindRepository in charge of finding documents by criteria now detects when a request goes over the ES max result window and will:
When the requested page is under the ES max result window, the repository does a simple search with size/from pagination and do not use the search_after
To be efficient a cache is used (with a TTL of 1H) to store every page last sort values for every query.
The cache is structured as such: QueryWithoutPagination(String) => Page{size,from} => LastSortValues(Object[]) (see ESGenericFindRepository).
If the previous page for the current query is not in cache, the repository will paginate from the last page under the max result window to the current page and store each sort values in cache for future use. In classical scenarios, the BrAPI client will fetch a page for a given query/criteria, the repository will store this page's last document sort values and when the client request the next page, the repository has in cache everything it needs to fetch the current page.
master