Pagination on "active" resources

Note: This is one of multiple posts about issues I’ve discovered while working depth with the Pipedrive API. A description of the use case and collection of all issues can be found at · GitHub.

There seems to be no “stable” way to handle pagination on large, “active” resources. Example:

  • I want to fetch multiple pages of activities.
  • I fetch page 1 from the API.
  • While fetching page 1, a new activity gets added.
  • When I fetch page 2, I get a duplicate activity that I’ve already fetched at page 1 and the newly added activity never gets fetched.
  • The problem only becomes worse as the result set and/or number of active users increases.

Of course, it’s possible to work around this using a filter, i.e. before running the paginated requests, create a filter with update_time = the-start-time-of-the-process and then delete the filter once it’s done. But this is quite cumbersome => there should be a better way (e.g. something that functions like a cursor in databases).

Thanks for sharing the details. I have already captured it as feedback to the relevant team :slight_smile: