Note: This is one of multiple posts about issues I’ve discovered while working depth with the Pipedrive API. A description of the use case and collection of all issues can be found at pipedrive-api-notes.md · GitHub.
There seems to be no “stable” way to handle pagination on large, “active” resources. Example:
- I want to fetch multiple pages of
activities. - I fetch page 1 from the API.
- While fetching page 1, a new activity gets added.
- When I fetch page 2, I get a duplicate activity that I’ve already fetched at page 1 and the newly added activity never gets fetched.
- The problem only becomes worse as the result set and/or number of active users increases.
Of course, it’s possible to work around this using a filter, i.e. before running the paginated requests, create a filter with update_time = the-start-time-of-the-process and then delete the filter once it’s done. But this is quite cumbersome => there should be a better way (e.g. something that functions like a cursor in databases).
