Handling Pagination

API
1

I’m trying to build a messaging app extension. I believe I’ve handled the installation flow correctly and the sending a message flow also seems to work quite well.

The issue I’m having is when I test it for a user who has a conversation with a lot of messages (1000+): the JSON response I send to the getConversations endpoint has all the messages but only a fraction of them show up in the inbox.

This doesn’t seem to be an issue when a room has a little over 100 messages. I haven’t tested what happens when a user has many conversations, but I expect a similar error to occur then.

Is this about implementing pagination? If so, how would I paginate my response to getConversations so that both the conversations and messages are paginated 30 by 30 as suggested?

I’m sorry if this is a redundant issue, I tried to find a specific answer in the docs and the forums but couldn’t find any.

2

Also there’s an issue in the documentation. Here in the docs it says the query parameter after is an integer, but in actuality it’s a string – precisely the next_messages_cursor we pass in getConversations.

3

Hey @felatun, welcome to the community! :tada:

getConversations endpoint is called after a new channel is created. The request contains also

  • conversations_limit with value 30
  • messages_limit with value 30

We expect that the response will contain 30 conversations max and every conversation will contain 30 messages at maximum. This is because we would like to sync just the recent conversations, we do not want to copy all the user’s data.

If the conversation has more than 30 messages, you can send next_messages_cursor with the conversation and we will use that cursor for calling your API in case the user needs more than 30 messages (there is an infinite loader on the UX).

If you do not send the next_messages_cursor with the conversation it means there are no more messages.

Hope it helps!

4

Also there’s an issue in the documentation. Here in the docs it says the query parameter after is an integer, but in actuality it’s a string – precisely the next_messages_cursor we pass in getConversations .

You are correct. It is a string, thank you for the report. I will hand over it to the responsible department.

5

Hello! Thank you for your response.

I can see that whenever I try to install & test the app, I first get callback request for OAuth, followed by the request to create the channel. Then I receive the Pipedrive request:

GET /api/channels/<provider-channel-id>/conversations?conversations_limit=30&messages_limit=30

And I’m assuming after is null by default. I’m testing my implementation for a static user with a mock data set, containing of 200 conversations, each of which has somewhere between 1 and 1000 messages.

I didn’t implement the messages_limit logic yet but I don’t have any issues with this. The getConversations endpoint gets hit once, I return the first 30 conversations with all of their message content, and the messages show up correctly on the Pipedrive Messaging tab. There are about 1500 messages in total. I am planning to implement the message cursors once I’ve succeeded in implementing the conversation pagination logic.

Now, the API documentation says:

In the same getConversations endpoint, inside the additional_data object, the application will send the after property to our API, which is a cursor that can be used to fetch more conversations in the same endpoint (after query parameter).

My question is, despite passing this after parameter in the getConversations endpoint correctly (I include the unique conversation id of the 30th conversation in string format under the additional data field), my getConversations endpoint is not hit again.

Any ideas as to why that might be the case? Is the process of implementing messaging cursors a prerequisite? I thought they were only for performance, seeing as despite not handling it, I can see 30+ in conversations, in fact even up to 939 in one.

6

Hey,

My question is, despite passing this after parameter in the getConversations endpoint correctly (I include the unique conversation id of the 30th conversation in string format under the additional data field), my getConversations endpoint is not hit again.

After creating a new channel and calling the getConversations with 30/30 limits, there is no such logic to load more conversations/messages right away from you. This is because the user usually does not care about old conversations and messages.

In case the user needs more than 30 messages in some conversation he can scroll up in the UX and there will be an infinite loader - so it will call your API (getConversationById) with after value you sent to us as next_messages_cursor when the conversation was created. It means that the only way how to load more messages to the conversation is on the user’s behalf - when he needs them.

BTW please do not ignore message/conversation limits sent to you from us, it is BETA so it’s not probably validated by us now but it might be at some point and it can cause some nasty bugs then.

7

I updated my getConversations endpoint to return only the most recent number of messages given to me by the query parameter messages_limit, which seems to be 30. I’ve now moved on to getConversationsById endpoint to implement fetching older messages.

The endpoint is hit correctly when I scroll up in the messages screen but the loader doesn’t stop loading. When I check my logs I see the following request:

Incoming request: GET /api/channels/<provider-channel-id>/conversations/<conversation-id>?after=<next-messages-cursor>&messages_limit=30

I can see that the provider channel ID and conversation ID values are set correctly for the conversation I’m trying to fetch more messages for. The after parameter is simply the cursor I pass in the previous getConversations endpoint under the field next_messages_cursor.

I create a correctly-formatted JSON response for this request:

{
  "success": true,
  "data": [
    {
      "link": <conversation-url>,
      "id": <conversation-id>,
      "status": "open",
      "created_at": "2024-01-13T16:59:30.352Z",
      "seen": true,
      "next_messages_cursor": <the-next-next-messages-cursor>,
      "messages": [
        {
          "id": <unique-id-of-message>,
          "status": "sent",
          "created_at": "2024-01-13T17:00:02.316Z",
          "message": "This is message number 972",
          "sender_id": "pipedrive7",
          "reply_by": null,
          "attachments": []
        },
        {
          "id": <unique-id-of-message>,
          "status": "sent",
          "created_at": "2024-01-13T17:00:01.210Z",
          "message": "This is message number 971",
          "sender_id": "pipedrive8",
          "reply_by": null,
          "attachments": []
        },
        ...
        {
          "id": <unique-id-of-message>,
          "status": "sent",
          "created_at": "2024-01-13T16:59:31.425Z",
          "message": "This is message number 944",
          "sender_id": "pipedrive7",
          "reply_by": null,
          "attachments": []
        },
        {
          "id": <unique-id-of-message>,
          "status": "sent",
          "created_at": "2024-01-13T16:59:30.352Z",
          "message": "This is message number 943",
          "sender_id": "pipedrive8",
          "reply_by": null,
          "attachments": []
        }
      ],
      "participants": [
        {
          "id": "pipedrive7",
          "name": "pipedrive7",
          "role": "source_user",
          "avatar_url": null,
          "avatar_expires": false,
          "fetch_avatar": false
        },
        {
          "id": "pipedrive8",
          "name": "pipedrive8",
          "role": "end_user",
          "avatar_url": null,
          "avatar_expires": false,
          "fetch_avatar": false
        }
      ]
    }
  ],
  "additional_data": {
    "after": "not null"
  }
}

Obviously, everything in the JSON is in string format as required. Despite sending this JSON response in the api route with res.json(response) and seeing this correctly formatted JSON on my enpoint URL, the messages do not properly load in the front end. How come?

8

Hey,
please try to remove the array wrapper in the JSON. It makes sense to use it if you are fetching a list of conversations, but with this getConversationById endpoint we expect just one conversation in response.

{
  "success": true,
  "data": {
    "id":"some-id",
    ...
  }
}

getConversationById doc