We just discovered a few deals that were updated last Friday (February 15th) at around 13:33, that doesn’t seem to have triggered the webhook.
Last known incident was at 14:36.
I can see here https://status.pipedrive.com/ there was a problem with the “Web App” at around that time ([Resolved] Web application partially slow or down).
Is that the problem?
Our problem seems to have lasted longer than this, so we need to know if we need to go through all deals that has been updated at that time and make sure we got the updates.
Yes, there was an issue on February and that’s most likely why your events got lost. I’m sorry about that, and I hope it didn’t cause serious problems.
It’s a rare case, but we’re working to make it even less likely.
If I were you I would probably check the deals updated at that time to make sure the data was sent to you correctly.
I can create a filter to find all deals with an update time from last Friday and then manually update something on each deal to trigger the webhook.
We do however do some stuff if specific fields has been updated, so it will be a challenge to figure out what exactly has been updated.
About ~120 deals on different accounts, so I would rather not go through them manually.
Do you have a suggestion?
I can get deal updates for each deal from the filter using: https://developers.pipedrive.com/docs/api/v1/#!/Deals/get_deals_id_flow and then loop through them to see if the update time matches last friday, type is matching and see if it’s a field where we need to do something special.
That seems like a lot of work though. Can I somehow get all updates made in the time window the problem happended?
And then we get to the Product updates, that I assume also didn’t come through. As they don’t have an update time, I can’t create a meaningful filter.
I can handle Organization updates manually as there aren’t that many.
I started looking through a couple of deals manually.
It is mostly activities added to the deals, which our webhooks doesn’t care about.
Is there a way to make a filter to find deals that had the fields updated at a specific date?
One issue that you could have is that you only get the latest update time. So if a deal was updated during the downtime, and then again today (for example), you would only get the updated time of today.
Could you rerun the automatism for all the latest updated deals?
I have no idea how to do this. We need a list of all the updates that happened in the window while the system was down, but I can’t figure out a way to do this in a sensible manner.
Only way is to go through all deals updated AFTER Feb 15th 13:30 and analyze the complete update log, using deal flow API endpoint mentioned above.
Ok… Here’s one to find what deals need to be double-checked.
Find out first what deals were updated during the downtime or after, by using the /searchResults/field, with the following parameters: term: 2019-02-1 field_type: dealField field_key: update_time exact_match: 0 return_item_ids: 1 start: 0
This will return all the deals updated from 2019-02-10 until today (if you run this today or tomorrow). From this list, you can already exclude those updated before the 15th.
(If you prefer, you can also make 4 calls, with term: 2019-02-15, term: 2019-02-16, term: 2019-02-17, term: 2019-02-18)
For each of the returned deals, you can check the /deals/{id}/flow endpoint that you already mentioned, and only take into consideration those that were actually updated in a time close to the downtime - I would also add 1 or 2 hours of buffer, just to be safe.
Making 4 calls is better, because almost nothing happens during the weekend, making it only two days that needs to be checked.
On one account, we are looking at 186 deals Friday and 141 deals today that needs to be checked. Multiply that by 5 accounts.
We could need to make a script that can loop through all the deals (throttled so we don’t hit the API call limit), analyze the data based on the update time and evaluate the field for each account. It’s too much work, although it would be nice to have if this problem happens again.
What I would love was a full replay log for each webhook
I think you’re right, but it should be possible to recreate for someone with real access to the system
Yeah, I will have to do that. Fortunately I can limit it to deals with update time last Friday, as deals that has a later update time will already have had something updated
Even more fortunate, I have already prepared scripts for this exact case
So, I asked Pipedrive to provide a complete list of updates that happened in the time the system was down.
That’s all fine and I’m looking through the csv files I got
But what do we do next time this happens? It was only because of a very obvious missing update we even noticed this problem existed.
A “webhook log API endpoint” would be very usefull, as suggested here:
If we can’t rely on all updates getting through webhooks, we need to rethink all cases where we compare the “current” vs “previous” state. The whole “previous” chunk of the update could even be removed.
I know it’s a shitty situation that shouldn’t happen, but a lot of our system is designed around updates coming through so we can react accordingly.
Thanks for letting me vent It’s just getting a bit frustrating spending time building workarounds around the Pipedrive system.
You all have a nice day
@Troels_Gorm_Rasmussen just to let you know… We have this on the plan. We want to improve the whole delivery system to guarantee reliability
I can’t give you an exact date yet, but we know how important it is and we’re trying to push it.
