Urgent P1 - 504 errors from API causing idempotency issues

Hi Dwolla!

We’ve run into some payment idempotency issues. We ran a batch of payments and have been running into issues with the API. We provide unique idempotency for each transaction and have observed some strange behavior today in regard to those keys. We got a random bout of 504 gateway errors, and did a backoff retry for each of these transactions. One transaction ended up being created twice, and others are properly returning a 409 but aren’t appearing in our Dwolla UI.

Any ideas on how to resolve this? This is a P1 issue for us and causing a significant outage. Any assistance would be appreciated!

I also filed more specific information using the support email. Any chance we can get this escalated. It’s causing significant impact to our business.

Thanks!

-Matt

Hi @mdobs , Thanks for reaching out and providing detailed information about the payment idempotency issues you’re experiencing. We understand this is a P1 issue causing a significant outage for your business, and we sincerely apologize for the disruption.

We appreciate you also sending a separate email with more specifics – this will be very helpful in our investigation. We’ve received it and are already looking into the details you provided there. We’ve escalated this internally and have assigned a high-priority ticket to it.

We understand the urgency of this situation and are working diligently to resolve it. We will keep you updated on our progress and let you know as soon as we have any findings or require further information from your side.

In the meantime, if you observe any other unusual behavior, please don’t hesitate to let us know immediately!

Thank you! Very much appreciated!

This topic was automatically closed after 365 days. New replies are no longer allowed.