Out-of-order funding source webhooks can desync removal status

API Support Funding Sources
1

According to this article, webhook delivery order is not guaranteed. This isn’t an issue for most webhook events since they tend to converge to an end state (e.g. transfers start as pending and become completed or errored), but it is an issue for funding sources. When a funding source was previously deactivated and a new funding source is created with the same routing & account numbers, Dwolla reactivates the deactivated funding source and returns identical API data for it. If customer_funding_source_added and customer_funding_source_removed events are delivered out of order, or if the delivery of a customer_funding_source_removed event is delayed to a time after an additional create-funding-source API request has been issued, our application could store the wrong removed status for a funding source.

This wouldn’t be an issue in most cases since creation and removal pass through our application server before we make API requests to Dwolla, except that Dwolla also issues customer_funding_source_removed events when funding sources are removed automatically due to certain ACH failure codes. There doesn’t appear to be any way to distinguish these two types of funding source removals.

Do you have any guidance on how we might resolve this issue? Here are some approaches that I’ve investigated without success:

  • Looking for some way to distinguish whether funding sources were removed automatically or as the result of an API request. I could find no way to do so.
  • Looking in the API for some piece of data that could be used to totally order events. I could find no such data.
  • Looking in the API for some piece of data that indicates when a funding source was removed and/or reactivated. I could find no such data.
  • Ignoring customer_funding_source_removed events whose webhook timestamp is newer than the funding source’s created field. The created field does not change when a funding source is reactivated, and the webhook timestamp is a delivery timestamp, meaning it could be skewed by multiple delivery attempts.

In the meantime, the best solution I could come up with is to always fetch the current state of a funding source when any funding-source-related webhook is delivered, ignore the topic, and just save the current state of the resource. This seems like it’s not the intended way to interact with Dwolla’s webhooks, so I’m hoping you can offer some guidance in this issue.

1 Like
2

Hi @max.rothman! Thanks for the thorough explanation of the issue — you’ve clearly thought through the possible pitfalls. With the current webhook delivery model, the approach you outlined (retrieving the current state of the funding source whenever you receive a funding-source-related webhook) is indeed the recommended way to handle this. That way, your system is always storing the most up-to-date state, regardless of delivery timing or order.

That said, I’d like to clarify a couple of points that may help reduce your concern about the frequency of this scenario:

  • Funding source reactivation behavior: Dwolla will only reactivate an existing funding source if it was removed and re-added within 48 hours of its initial creation, and only if it was originally an unverified funding source. For verified funding sources, removing and re-adding will always result in a brand-new funding source. This makes the case you’re describing less common.

  • Webhook ordering risk: While webhook delivery order isn’t guaranteed, the risk of a customer_funding_source_removed event being delivered after a new funding source is added is quite low in practice. The only scenario where this could happen is if a funding source is automatically removed (due to certain ACH return codes) and then re-added before the webhook for the removal is delivered. For user-driven removal and re-adding, delayed out-of-order delivery is very unlikely.

So, while this is technically an edge case, your solution of fetching the funding source state when processing removed or added events is both safe and appropriate. It also ensures your application has a consistent source of truth aligned with the Dwolla API.

Additionally, adding a “removed” timestamp to the funding-source is something we are aware as being helpful to our clients for debugging. We have it on our backlog, however, I can’t provide a timeline on when this would be available.

Hope this helps clarify things! Please let us know if you have any further questions or edge cases you’d like to walk through. I’d be happy to jump in!

3

Thanks for the info.

Thanks for this clarification. Is this behavior documented anywhere? I didn’t see it while building my implementation.

I agree that the scenario you’re describing is unlikely, but I think it becomes more likely once webhook retries are considered. A webhook delivery failure can cause a webhook to be delivered hours or days later, greatly enlarging the window of opportunity for such an issue to occur. Furthermore, since Dwolla does not have the facility to send alerts for failed webhook deliveries, we’ve opted to put received webhooks on our own queue before being processed. If a webhook ends up on that queue’s DLQ there could be a significant delay before it’s processed, so the lack of a removed timestamp adds risk to our ability to redrive that DLQ without the “ignore the topic and just fetch” solution.

If I might suggest another improvement, it would be extremely helpful to have a timestamp for when an event occurred, separate from when webhook delivery was attempted. Dwolla seems to have the ability to order events in time in the UI, but no data that could be used to do this is exposed on the APIs.

4

Hi @max.rothman, thanks for the thoughtful follow up!

On your first question: we’ve recently migrated our documentation to a new platform, and some supplemental details have not yet been carried over to the API reference sections. The information I shared about funding source reactivation (only applying within 48 hours for unverified sources) is part of that group. We’ll make a note to update the Funding Source Removal section of the API reference with this clarification so it’s more discoverable going forward.

On event timing: you’re correct that retries can increase the window in which out-of-order delivery might be observed. One important note is that webhook events already include a created timestamp, which reflects when the event was generated by Dwolla. Additionally, when you retrieve the webhook in the API, you can also see the timestamps of the attempts.

  • The created field in the event payload ("created": "2025-09-12T18:49:23.093Z") reflects when the event itself was generated by Dwolla. This value stays constant even if the webhook is retried, and is the “event occurred at” timestamp you’re asking for.

  • The attempt timestamps you see when retrieving the webhook resource (attempt.request.timestamp) reflect when a particular delivery attempt was made. These will vary for each retry.

Here’s an example response from the API:

GET https://api-sandbox.dwolla.com/webhooks/4d817dde-f7c6-49b2-96c8-88c98c4ec675
...
{
    "_links": {
        "event": {
            "href": "https://api-sandbox.dwolla.com/events/4d319697-7fde-4197-8924-247080658ea8",
            "type": "application/vnd.dwolla.v1.hal+json",
            "resource-type": "event"
        },
        "account": {
            "href": "https://api-sandbox.dwolla.com/accounts/0ee84069-47c5-455c-b425-633523291dc3",
            "type": "application/vnd.dwolla.v1.hal+json",
            "resource-type": "account"
        },
        "retry": {
            "href": "https://api-sandbox.dwolla.com/webhooks/4d817dde-f7c6-49b2-96c8-88c98c4ec675/retries",
            "type": "application/vnd.dwolla.v1.hal+json",
            "resource-type": "retry"
        },
        "self": {
            "href": "https://api-sandbox.dwolla.com/webhooks/4d817dde-f7c6-49b2-96c8-88c98c4ec675",
            "type": "application/vnd.dwolla.v1.hal+json",
            "resource-type": "webhook"
        },
        "subscription": {
            "href": "https://api-sandbox.dwolla.com/webhook-subscriptions/ed0db938-d7f0-4694-90ee-b12053182dc0",
            "type": "application/vnd.dwolla.v1.hal+json",
            "resource-type": "webhook-subscription"
        }
    },
    "id": "4d817dde-f7c6-49b2-96c8-88c98c4ec675",
    "topic": "customer_created",
    "accountId": "0ee84069-47c5-455c-b425-633523291dc3",
    "eventId": "4d319697-7fde-4197-8924-247080658ea8",
    "subscriptionId": "ed0db938-d7f0-4694-90ee-b12053182dc0",
    "attempts": [
        {
            "id": "5f513805-058f-43f9-aab3-30cf61eeec0d",
            "request": {
                "timestamp": "2025-09-12T18:51:17.797Z",
                "url": "https://eotgms3j77wxf68.m.pipedream.net",
                "headers": [
                    {
                        "name": "Content-Length",
                        "value": "602"
                    },
                    {
                        "name": "Content-Type",
                        "value": "application/json"
                    },
                    {
                        "name": "User-Agent",
                        "value": "dwolla-webhooks/1.1"
                    },
                    {
                        "name": "X-Dwolla-Topic",
                        "value": "customer_created"
                    },
                    {
                        "name": "X-Request-Signature-SHA-256",
                        "value": "bffa2ceffd343e95e484319cf00b6dc113d954fa0d90d25c225cc01f8e1899d5"
                    }
                ],
                "body": "{\"id\":\"4d319697-7fde-4197-8924-247080658ea8\",\"resourceId\":\"092d7a33-8f09-44a3-a788-268b80101d32\",\"topic\":\"customer_created\",\"timestamp\":\"2025-09-12T18:49:23.093Z\",\"_links\":{\"self\":{\"href\":\"https://api-sandbox.dwolla.com/events/4d319697-7fde-4197-8924-247080658ea8\"},\"account\":{\"href\":\"https://api-sandbox.dwolla.com/accounts/0ee84069-47c5-455c-b425-633523291dc3\"},\"resource\":{\"href\":\"https://api-sandbox.dwolla.com/customers/092d7a33-8f09-44a3-a788-268b80101d32\"},\"customer\":{\"href\":\"https://api-sandbox.dwolla.com/customers/092d7a33-8f09-44a3-a788-268b80101d32\"}},\"created\":\"2025-09-12T18:49:23.093Z\"}"
            },
            "response": {
                "timestamp": "2025-09-12T18:51:18.064Z",
                "headers": [],
                "statusCode": 200,
                "body": ""
            }
        },
        {
            "id": "816be70a-35f9-4ab8-b164-6169a81acc92",
            "request": {
                "timestamp": "2025-09-12T18:52:46.014Z",
                "url": "https://eotgms3j77wxf68.m.pipedream.net",
                "headers": [
                    {
                        "name": "Content-Length",
                        "value": "602"
                    },
                    {
                        "name": "Content-Type",
                        "value": "application/json"
                    },
                    {
                        "name": "User-Agent",
                        "value": "dwolla-webhooks/1.1"
                    },
                    {
                        "name": "X-Dwolla-Topic",
                        "value": "customer_created"
                    },
                    {
                        "name": "X-Request-Signature-SHA-256",
                        "value": "bffa2ceffd343e95e484319cf00b6dc113d954fa0d90d25c225cc01f8e1899d5"
                    }
                ],
                "body": "{\"id\":\"4d319697-7fde-4197-8924-247080658ea8\",\"resourceId\":\"092d7a33-8f09-44a3-a788-268b80101d32\",\"topic\":\"customer_created\",\"timestamp\":\"2025-09-12T18:49:23.093Z\",\"_links\":{\"self\":{\"href\":\"https://api-sandbox.dwolla.com/events/4d319697-7fde-4197-8924-247080658ea8\"},\"account\":{\"href\":\"https://api-sandbox.dwolla.com/accounts/0ee84069-47c5-455c-b425-633523291dc3\"},\"resource\":{\"href\":\"https://api-sandbox.dwolla.com/customers/092d7a33-8f09-44a3-a788-268b80101d32\"},\"customer\":{\"href\":\"https://api-sandbox.dwolla.com/customers/092d7a33-8f09-44a3-a788-268b80101d32\"}},\"created\":\"2025-09-12T18:49:23.093Z\"}"
            },
            "response": {
                "timestamp": "2025-09-12T18:52:46.321Z",
                "headers": [],
                "statusCode": 200,
                "body": ""
            }
        },
        {
            "id": "a4a7613f-57f6-4c16-a1ef-78d827435525",
            "request": {
                "timestamp": "2025-09-12T18:49:23.366Z",
                "url": "https://eotgms3j77wxf68.m.pipedream.net",
                "headers": [
                    {
                        "name": "Content-Length",
                        "value": "602"
                    },
                    {
                        "name": "Content-Type",
                        "value": "application/json"
                    },
                    {
                        "name": "User-Agent",
                        "value": "dwolla-webhooks/1.1"
                    },
                    {
                        "name": "X-Dwolla-Topic",
                        "value": "customer_created"
                    },
                    {
                        "name": "X-Request-Signature-SHA-256",
                        "value": "bffa2ceffd343e95e484319cf00b6dc113d954fa0d90d25c225cc01f8e1899d5"
                    }
                ],
                "body": "{\"id\":\"4d319697-7fde-4197-8924-247080658ea8\",\"resourceId\":\"092d7a33-8f09-44a3-a788-268b80101d32\",\"topic\":\"customer_created\",\"timestamp\":\"2025-09-12T18:49:23.093Z\",\"_links\":{\"self\":{\"href\":\"https://api-sandbox.dwolla.com/events/4d319697-7fde-4197-8924-247080658ea8\"},\"account\":{\"href\":\"https://api-sandbox.dwolla.com/accounts/0ee84069-47c5-455c-b425-633523291dc3\"},\"resource\":{\"href\":\"https://api-sandbox.dwolla.com/customers/092d7a33-8f09-44a3-a788-268b80101d32\"},\"customer\":{\"href\":\"https://api-sandbox.dwolla.com/customers/092d7a33-8f09-44a3-a788-268b80101d32\"}},\"created\":\"2025-09-12T18:49:23.093Z\"}"
            },
            "response": {
                "timestamp": "2025-09-12T18:49:23.659Z",
                "headers": [],
                "statusCode": 200,
                "body": ""
            }
        }
    ]
}

So if you need to consistently order events or reason about when they occurred versus when they were delivered, you’ll want to rely on the created timestamp from the event itself.

Hope this is helpful!