Webhook Improvements: A Deeper Dive

(Cory Anderson) #1

Webhook Improvements: A Deeper Dive.

Let’s face it. Webhooks are key to maintaining a robust application-- allowing applications to keep up-to-date with the state of resources created within the Dwolla Platform. Applications can combine the concepts of asynchronous communication and reactive application architectures to solve unique and challenging problems. We’ve seen several customers build really interesting notification solutions, information radiators and visualization frameworks by utilizing webhooks.

Now let’s imagine having an application that heavily relies on consistent delivery of these webhooks. It uses them for everything from updating components on its front end to notifying end users of status changes. Applications need to have peace of mind to automate day-to-day activities without polling the API.

So what happens when webhooks don’t deliver quickly enough?

This was the question we spent a lot of time thinking about. In late Q4 of 2018 we received reports of webhooks taking nearly 60 minutes to deliver during peak load time. For API clients that rely on timely delivery to update their applications, this was causing some strain on their user experience and even led some API clients to fall back to poll the API at these peak times.

These bottlenecks were because we used a single, shared queue across all partners. A single slow-to-respond or high-volume API client would affect webhooks for everyone leveraging our API.

The old webhook architecture

So we began our task of re-architecting our webhooks system.

Making Changes to Webhooks

As of March 2019, we have been able to improve performance and reduce bottlenecks to keep our customers’ applications working as efficiently as possible. Our new and improved webhook sending service is currently live in both the Sandbox and Production environments, but how did we go about this?

Instead of a single queue, the Dwolla Platform now provisions customer-independent infrastructure for every application with a webhook subscription. This change allows Dwolla’s Platform to tailor webhooks to customer needs, scaling from single-threaded delivery for small workloads, to thousands of parallel requests so customers can be notified the moment an event happens. The send rate can be configured on a per-API-client basis, meaning our platform can scale with the needs of your business.

The new webhook architecture

For API clients with an existing webhook subscription, there was no need to create a new subscription. Over the last several weeks, we migrated all existing webhook subscriptions to the new architecture. No changes were required for existing webhook subscribers to benefit from these improvements.

To top it all off, our engineering team open-sourced a sample webhook-receiver built to consume our own webhooks internally. This project deploys a webhook handler via the Serverless Framework to the cloud of your choice (currently configured for AWS). Using this project, you can save time in bringing your idea for a reactive application to fruition. The project can be forked on GitHub and modified for your team’s purposes. If you’re more of the hands-on type, it’s easy to create a webhook subscription using Dwolla’s SDKs. Here’s how you’d do it with our Dwolla Node SDK;

var requestBody = {
url: 'http://myawesomeapplication.com/destination',
secret: 'your webhook secret'

.post('webhook-subscriptions', requestBody)
.then(res => res.headers.get('location'));

Interested to know more on how we went about rearchitecting our webhooks to serverless? Check out the presentation that one of our Principal Software Engineers created for a recent Des Moines Javascript meetup.

As an added bonus, you can find links to all of our open-sourced projects that relate to this project.

  • webhook-provisioner - Provision AWS resources to create, update, and delete API-client-specific AWS resources
  • webhook-handler - POST webhooks to APIs
  • webhook-receiver - A sample application to receive and verify webhooks
  • cloudwatch-alarm-to-slack - Forward CloudWatch Alarms to Slack via a Slack-bot
  • sqs-mv - Move SQS messages from one queue to another
  • generator-serverless - Yeoman generator for serverless functions

What’s next?

As 2019 continues, our product roadmap is chock full of feature enhancements to delight and satisfy our API customers, with the intent of bringing additional value to those businesses that currently leverage Dwolla’s Platform.

Let’s chat, what kind of product improvements are you looking for? How can we continue to make the platform better?