Asynchronous processing pitfalls and ActiveRecord locking

28 Sep 2015 My app is humming along, but there are a few sections that have to do a lot of heavy lifting, whether that be image processing, updating several different related objects, or looping through all of a has_many association to manipulate every record.

These types of things are perfect places to push some logic to a background process. In other words, make it asynchronous. So, if a user clicks a button to update several objects, unless the user needs to see those updates completed immediately, then a tool like Sidekiq can help run some expensive logic as a background process and allow the user to continue surfing the site while that process runs.

There are plenty of tutorials about how to set this up, so that’s not what I want to talk about here.

One of the potential pitfalls of processing things asynchronously, particularly if the processing involves updating database records, is multiple records being updated at the same time. For example, one user updates a Project in a background process, and another user updates the same Project somewhere else in the app that doesn’t use a background process. We want to last update to be the one that sticks, but if both updates are happening at the same time, things could get hairy.

This is the dreaded “race conditions” issue, and luckily there’s a pretty simple fix.

On ActiveRecord - and I believe this will work with any SQL database but definitely at least on Postgres and MySQL - I can lock a specific row while editing, then unlock it afterwards, so that each edit takes place in its own little bubble.

Account.transaction do
  acc = Account.lock.find(id)
  acc.balance -= 25
  acc.save!
end

This finds an account and locks it, then updates the balance, saves, and unlocks.

This is really important when dealing with numbers, money, and balances. Let me share a parable to explain:

There is an Account with $100 balance. One user in that account makes a transaction for$ 75.

A split second later, another user in that account makes a transaction for $90.

What should happen is the second user’s transaction should be rejected for lack of funds.

What actually happens without locking is that for each transaction, the first step is that the server runs account = Account.find(id) to retrieve the account, and since this is happening before either transaction has finished, the balance on the account in both cases is $100. So, after the first user's transaction, the balance is updated to$ 25 and the account is saved. But on the second user’s transaction, the account object that was pulled from the database still shows a balance of $100, so the server deducts the$ 90 charge from the balance and saves it, thereby allowing the transaction to go through and setting the saved account balance as $10.

Is your head spinning yet?

When we lock the account per transaction, we avoid this issue by only allowing the account to be edited on transaction at a time.

It’s a cool concept but also can be tricky to wrap your head around. Here’s a blog post I borrowed heavily from for my post. Or check out the Rails API docs.