A Background on Que
For the last year, my favorite Ruby queueing library has been Que by Chris Hanks. Que uses PostgreSQL advisory locks to deliver a very high-throughput queue on top of Postgres. This works very well, even at moderate scale.
Que is great for getting started because it lets you avoid taking another dependency on a separate queue (or something like Redis).
With Que, you also get all the transactional ACID guarantees of Postgres with your jobs. This means that jobs are guaranteed to run at least once, and can be enqueued in a transaction along with other database changes.
In a distributed system, this gives you access to some powerful patterns to reduce overall complexity. For example, if you need to make a call to another service, but only want that to happen after a user has been fully saved to the database, you can transactionally enqueue a job to call that other system along with changes to the user row. By doing this, you avoid having to handle the case where a job was added to the queue, but a later error or crash caused the transaction to abort. I used this pattern extensively at Heroku, most notably in the system that reliably replicates routing data between different internal services.
Que for Go
These days, I spend a lot of my time writing Golang code. At times, I've wished for something like Que that would let me get the same kind of transactional guarantees when running jobs after making changes to the database. I've also worked on Ruby apps using Que and wished that I could run some of my jobs in a higher-performance language like Go. Now, you can have both!
I decided to port the Que library into a Golang package called que-go. The que-go package is fully interoperable with the Ruby Que queueing library. It's also written in nice, idiomatic Go. Or at least my best attempt at it. You could enqueue jobs in Ruby and work them in Go, or vice versa. Or you can go the 100% Go route :)
Porting Que turned out to be pretty straightforward because the majority of the complexity is wrapped up in the SQL statements, which Chris Hanks has already worked hard to fine-tune.
Please check out the que-go docs and let me know how it goes!
Thanks to Chris Hanks for writing Que in the first place.