What is Q4M?

Q4M (Queue for MySQL) is a message queue licensed under GPL that works as a pluggable storage engine of MySQL 5.1, designed to be robust, fast, flexible. It is already in production quality, and is used by several web services (see Users of Q4M).
To start using Q4M, download either a binary or source distribution from the install page, and follow the installation instructions. A small tutorial is also avialable. You may use SQL to access Q4M queues, or there is a wrapper module available for perl (Queue::Q4M).
For more information, please read the developer's weblog (Kazuho at Work) or subscribe to the mailing list.

What's New

0.9.5 release (Feb 25 2011)
fix SIGSEGV/SIGBUS on TRUNCATE TABLE
fix race condition that caused (in rare situation) .Q4M file to reappear after DROP TABLE
0.9.4 release (Jun 28 2010)
support for concurrent compaction
added queue_compact(table_name) function to trigger compaction manually
stability improvements (see ChangeLog)
0.9 release (Jan 05 2010)
fix deadlock issue while performing compaction
0.8.9 release (Oct 26 2009)
fix memory corruption when FLOAT or DOUBLE fields where used, fix deadlock under some situations calling queue_wait with multiple tables, fix compile error on GCC 4.3, etc.
slides at MySQL Conference & Expo 2009 (Apr 22 2009)
presentation slides used at MySQL Conference & Expo 2009 is available here.

Design Goals of Q4M

Robust
Q4M logs all message transfers to disk. A power failure or OS crash would never loose a message.
Fast
Although Q4M syncs every operation to disk, it runs fast by using group commits, and checksums for data recovery. Even with HDD write cache disabled, it is able to transfer thousands of messages per second on an ordinal linux server.
Flexible
Since Q4M works as a storage engine of MySQL, ordinally SQL can be used for administration. It is also possible to JOIN Q4M queues against other MySQL tables. Or you may use triggers for sending data into queue.

Users of Q4M

Q4M is used by some of the most popular web services in Japan. The users include:

Limitations and Known Issues

Limitations, known issues under version 0.5.
  • row size is limited to 512MB
  • table size is limited to 2**63 bytes
  • no support for UPDATE and REPLACE
  • no support for indexes
  • removal of multiple rows from a single DELETE statement is not atomic
  • may cause server crash under extremely low memory condition
  • may lose data if OS crashes while Q4M is performing table compaction (since the directory is not fsynced)