Moneta: A unified interface for key/value stores

Moneta provides a standard interface for interacting with various kinds of key/value stores. Moneta supports the well-known NoSQL and document based stores.

A short overview of the features:

If you are not yet convinced, you might ask why? What are the goals of the project?

Moneta is tested thoroughly using Travis-CI.


Getting started

Install Moneta via Rubygems

~~~ $ gem install moneta ~~~

or add it to your Gemfile

~~~ ruby gem 'moneta' ~~~

Now you are ready to go:

~~~ ruby require 'moneta'

Create a simple file store

store = Moneta.new(:File, :dir => 'moneta')

Store some entries

store = 'value'

Read entry

store.key?('key') # returns true store # returns 'value'

store.close ~~~



Supported backends

Out of the box, it supports the following backends. Use the backend name symbol in the Moneta constructor (e.g. Moneta.new(:Memory)).

Some of the backends are not exactly based on key/value stores, e.g. the relational ones. These are useful if you already use the corresponding backend in your application. You get a key/value store for free then without installing any additional services and you still have the possibility to upgrade to a real key/value store.

Backend feature matrix

NOTE: <a name=“backend-matrix”>The backend matrix</a> is much more readable on rubydoc.info than on github. Go there!

AdapterRequired gemsMulti-thread safe[1]Multi-process safe[2]Atomic increment[8]Atomic create[9]Native expires[3]PersistentDescription
Persistent stores
MongomongoMongoDB database
RedisredisRedis database
ActiveRecordactiverecordActiveRecord ORM
File-File store
SequelsequelSequel ORM
TokyoTyranttokyotyrant or ruby-tokyotyrantTokyoTyrant database
PStore-[10]PStore store
YAML-[10]YAML store
Sqlitesqlite3?[10]Sqlite3 database
Daybreakdaybreak(✓)[7]Incredibly fast pure-ruby key/value store Daybreak
DBM-Berkeley DB using DBM interface or NDBM (Depends on Ruby environment)
GDBM-GDBM database
LevelDBleveldbLevelDB database
SDBM-SDBM database
TDBtdbTDB database
KyotoCabinetkyotocabinetKyotoCabinet database
TokyoCabinettokyocabinetTokyoCabinet database
DataMapperdm-core, dm-migrationsDataMapper ORM
Couchfaraday, multi_jsonCouchDB database
HBasehbaserb?HBase database
Cassandracassandra?Cassandra distributed database
LocalMemCachelocalmemcacheLocalMemCache database
Fogfog?Fog cloud store
Riakriak-clientRiak database
Non persistent stores
MemcachedDallidalli[4]Memcached database with Dalli library
Memcacheddalli or memcached?[4]Memcached database
MemcachedNativememcached[4]Memcached database with native library
Cookie-(✓)[6]Cookie in memory store
LRUHash-(✓)[6]LRU memory store
Memory-(✓)[6]Memory store
Null-No database
Client-?[5]?[5]?[5]?[5]Moneta client adapter
RestClientfaraday?[5]Moneta REST client adapter

Proxies

In addition it supports proxies (Similar to Rack middlewares) which add additional features to storage backends:

Serializers and compressors (Moneta::Transformer)

Supported serializers:

Supported value compressors:

Special transformers:


Moneta API

The Moneta API is purposely extremely similar to the Hash API with a few minor additions. Every method takes also a optional option hash. In order so support an identical API across stores, Moneta does not support iteration or partial matches.

~~~

initialize(options) options differs per-store, and is used to set up the store.

retrieve a key. If the key is not available, return nil.

load(key, options = {}) retrieve a key. If the key is not available, return nil.

fetch(key, options = {}, &block) retrieve a key. If the key is not available, execute the

                                      block and return its return value.

fetch(key, value, options = {}) retrieve a key. If the key is not available, return the value,

[]=(key, value) set a value for a key. If the key is already used, clobber it.

                                      keys set using []= will never expire.

store(key, value, options = {}) same as []=, but you can supply options.

delete(key, options = {}) delete the key from the store and return the current value.

key?(key, options = {}) true if the key exists, false if it does not.

increment(key, amount = 1, options = {}) increment numeric value. This is an atomic operation

                                      which is not supported by all stores. Returns current value.

decrement(key, amount = 1, options = {}) increment numeric value. This is an atomic operation

                                      which is not supported by all stores. Returns current value.
                                      This is just syntactic sugar for incrementing with a negative value.

create(key, value, options = {}) create entry. This is an atomic operation which is not supported by all stores.

                                      Returns true if the value was created.

clear(options = {}) clear all keys in this store.

close close database connection.

features return array of features, e.g. [:create, :expires, :increment]

supports?(feature) returns true if store supports a given feature

~~~

Creating a Store

There is a simple interface to create a store using Moneta.new. You will get automatic key and value serialization which is provided by Moneta::Transformer. This allows you to store arbitrary Ruby objects. You can tune some options when you call Moneta.new. However for very fine tuning use Moneta.build.

~~~ ruby store = Moneta.new(:Memcached, :server => 'localhost:11211') store = 'value' store = {:a => 1, :b => 2} store = MarshallableRubyObject.new ~~~

If you want to have control over the proxies, you have to use Moneta.build:

~~~ ruby store = Moneta.build do # Adds expires proxy use :Expires

# Transform key using Marshal and Base64 and value using Marshal use :Transformer, :key => [:marshal, :base64], :value => :marshal

# IMPORTANT: adapter must be defined last for the builder to function properly.

# Memory backend adapter :Memory end ~~~

You can also directly access the underlying adapters if you don't want to use the Moneta stack.

~~~ ruby db = Moneta::Adapters::File.new(:dir => 'directory') db = {:a => 1, :b => 2} # This will fail since you can only store Strings

However for Mongo and Couch this works

The hash will be mapped directly to a Mongo/Couch document.

db = Moneta::Adapters::Couch.new db = {:a => 1, :b => 2}

db = Moneta::Adapters::Mongo.new db = {:a => 1, :b => 2} ~~~

Expiration

The Cassandra, Memcached, Redis and Mongo backends support expiration natively.

~~~ ruby cache = Moneta::Adapters::Memcached.new

Or using the builder…

cache = Moneta.build do adapter :Memcached end

Expires in 60 seconds

cache.store(key, value, :expires => 60)

Never expire

cache.store(key, value, :expires => 0) cache.store(key, value, :expires => false)

Update expires time if value is found

cache.load(key, :expires => 30) cache.key?(key, :expires => 30)

Or remove the expiration if found

cache.load(key, :expires => false) cache.key?(key, :expires => 0) ~~~

You can add the expires feature to other backends using the Moneta::Expires proxy. But be aware that expired values are not deleted automatically if they are not looked up.

~~~ ruby

Using the :expires option

cache = Moneta.new(:File, :dir => '…', :expires => true)

or manually by using the proxy…

cache = Moneta::Expires.new(Moneta::Adapters::File.new(:dir => '…'))

or using the builder…

cache = Moneta.build do use :Expires adapter :File, :dir => '…' end ~~~

Atomic operations

Atomic incrementation and raw access

The stores support the #increment which allows atomic increments of unsigned integer values. If you increment a non existing value, it will be created. If you increment a non integer value an exception will be raised.

~~~ ruby store.increment('counter') # returns 1, counter created store.increment('counter') # returns 2 store.increment('counter', -1) # returns 1 store.increment('counter', 13) # returns 14 store.increment('counter', 0) # returns 14 store.decrement('counter') # returns 13 store = 'Moneta' store.increment('name') # raises an Exception ~~~

If you want to access the counter value you have to use raw access to the datastore. This is only important if you have a Moneta::Transformer somewhere in your proxy stack which transforms the values e.g. with Marshal.

~~~ ruby store.increment('counter') # returns 1, counter created store.load('counter', :raw => true) # returns 1

store.store('counter', '10', :raw => true) store.increment('counter') # returns 11 ~~~

Fortunately there is a nicer way to do this using some syntactic sugar!

~~~ ruby store.increment('counter') # returns 1, counter created store.raw # returns 1 store.raw.load('counter') # returns 1

store.raw = '10' store.increment('counter') # returns 11 ~~~

You can also keep the raw store in a variable and use it like this:

~~~ ruby counters = store.raw

counters.increment('counter') # returns 1, counter created counters # returns 1 counters.load('counter') # returns 1

counters = '10' counters.increment('counter') # returns 11 ~~~

Atomic create

The stores support the #create which allows atomic creation of entries. #create returns true if the value was created.

~~~ ruby store.create('key', 'value') # returns true store.create('key', 'other value') # returns false ~~~

Shared/distributed synchronization primitives

Moneta provides shared/distributed synchronization primitives which are shared database-wide between all clients.

Moneta::Mutex allows a single thread to enter a critical section.

~~~ ruby mutex = Moneta::Mutex.new(store, 'mutex_key')

mutex.synchronize do mutex.locked? # returns true

# Synchronized access to counter store += 1 end

begin mutex.lock mutex.locked? # returns true # … ensure mutex.unlock end ~~~

Moneta::Semaphore allows max_concurrent threads to enter a critical section.

~~~ ruby semaphore = Moneta::Semaphore.new(store, 'semaphore_counter', max_concurrent)

semaphore.synchronize do semaphore.locked? # returns true # … end

begin semaphore.enter semaphore.locked? # returns true # … ensure semaphore.leave end ~~~

Weak atomic operations

If an underlying adapter doesn't provide atomic #create or #increment and #decrement you can use the proxies Moneta::WeakIncrement and Moneta::WeakCreate to add support without atomicity.

But then you have to ensure that the store is not shared by multiple processes and thread-safety is provided by Moneta::Lock.

Syntactic sugar and option merger

For raw data access as described before the class Moneta::OptionMerger is used. It works like this:

~~~ ruby

All methods after 'with' get the options passed

store.with(:raw => true).load('key')

You can also specify the methods

store.with(:raw => true, :only => :load).load('key') store.with(:raw => true, :except => [:key?, :increment]).load('key')

Syntactic sugar for raw access

store.raw.load('key')

Access substore where all keys get a prefix

substore = store.prefix('sub') substore = 'value' store # returns nil store # returns 'value'

Set expiration time for all keys

short_lived_store = store.expires(60) short_lived_store = 'value' ~~~

Add proxies to existing store

You can add proxies to an existing store. This is useful if you want to compress only a few values for example.

~~~ ruby compressed_store = store.with(:prefix => 'compressed') do use :Transformer, :value => :zlib end

store = 'this value will not be compressed' compressed_store = 'value will be compressed' ~~~


Framework Integration

Inspired by redis-store there exist integration classes for Rails and Rack}[http://rack.github.com/]/{Rack-Cache. You can also use all the Rack middlewares together with Rails and the Sinatra framework. There exist the following integration classes:

Rack

Session store

You can use Moneta as a Rack session store. Use it in your config.ru like this:

~~~ ruby require 'rack/session/moneta'

Use only the adapter name

use Rack::Session::Moneta, :store => :Redis

Use Moneta.new

use Rack::Session::Moneta, :store => Moneta.new(:Memory, :expires => true)

Use the Moneta builder

use Rack::Session::Moneta do use :Expires adapter :Memory end ~~~

Moneta middleware

There is a simple middleware which places a Moneta store in the Rack environment at env['rack.moneta_store']. It supports per-request caching if you add the option :cache => true. Use it in your config.ru like this:

~~~ ruby

Add Rack::MonetaStore somewhere in your rack stack

use Rack::MonetaStore, :Memory, :cache => true

run lambda { |env| env # is a Moneta store with per-request caching }

Pass it a block like the one passed to Moneta.build

use Rack::MonetaStore do use :Transformer, :value => :zlib adapter :Cookie end

run lambda { |env| env # is a Moneta store without caching } ~~~

REST server

If you want to expose your Moneta key/value store via HTTP, you can use the Rack/Moneta REST service. Use it in your config.ru like this:

~~~ ruby require 'rack/moneta_rest'

map '/moneta' do run Rack::MonetaRest.new(:Memory) end

Or pass it a block like the one passed to Moneta.build

run Rack::MonetaRest.new do use :Transformer, :value => :zlib adapter :Memory end ~~~

Rack-Cache

You can use Moneta as a Rack-Cache store. Use it in your config.ru like this:

~~~ ruby require 'rack/cache/moneta'

use Rack::Cache, :metastore => 'moneta://Memory?expires=true', :entitystore => 'moneta://Memory?expires=true'

Or used named Moneta stores

Rack::Cache::Moneta = Moneta.build do use :Expires adapter :Memory end use Rack::Cache, :metastore => 'moneta://named_metastore', :entity_store => 'moneta://named_entitystore' ~~~

Cookies

Use Moneta to store cookies in Rack. It uses the Moneta::Adapters::Cookie. You might wonder what the purpose of this store or Rack middleware is: It makes it possible to use all the transformers on the cookies (e.g. :prefix, :marshal and :hmac for value verification).

~~~ ruby require 'rack/moneta_cookies'

use Rack::MonetaCookies, :domain => 'example.com', :path => '/path' run lambda { |env| req = Rack::Request.new(env) req.cookies #=> is now a Moneta store! env #=> is now a Moneta store! req.cookies #=> retrieves 'key' req.cookies = 'value' #=> sets 'key' req.cookies.delete('key') #=> removes 'key' [200, {}, []] } ~~~

Rails

Session store

Add the session store in your application configuration config/environments/*.rb.

~~~ ruby require 'moneta'

Only by adapter name

config.cache_store :moneta_store, :store => :Memory

Use Moneta.new

config.cache_store :moneta_store, :store => Moneta.new(:Memory)

Use the Moneta builder

config.cache_store :moneta_store, :store => Moneta.build do use :Expires adapter :Memory end ~~~

Cache store

Add the cache store in your application configuration config/environments/*.rb. Unfortunately the Moneta cache store doesn't support matchers. If you need these features use a different server-specific implementation.

~~~ ruby require 'moneta'

Only by adapter name

config.cache_store :moneta_store, :store => :Memory

Use Moneta.new

config.cache_store :moneta_store, :store => Moneta.new(:Memory)

Use the Moneta builder

config.cache_store :moneta_store, :store => Moneta.build do use :Expires adapter :Memory end ~~~


Advanced

Build your own key value server

You can use Moneta to build your own key/value server which is shared between multiple processes. If you run the following code in two different processes, they will share the same data which will also be persistet in the database shared.db.

~~~ ruby require 'moneta'

store = Moneta.build do use :Transformer, :key => :marshal, :value => :marshal use :Shared do use :Cache do cache do adapter :LRUHash end backend do adapter :GDBM, :file => 'shared.db' end end end end ~~~

If you want to go further, you might want to take a look at Moneta::Server and Moneta::Adapters::Client which are used by Moneta::Shared and provide the networking communication. But be aware that they are experimental and subjected to change. They provide an acceptable performance (for being ruby only), but don't have a stable protocol yet.

You might wonder why I didn't use DRb to implement server and client - in fact my first versions used it, but with much worse performance and it was real fun to implement the networking directly :) There is still much room for improvement and experiments, try EventMachine, try Kgio, …

ToyStore ORM

If you want something more advanced to handle your objects and relations, use John Nunemaker's ToyStore which works together with Moneta. Assuming that Person is a ToyStore::Object you can add persistence using Moneta as follows:

~~~ ruby

Use the Moneta Redis backend

Person.adapter :memory, Moneta.new(:Redis) ~~~


Testing and Benchmarks

Testing is done using Travis-CI. Currently we support Ruby 1.8.7 and 1.9.3.

Benchmarks for each store are done on Travis-CI for each build. Take a look there to compare the speed of the different key value stores for different key/value sizes and size distributions. Feel free to add your own configurations! The impact of Moneta should be minimal since it is only a thin layer on top of the different stores.


How to contribute?

Always feel free to open an issue on github.com/minad/moneta/issues if something doesn't work as you expect it to work. Feedback is also very welcome!

My only request about patches is that you please try to test them before submitting.

Contribute an adapter

If you want support for another adapter you can at first at it to the list of missing adapters at github.com/minad/moneta/issues/16

If you choose to implement an adapter please also add tests. Usually you only have to add a few lines to script/generate-specs to generate appropriate tests for your adapter. Please check also if travis.yml needs changes, for example if you need to start additional services.

Check if the default settings in Moneta#new are appropriate for your adapter. If not specify a better one.

Don't forget to edit the README.md and the CHANGES.


Alternatives


Authors