This is not an ORM. People that are wrapping ORM’s around Redis are missing the point.
The killer feature of Redis is that it allows you to perform atomic operations on individual data structures, like counters, lists, and sets. The atomic part is HUGE. Using an ORM wrapper that retrieves a “record”, updates values, then sends those values back, removes the atomicity, cutting the nuts off the major advantage of Redis. Just use MySQL, k?
This gem provides a Rubyish interface to Redis, by mapping Redis types to Ruby objects, via a thin layer over Ezra’s redis gem. It offers several advantages over the lower-level redis-rb API:
-
Easy to integrate directly with existing ORMs - ActiveRecord, DataMapper, etc. Add counters to your model!
-
Complex data structures are automatically Marshaled (if you set :marshal => true)
-
Integers are returned as integers, rather than ‘17’
-
Higher-level types are provided, such as Locks, that wrap multiple calls
This gem originally arose out of a need for high-concurrency atomic operations; for a fun rant on the topic, see An Atomic Rant, or scroll down to “Atomicity” in this README.
There are two ways to use Redis::Objects, either as an include in a model class (to integrate with ORMs or other classes), or by using new with the type of data structure you want to create.
gem install redis-objects
Using Redis::Objects this way makes it trivial to integrate Redis types with an existing ActiveRecord model, DataMapper resource, or other class. Redis::Objects will work with any class that provides an id method that returns a unique value. Redis::Objects will automatically create keys that are unique to each object, in the format:
model_name:id:field_name
Redis::Objects needs a handle created by Redis.new. (If you’re on Rails, config/initializers/redis.rb is a good place for this.)
require 'redis' require 'redis/objects' Redis::Objects.redis = Redis.new(:host => '127.0.0.1', :port => 6379)
Remember you can use Redis::Objects in any Ruby code. There are no dependencies on Rails. Standalone, Sinatra, Resque - no problem.
You can include Redis::Objects in any type of class:
class Team < ActiveRecord::Base include Redis::Objects lock :trade_players, :expiration => 15 # sec counter :hits counter :runs counter :outs counter :inning, :start => 1 list :on_base set :outfielders value :at_bat sorted_set :rank, :global => true hash_key :pitchers_faced # "hash" is taken by Ruby end
Familiar Ruby array operations Just Work (TM):
@team = Team.find_by_name('New York Yankees') @team.on_base << 'player1' @team.on_base << 'player2' @team.on_base << 'player3' @team.on_base # ['player1', 'player2', 'player3'] @team.on_base.pop @team.on_base.shift @team.on_base.length # 1 @team.on_base.delete('player2')
Sets work too:
@team.outfielders << 'outfielder1' @team.outfielders << 'outfielder2' @team.outfielders << 'outfielder1' # dup ignored @team.outfielders # ['outfielder1', 'outfielder2'] @team.outfielders.each do |player| puts player end player = @team.outfielders.detect{|of| of == 'outfielder2'}
And you can do intersections between objects (kinda cool):
@team1.outfielders | @team2.outfielders # outfielders on both teams @team1.outfielders & @team2.outfielders # in baseball, should be empty :-)
Counters can be atomically incremented/decremented (but not assigned):
@team.hits.increment # or incr @team.hits.decrement # or decr @team.hits.incr(3) # add 3 @team.runs = 4 # exception
Finally, for free, you get a redis method that points directly to a Redis connection:
Team.redis.get('somekey') @team = Team.new @team.redis.get('somekey') @team.redis.smembers('someset')
You can use the redis handle to directly call any Redis API command.
There is a Ruby class that maps to each Redis type, with methods for each Redis API command. Note that calling new does not imply it’s actually a “new” value - it just creates a mapping between that object and the corresponding Redis data structure, which may already exist on the redis-server.
Redis::Objects needs a handle to the redis server. For standalone use, you can either set the $redis global variable:
$redis = Redis.new(:host => 'localhost', :port => 6379) @list = Redis::List.new('mylist')
Or you can pass the Redis handle into the new method for each type:
redis = Redis.new(:host => 'localhost', :port => 6379) @list = Redis::List.new('mylist', redis)
Create a new counter. The counter_name is the key stored in Redis.
require 'redis/counter' @counter = Redis::Counter.new('counter_name') @counter.increment @counter.decrement puts @counter.value
This gem provides a clean way to do atomic blocks as well:
@counter.increment do |val| raise "Full" if val > MAX_VAL # rewind counter end
See the section on “Atomicity” for cool uses of atomic counter blocks.
A convenience class that wraps the pattern of {using setnx to perform locking}[http://redis.io/commands/setnx].
require 'redis/lock' @lock = Redis::Lock.new('image_resizing', :expiration => 15, :timeout => 0.1) @lock.lock do # do work end
This can be especially useful if you’re running batch jobs spread across multiple hosts.
Simple values are easy as well:
require 'redis/value' @value = Redis::Value.new('value_name') @value.value = 'a' @value.delete
Complex data is no problem with :marshal => true:
@account = Account.create!(params[:account]) @newest = Redis::Value.new('newest_account', :marshal => true) @newest.value = @account.attributes puts @newest.value['username']
Lists work just like Ruby arrays:
require 'redis/list' @list = Redis::List.new('list_name') @list << 'a' @list << 'b' @list.include? 'c' # false @list.values # ['a','b'] @list << 'c' @list.delete('c') @list[0] @list[0,1] @list[0..1] @list.shift @list.pop @list.clear # etc
You can bound the size of the list to only hold N elements like so:
# Only holds 10 elements, throws out old ones when you reach :maxlength. @list = Redis::List.new('list_name', redis_handle, :maxlength => 10)
Complex data types are no problem with :marshal => true:
@list = Redis::List.new('list_name', :marshal => true) @list << {:name => "Nate", :city => "San Diego"} @list << {:name => "Peter", :city => "Oceanside"} @list.each do |el| puts "#{el[:name]} lives in #{el[:city]}" end
Hashes work like a Ruby Hash, with a few Redis-specific additions. (The class name is “HashKey” not just “Hash”, due to conflicts with the Ruby core Hash class in other gems.)
require 'redis/hash_key' @hash = Redis::HashKey.new('hash_name') @hash['a'] = 1 @hash['b'] = 2 @hash.each do |k,v| puts "#{k} = #{v}" end @hash['c'] = 3 puts @hash.all # {"a"=>"1","b"=>"2","c"=>"3"} @hash.clear
Redis also adds incrementing and bulk operations:
@hash.incr('c', 6) # 9 @hash.bulk_set('d' => 5, 'e' => 6) @hash.bulk_get('d','e') # "5", "6"
Remember that numbers become strings in Redis. Unlike with other Redis data types, redis-objects can’t guess at your data type in this situation.
Sets work like the Ruby Set class:
require 'redis/set' @set = Redis::Set.new('set_name') @set << 'a' @set << 'b' @set << 'a' # dup ignored @set.member? 'c' # false @set.members # ['a','b'] @set.members.reverse # ['b','a'] @set.each do |member| puts member end @set.clear # etc
You can perform Redis intersections/unions/diffs easily:
@set1 = Redis::Set.new('set1') @set2 = Redis::Set.new('set2') @set3 = Redis::Set.new('set3') members = @set1 & @set2 # intersection members = @set1 | @set2 # union members = @set1 + @set2 # union members = @set1 ^ @set2 # difference members = @set1 - @set2 # difference members = @set1.intersection(@set2, @set3) # multiple members = @set1.union(@set2, @set3) # multiple members = @set1.difference(@set2, @set3) # multiple
Or store them in Redis:
@set1.interstore('intername', @set2, @set3) members = @set1.redis.get('intername') @set1.unionstore('unionname', @set2, @set3) members = @set1.redis.get('unionname') @set1.diffstore('diffname', @set2, @set3) members = @set1.redis.get('diffname')
And use complex data types too, with :marshal => true:
@set1 = Redis::Set.new('set1', :marshal => true) @set2 = Redis::Set.new('set2', :marshal => true) @set1 << {:name => "Nate", :city => "San Diego"} @set1 << {:name => "Peter", :city => "Oceanside"} @set2 << {:name => "Nate", :city => "San Diego"} @set2 << {:name => "Jeff", :city => "Del Mar"} @set1 & @set2 # Nate @set1 - @set2 # Peter @set1 | @set2 # all 3 people
Due to their unique properties, Sorted Sets work like a hybrid between a Hash and an Array. You assign like a Hash, but retrieve like an Array:
require 'redis/sorted_set' @sorted_set = Redis::SortedSet.new('number_of_posts') @sorted_set['Nate'] = 15 @sorted_set['Peter'] = 75 @sorted_set['Jeff'] = 24 # Array access to get sorted order @sorted_set[0..2] # => ["Nate", "Jeff", "Peter"] @sorted_set[0,2] # => ["Nate", "Jeff"] @sorted_set['Peter'] # => 75 @sorted_set['Jeff'] # => 24 @sorted_set.score('Jeff') # same thing (24) @sorted_set.rank('Peter') # => 2 @sorted_set.rank('Jeff') # => 1 @sorted_set.first # => "Nate" @sorted_set.last # => "Peter" @sorted_set.revrange(0,2) # => ["Peter", "Jeff", "Nate"] @sorted_set['Newbie'] = 1 @sorted_set.members # => ["Newbie", "Nate", "Jeff", "Peter"] @sorted_set.members.reverse # => ["Peter", "Jeff", "Nate", "Newbie"] @sorted_set.rangebyscore(10, 100, :limit => 2) # => ["Nate", "Jeff"] @sorted_set.members(:with_scores => true) # => [["Newbie", 1], ["Nate", 16], ["Jeff", 28], ["Peter", 76]] # atomic increment @sorted_set.increment('Nate') @sorted_set.incr('Peter') # shorthand @sorted_set.incr('Jeff', 4)
The other Redis Sorted Set commands are supported as well; see Sorted Sets API.
You are probably not handling atomicity correctly in your app. For a fun rant on the topic, see An Atomic Rant.
Atomic counters are a good way to handle concurrency:
@team = Team.find(1) if @team.drafted_players.increment <= @team.max_players # do stuff @team.team_players.create!(:player_id => 221) @team.active_players.increment else # reset counter state @team.drafted_players.decrement end
Atomic block - a cleaner way to do the above. Exceptions or return nil rewind counter back to previous state:
@team.drafted_players.increment do |val| raise Team::TeamFullError if val > @team.max_players @team.team_players.create!(:player_id => 221) @team.active_players.increment end
Similar approach, using an if block (failure rewinds counter):
@team.drafted_players.increment do |val| if val <= @team.max_players @team.team_players.create!(:player_id => 221) @team.active_players.increment end end
Class methods work too - notice we override ActiveRecord counters:
Team.increment_counter :drafted_players, team_id Team.decrement_counter :drafted_players, team_id, 2 Team.increment_counter :total_online_players # no ID on global counter
Class-level atomic block (may save a DB fetch depending on your app):
Team.increment_counter(:drafted_players, team_id) do |val| TeamPitcher.create!(:team_id => team_id, :pitcher_id => 181) Team.increment_counter(:active_players, team_id) end
Locks with Redis. On completion or exception the lock is released:
class Team < ActiveRecord::Base lock :reorder # declare a lock end @team.reorder_lock.lock do @team.reorder_all_players end
Class-level lock (same concept)
Team.obtain_lock(:reorder, team_id) do Team.reorder_all_players(team_id) end
Lock expiration. Sometimes you want to make sure your locks are cleaned up should the unthinkable happen (server failure). You can set lock expirations to handle this. Expired locks are released by the next process to attempt lock. Just make sure you expiration value is sufficiently large compared to your expected lock time.
class Team < ActiveRecord::Base lock :reorder, :expiration => 15.minutes end
Copyright © 2009-2010 Nate Wiger. All Rights Reserved. Released under the Artistic License.