activity cache replication

If there is a communications fault for master redis instance, normally activity-cache0, haproxy will automatically start using the slave, activity-cache1. However when the redis master re-joins the cluster haproxy does not switch back to the master.

This procedure should reset the slave-master setup to normal, activity-cache0=master (now called 0) and activity-cache1=slave (now called 1).

Obviously check the for the private IPs of activity-cache{0,1} in case they've changed for some reason.

1. first swap master/slave relationship;

   on 1...

   redis-cli slaveof no one

   on 0...

   redis-cli slaveof 10.224.14.104 6379 redis-cli sync
2. wait until replication is normalized

   on 0

   redis-cli info | grep master_

   if master_sync_in_progress=0 then no sync is in progress

   if master_last_io_seconds_ago>0 but only a few seconds then a sync has been done recently

   A better definitive check would be nice

3. swap master/slave back again

   on 0...

   redis-cli slaveof no one

   on 1...

   redis-cli slaveof 10.224.14.103 6379 redis-cli sync
4. reload haproxy

   on proxy0...

   oldpid=$(ps -A | awk '/redis/ { print $1 }') service haproxy reload kill $oldpid

Checking the activity cache is easy, log in and upload a doc and it should appear in your activity stream.