Documentation for the BGPRedisSaver application

Table of contents

1. Table of contents
2. Overview
3. Before you start
   1. Servers
   2. Create Cassandra Tables & Keyspace
   3. Install Libraries
   4. Edit CMakeLists for your own infrastructure
4. Use the application

Overview

This application performs periodic deletion of data from a Redis database. The deleted data is transferred to a Cassandra database.

To do so, we use Redis' ordered sets (ZSets) to store key-score pairs. Once a set of keys reaches a certain treshold (in terms of cardinality), we choose one or sevral keys in the set, depending on their scores, to go back under the threshold. Then, the related data is deleted from its structure. Also, the key is deleted from the ordered set so we can access the new cardinality.

The keys we find in the ZSet can be :

• Static : The related data will be a single data stored in a Hash (HSet) or directly stored in Redis (stored as key-value).
• Non-Static : There are multiple related data meaning the key is modified to find the key of the list of data that has to be deleted. In this case, we simply had a prefix to the key and it gives the key of the list.

On Cassandra's side, we use 4 tables :

• A default table for unknown value we could try to save (named 'Default_Table'),
• A table for Paths (named 'Path'),
• A table for Routing events (named 'RoutingEvent'),
• A table for AS events (named 'ASEvent').

Before you start

Servers

Before you start to use the application, you have to ensure that you have both Redis AND Cassandra servers running (you can use remote servers).

Redis :

• Installation : https://redis.io/download
• Running :
  • Linux : sudo service redis-server start
  • Windows : Go to the directory where you installed Redis (usually C:\Redis) and run redis-server.exe

Cassandra :

• Installation : https://cassandra.apache.org/doc/latest/getting_started/installing.html
• Running : Go to the directory where you installed Cassandra (usually "Program Files"\apache-cassandra-3.11.10\bin) and run "cassandra.bat -f"

Create Cassandra Tables & Keyspace

Before using this application, you need to create Cassandra's keyspace and tables. To do so, you can execute the script named BGPCassandraDatabaseCreation :

cqlsh --file="path\to\BGPCassandraDatabaseCreation"

Install Libraries

You must also ensure that you installed the following libraries :

• Redis++ : https://github.com/sewenew/redis-plus-plus
• DataStax's c++ Cassandra Driver : https://github.com/datastax/cpp-driver

Edit CMakeLists for your own infrastructure

I had issues to include the DataStax c++ driver for Cassandra. You will probably have to edit the CMakeLists.txt file and to provide an access to this library.

Use the application

You can change some parameters via the command line when launching the application. These options can be found by running :

/path/to/BGPRedisSaver --help

Options :

• -R <host> <port> Connection information for Redis database
• -C <host> <port> Connection information for Cassandra database
• -P <bool> Authorize (or not) to print information
• -N <BATCH_MAX_SIZE> Number of accumulated values before effective insertion in Cassandra
• -B <sleep> Integer defining the duration of the break between the analyse of two sets.
• --help Show this page of help
• -S <set1> [<set2>] [...] Where <set> is <keys,values,size[,static[,dstTable]]> with :
  • keys : Name of the keys' set.
  • values : Name of the values' set.
  • size : size threshold for this set (number of keys).
  • static : boolean defining the mode (static/non-static)
  • -dstTable : Cassandra table where the data will be transferred.

To stop the application, you have to publish the message "STOP" on the "ChannelEndBGPRedisSaver" redis channel. To do so, you can run the following command :

redis-cli PUBLISH ChannelEndBGPRedisSaver STOP