Some more doc

README.md

@@ -1,59 +1,33 @@
 # silo
-Silo is a general purpose storage engine
+Silo is a general purpose storage engine. One of the core functionalities within Silo is the ability to migrate/sync storage while it is still in use.
 
-## Migration
-
-Example of a basic migration
-
-![alt text](./graph.png?raw=true)
-
-This example adds a device reading from the destination. The block order is by least volatile, but with a priority for the blocks needed for reading.
-
-![alt text](./images/example1.png?raw=true)
+## Sources
 
-Same as above, but with concurrency set to 32. As long as the destination can do concurrent writes, everything will flow.
+All storage sources within Silo implement `storage.StorageProvider`. You can find some example sources at [pkg/storage/sources](pkg/storage/sources/README.md).
 
-![alt text](./images/example2.png?raw=true)
+## Expose
 
+If you wish to expose a Silo storage device to an external consumer, one way would be to use the NBD kernel driver. See [pkg/expose/sources](pkg/storage/expose/README.md).
 
-## Sources
+## Block orders
 
-Silo storage sources.
+When you wish to move storage from one place to another, you'll need to specify an order. This can be dynamically changing. For example, there is a volatility monitor which can be used to migrate storage from least volatile to most volatile. Also you may wish to prioritize certain blocks for example if the destination is trying to read them. [pkg/storage/blocks](pkg/storage/blocks/README.md).
 
-* File source
-* Memory source
+## Migration
 
-## Modules
+Migration of storage is handled by a `Migrator`. For more information on this see [pkg/storage/migrator](pkg/storage/migrator/README.md).
 
-Modules can be chained together to create various storage workflows / setups.
+Example of a basic migration. Here we have block number on the Y axis, and time on the X axis.
+We start out by performing random writes to regions of the storage. We start migration at 500ms and you can see that less volatile blocks are moved first (in blue). Once that is completed, dirty blocks are synced up (in red).
 
-| Module                | Description |
-| --------------------- | ----------- |
-| ArtificialLatency     | Adds a fixed latency to reads / writes for testing/benchmarking.            |
-| Cache                 | This contains a source, and a store of what data is contained in the cache. If the data is not in the cache for a read, a cache miss error is returned. |
-| DirtyTracker          | This tracks write operations for some period, so we know which areas of the source are dirty. |
-| FilterRedundantWrites | This can filter out any redundant writes (Write data is same as source data), and can also cut large writes into smaller writes which only contains changed data. |
-| Lockable              | This can lock a source for some period. |
-| Metrics               | General metrics module to track read/write ops,bytes and latency |
-| Nothing               | Empty module - /dev/null |
-| ReadDirtyTracker      | Similar to a DirtyTracker, but only starts tracking writes to a block after it's been read ONCE. |
-| ShardedStorage        | This shards storage into multiple storage, which for example allows concurrent writes across shards. |
-| Splitter              | This allows the insertion of cache type sources. Any cache hit reads return immediately, whilst misses result in a read from the source, which then also gets written to any of the cache type sources. |
-| VolatilityMonitor     | This tracks writes to blocks, and implements BlockOrder - it returns blocks in least volatile to most volatile order. |
-| WaitingCache          | With this cache, reads WAIT until a write to all blocks concerned have taken place. |
+![alt text](./graph.png?raw=true)
 
-## BlockOrder
+This example adds a device reading from the destination. The block order is by least volatile, but with a priority for the blocks needed for reading. You can also see on the graph that the average read latency drops as more of the storage is locally available at dest.
 
-BlockOrder allows specifying black order in migration. They can be chained together.
+![alt text](./images/example1.png?raw=true)
 
-| BlockOrder            | Description |
-| --------------------- | ----------- |
-| AnyBlockOrder         | This returns blocks in any order. |
-| PriorityBlockOrder    | This returns blocks in the order they were given priority, and then will defer to next() |
-| VolatilityMonitor     | THis returns blocks from least volatile to most volatile. |
+Same as above, but with concurrency set to 32. As long as the destination can do concurrent writes, everything will flow.
 
-## Expose
+![alt text](./images/example2.png?raw=true)
 
-These expose a silo storage as a device.
 
-* NBD device

pkg/storage/modules/README.md

@@ -0,0 +1,18 @@
+## Modules
+
+Modules can be chained together to create various storage workflows / setups.
+
+| Module                | Description |
+| --------------------- | ----------- |
+| ArtificialLatency     | Adds a fixed latency to reads / writes for testing/benchmarking.            |
+| Cache                 | This contains a source, and a store of what data is contained in the cache. If the data is not in the cache for a read, a cache miss error is returned. |
+| DirtyTracker          | This tracks write operations for some period, so we know which areas of the source are dirty. |
+| FilterRedundantWrites | This can filter out any redundant writes (Write data is same as source data), and can also cut large writes into smaller writes which only contains changed data. |
+| Lockable              | This can lock a source for some period. |
+| Metrics               | General metrics module to track read/write ops,bytes and latency |
+| Nothing               | Empty module - /dev/null |
+| ReadDirtyTracker      | Similar to a DirtyTracker, but only starts tracking writes to a block after it's been read ONCE. |
+| ShardedStorage        | This shards storage into multiple storage, which for example allows concurrent writes across shards. |
+| Splitter              | This allows the insertion of cache type sources. Any cache hit reads return immediately, whilst misses result in a read from the source, which then also gets written to any of the cache type sources. |
+| VolatilityMonitor     | This tracks writes to blocks, and implements BlockOrder - it returns blocks in least volatile to most volatile order. |
+| WaitingCache          | With this cache, reads WAIT until a write to all blocks concerned have taken place. |

pkg/storage/protocol/README.md

@@ -1,10 +1,53 @@
 # Silo protocol
 
+## Usage
+
+To use Silo connections, you must first use `ToProtocol` and `FromProtocol` to serialize and deserialize storage requests to the protocol.
+Both of these take a `Protocol` argument. One such implementation of `Protocol` is `ProtocolRW` which can use any `Reader` and `Writer` to communicate.
+
+So you can easily use a TCP connection, pipe, QUIC connection etc etc.
+
+The protocol itself is a bidirectional packet based protocol.
+
 There is a source (src), and a destination (dst) in a Silo connection for migration.
 
 All packets contain a DeviceID, and a transactionID.
 Several Devices can be multiplexed down a single connection, and several operations for example WriteAt can operate concurrently.
 
+## Simple example
+
+    r1, w1 := io.Pipe()
+  	r2, w2 := io.Pipe()
+
+    destDev := make(chan uint32, 8)
+
+    prSource := NewProtocolRW(context.TODO(), r1, w2, nil)
+    prDest := NewProtocolRW(context.TODO(), r2, w1, func(p Protocol, dev uint32) {
+      destDev <- dev
+    })
+
+    sourceToProtocol := NewToProtocol(uint64(size), 1, prSource)
+
+    storeFactory := func(di *DevInfo) storage.StorageProvider {
+      store = sources.NewMemoryStorage(int(di.Size))
+      return store
+    }
+
+    destFromProtocol := NewFromProtocol(1, storeFactory, prDest)
+
+There's a lot going on here, so lets go through what is happening.
+First we setup a couple of `io.Pipe()` to simulate some transport.
+We then wrap these pipes in `NewProtocolRW()`. The last argument here is a callback when a new device is detected (Any packet is received on a device not seen before). This callback can be used to start processing requests for that device for example. (In the example we just write to a channel).
+
+Next up, we create a `NewToProtocol()`. This implements `storage.StorageProvider` and can be used as storage within silo. Any storage requests are then serialized and sent through the protocol.
+
+Finally, we have a `NewFromProtocol()`, which deserializes storage requests.
+When a `DevInfo` is received, the storeFactory callback is called to initialize some `storage.StorageProvider` to handle subsequent storage requests.
+
+
+
+## Low level protocol
+
 The first packet for a device should always be a DevInfo packet.
 
 | Packet                | Direction | Description |

pkg/storage/sources/README.md

@@ -0,0 +1,9 @@
+# Sources
+
+## Memory
+
+MemoryStorage can be setup using `NewMemoryStorage(size)`. A RWMutex is used to ensure safety. If you wish to support concurrent writes, one way would be to use a `ShardedStorage` module to split the memory into several blocks which can then be written to concurrently.
+
+## File
+
+FileStorage can be setup using `NewFileStorage(f, size)` for an existing file, and `NewFileStorageCreate(f, size)` if you wish to create a new file.