Skip to content

Channel Index Code Structure

Adam Fraser edited this page Sep 15, 2015 · 2 revisions

###Code Structure

The in-memory cache used the following pattern:

  • For each database, initialize a changeCache
  • The changeCache maintains a collection of channelCache instances, which manage the in-memory storage for each channel
  • Feed events are sent to the changeCache
  • For each incoming doc, changeCache does sequence buffering (processEntry), then calls channelCache.addToCache for each channel on the doc

For 1.2, we want to support both options: in-memory cache, and a key value store index. To support this we've added the following:

ChangeIndex
Interface, defines interaction with the channel store. Each database has an instance of changeIndex, and the old changeCache has been refactored to implement the ChangeIndex interface.

kvChangeIndex
Implementation of changeIndex used for distributed index ("key value change index"). Responsibilities:

  • Maintains two collections of channel indexes - channelIndexWriters, and channelIndexReaders. These are separated to avoid synchronization/locking overhead, as well as separation of functional handling.
  • Detecting changes in the index and notifying active _changes feeds
  • Receiving feed events, and calling the appropriate channelIndexWriters

kvChannelIndex
Manages writes and reads from ChannelStorage, and manages the channel clock value

ChannelStorage
Interface - defines operations used to read and write from bucket. Currently only one implementation of the interface exists (BitFlagStorage), but intended to isolate the index storage format to simplify any changes we decide to make based on performance results

BitFlagStorage
ChannelStorage implementation that tracks channel presence using an array of bytes, where each byte represents presence/absence of the sequence in the channel, and whether it's a removal.

SequenceClock

Interface to manage the vector clock storage and manipulation. Two implementations - SequenceClockImpl (for single-threaded use), and SyncSequenceClockImpl (synchronized).

SequenceID

The existing SequenceID struct has been extended to support either single integer sequence or vector clock sequence. SequenceID includes a flag (SeqType) to indicate whether t represents an integer or clock-based sequence. Originally made an attempt to convert SequenceID to an interface and have two different sequence types as implementations of the interface, but would have introduced a great deal of instability/refactoring around the SequenceID marshalling/unmarshalling for _changes handling.

index_changes.go
Defines the changes feed implementation when using a kvChangeIndex. In particular VectorMultiChangesFeed, which corresponds to the existing changes.go/MultiChangesFeed. The general approach is the same, but there were enough differences between the two changes feeds that it made more sense to split them out, instead of making changes.go even harder to read with a lot of switch statements.

Clone this wiki locally