Add ability to load triemux from Content Store #490
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
theseanything
force-pushed
the
populate-triemux-from-cs
branch
3 times, most recently
from
4eb7a0b
to
5aec294
Compare
This struct and type contains the logic for routes contained previously in the Router API and be used to help choose which handler to use.
Used for interacting with content store's postgres db.
This implements the logic to query the content store and populate the triemux.
This makes a connection pool to the database url specified with he env var CONTENT_STORE_DATABASE_URL.
This provides a simple feature flag to turn off routing using content store and allows us to canary a percentage of requests to use the content store mux. A value of 0.0 will switch off the feature entirely and prevent the Router from trying to create a connection pool.
This uses that the CsMuxSampleRate to controller the percentage of requests served by the mux built using the content store data.
This adds a helper function to update csmux with a new version containing updated routes.
This fixes a golint error.
This is a fail safe in-case the listen/notify mechanism.
theseanything
force-pushed
the
populate-triemux-from-cs
branch
from
eaa62b5
to
0c38590
Compare
This ensures we don't reload more than we need to. If a notification already exists to reload, then we don't need to send another notification, as all updates will be captured in the next reload.
theseanything
force-pushed
the
populate-triemux-from-cs
branch
from
6de67a8
to
e54299c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the ability to build and use a triemux from Content Store's postgres database instead of MongoDB. Moving away from MongoDB, which is running an outdated and unsupported version, resolves multiple challenges, including the inability to shift workloads to ARM and difficulties in maintaining a local development environment. By leveraging PostgreSQL, the Content Store can become the single source of truth for routes, simplifying the system architecture and removing the dependency on MongoDB.
This change also paves the way for retiring the Router API, allowing for a leaner, more efficient architecture. Instead of querying an API endpoint on the Content Store, this approach directly queries the PostgreSQL database. This is beneficial because the large number of routes involved would add significant latency if serialized and deserialized through an API layer. A risk to note is that schema changes in the Content Store could impact route loading in Router, though such changes are infrequent, and ongoing work may eventually make it possible to remove Router altogether.
Traffic control is managed through an environment variable, CSMUX_SAMPLE_RATE, which accepts values between 0.0 and 1.0. This variable sets the sampling rate for serving requests via the triemux built from the Content Store. A value of 0.0 disables triemux loading from the Content Store entirely, while higher values allow gradual adoption, enabling canarying of traffic to minimize rollout risks.
The Content Store PostgreSQL database URL is specified through the CONTENT_STORE_DATABASE_URL environment variable. Route re-loading is triggered automatically by PostgreSQL’s LISTEN/NOTIFY mechanism, which responds to inserts, updates, or deletions in the content_items or publish_intents tables. alphagov/content-store#1340
This resolves: #497