Releases: reddit/baseplate.py
v1.4.0
New Features
Tagged statsd metrics
Baseplate.py now includes an optional new metrics observer that emits
InfluxStatsD tagged metrics. This makes metric paths more discoverable and
allows for much cleaner metric queries. Additionally, you can change the
default list of tags to further slice your metrics as needed in your
application.
See upgrade instructions below and #452 for history
JSON Structured Logging
The default format of log entries emitted by Baseplate.py applications is now
JSON. This allows a variety of metadata to be included in log entries without
cluttering up the main message.
See the logging observer documentation for full details of the available
fields.
See upgrade instructions below and #430 and #466 for history
New healthcheck protocol
Baseplate.py now supports different kinds of health checks. Currently
supported healthcheck probe types are: readiness, liveness, and startup. This
can be used to fine tune when the infrastructure should route traffic to your
application and when it should restart an unresponsive application.
See upgrade instructions below and #460 and #468 for history
Instrumented HTTP client
Requests is a library for making HTTP requests. Baseplate.py now provides two
wrappers for Requests: the "external" client is suitable for communication with
third party, potentially untrusted, services; the "internal" client is suitable
for talking to first-party services and automatically includes trace and edge
context data in requests. Baseplate.py uses Advocate to prevent the external
client from talking to internal services and vice versa.
See baseplate.clients.requests documentation and #465 and #479 for history
Changes
- The KombuProducer client now passes a
secretsparameter through to
connection_from_config(#451) - Baseplate.thrift now contains a standardized typedef for timestamp fields.
Use this in your services to communicate the timestamp format expected. (#458) - A new convenience function
baseplate.lib.datetime.epoch_milliseconds_to_datetimeparses
milliseconds since the UNIX epoch into a datetime object (#461). - When using a pre-release of Thrift v0.14, the Thrift connection pool will now
test if the connection was closed by the remote side before returning a
connection to the application for use. This should reduce the frequency of
spurious RPC failures. (also in 1.3.5, #464) - Many additions and changes to the documentation. Check out the new Startup
Sequence and FAQ pages. (#467, #469, #471, #473) - Unblock use of
lock()on redis client wrapper (#480)
Bug fixes
- Fix bug where thrift pool gains an extra slot after the retry policy is
exhausted while trying to connect (also in 1.3.1, #447) - The event publisher sidecar will no longer crash when events are rejected
because of validation errors (also in 1.3.1, #450) - Fix extra connection slots being returned to thrift connection pool when
server timeout occurs while waiting for a slot (also in 1.3.2, #431) - Properly handle base64-encoded edge context headers in Pyramid services (also
in 1.3.2, #449, #456) - Fix breaking behavior introduced in 1.2 around server timeouts. Rather than
defaulting to a 10 second timeout if not specified in config, timeouts are
defaulted to off and a deprecation warning is emitted. Please do configure
timeouts for your service as they're an important mitigation for overloads
during production incidents. (also in 1.3.3, #446)
Upgrading
This release should not contain any breaking changes but following are notes on
how to upgrade to take advantage of the new features.
Tagged statsd metrics
For reddit employees, search "Baseplate Tags Guide (v1.4+)" in Confluence to
get a comprehensive upgrade guide or come to #baseplate-tags-support in Slack.
In your application's configuration file, remove metrics.namespace
configuration and add metrics.tagging = true. See the tagged metrics
observer documentation for full information.
JSON Structured Logging
By default, no changes should be necessary to get JSON formatted logs. However,
if your service is overriding the default logging.Formatter (e.g. by
configuring it in your INI file) you may need to amend your
configuration. See python-json-logger's documentation for more information. Note that
Baseplate.py supplies a custom formatter subclass that you can find at baseplate.lib.log_formatter.CustomJsonFormatter.
New healthcheck protocol
To support the new healthcheck, you'll need to make a few small changes:
In your Thrift IDL, change the base your service extends to BaseplateServiceV2:
--- a/myservice/myservice.thrift
+++ b/myservice/myservice.thrift
-service MyService extends baseplate.BaseplateService {
+service MyService extends baseplate.BaseplateServiceV2 {Then add the new request parameter to your service's is_healthy method and
use its probe field:
--- a/myservice/__init__.py
+++ b/myservice/__init__.py
@@ -6,12 +6,15 @@ import re
from typing import Dict
from typing import List
+from typing import Optional
from baseplate import Baseplate
from baseplate import RequestContext
from baseplate.clients.redis import RedisClient
from baseplate.frameworks.thrift import baseplateify_processor
from baseplate.lib import config
+from baseplate.thrift.ttypes import IsHealthyProbe
+from baseplate.thrift.ttypes import IsHealthyRequest
from thrift.Thrift import TProcessor
from .myservice_thrift import MyService
@@ -45,8 +48,13 @@ class Handler:
- def is_healthy(self, context: RequestContext) -> bool:
- return context.redis.ping()
+ def is_healthy(
+ self, context: RequestContext, request: Optional[IsHealthyRequest] = None
+ ) -> bool:
+ if request.probe == IsHealthyProbe.LIVENESS:
+ return True
+ else: # treat all other probe types as READINESS
+ return context.redis.ping()v1.3.5
v1.3.4
v0.30.8
v1.3.3
Bug fixes
- Fix breaking behavior introduced in 1.2 around server timeouts. Rather than defaulting to a 10 second timeout if not specified in config, timeouts are defaulted to off and a deprecation warning is emitted. Please do configure timeouts for your service as they're an important mitigation for overloads during production incidents. (#446)
v1.3.2
v1.3.1
Bug Fixes
v1.3.0
New Features
Standardized Thrift Error Exceptions
This new standardized exception type and accompanying common error code enumeration makes it easy for you to consume errors from multiple services and for them to understand what you're trying to say with your errors.
To use this, add baseplate.Error to the throws declaration on one of your service's methods:
service MyService extends baseplate.BaseplateService {
void do_something() throws (1: baseplate.Error error),
}and then raise the exception in your code:
from baseplate.thrift.ttypes import Error, ErrorCode
class Handler:
def do_something(self, context):
raise Error(code=ErrorCode.IM_A_TEAPOT)Note that changing exception types is a breaking change in your service's interface, so if you want to move over you'll need to add the new type, update all callers, and then remove the old one. Also note that clients will need to be running Baseplate.py 1.3+ (or have the latest Thrift IDL in your Baseplate.go service) to be able to consume these errors.
See the Thrift IDL for more details.
See #428
StatsD Metrics Sampling
If your service is handling a lot of traffic, it's also probably generating a lot of metrics. This can overwhelm your metrics aggregator and cause incorrect measurements. You can now configure the rate at which metrics are sampled. This is done at the whole-request level so a single request will either generate all metrics or none.
[app:main]
...
metrics_observer.sample_rate = 75%
...Note that this will not affect metrics generated manually with context.metrics.counter() etc. If you want those metrics to be sampled at the same rate, consider using local spans for timers and incr_tag() for counters instead.
See the documentation for more information.
Easier creation of server spans
Baseplate requires a span context to do a lot of its work. In simple one-off tasks like cron jobs and migration scripts this can be extra overhead to set up. To simplify this common pattern, there's now a context manager, server_context that does all the setup for you.
context = baseplate.make_context_object()
with baseplate.make_server_span(context, "foo"):
context.foo.bar()becomes:
with baseplate.server_context("foo") as context:
context.foo.bar()See #419
Error handler callback for Kombu queue consumer
Baseplate's Kombu queue consumer would previously re-queue messages where processing raised an exception. This new callback allows you to take control of what happens in that situation and manage the failed message yourself.
See the documentation for more details.
See #413
Automatic identification of services calling you
Baseplate now sends a new header to services when making requests that identifies who's making the call. This can be used for metrics, ratelimiting, etc.
By default, Baseplate will auto-detect the name of the Python module that created the Baseplate() object and use that as the service's identity when calling other services. If this doesn't work or the name is not what you'd like to present, you can change it in your service's configuration file:
[app:main]
baseplate.service_name = fooNote that this requires your application to pass its config to the Baseplate() object as has been standard since v1.2. See the upgrade instructions below for further details.
For now, this is exposed as a new span tag, peer.service, that can be accessed by observers (including custom ones in your application).
See #422
Changes
- redis-py 3.0+ is now supported (also in 1.2.1, #393)
- Edge context additions:
- Add
pool_pre_pingto SQLAlchemy configuration (#409)
Bug Fixes
- Fix crash in Pyramid integration when request parsing fails very early (also in 1.2.1, #395)
- Fix serialization of booleans when submitting traces to Wavefront, allowing it to parse those values properly. (also in 1.2.1, #401, #402)
- Various typos and mistakes in the documentation (also in 1.2.1, #406, #407)
- Fix thrift connection pool resource leaks that can happen when server timeouts fire while the application is allocating or releasing connections. (also in 1.2.2, #421)
- Fix some crashes that might happen if an observer raises an exception during on_start() (also in 1.2.3, #415)
- Send queue consumer healthcheck logging through the proper logging system rather than to stderr (also in 1.2.3, #424)
- Fix the response type of the queue consumer healthcheck to match baseplate standard (also in 1.2.3, #425)
- Fix regression in deprecated queue consumer when internal work queue is empty (also in 1.2.4, #432)
Upgrading
There are no breaking changes in this release. You can use the upgrader tool to upgrade and get advice on the following deprecation.
Application wire-up and configuration
Since v1.2 it's been possible to pass the application configuration to the Baseplate() constructor rather than to each subsequent call. The alternative way of doing things is now deprecated (and will be removed in 2.0) and for features like Thrift service identity to work properly, please ensure your application does something like this during wire-up:
def make_processor(app_config):
baseplate = Baseplate(app_config)
baseplate.configure_observers()
baseplate.configure_context(...)