Skip to content
This repository has been archived by the owner on May 26, 2023. It is now read-only.

Commit

Permalink
Add locks around zstd compress calls
Browse files Browse the repository at this point in the history
The python library is explicitly NOT thread-safe. This will still let us
not block on compression. If ever needed, we can ask zstd to compress with
multiple threads anyway.

Signed-off-by: Igor Kotrasinski <i.kotrasinsk@gmail.com>
  • Loading branch information
Igor Kotrasinski committed Jan 17, 2021
1 parent 27f3f51 commit 2747dcb
Showing 1 changed file with 9 additions and 2 deletions.
11 changes: 9 additions & 2 deletions replayserver/bookkeeping/storage.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
import json
import zstandard as zstd
import asyncio
import threading

from replayserver.errors import BookkeepingError
from replayserver.logging import short_exc
Expand Down Expand Up @@ -40,7 +41,11 @@ class ReplaySaver:
def __init__(self, paths, database):
self._paths = paths
self._database = database
self._compressor = zstd.ZstdCompressor(level=10)
# TODO - consider multi-threaded compression if we end up needing more
# performance.
self._compressor = zstd.ZstdCompressor(level=10,
write_checksum=True)
self._compressor_lock = threading.Lock()

@classmethod
def build(cls, database, config):
Expand Down Expand Up @@ -94,7 +99,9 @@ def _write_replay(self, rfile, info, data):
try:
rfile.write(json.dumps(info).encode('UTF-8'))
rfile.write(b"\n")
data = self._compressor.compress(data)
# zstandard is explicitly NOT thread-safe
with self._compressor_lock:
data = self._compressor.compress(data)
rfile.write(data)
# json should always produce ascii, but just in case...
except UnicodeEncodeError:
Expand Down

0 comments on commit 2747dcb

Please sign in to comment.