-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
python-cache suggests using pylibmc in a way that is not threadsafe #3
Comments
That sounds workable. I'd happily merge that. |
I ended up deciding to switch to dogpile.cache. Its API has a lot of the same advantages of python-cache, but it also takes care of the "dogpile problem" i.e. avoiding many concurrent attempts to recompute an expired cache, and it deals with making pylibmc thread safe for you. Here's a couple of posts about its rationale from its author, Mike Bayer (also the author of SQLAlchemy, and a couple other libraries we use): Thoughts on beaker python-cache served us well for a long time while our request volume was reasonably low, but at this point, it doesn't make sense to me to put time into updating its API and adding the necessary tests. I wanted to make you, and possibly other users, aware of this issue, but it might make sense to just point people at dogpile.cache in the future unless you think this lib still has important advantages over it. |
See pylibmc's documentation on pooling:
http://sendapatch.se/projects/pylibmc/pooling.html
A single pylibmc client can not be used concurrently by different threads. It's common for python web frameworks and web servers to use several threads to respond to incoming requests, so following python-cache's usage example makes it very easy to create invalid concurrent operations.
Beaker deals with this by using a ThreadMappedPool:
https://github.com/bbangert/beaker/blob/95cf84aff9242a86fdfced1cf9747bbe6fdc4029/beaker/ext/memcached.py#L179
It might be best for python-cache to take a pool (maybe a ThreadMappedPool, but a ClientPool might be preferable, see the pooling docs) instead of taking a client.
The text was updated successfully, but these errors were encountered: