500 Error

[ComradeDaria]

Hi, I'm afraid the API is down. It returns a 500 error.

[tae898]

Same here

[henningsmith]

I'm looking in to it now - hope to have it back up soon

[henningsmith]

Back up and running

[tae898]

@henningsmith yay thanks!

[ComradeDaria]

Great, thank you!

[ComradeDaria]

Hi, I'm sorry, but this just happened again. Got a 500: Internal Server Error from the API and the demo page.

[henningsmith]

I restarted the services again, but it seems someone is running a lot of queries against the system continuously.

[amirouche]

Internal server error is usually a problem with the backend code. I never seen such error raised because of high load. Is there a traceback available ?

[henningsmith]

There were 2 separate types of errors today:

Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]: [2021-08-15 17:00:46,840] ERROR in app: Exception on /a/[/r/EtymologicallyRelatedTo/,/c/en/drag/,/c/nds/dragge/] [GET]
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]: Traceback (most recent call last):
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:     response = self.full_dispatch_request()
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:     rv = self.handle_user_exception(e)
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:     reraise(exc_type, exc_value, tb)
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:     raise value
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:     rv = self.dispatch_request()
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:     return self.view_functions[rule.endpoint](**req.view_args)
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:   File "/home/conceptnet/conceptnet5/web/conceptnet_web/web.py", line 133, in browse_node
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:     results = responses.lookup_paginated(path, offset=offset, limit=limit)
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:   File "/home/conceptnet/conceptnet5/conceptnet5/api.py", line 150, in lookup_paginated
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:     found = FINDER.lookup(term, limit=(limit + 1), offset=offset)
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:   File "/home/conceptnet/conceptnet5/conceptnet5/db/query.py", line 142, in lookup
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:     return self.lookup_assertion(uri)
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:   File "/home/conceptnet/conceptnet5/conceptnet5/db/query.py", line 194, in lookup_assertion
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]:     cursor = self.connection.cursor()
Aug 15 17:00:46 ip-172-31-6-244 uwsgi[1208]: psycopg2.InterfaceError: connection already closed

and

Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]: [2021-08-16 14:18:39,630] ERROR in app: Exception on /c/zh/文件夾 [GET]
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]: Traceback (most recent call last):
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     response = self.full_dispatch_request()
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     rv = self.handle_user_exception(e)
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask_cors/extension.py", line 161, in wrapped_function
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     return cors_after_request(app.make_response(f(*args, **kwargs)))
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     reraise(exc_type, exc_value, tb)
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     raise value
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 1948, in full_dispatch_request
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     rv = self.preprocess_request()
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask/app.py", line 2242, in preprocess_request
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     rv = func()
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask_limiter/extension.py", line 534, in __check_request_limit
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     six.reraise(*sys.exc_info())
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/six.py", line 703, in reraise
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     raise value
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask_limiter/extension.py", line 517, in __check_request_limit
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     self.__evaluate_limits(endpoint, all_limits)
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/flask_limiter/extension.py", line 393, in __evaluate_limits
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     if not self.limiter.hit(lim.limit, *args):
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/limits/strategies.py", line 132, in hit
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     self.storage().incr(item.key_for(*identifiers), item.get_expiry())
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/limits/storage.py", line 171, in incr
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     self.__schedule_expiry()
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/home/conceptnet/env/lib/python3.6/site-packages/limits/storage.py", line 159, in __schedule_expiry
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     self.timer.start()
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:   File "/usr/lib/python3.6/threading.py", line 846, in start
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]:     _start_new_thread(self._bootstrap, ())
Aug 16 14:18:39 ip-172-31-6-244 uwsgi[1193]: RuntimeError: can't start new thread

[krgallagher]

Hi, sorry to bother, but there is another 500 server error. Many thanks!

[henningsmith]

Was the _start_new_thread error again...

[amirouche]

Internal server error is usually a problem with the backend code. I never seen such error raised because of high load.

The proof of the pudding is in the eating. That sentence shows my lack of experience with high loads.

Is there a traceback available ?

I will look into the traceback from #306 (comment) both seem familiar and related to high concurrency, possibly a mis-configuration related to how the flask application is running. In the the documentation build process, it is noted that the application can be run with:

python conceptnet_web/api.py

It prolly not how it is run in production. I guess you use gunicorn. The information is in the AMI but I can not spin up that at the moment. Also, I will need to number of available cores, such as displayed by htop (on the top the number of bars that grow green) or using the following shell code:

cat /proc/cpuinfo | grep processor | wc -l

If you have multiple vm, I need the gunicorn start up command (prolly in a systemd file or supervisord configuration) and the number of available processor. Regarding gunicorn I am interested in the worker class, number of workers and possibly the number of threads.

Edit: Production use uwsgi as show in the traceback. Sorry for the noise.

[stargazing-dino]

Hello ! I really appreciate all the work behind this service.

Looks like we're getting 500s again. Both on https://conceptnet.io/ and from external requests.

[Dei6]

InternalServerError: 500 Internal Server Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.

[TaichiY496]

Sorry, I ran too many queries last Wednesday!
And it seems still not recovered…

[Rayckey]

Unfortunately, it seems to be having the same issues again, any queries I made have the same error: "details': 'Internal server error', 'status': 500"