scrapydweb fresh install won't run

[wmullaney]

I receive the following error when trying to run scrapydweb:

TypeError: init() got an unexpected keyword argument '_warn_on_bytestring'

I have had an extremely hard time installing scrapydweb. It's impossible using the current documentation. I am using a clean ubuntu 20.04 instance and have followed the documentation but I guess it's outdated. Anyone out there able to install this and get it running in 2022 on Ubuntu 20.04? If so mind helping me get this going? Thanks :)

[canahmetmaple]

It's probably because of your python version

[spechtx]

i second this. a fresh install like described in the docs is not working.

this is the error log after
pip install scrapydweb
scrapydweb

`(.venv) scrapy@:~/scrapydweb$ scrapydweb
[2022-12-21 23:31:53,636] INFO in apscheduler.scheduler: Scheduler started
[2022-12-21 23:31:53,649] INFO in scrapydweb.run: ScrapydWeb version: 1.4.0
[2022-12-21 23:31:53,649] INFO in scrapydweb.run: Use 'scrapydweb -h' to get help
[2022-12-21 23:31:53,649] INFO in scrapydweb.run: Main pid: 4523
[2022-12-21 23:31:53,649] DEBUG in scrapydweb.run: Loading default settings from /home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/scrapydweb/default_settings.py
Traceback (most recent call last):
File "/home/scrapy/scrapydweb/.venv/bin/scrapydweb", line 8, in
sys.exit(main())
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/scrapydweb/run.py", line 32, in main
app = create_app()
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/scrapydweb/init.py", line 81, in create_app
handle_db(app)
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/scrapydweb/init.py", line 123, in handle_db
db.create_all()
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/flask_sqlalchemy/extension.py", line 868, in create_all
self._call_for_binds(bind_key, "create_all")
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/flask_sqlalchemy/extension.py", line 839, in _call_for_binds
engine = self.engines[key]
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/flask_sqlalchemy/extension.py", line 628, in engines
app = current_app._get_current_object() # type: ignore[attr-defined]
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/werkzeug/local.py", line 513, in _get_current_object
raise RuntimeError(unbound_message) from None
RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
the current application. To solve this, set up an application context
with app.app_context(). See the documentation for more information.
2022-12-21 23:31:53,668 INFO sqlalchemy.engine.Engine BEGIN (implicit)
[2022-12-21 23:31:53,668] INFO in sqlalchemy.engine.Engine: BEGIN (implicit)
2022-12-21 23:31:53,671 INFO sqlalchemy.engine.Engine SELECT metadata.id AS metadata_id, metadata.version AS metadata_version, metadata.last_check_update_timestamp AS metadata_last_check_update_timestamp, metadata.main_pid AS metadata_main_pid, metadata.logparser_pid AS metadata_logparser_pid, metadata.poll_pid AS metadata_poll_pid, metadata.pageview AS metadata_pageview, metadata.url_scrapydweb AS metadata_url_scrapydweb, metadata.url_jobs AS metadata_url_jobs, metadata.url_schedule_task AS metadata_url_schedule_task, metadata.url_delete_task_result AS metadata_url_delete_task_result, metadata.username AS metadata_username, metadata.password AS metadata_password, metadata.scheduler_state AS metadata_scheduler_state, metadata.jobs_per_page AS metadata_jobs_per_page, metadata.tasks_per_page AS metadata_tasks_per_page, metadata.jobs_style AS metadata_jobs_style
FROM metadata
WHERE metadata.version = ?
LIMIT ? OFFSET ?
[2022-12-21 23:31:53,671] INFO in sqlalchemy.engine.Engine: SELECT metadata.id AS metadata_id, metadata.version AS metadata_version, metadata.last_check_update_timestamp AS metadata_last_check_update_timestamp, metadata.main_pid AS metadata_main_pid, metadata.logparser_pid AS metadata_logparser_pid, metadata.poll_pid AS metadata_poll_pid, metadata.pageview AS metadata_pageview, metadata.url_scrapydweb AS metadata_url_scrapydweb, metadata.url_jobs AS metadata_url_jobs, metadata.url_schedule_task AS metadata_url_schedule_task, metadata.url_delete_task_result AS metadata_url_delete_task_result, metadata.username AS metadata_username, metadata.password AS metadata_password, metadata.scheduler_state AS metadata_scheduler_state, metadata.jobs_per_page AS metadata_jobs_per_page, metadata.tasks_per_page AS metadata_tasks_per_page, metadata.jobs_style AS metadata_jobs_style
FROM metadata
WHERE metadata.version = ?
LIMIT ? OFFSET ?
2022-12-21 23:31:53,672 INFO sqlalchemy.engine.Engine [generated in 0.00070s] ('1.4.0', 1, 0)
[2022-12-21 23:31:53,672] INFO in sqlalchemy.engine.Engine: [generated in 0.00070s] ('1.4.0', 1, 0)
2022-12-21 23:31:53,673 INFO sqlalchemy.engine.Engine ROLLBACK
[2022-12-21 23:31:53,673] INFO in sqlalchemy.engine.Engine: ROLLBACK
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: metadata

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/scrapydweb/common.py", line 85, in handle_metadata
metadata = Metadata.query.filter_by(version=version).first()
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2824, in first
return self.limit(1)._iter().first()
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter
result = self.session.execute(
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/orm/session.py", line 1714, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement
ret = self._execute_context(
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context
self.handle_dbapi_exception(
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2124, in handle_dbapi_exception
util.raise(
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 211, in raise
raise exception
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context
self.dialect.do_execute(
File "/home/scrapy/scrapydweb/.venv/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: metadata
[SQL: SELECT metadata.id AS metadata_id, metadata.version AS metadata_version, metadata.last_check_update_timestamp AS metadata_last_check_update_timestamp, metadata.main_pid AS metadata_main_pid, metadata.logparser_pid AS metadata_logparser_pid, metadata.poll_pid AS metadata_poll_pid, metadata.pageview AS metadata_pageview, metadata.url_scrapydweb AS metadata_url_scrapydweb, metadata.url_jobs AS metadata_url_jobs, metadata.url_schedule_task AS metadata_url_schedule_task, metadata.url_delete_task_result AS metadata_url_delete_task_result, metadata.username AS metadata_username, metadata.password AS metadata_password, metadata.scheduler_state AS metadata_scheduler_state, metadata.jobs_per_page AS metadata_jobs_per_page, metadata.tasks_per_page AS metadata_tasks_per_page, metadata.jobs_style AS metadata_jobs_style
FROM metadata
WHERE metadata.version = ?
LIMIT ? OFFSET ?]
[parameters: ('1.4.0', 1, 0)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)`

[eracle]

Which python version are you using?
Check this:
#205

[spechtx]

Thanks eracle!

For anyone interested the solution is here: #205 (comment)

[my8100]

Please have a try for the latest scrapydweb v1.4.1, which supports Python 3.6 to 3.9

[my8100]

Fixed in scrapydweb v1.4.1

[SiMu-EVA]

在 scrapydweb v1.4.1 中修复

并没有，还是会报这个错：
[root@localhost scrapydweb]# scrapydweb
[2023-09-13 17:51:32,029] INFO in apscheduler.scheduler: Scheduler started
[2023-09-13 17:51:32,039] INFO in scrapydweb.run: ScrapydWeb version: 1.4.1
[2023-09-13 17:51:32,039] INFO in scrapydweb.run: Use 'scrapydweb -h' to get help
[2023-09-13 17:51:32,039] INFO in scrapydweb.run: Main pid: 445615
[2023-09-13 17:51:32,039] DEBUG in scrapydweb.run: Loading default settings from /usr/local/lib/python3.9/site-packages/scrapydweb-1.4.1-py3.9.egg/scrapydweb/default_settings.py
Traceback (most recent call last):
File "/usr/local/bin/scrapydweb", line 33, in
sys.exit(load_entry_point('scrapydweb==1.4.1', 'console_scripts', 'scrapydweb')())
File "/usr/local/lib/python3.9/site-packages/scrapydweb-1.4.1-py3.9.egg/scrapydweb/run.py", line 32, in main
app = create_app()
File "/usr/local/lib/python3.9/site-packages/scrapydweb-1.4.1-py3.9.egg/scrapydweb/init.py", line 81, in create_app
handle_db(app)
File "/usr/local/lib/python3.9/site-packages/scrapydweb-1.4.1-py3.9.egg/scrapydweb/init.py", line 123, in handle_db
db.create_all()
File "/usr/local/lib/python3.9/site-packages/flask_sqlalchemy/extension.py", line 900, in create_all
self._call_for_binds(bind_key, "create_all")
File "/usr/local/lib/python3.9/site-packages/flask_sqlalchemy/extension.py", line 871, in _call_for_binds
engine = self.engines[key]
File "/usr/local/lib/python3.9/site-packages/flask_sqlalchemy/extension.py", line 687, in engines
app = current_app._get_current_object() # type: ignore[attr-defined]
File "/usr/local/lib/python3.9/site-packages/werkzeug/local.py", line 508, in _get_current_object
raise RuntimeError(unbound_message) from None
RuntimeError: Working outside of application context.

This typically means that you attempted to use functionality that needed
the current application. To solve this, set up an application context
with app.app_context(). See the documentation for more information.
2023-09-13 17:51:32,057 INFO sqlalchemy.engine.Engine BEGIN (implicit)
[2023-09-13 17:51:32,057] INFO in sqlalchemy.engine.Engine: BEGIN (implicit)
2023-09-13 17:51:32,060 INFO sqlalchemy.engine.Engine SELECT metadata.id AS metadata_id, metadata.version AS metadata_version, metadata.last_check_update_timestamp AS metadata_last_check_update_timestamp, metadata.main_pid AS metadata_main_pid, metadata.logparser_pid AS metadata_logparser_pid, metadata.poll_pid AS metadata_poll_pid, metadata.pageview AS metadata_pageview, metadata.url_scrapydweb AS metadata_url_scrapydweb, metadata.url_jobs AS metadata_url_jobs, metadata.url_schedule_task AS metadata_url_schedule_task, metadata.url_delete_task_result AS metadata_url_delete_task_result, metadata.username AS metadata_username, metadata.password AS metadata_password, metadata.scheduler_state AS metadata_scheduler_state, metadata.jobs_per_page AS metadata_jobs_per_page, metadata.tasks_per_page AS metadata_tasks_per_page, metadata.jobs_style AS metadata_jobs_style
FROM metadata
WHERE metadata.version = ?
LIMIT ? OFFSET ?
[2023-09-13 17:51:32,060] INFO in sqlalchemy.engine.Engine: SELECT metadata.id AS metadata_id, metadata.version AS metadata_version, metadata.last_check_update_timestamp AS metadata_last_check_update_timestamp, metadata.main_pid AS metadata_main_pid, metadata.logparser_pid AS metadata_logparser_pid, metadata.poll_pid AS metadata_poll_pid, metadata.pageview AS metadata_pageview, metadata.url_scrapydweb AS metadata_url_scrapydweb, metadata.url_jobs AS metadata_url_jobs, metadata.url_schedule_task AS metadata_url_schedule_task, metadata.url_delete_task_result AS metadata_url_delete_task_result, metadata.username AS metadata_username, metadata.password AS metadata_password, metadata.scheduler_state AS metadata_scheduler_state, metadata.jobs_per_page AS metadata_jobs_per_page, metadata.tasks_per_page AS metadata_tasks_per_page, metadata.jobs_style AS metadata_jobs_style
FROM metadata
WHERE metadata.version = ?
LIMIT ? OFFSET ?
2023-09-13 17:51:32,060 INFO sqlalchemy.engine.Engine [generated in 0.00030s] ('1.4.1', 1, 0)
[2023-09-13 17:51:32,060] INFO in sqlalchemy.engine.Engine: [generated in 0.00030s] ('1.4.1', 1, 0)
2023-09-13 17:51:32,061 INFO sqlalchemy.engine.Engine ROLLBACK
[2023-09-13 17:51:32,061] INFO in sqlalchemy.engine.Engine: ROLLBACK

[my8100]

Can you show me the result of “pip list”, or create a new virtual env and try again?