Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: import and export database with filters #189

Open
mastersang opened this issue Jul 27, 2021 · 0 comments
Open

Feature request: import and export database with filters #189

mastersang opened this issue Jul 27, 2021 · 0 comments

Comments

@mastersang
Copy link

We have multiple environments for Scrapydweb (dev, test, prod, etc). Currently in our CI/CD pipeline between the environments we have to upload the egg files manually onto each environment, then we have to add timer tasks also manually. This quickly becomes very tedious and prone to human errors as we have more and more spiders and timer tasks running.

Cloning the entire database is also not a solution for us because there are times when we want to roll out a tested project onto production without rolling out the others.

Therefore, I would like a feature from the UI where we can export the database with filters on the projects, spiders, and timer tasks in addition to an import feature. Then our CI/CD pipeline will be something like:
Deploy projects on Dev -> Execute jobs to test them out -> Export projects on Dev -> Import projects on Test -> Create timer tasks for testing -> Export projects and timer tasks on Test -> Import both on Prod.

I think this would make working between different environments much easier to manage and greatly reduce the possibility of human errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant