You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have multiple environments for Scrapydweb (dev, test, prod, etc). Currently in our CI/CD pipeline between the environments we have to upload the egg files manually onto each environment, then we have to add timer tasks also manually. This quickly becomes very tedious and prone to human errors as we have more and more spiders and timer tasks running.
Cloning the entire database is also not a solution for us because there are times when we want to roll out a tested project onto production without rolling out the others.
Therefore, I would like a feature from the UI where we can export the database with filters on the projects, spiders, and timer tasks in addition to an import feature. Then our CI/CD pipeline will be something like:
Deploy projects on Dev -> Execute jobs to test them out -> Export projects on Dev -> Import projects on Test -> Create timer tasks for testing -> Export projects and timer tasks on Test -> Import both on Prod.
I think this would make working between different environments much easier to manage and greatly reduce the possibility of human errors.
The text was updated successfully, but these errors were encountered:
We have multiple environments for Scrapydweb (dev, test, prod, etc). Currently in our CI/CD pipeline between the environments we have to upload the egg files manually onto each environment, then we have to add timer tasks also manually. This quickly becomes very tedious and prone to human errors as we have more and more spiders and timer tasks running.
Cloning the entire database is also not a solution for us because there are times when we want to roll out a tested project onto production without rolling out the others.
Therefore, I would like a feature from the UI where we can export the database with filters on the projects, spiders, and timer tasks in addition to an import feature. Then our CI/CD pipeline will be something like:
Deploy projects on Dev -> Execute jobs to test them out -> Export projects on Dev -> Import projects on Test -> Create timer tasks for testing -> Export projects and timer tasks on Test -> Import both on Prod.
I think this would make working between different environments much easier to manage and greatly reduce the possibility of human errors.
The text was updated successfully, but these errors were encountered: