Sicarator

Instant Setup & Best Quality for Data Projects!

Sicarator is a CLI generator for Data Science projects. It is built with Yeoman and maintained by Sicara.

To know more about the story behind Sicarator and its philosophy, you can read this blog article.

Main components of the generated project

• Python development environment with:
  • Poetry for dependency management
  • Pytest for testing
  • Ruff for static analysis & code formatting
  • Mypy for type checking
  • Pre-commit for git hooks
• Continuous Integration of your choice:
  • CircleCI
  • Github Actions
  • Gitlab CI/CD
  • Azure Pipelines
• (optional) API:
  • built with FastAPI
  • containerized with Docker
  • deployed on one of the following infrastructures:
    • AWS auto-scaled infrastructure (API Gateway, ASG, ECS, EC2)
    • GCP serverless infrastructure (Cloud Run, Artifact Registry)
    • more to come...
  • provisioned with Terraform
  • testable with a Postman collection
• (optional) Data Versioning and Pipelines with DVC and Typer
• (optional) Data Visualization with Streamlit
• (optional) Experiment Tracking with DVC + Streamlit

Requirements

Pyenv

Install pyenv to manage your Python versions and virtual environments:

curl -sSL https://pyenv.run | bash

If you are on MacOS and experiencing errors on Python install with PyEnv, follow this comment

Poetry

Install Poetry to manage your dependencies and tooling configs:

curl -sSL https://install.python-poetry.org | python - --version 1.7.0

If you have not previously installed any Python version, you may need to install one and set it as your default Python version before installing Poetry:

pyenv update  # fetch the latest list of Python versions
pyenv install 3.11  # install the latest Python 3.11.x
pyenv global 3.11  # set your new global default Python version

Node.js

Install a stable version of Node.js (v18.x.x) if you don't have one. The Sicarator is indeed a Yeoman generator, which is a Node.js module.

For linux:

curl -fsSL https://deb.nodesource.com/setup_18.x | sudo -E bash -
sudo apt-get install -y nodejs

Yeoman

Install Yeoman:

npm install -g yo

Usage

• Before generating a new project, make sure to install last version of Sicarator by re-running:

  npm install -g sicara/sicarator

• Generate a new project with:

  yo sicarator

Troubleshooting

I don't have the correct permissions to install yeoman

Example error:

npm ERR! code EACCES
npm ERR! syscall rename
npm ERR! path /usr/lib/node_modules/generator-sicarator
npm ERR! dest /usr/lib/node_modules/.generator-sicarator-cELCsz5l
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, rename '/usr/lib/node_modules/generator-sicarator' -> '/usr/lib/node_modules/.generator-sicarator-cELCsz5l'
...

Solution: give yourself the ownership of npm and node_modules :

sudo chown -R $(whoami) ~/.npm
sudo chown -R $(whoami) /usr/lib/node_modules # or /usr/local/lib/node_modules depending on where node modules are installed

I have the correct git credentials, but installation is throwing a git error

Example error:

git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Solution: you are probably using sudo to run the installation, which doesn't use the same git credentials. See solution above to install the project with your current user profile.

Contribute to the project

Install the project

• Install the project with:

  git clone git@github.com:sicara/sicarator.git

  and install dependencies

  npm install

• To be able to run current dev version of the project, you can use the following command in your project directory:

  npm link

  • Next time you run yo sicarator, dev version will be used.
  • If you want to go back to the production version you can re-run:

    npm install -g sicara/sicarator

Dev tools

• For debugging, you can run:

  npx --node-options="--inspect" yo sicarator

• To run the linter (eslint), you can use:

  npm run lint

• To run the tests, you can use:

  npm test