rovers is a service to retrieve repository URLs from multiple repository hosting providers and store the URLs and metadata to a PostgreSQL database while issuing messages to a RabbitMQ queue so you can connect other processes to it.
Get the latest rovers image:
docker pull srcd/roversInstall docker-compose.
Start Rabbit and Postgres
docker-compose -d up rovers-postgres rovers-rabbitmqExport as environment variables the API keys (see section). Then, you can execute rovers:
docker-compose up --no-deps roversIf you need to run just some of the available supported-providers, you can use this command replacing the flag --provider with those providers you want to use:
docker-compose run --rm --no-deps --service-ports rovers /bin/sh -c "rovers initdb; rovers repos --provider=github --provider=bitbucket"After that, rovers will generate a lot of 'mentions' (git repositories found on the internet), and sending them to the 'rovers' queue in Rabbit.
To stop and remove all the containers running docker-compose down.
Finally, you can use Borges to fetch the content of these repositories.
All the supported providers are used by default. In case you need to run only some of them you must use the --provider flag:
rovers repos --provider=github --provider=bitbucketTo be able to fetch github and cgit repositories, you should create several API keys:
- Get Github token: https://github.com/settings/tokens
- Get Bing token (Bing is the search engine used to fetch cgit repositories from internet): https://azure.microsoft.com/en-us/pricing/details/cognitive-services/search-api/web/
Uses the GitHub API to get new repositories. Requires a GitHub API token, which will only need repository read access. You can set the token through the environment variable:
$ export CONFIG_GITHUB_TOKEN=github-tokenUses the Bitbucket API to get new repositories as an anonymous user.
Note: the first time this provider runs it may take a while to find git repositories, as the vast majority of repositories at the beginning of the API results are mercurial repositories.
cgit is a popular service to create your own self-hosted Git repository provider. Rovers uses Bing search to discover cgit instances online and tracks them to get new repositories. Requires a Bing API key. You can set the key through the environment variable:
$ export CONFIG_BING_KEY=bing-api-keygo get -u github.com/src-d/rovers/...
Run rovers --help to get help about the supported commands and their options.
The commands initdb, replay and repos also have their own --help switch
for specific options related to them.
To initialize the database schemas. You need to run this command only once.
rovers initdb
To start collecting repository URLs
rovers repos --provider=github
You can configure rovers using environment variables:
Providers:
CONFIG_GITHUB_TOKENto set the github api key.CONFIG_BING_KEYto set the cgit api key.
Broker:
CONFIG_BROKER_URLto set the broker url, by defaultamqp://guest:guest@localhost:5672
Database:
CONFIG_DBUSER: database username, by default if not settestingCONFIG_DBPASS: database user password, by default if not settestingCONFIG_DBHOST: database host, by default if not set0.0.0.0CONFIG_DBPORT: database port, by default if not set5432CONFIG_DBNAME: database name, by default if not settestingCONFIG_DBSSLMODE: ssl mode to use, by default if not setdisableCONFIG_DBTIMEOUT: connection timeout, by default if not set30sCONFIG_DBAPPNAME: application name
rm Makefile.main; rm -rf .cito make sure you will have the last Makefile changes.make dependenciesto download vendor dependencies.make packagesto generate binaries for several platforms.
You will find the built binaries under ./build.
This service uses PostgreSQL and RabbitMQ.
To execute tests locally you need to run RabbitMQ and PostgreSQL.
docker-compose up -d rovers-postgres rovers-rabbitmq
make testYou can use the official Helm chart to deploy Rovers in your Kubernetes cluster.