From e8f31321aa0307d999b540e98455eb27b20c9995 Mon Sep 17 00:00:00 2001 From: "amaury@reacher.email" Date: Sun, 15 Dec 2024 18:45:32 +0000 Subject: [PATCH 1/4] GITBOOK-16: Commercial License Trial --- docs/SUMMARY.md | 16 ++- .../migrations}/bulk.md | 4 +- .../docker-environment-variables.md | 2 +- .../migrating-from-0.7-to-0.10-beta.md | 10 +- docs/advanced/openapi/v1-bulk.md | 4 - docs/advanced/openapi/v1-check_email.md | 4 - docs/advanced/run-your-own-proxy.md | 5 + docs/self-hosting/debugging-reacher.md | 12 +- docs/self-hosting/install.md | 77 +++++++++---- .../{licensing.md => licensing/README.md} | 4 +- .../licensing/commercial-license-trial.md | 24 ++++ docs/self-hosting/proxies.md | 49 +++++--- .../reacher-configuration-v0.10.md | 106 ++++++++++++------ docs/self-hosting/saas-vs-self-host.md | 24 ++-- docs/self-hosting/scaling-for-production.md | 54 --------- .../scaling-for-production/README.md | 23 ++++ .../option-1-manage-scaling-yourself.md | 54 +++++++++ ...ion-2-rabbitmq-based-queue-architecture.md | 73 ++++++++++++ 18 files changed, 380 insertions(+), 165 deletions(-) rename docs/{self-hosting => advanced/migrations}/bulk.md (97%) rename docs/{self-hosting => advanced/migrations}/docker-environment-variables.md (96%) create mode 100644 docs/advanced/run-your-own-proxy.md rename docs/self-hosting/{licensing.md => licensing/README.md} (74%) create mode 100644 docs/self-hosting/licensing/commercial-license-trial.md rename docs/{advanced/migrations => self-hosting}/reacher-configuration-v0.10.md (58%) delete mode 100644 docs/self-hosting/scaling-for-production.md create mode 100644 docs/self-hosting/scaling-for-production/README.md create mode 100644 docs/self-hosting/scaling-for-production/option-1-manage-scaling-yourself.md create mode 100644 docs/self-hosting/scaling-for-production/option-2-rabbitmq-based-queue-architecture.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 1820935db..03d00806d 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -11,12 +11,14 @@ * [SaaS vs Self-Host](self-hosting/saas-vs-self-host.md) * [Install Reacher in 20min](self-hosting/install.md) -* [Licensing](self-hosting/licensing.md) -* [Scaling for Production](self-hosting/scaling-for-production.md) +* [Scaling for Production](self-hosting/scaling-for-production/README.md) + * [Option 1: Manage scaling yourself](self-hosting/scaling-for-production/option-1-manage-scaling-yourself.md) + * [Option 2: RabbitMQ-based Queue Architecture](self-hosting/scaling-for-production/option-2-rabbitmq-based-queue-architecture.md) +* [Licensing](self-hosting/licensing/README.md) + * [Commercial License Trial](self-hosting/licensing/commercial-license-trial.md) * [Proxies](self-hosting/proxies.md) -* [Bulk Verification (v0.7)](self-hosting/bulk.md) +* [Reacher Configuration](self-hosting/reacher-configuration-v0.10.md) * [Debugging Reacher](self-hosting/debugging-reacher.md) -* [Docker Environment Variables (v0.7)](self-hosting/docker-environment-variables.md) ## Advanced @@ -24,6 +26,8 @@ * [/v0/check\_email](advanced/openapi/v0-check_email.md) * [/v1/check\_email](advanced/openapi/v1-check_email.md) * [/v1/bulk](advanced/openapi/v1-bulk.md) +* [Run your own Proxy](advanced/run-your-own-proxy.md) * [Migrations](advanced/migrations/README.md) - * [Migrating from 0.7 to 0.10 (beta)](advanced/migrations/migrating-from-0.7-to-0.10-beta.md) - * [Reacher Configuration (v0.10)](advanced/migrations/reacher-configuration-v0.10.md) + * [Migrating from 0.7 to 0.10](advanced/migrations/migrating-from-0.7-to-0.10-beta.md) + * [Bulk Verification (v0.7)](advanced/migrations/bulk.md) + * [Docker Environment Variables (v0.7)](advanced/migrations/docker-environment-variables.md) diff --git a/docs/self-hosting/bulk.md b/docs/advanced/migrations/bulk.md similarity index 97% rename from docs/self-hosting/bulk.md rename to docs/advanced/migrations/bulk.md index c8f9f622b..0035e4670 100644 --- a/docs/self-hosting/bulk.md +++ b/docs/advanced/migrations/bulk.md @@ -1,10 +1,10 @@ -# Bulk Verification +# Bulk Verification (v0.7) The default Reacher API only exposes one endpoint, `/v0/check_email`, which allows verifying one email at a time. The optional Bulk Verification API allows you to queue up a list of emails in one go. ## Prerequisites -* A self-hosted setup, see [install.md](install.md "mention"). +* A self-hosted setup, see [install.md](../../self-hosting/install.md "mention"). * A PostgreSQL database, you can start for free with [Supabase](https://supabase.com/) (no affiliation). ## Get Started diff --git a/docs/self-hosting/docker-environment-variables.md b/docs/advanced/migrations/docker-environment-variables.md similarity index 96% rename from docs/self-hosting/docker-environment-variables.md rename to docs/advanced/migrations/docker-environment-variables.md index eaef55b04..7643319c4 100644 --- a/docs/self-hosting/docker-environment-variables.md +++ b/docs/advanced/migrations/docker-environment-variables.md @@ -1,7 +1,7 @@ # Docker Environment Variables (v0.7) {% hint style="info" %} -This page only applies to Reacher version 0.7. For the `beta` v0.10 version, see [reacher-configuration-v0.10.md](../advanced/migrations/reacher-configuration-v0.10.md "mention"). +This page only applies to Reacher version 0.7. For the `beta` v0.10 version, see [reacher-configuration-v0.10.md](../../self-hosting/reacher-configuration-v0.10.md "mention"). {% endhint %} Reacher's software is available on [Docker Hub](https://hub.docker.com/r/reacherhq/backend/tags). You can get started using the default parameters: diff --git a/docs/advanced/migrations/migrating-from-0.7-to-0.10-beta.md b/docs/advanced/migrations/migrating-from-0.7-to-0.10-beta.md index d1ac5c808..2ad406675 100644 --- a/docs/advanced/migrations/migrating-from-0.7-to-0.10-beta.md +++ b/docs/advanced/migrations/migrating-from-0.7-to-0.10-beta.md @@ -1,19 +1,15 @@ -# Migrating from 0.7 to 0.10 (beta) - -{% hint style="info" %} -v0.10 is currently still in `beta` phase. -{% endhint %} +# Migrating from 0.7 to 0.10 Reacher v0.10 introduces the `/v1/*` endpoints, namely: -* `/v1/check_email`: Performs a single email verification while respecting the optional throttle and concurrency settings set in [reacher-configuration-v0.10.md](reacher-configuration-v0.10.md "mention"). +* `/v1/check_email`: Performs a single email verification while respecting the optional throttle and concurrency settings set in [reacher-configuration-v0.10.md](../../self-hosting/reacher-configuration-v0.10.md "mention"). * `/v1/bulk`, `/v1/bulk/{job_id}`, `/v1/bulk/{job_id}/results`: Create a bulk verification job, and query its progress and status. Docs coming soon. The `/v0/check_email` endpoint **DOES NOT** change, neither in API nor in behavior. More specifically, even if you specify throttle and concurrency settings in the newly introduced Reacher Configuration, they will not be taken into account by the `/v0/check_email` endpoint, which will perform email verification as soon as it receives the request. ## Environment Variables -With the introduction of [reacher-configuration-v0.10.md](reacher-configuration-v0.10.md "mention"), some of the Environment Variables have changed names. +With the introduction of [reacher-configuration-v0.10.md](../../self-hosting/reacher-configuration-v0.10.md "mention"), some of the Environment Variables have changed names.
Old nameNew nameDescription
RCH_HTTP_HOSTRCH__HTTP_HOSTThe host name to bind the HTTP server to.
PORTRCH__HTTP_PORTThe port to bind the HTTP server to, often populated by the cloud provider.
RCH_SENTRY_DSNRCH__SENTRY_DSNIf set, bug reports will be sent to this Sentry DSN.
RCH_HEADER_SECRETRCH__HEADER_SECRETIf set, then all HTTP requests must have the x-reacher-secret header set to this value. This is used to protect the backend against public unwanted HTTP requests.
RCH_FROM_EMAILRCH__FROM_EMAILEmail to use in the <MAIL FROM:> SMTP step. Can be overwritten by each API request's from_email field.
RCH_HELLO_NAMERCH__HELLO_NAMEName to use in the <EHLO> SMTP step. Can be overwritten by each API request's hello_name field.
RCH_SMTP_TIMEOUTRCH__SMTP_TIMEOUTTimeout for each SMTP connection.
RCH_WEBDRIVER_ADDRRCH__WEBDRIVER_ADDRSet to a running WebDriver process endpoint (e.g. http://localhost:9515) to use a headless navigator to password recovery pages to check Yahoo and Hotmail/Outlook addresses. We recommend chromedriver as it allows parallel requests.
For Bulk Verification:
RCH_ENABLE_BULKRCH__WORKER__ENABLE
DATABASE_URLRCH__WORKER__POSTGRES__DB_URL[Bulk] Database connection string for storing results and task queue
RCH_DATABASE_MAX_CONNECTIONSRemoved[Bulk] Connections created for the database pool
RCH_MINIMUM_TASK_CONCURRENCYRemoved[Bulk] Minimum number of concurrent running tasks below which more tasks are fetched
RCH_MAXIMUM_CONCURRENT_TASK_FETCHRemoved[Bulk] Maximum number of tasks fetched at once
diff --git a/docs/advanced/openapi/v1-bulk.md b/docs/advanced/openapi/v1-bulk.md index c582a5724..c21b76e87 100644 --- a/docs/advanced/openapi/v1-bulk.md +++ b/docs/advanced/openapi/v1-bulk.md @@ -1,9 +1,5 @@ # /v1/bulk -{% hint style="info" %} -This endpoint is available starting from Reacher v0.10. -{% endhint %} - {% swagger src="https://raw.githubusercontent.com/reacherhq/check-if-email-exists/refs/heads/master/backend/openapi.json" path="/v1/bulk" method="post" %} [https://raw.githubusercontent.com/reacherhq/check-if-email-exists/refs/heads/master/backend/openapi.json](https://raw.githubusercontent.com/reacherhq/check-if-email-exists/refs/heads/master/backend/openapi.json) {% endswagger %} diff --git a/docs/advanced/openapi/v1-check_email.md b/docs/advanced/openapi/v1-check_email.md index 6b98018d1..053c3bca9 100644 --- a/docs/advanced/openapi/v1-check_email.md +++ b/docs/advanced/openapi/v1-check_email.md @@ -1,9 +1,5 @@ # /v1/check\_email -{% hint style="info" %} -This endpoint is available starting from Reacher v0.10. -{% endhint %} - {% swagger src="https://raw.githubusercontent.com/reacherhq/check-if-email-exists/refs/heads/master/backend/openapi.json" path="/v1/check_email" method="post" %} [https://raw.githubusercontent.com/reacherhq/check-if-email-exists/refs/heads/master/backend/openapi.json](https://raw.githubusercontent.com/reacherhq/check-if-email-exists/refs/heads/master/backend/openapi.json) {% endswagger %} diff --git a/docs/advanced/run-your-own-proxy.md b/docs/advanced/run-your-own-proxy.md new file mode 100644 index 000000000..7aa46b3ad --- /dev/null +++ b/docs/advanced/run-your-own-proxy.md @@ -0,0 +1,5 @@ +# Run your own Proxy + +Reacher integrates seamlessy with [proxies.md](../self-hosting/proxies.md "mention"), and we propose some 3rd-party proxies to use. However, you may also choose to run your own Proxy. + +If you're interested in this feature, please contact [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention"). diff --git a/docs/self-hosting/debugging-reacher.md b/docs/self-hosting/debugging-reacher.md index bf58be833..5208edf4b 100644 --- a/docs/self-hosting/debugging-reacher.md +++ b/docs/self-hosting/debugging-reacher.md @@ -1,7 +1,17 @@ # Debugging Reacher +## How to debug Reacher? + +The reflex to have when debugging Reacher is to set the `-e RUST_LOG=debug` flag to Docker. This will show all debug logs useful to pinpoint where the error happened. + +If you still don't understand the error after setting that flag, send an email to [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention") along with those debug logs. + ## How to check if port 25 is open? +{% hint style="info" %} +This only concerns users installing Reacher on a server without using [proxies.md](proxies.md "mention"). +{% endhint %} + When choosing a server to install Reacher on, you need to make sure that the server itself has port `25` open AND the chosen cloud provider allows outbound port `25` connections on its network. To test this, there are two methods: @@ -113,7 +123,7 @@ Here are details about some of the most well-known providers: * :warning: Digital Ocean: Your account needs to be 60d old, then you can [apply](https://www.digitalocean.com/community/questions/how-i-can-open-port-25-please?answer=67100) to open port 25 (Mar 2020). * ❌ GCP: Port 25 closed, [source](https://cloud.google.com/compute/docs/tutorials/sending-mail). * ❌ Heroku: Starting from July 2021, Heroku blocks port 25 intermittently according to [this document](https://help.heroku.com/IR3S6I5X/problem-in-sending-e-mails-through-smtp). -* :warning: Hetzner: Port 25 open according to [unofficial source](https://www.reddit.com/r/hetzner/comments/lb2o13/does\_hetzner\_block\_port\_25/) (Feb 2021), but seems now that you need to request manually. +* :warning: Hetzner: Port 25 open according to [unofficial source](https://www.reddit.com/r/hetzner/comments/lb2o13/does_hetzner_block_port_25/) (Feb 2021), but seems now that you need to request manually. * :warning: Linode: Port 25 closed for new accounts, but can be opened if reverse DNS is set up correctly, [source](https://www.linode.com/docs/guides/running-a-mail-server/#sending-email-on-linode=) (Apr 2022). * :white\_check\_mark: OVH: Port 25 open on new instances, but outbound port 25 traffic is monitored to prevent spam (May 2022). * :warning: Vultr: Create support ticket to open port 25, [source](https://www.vultr.com/docs/what-ports-are-blocked/), though [recent reports](https://github.com/LukeSmithxyz/emailwiz/issues/172) (May 2022) show that they won’t do it anymore. diff --git a/docs/self-hosting/install.md b/docs/self-hosting/install.md index 470d83c0e..69de18a7e 100644 --- a/docs/self-hosting/install.md +++ b/docs/self-hosting/install.md @@ -1,29 +1,43 @@ # Install Reacher in 20min -Reacher is built with self-hosting as a primary feature, giving you full control over how the service operates on your infrastructure. It can be deployed on a single server in under 20 minutes. This tutorial shows you how. +Reacher is designed for seamless self-hosting, giving you full control over its operation on your infrastructure. This guide demonstrates how to install and run Reacher on your local computer in under 20 minutes, using a Dockerfile that is provided you as part of the [commercial-license-trial.md](licensing/commercial-license-trial.md "mention"). -{% hint style="info" %} -You can run this tutorial without a Commercial License, as a Free Trial. Read more about the Free Trial in [licensing.md](licensing.md "mention"). -{% endhint %} +## Prerequisites + +* An account on [https://reacher.email](https://reacher.email) (required for the Commercial License Trial and access to the Dockerfile). +* Docker installed on your system (follow the [Docker installation guide](https://docs.docker.com/get-docker/) for your OS). + +## Tutorial Scope: Install Reacher on your local computer + +Reacher’s stateless architecture enables easy horizontal scaling by deploying multiple containers, each running a Reacher instance for parallel email verifications. However, for simplicity, this tutorial focuses on a local installation. A further section focuses on [scaling-for-production](scaling-for-production/ "mention"). + +The provided Dockerfile includes a pre-configured proxy, resolving the common ISP restriction on outgoing requests to port 25 used by Reacher to perform SMTP verifications. -## Tutorial Scope: Install Reacher on a single server +
-Reacher is **stateless** by design, meaning you can deploy multiple containers, each running a separate instance of Reacher, to perform email verifications in parallel. This architecture enables easy horizontal scaling. +Understand the features and limitations of the Commercial License Trial. -However, for the sake of this tutorial, we will install Reacher on a single dedicated server. This allows minimal setup to get Reacher working, and ensures that the chosen cloud provider allows outgoing port 25 requests. +The Dockerfile provided as part of the Commercial License Trial is designed to enable quick setup for email verifications. Below are its key features and limitations: -If you're interested in ideas for a production deployment setup, skip to [scaling-for-production.md](scaling-for-production.md "mention"). +* **Built-in Proxy Configuration**: we use [**Proxy4Smtp**](https://www.proxy4smtp.com), a 3rd-party proxy with carefully maintained IPs optimized for SMTP verifications. This ensures reliable email verification even in cloud environments with restricted SMTP access. Learn more in [proxies.md](proxies.md "mention"). +* **Daily Verification Limit**: capped at 60 per minute at **10,000 per day**. +* **Usage Tracking**: verification results are anonymized and sent back to Reacher, and used to monitor daily usage and detect potential abuse. + +You can also read more in [commercial-license-trial.md](licensing/commercial-license-trial.md "mention"). + +
## Step-by-Step Tutorial -1. Install Docker on your server. You can follow [Docker's guide](https://docs.docker.com/engine/install/) for your OS. -2. Run Reacher's latest (v0.7) [Docker image](https://hub.docker.com/r/reacherhq/backend): +1. Navigate to the **Commercial License Trial** tab of your Reacher Dashboard ([go there directly](https://app.reacher.email/en/dashboard/commercial_license)). You'll see a command to run Reacher's latest (v0.10) [Docker image](https://hub.docker.com/r/reacherhq/backend): ```bash -docker run -p 8080:8080 reacherhq/backend:latest # v0.7 +docker run -e RCH__COMMERCIAL_LICENSE_TRIAL__API_TOKEN= -p 8080:8080 reacherhq/commercial-license-trial:latest # v0.10 ``` -You should see the following output: +Replace `` with your unique API token shown in the dashboard. + +Expected output: ```bash 2024-09-19T12:58:32.918254Z INFO reacher: Running Reacher version="0.10.0" @@ -34,22 +48,45 @@ ChromeDriver was started successfully. 2024-09-19T12:58:32.976589Z INFO reacher: Server is listening host=0.0.0.0 port=80 ``` -Advanced users can set additional [docker-environment-variables.md](docker-environment-variables.md "mention"). +If you see an error message, such as `` Error: missing field `api_token` ``, double-check the `-e RCH__COMMERCIAL_LICENSE_TRIAL__API_TOKEN` flag you passed. If you see other errors, either try [debugging-reacher.md](debugging-reacher.md "mention") or send an email to [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention"). -3. Make sure that you can verify an email remotely by running the following command from your local machine. +Advanced users can also set additional [reacher-configuration-v0.10.md](reacher-configuration-v0.10.md "mention"). -```bash -curl -X POST \ - -H'Content-Type: application/json' \ +4. Verify an email by running the following command in another terminal. + +
curl -X POST \
+	-H'Content-Type: application/json' \
 	-d'{"to_email":"amaury@reacher.email"}' \
-	http://:8080/v0/check_email
+	http://localhost:8080/v1/check_email
+
+ +Advanced users can pass additional configuration fields to the [v1-check\_email.md](../advanced/openapi/v1-check_email.md "mention") endpoint. + +4. If successful, you'll see JSON object with an `is_reachable` field. + +```json +{ + "input": "amaury@reacher.email", + "is_reachable": "safe", + // --snip-- +} ``` +You can read more about all the fields in [is-reachable.md](../getting-started/is-reachable.md "mention"). + {% hint style="warning" %} -If this step hangs for a long time, or returns a JSON result with `is_reachable="unknown"`, it generally means that port 25 is restricted. See [debugging-reacher.md](debugging-reacher.md "mention")on how to fix this. +If this step hangs for a long time, or returns a JSON result with `is_reachable="unknown"`, see [debugging-reacher.md](debugging-reacher.md "mention")on how to fix this. {% endhint %} -4. If you see a JSON output with an `is_reachable` field, then you're set, congratulations! :tada: +If you go back to check the terminal with the Docker command, you should see corresponding logs: + +```log +// --snip-- +2024-12-15T11:33:36.169891Z INFO reacher: Starting verification email="amaury@reacher.email" +2024-12-15T11:33:45.015130Z INFO reacher: Done verification email="amaury@reacher.email" is_reachable=Safe +``` + +Congratulations! You just successfully verified an email from your computer. Now it's time to think about [scaling-for-production](scaling-for-production/ "mention"). ## Troubleshooting diff --git a/docs/self-hosting/licensing.md b/docs/self-hosting/licensing/README.md similarity index 74% rename from docs/self-hosting/licensing.md rename to docs/self-hosting/licensing/README.md index 594963af2..e88bf53f7 100644 --- a/docs/self-hosting/licensing.md +++ b/docs/self-hosting/licensing/README.md @@ -16,7 +16,7 @@ However, companies **cannot use** the open-source version if they plan to integr The Commercial License allows companies to integrate Reacher into their proprietary software or services without being bound by the open-source license terms (such as making their own code publicly available). By obtaining a Commercial License, businesses can use Reacher with full control over how the software is implemented and distributed within their products. -Check Reacher's [pricing page](https://reacher.email/pricing) to buy a Commercial License. +Check Reacher's [pricing page](https://reacher.email/pricing) to buy a Commercial License, or read the [full terms of the Commercial License](https://github.com/reacherhq/policies/blob/master/license/commercial.en.md). {% hint style="info" %} Both the open-source and Commercial License give you access to the same Reacher software. @@ -24,4 +24,4 @@ Both the open-source and Commercial License give you access to the same Reacher ## Does Reacher offer a Free Trial? -Yes, Reacher offers a **free trial** that allows you to use the self-hosted software for a limited time. During this period, you can test Reacher’s features for **internal testing** and **non-commercial purposes**. However, the trial is not intended for full-scale production use or commercial applications. To begin your trial, please contact [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention"), and you can then follow the guide to [install.md](install.md "mention"). +Yes, Reacher offers a free trial for the Commercial License, see [commercial-license-trial.md](commercial-license-trial.md "mention"). diff --git a/docs/self-hosting/licensing/commercial-license-trial.md b/docs/self-hosting/licensing/commercial-license-trial.md new file mode 100644 index 000000000..82bb27882 --- /dev/null +++ b/docs/self-hosting/licensing/commercial-license-trial.md @@ -0,0 +1,24 @@ +# Commercial License Trial + +The Commercial License Trial allows you to test the self-hosted software for a limited period of time, for **internal testing** and **non-commercial purposes**. + +## Features and Limitations + +As part of the Commercial License Trial, you'll receive a Dockerfile designed to enable quick setup for email verifications. Below are its key features and limitations: + +* **Built-in Proxy Configuration**: we embed [**Proxy4Smtp**](https://www.proxy4smtp.com), a 3rd-party proxy with carefully maintained IPs optimized for SMTP verifications. All verifications using the Dockerfile go through this proxy. This ensures reliable email verification even in cloud environments with restricted SMTP access. Learn more in [proxies.md](../proxies.md "mention"). +* **Daily Verification Limit**: capped at **60 per minute** and **10,000 per day.** +* **Usage Tracking**: verification results are anonymized and sent back to Reacher, and used to monitor daily usage and detect potential abuse. +* **For testing purposes only**. The Dockerfile can only be used internally, for testing purposes, and can in no case be used in production environments for commercial applications. + +{% hint style="danger" %} +All abuse of the Commercial License Trial will result in an immediate account ban. +{% endhint %} + +## Get Started + +To start your Commercial License Trial, sign up on [https://reacher.email](https://app.reacher.email/en/signup). Then, navigate to the "Commercial License Trial" tab of the Reacher Dashboard ([go there directly](https://app.reacher.email/en/dashboard/commercial_license)) and follow instructions. + +## After the Trial + +The Commercial License Trial cannot be used for full-scale production use or commercial applications. Once you've tested Reacher extensively, purchase a Commercial License to gain access to an unrestricted Dockerfile. diff --git a/docs/self-hosting/proxies.md b/docs/self-hosting/proxies.md index f044bea5d..cd8333e0c 100644 --- a/docs/self-hosting/proxies.md +++ b/docs/self-hosting/proxies.md @@ -4,28 +4,45 @@ Maintaining a good IP reputation is hard. Reacher integrates seamlessly with SOC ## What is a SOCKS5 Proxy? -A **SOCKS5 proxy** is a flexible proxy protocol that supports various types of traffic, including SMTP. When using it for email verifications, the reputation of the **proxy’s IP** is what matters, not your own IP. The proxy handles requests on your behalf, which helps protect your actual IP while ensuring verifications go through successfully. This is crucial for maintaining deliverability and avoiding issues like blacklisting. +A **SOCKS5 proxy** is a flexible proxy protocol that supports various types of traffic, including SMTP. When using it for email verifications, the reputation of the **proxy’s IP** is what matters, not your own IP. This is crucial for maintaining deliverability and avoiding issues like blacklisting. -{% hint style="warning" %} +Choosing a reputable 3rd-party proxy will greatly improve the quality of your email verification results. For a list of recommended proxies, see [#which-3rd-party-proxies-does-reacher-recommend](proxies.md#which-3rd-party-proxies-does-reacher-recommend "mention") + +{% hint style="info" %} SMTP email verifications are not possible via a traditional HTTP proxy. {% endhint %} -## Verify an Email Using a Proxy +## Setting up a Proxy -Reacher's API allows making an email verification using a proxy. Send a request with the following fields: +Once you've purchased a proxy, run the Docker command (see how in [install.md](install.md "mention")) and pass the following flags: -```json -{ - "to_email": "someone@gmail.com", - "proxy": { // (optional) SOCK5 proxy to run the verification through, default is empty - "host": "my-proxy.io", - "port": 1080, - "username": "username", // (optional) Proxy username - "password": "password" // (optional) Proxy password - } -} -``` +* `-e RCH__PROXY__HOST=`: The IP or hostname of the proxy server. +* `-e RCH__PROXY__PORT=`: The corresponding port. +* `-e RCH__PROXY__USERNAME=`: Optional. A username to authenticate the proxy. +* `-e RCH__PROXY__PASSWORD=`: Optional. The corresponding password. + +{% hint style="info" %} +If you're using the [commercial-license-trial.md](licensing/commercial-license-trial.md "mention"), these fields are already populated with the built-in proxy. However, you can still pass these flags manually pointing to a proxy of your own choosing, and they will overwrite the built-in settings. +{% endhint %} ## Which 3rd-party proxies does Reacher recommend? -Get in touch with [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention") if you are interested in using a proxy. +Reacher has been working closely with [Proxy4Smtp](https://www.proxy4smtp.com) since early 2024. The service is run by Jon, an email verification expert who has had over 10 years experience in the industry. He understands the complications and challenges that arise from large scale SMTP connections, both with B2C and B2B emails. His proxies integrate seamlessly with Reacher. + +Ask [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention")for an introduction to Jon. + +## Configure Reacher to respect Proxy limits + +If using proxies from [Proxy4Smtp](https://www.proxy4smtp.com/), we highly recommend respecting the following concurrency and throttling limits, to avoid getting the IPs that you purchased blacklisted: + +* **10,000 email verifications per IP per day**. You may buy as much IPs as you wish, but the total number of email verifications should respect this limit to avoid getting blacklisted. +* **60 email verifications per IP per minute**. Usage spikes are easily detectable by email providers, which will proceed to flag these IPs, so we highly recommend to smoothen out the email verification stream. +* **5 concurrent verifications per IP at any time**. Given that each email verification takes in average a handful of seconds, this setting should work well with the above setting of 60 verifications per minute. + +To estimate how many IPs you should buy, we recommend starting from your expected monthly volume, and calculate the number of IPs you need from there. For example, if you're aiming for 10 millions verifications per month, we recommend buying 33 or 34 IPs: + +``` +10,000,000 emails per month / 30 = 33,000 emails per day / 10000 = 33 IPs +``` + +The throttling limits of 10000 per day and 60 per minute are already baked in as defaults into Reacher. However, if you're managing scaling yourself ([option-1-manage-scaling-yourself.md](scaling-for-production/option-1-manage-scaling-yourself.md "mention")), make sure that at any time you have as many Reacher instances as IPs you bought. This will make sure that the concurrency of 5 is respected. diff --git a/docs/advanced/migrations/reacher-configuration-v0.10.md b/docs/self-hosting/reacher-configuration-v0.10.md similarity index 58% rename from docs/advanced/migrations/reacher-configuration-v0.10.md rename to docs/self-hosting/reacher-configuration-v0.10.md index 010a1bfc9..f4689c77f 100644 --- a/docs/advanced/migrations/reacher-configuration-v0.10.md +++ b/docs/self-hosting/reacher-configuration-v0.10.md @@ -1,12 +1,12 @@ -# Reacher Configuration (v0.10) +# Reacher Configuration {% hint style="info" %} -This configuration has been introduced in v0.10, which is still in `beta`. For the stable 0.7 version, please see [docker-environment-variables.md](../../self-hosting/docker-environment-variables.md "mention"). +This configuration is for the current 0.10 version. For the older 0.7 version, please see [docker-environment-variables.md](../advanced/migrations/docker-environment-variables.md "mention"). {% endhint %} -Previously, in v0.7, configuration was done solely via environment variables. Given the growing amount of configurable parameters, we now offer a file-based configuration too, on top of environment variables. +You can find below the exhaustive list of configurable parameters to optimize Reacher. -You can find below the exhaustive list of configurations, as well as their corresponding environment variable. +To tweak a configuration, look at the "Env variable" name in the comments, and pass in the `-e ENV_VAR=VALUE` flag to Docker. See [#examples-with-docker](reacher-configuration-v0.10.md#examples-with-docker "mention"). ```toml # Backend configuration. @@ -55,6 +55,11 @@ webdriver_addr = "http://localhost:9515" # Env variable: RCH__SMTP_TIMEOUT # smtp_timeout = 45 +# Optional Sentry DSN. If set, all errors will be sent to Sentry. +# +# Env variable: RCH__SENTRY_DSN +# sentry_dsn = "" + # Uncomment the lines below to route all SMTP verification requests # through a specified proxy. Note that the proxy must be a SOCKS5 proxy to work # with the SMTP protocol. This proxy will not be used for headless @@ -94,12 +99,47 @@ hotmailb2c = "headless" # recommended. yahoo = "headless" +# Throttle the maximum number of requests per second, per minute, per hour, and +# per day for this worker. +# All fields are optional; comment them out to disable the limit. +# +# We however recommend setting the throttle for at least the per-minute and +# per-day limits to prevent the IPs from being blocked by the email providers. +# The default values are set to 60 requests per minute and 10,000 requests per +# day. +# +# Important: these throttle configurations only apply to /v1/* endpoints, and +# not to the previous /v0/check_email endpoint. The latter endpoint always +# executes the verification immediately, regardless of the throttle settings. +# +# Env variables: +# - RCH__THROTTLE__MAX_REQUESTS_PER_SECOND +# - RCH__THROTTLE__MAX_REQUESTS_PER_MINUTE +# - RCH__THROTTLE__MAX_REQUESTS_PER_HOUR +# - RCH__THROTTLE__MAX_REQUESTS_PER_DAY +[throttle] +# max_requests_per_second = 20 +max_requests_per_minute = 60 +# max_requests_per_hour = 1000 +max_requests_per_day = 10000 + +# Configuration for a queue-based architecture for Reacher. This feature is +# currently in **beta**. The queue-based architecture allows Reacher to scale +# horizontally by running multiple workers that consume emails from a RabbitMQ +# queue. +# +# To enable the queue-based architecture, set the "enable" field to "true" and +# configure the RabbitMQ connection below. The "concurrency" field specifies +# the number of concurrent emails to verify for this worker. +# +# For more information, see the documentation at: +# https://docs.reacher.email/self-hosting/scaling-for-production [worker] # Enable the worker to consume emails from the RabbitMQ queues. If set, the # RabbitMQ configuration below must be set as well. # # Env variable: RCH__WORKER__ENABLE -enable = true +enable = false # RabbitMQ configuration. [worker.rabbitmq] @@ -111,48 +151,42 @@ url = "amqp://guest:guest@localhost:5672" # Env variable: RCH__WORKER__RABBITMQ__CONCURRENCY concurrency = 5 -# Throttle the maximum number of requests per second, per minute, per hour, and -# per day for this worker. -# All fields are optional; comment them out to disable the limit. -# -# Important: these throttle configurations only apply to /v1/* endpoints, and -# not to the previous /v0/check_email endpoint. The latter endpoint always -# executes the verification immediately, regardless of the throttle settings. +# Below are the configurations for the storage of the email verification +# results. We currently support the following storage backends: +# - Postgres # -# Env variables: -# - RCH__WORKER__THROTTLE__MAX_REQUESTS_PER_SECOND -# - RCH__WORKER__THROTTLE__MAX_REQUESTS_PER_MINUTE -# - RCH__WORKER__THROTTLE__MAX_REQUESTS_PER_HOUR -# - RCH__WORKER__THROTTLE__MAX_REQUESTS_PER_DAY -[worker.throttle] -# max_requests_per_second = 20 -# max_requests_per_minute = 100 -# max_requests_per_hour = 1000 -# max_requests_per_day = 20000 +# Uncomment the following line to configure the storage to use Postgres. +# [storage.postgres] -# Postgres configuration. Currently, a Postgres database is required to store -# the results of the verifications. This might change in the future, allowing -# for pluggable storage. -[worker.postgres] -# Env variable: RCH__WORKER__POSTGRES__DB_URL -db_url = "postgresql://localhost/reacherdb" - -# Optional Sentry DSN. If set, all errors will be sent to Sentry. +# # URL to connect to the Postgres database. # -# Env variable: RCH__SENTRY_DSN -# sentry_dsn = "" - +# Env variable: RCH__STORAGE__POSTGRES__DB_URL +# db_url = "postgresql://localhost/reacherdb" +# +# If you wish to store additional data along with the verification results, +# you can add a JSON object to the "extra" field. This object will be stored +# as a JSONB column in the database. This is for example useful to track who +# initiated the verification request in a multi-tenant system. +# +# Env variable: RCH__STORAGE__0__POSTGRES__TABLE_NAME +# extra = { "my_custom_key" = "my_custom_value" } ``` -## Usage with Docker +## Examples with Docker -You can continue using environment variables with Docker. For example, to overwrite the EHLO/HELO name, simply run: +To overwrite the EHLO/HELO name: ```bash docker run -e RCH__HELLO_NAME=my.company.com -p 8080:8080 reacherhq/backend:beta ``` -However, if you prefer to pass in a local `backend_config.toml` file instead, run: +To store all email verification results to a Postgres database: + +```bash +docker run -e RCH__STORAGE__POSTGRES__DB_URL="postgres://user:pass@mydomain.mycompany.com/my_db_name" -p 8080:8080 reacherhq/backend:beta +``` + +For advanced users, if you prefer to pass in the full [`backend_config.toml`](../../backend/backend_config.toml) file instead of individual environment variable flags, run: ```bash docker run -e RUST_LOG=reacher=debug -v /path/to/local/backend_config.toml:./backend_config.toml -p 8080:8080 reacherhq/backend:beta diff --git a/docs/self-hosting/saas-vs-self-host.md b/docs/self-hosting/saas-vs-self-host.md index 0486eef05..7e9ee3a13 100644 --- a/docs/self-hosting/saas-vs-self-host.md +++ b/docs/self-hosting/saas-vs-self-host.md @@ -1,26 +1,26 @@ # SaaS vs Self-Host -When using Reacher, you have two options: using the **SaaS version** provided by Reacher or **self-hosting** the service on your own infrastructure. Each approach has unique advantages depending on your specific needs, technical resources, and how much control you want over the service. +When using Reacher, you can choose between two options: the SaaS version hosted by Reacher or self-hosting the service on your infrastructure. Each option offers distinct advantages depending on your needs, technical resources, and desired level of control. ## What is Reacher SaaS? -Reacher SaaS is the cloud-hosted version of the service, which is available at [https://app.reacher.email](https://app.reacher.email), where everything is managed for you. You can verify emails directly via the Reacher Dashboard without needing to worry about infrastructure, updates, or scaling. +Reacher SaaS is the cloud-hosted version of the service, accessible at [https://app.reacher.email](https://app.reacher.email). It handles all infrastructure, updates, and scaling for you. You can verify emails directly via the Reacher Dashboard without managing technical details. ## What is Self-Hosting Reacher? -Reacher is also designed for self-hosting, allowing you to run the service on your own servers. This option gives you full control over the environment and data while using the same powerful verification engine that Reacher SaaS provides. +Reacher is also available for self-hosting, enabling you to run the service on your own servers. This approach provides full control over the environment and data while leveraging the same verification engine as Reacher SaaS. Reacher is designed to make self-hosting straightforward and efficient. {% hint style="success" %} -Reacher's goal is to make Self-Hosting easy. You can [install.md](install.md "mention"). +Reacher's goal is to make Self-Hosting easy. You can [install.md](install.md "mention")as part of your **Commercial License Trial**. {% endhint %} ## Key Differences -| Feature | SaaS | Self-Hosting | -| --------------------- | -------------------------------------- | -------------------------------------------------------------------------------- | -| **Volume** | Limited to 10k verifications per month | Unlimited verifications | -| **Cost** | Monthly subscription | Monthly subscription + your own server costs. Amortized costs for large volumes. | -| **Setup Time** | Instant | Requires installation and setup | -| **Maintenance** | Fully managed by Reacher | Managed by your IT team | -| **Data Ownership** | Data stored on Reacher servers | Full ownership, no data is sent to Reacher | -| **Bulk Verification** | No | Yes, see [bulk.md](bulk.md "mention") | +| Feature | SaaS | Self-Hosting | +| --------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------- | +| **Volume** | Limited to 10k verifications per month | Unlimited verifications | +| **Cost** | Monthly subscription | Monthly subscription + server costs (lower at scale) + [proxy](proxies.md) costs (optional) | +| **Setup Time** | Instant | Requires installation and setup | +| **Maintenance** | Fully managed by Reacher | Managed by your IT team | +| **Data Ownership** | Data stored on Reacher servers | Full ownership, no data is sent to Reacher | +| **Bulk Verification** | Not supported | Supported, see [bulk.md](../advanced/migrations/bulk.md "mention") | diff --git a/docs/self-hosting/scaling-for-production.md b/docs/self-hosting/scaling-for-production.md deleted file mode 100644 index 1b7712408..000000000 --- a/docs/self-hosting/scaling-for-production.md +++ /dev/null @@ -1,54 +0,0 @@ -# Scaling for Production - -{% hint style="info" %} -The architecture detalied below is currently only available on the v0.10 `beta` Docker version. -{% endhint %} - -Reacher's stateless design allows for efficient horizontal scaling. We propose here a queue-based architecture to handle more than 10 millions of email verifications per month. - -The architecture contains 4 components: - -
ComponentDescriptionDocker image
HTTP serverReceives incoming email verification requests, and post them into the queue.reacherhq/backend:beta
RabbitMQReacher uses a reliable, mature and open-source queue implementation.rabbitmq:4.0-management
WorkersOne or more consumers of the queue, which perform the actual email verification task.reacherhq/backend:beta
StorageA place to store all results, currently only PostgresDB is supported.postgres:14
- -Note that Reacher provides the same Docker image `reacherhq/backend` which can act as both a **Worker** and a **HTTP server**. - -

Reacher queue architecture

- -With this architecture, it's possible to horizontally scale the number of workers. However, to prevent spawning to many workers at once resulting in blacklisted IPs, we need to configure some concurrency and throttling parameters below. - -### Worker Configuration - -To enable the above worker architecture without getting blacklisted, we need to set some parameters in [reacher-configuration-v0.10.md](../advanced/migrations/reacher-configuration-v0.10.md "mention"): - -* `worker.enable`: true -* `worker.rabbitmq.url`: Points to the URL of the RabbitMQ instance. -* `worker.postgres.db_url`: A Postgres database to store the email verification results. - -Since spawning workers (generally on cloud providers) doesn't guarantee a reputable IP assigned to the worker, we propose to configure all workers to use a proxy. Proxies generally offer a pricing per IP per month; we recommend buying one IP for each 10000 email verifications you do per day. - -* `worker.proxy.{host,port}`: Set a proxy to route all SMTP requests through. You can optionally pass in `username` and `password` if required. - -We also propose some recommended values for concurrency and throttling parameters. These parameters ensure that the proxy that we use will have its IP well maintained. - -* `worker.rabbitmq.concurrency`: 5. Each worker can process 5 emails at a time. -* `worker.throttle.max_requests_per_minute`: 60. If this value is too high, the recipient SMTP server might see sudden spikes of email verifications, resulting in an IP blacklist. -* `worker.throttle.max_requests_per_day`: 10000. This is the recommended number of verifications per IP per day. Assuming our proxy has `N` IP addresses and `N` workers, each worker will perform 10000 verifications per day in average. - -You can scale up the number `N` as much as you need, by buying more IPs and spawning more workers. Remember, the rule of thumb is 10000 verifications per IP per day. For example, if you're aiming for 10 millions verifications per month, we recommend buying 33 or 34 IPs: - -``` -10,000,000 emails per month / 30 = 33,000 emails per day / 10000 = 33 IPs -``` - -Refer to [reacher-configuration-v0.10.md](../advanced/migrations/reacher-configuration-v0.10.md "mention")to see how to set these settings. - -## Understanding the architecture with Docker Compose - -We do not recommend using Docker Compose for a high-volume production setup. However, for understanding the architecture, the different Docker images, as well as how to configure the workers, this [`docker_compose.yaml`](../../docker-compose.yaml) file can be useful. - -## More questions? - -Contact [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention")if you have more questions about this architecture, such as: - -* deploying on Kubernetes (Ansible playbook, Pulumi) -* more specialized workers (e.g. some workers doing headless verification only, others doing SMTP only) diff --git a/docs/self-hosting/scaling-for-production/README.md b/docs/self-hosting/scaling-for-production/README.md new file mode 100644 index 000000000..1585f9290 --- /dev/null +++ b/docs/self-hosting/scaling-for-production/README.md @@ -0,0 +1,23 @@ +# Scaling for Production + +Reacher's stateless architecture enables efficient horizontal scaling, allowing companies to tailor deployments to their specific needs. Below are the scaling options and pathways for a production setup. + +1. [option-1-manage-scaling-yourself.md](option-1-manage-scaling-yourself.md "mention"): Leverage Reacher's statelessness and decide yourself how to deploy Reacher. +2. [option-2-rabbitmq-based-queue-architecture.md](option-2-rabbitmq-based-queue-architecture.md "mention"): Reacher includes a pre-integrated queue system based on [**RabbitMQ**](https://rabbitmq.com), enabling efficient task management and scaling. + +Alternative scaling solutions were also explored, chat with [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention") if you want to discuss more. + +## Scaling beyond the Commercial License Trial + +The documentation in this section mentions the Dockerfile provided as part of the [commercial-license-trial.md](../licensing/commercial-license-trial.md "mention"), which: + +* has a built-in proxy, +* limits the number of daily verifications to 10000. + +The strategies documented in this section apply both to the Commercial License Trial as well as a high-volume setup beyond this limit. Once you're ready to transition from the former to the latter, you must: + +1. **Purchase a Commercial License**. This will grant you access to an unrestricted Dockerfile. +2. **Purchase 3rd-party proxy IPs**. You will receive configuration details for the proxy to be passed into the unrestricted Dockerfile. + +Get in touch with [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention") when you're there. + diff --git a/docs/self-hosting/scaling-for-production/option-1-manage-scaling-yourself.md b/docs/self-hosting/scaling-for-production/option-1-manage-scaling-yourself.md new file mode 100644 index 000000000..4d0dc3e42 --- /dev/null +++ b/docs/self-hosting/scaling-for-production/option-1-manage-scaling-yourself.md @@ -0,0 +1,54 @@ +# Option 1: Manage scaling yourself + +Reacher is stateless by design. This means that you can spawn instances of Reacher concurrently, and they don't need coordinate to perform email verifications. + +However, in most cases, some coordination is desired. For example, to avoid IPs being blacklisted, we might want to limit to the number of spawned instances at any time (concurrency) or during a given period of time (throttling). See the last section in [proxies.md](../proxies.md "mention") for some actual numbers for these settings. + +The best scaling solution for you is the one that fits into your architecture. Below are several scaling solutions, including both traditional and advanced approaches. They are often not mutually exclusive. + +#### 1. Install Reacher on dedicated servers (proxies optional). + +* **Description**: Install Reacher on dedicated servers. This setup is straightforward and allows you to manage IP quality directly. +* **Pros**: + * Complete control over IP maintenance (if desired). + * Independence from 3rd-party proxies. +* **Cons**: + * Limited scalability compared to cloud-based solutions. + * Higher initial infrastructure cost. + +#### 2. Pure serverless architecture. + +* **Description**: Deploy Reacher as a stateless function using serverless platforms like AWS Lambda or Google Cloud Functions. +* **Pros**: + * Infinite horizontal scaling. + * No infrastructure management. +* **Cons**: + * Complexity in managing rate-limiting and proxy usage (see those limits in [proxies.md](../proxies.md "mention")). + * Higher cost for large-scale, constant usage. + +#### 3. **Amazon SQS with Worker Instances.** + +* **Description**: Use [Amazon SQS](https://aws.amazon.com/sqs/) (or similar message queues) to distribute email verification tasks to worker instances running Reacher. +* **Architecture**: + * Task requests are queued in SQS (or another broker). + * Reacher workers (Docker containers or serverless functions) poll the queue to process tasks. +* **Pros**: + * Efficient task distribution. + * Built-in concurrency control via SQS throttling. +* **Cons**: + * Requires managing worker instances. + +This solution is explor more in details in [option-2-rabbitmq-based-queue-architecture.md](option-2-rabbitmq-based-queue-architecture.md "mention"), using [RabbitMQ](https://rabbitmq.com) instead of Amazon SQS. + +#### **4. Kubernetes (k8s) Clusters** + +* **Description**: Run Reacher instances as pods within a Kubernetes cluster for high availability and scalability. +* **Architecture**: + * Use Kubernetes' Horizontal Pod Autoscaler (HPA) to scale Reacher pods based on task load. + * Integrate with RabbitMQ for task queuing and distribution (see [option-2-rabbitmq-based-queue-architecture.md](option-2-rabbitmq-based-queue-architecture.md "mention")) +* **Pros**: + * Highly scalable and resilient. + * Centralized monitoring and orchestration. +* **Cons**: + * Higher operational complexity. + * Requires Kubernetes expertise. diff --git a/docs/self-hosting/scaling-for-production/option-2-rabbitmq-based-queue-architecture.md b/docs/self-hosting/scaling-for-production/option-2-rabbitmq-based-queue-architecture.md new file mode 100644 index 000000000..74e1c5c72 --- /dev/null +++ b/docs/self-hosting/scaling-for-production/option-2-rabbitmq-based-queue-architecture.md @@ -0,0 +1,73 @@ +# Option 2: RabbitMQ-based Queue Architecture + +Reacher includes an optional, opinionated queue-based architecture designed to scale efficiently and handle high email verification volumes. This architecture comprises 4 main components and is highly configurable to meet specific business needs. + +
ComponentDescriptionDocker image
HTTP serverAccepts incoming email verification requests, and post them into the queue.reacherhq/commercial-license-trial
RabbitMQReacher uses a reliable, mature and open-source queue implementation.rabbitmq:4.0-management
WorkersOne or more consumers of the queue, which perform the actual email verification task.reacherhq/commercial-license-trial
StorageA place to store all results, currently only PostgresDB is supported.postgres:14
+ +Note that Reacher provides the same Docker image `reacherhq/commercial-license-trial` which can act as both a **Worker** and a **HTTP server**. + +

Reacher queue architecture

+ +With this architecture, it's possible to horizontally scale the number of workers. However, to prevent spawning too many workers at once resulting in blacklisted IPs, we need to configure some concurrency and throttling parameters below. + +### Worker Configuration + +#### **Enabling the Architecture** + +To enable the worker-based architecture, configure the following parameters in your deployment: + +* `worker.enable`: Set to `true` to activate the worker role. +* `worker.rabbitmq.url`: URL of the RabbitMQ instance for task queuing. +* `postgres.db_url`: URL of a PostgreSQL database to store verification results. + +#### Using Proxies for Workers + +Since spawning workers on cloud providers doesn't guarantee a reputable IP assigned to the worker, we configure all workers to use a proxy. + +* `proxy.{host,port}`: Set a proxy to route all SMTP requests through. You can optionally pass in `username` and `password` if required. + +{% hint style="info" %} +The Dockerfile provided in the Commercial License Trial already has these parameters set up. +{% endhint %} + +**Proxy Recommendation**: + +* Use one proxy IP per 10,000 email verifications per day. +* Purchase additional proxy IPs as needed to scale. +* Spawn as many workers as IPs you purchased. + +#### Concurrency and Throttling Parameters + +To prevent IP blacklisting, configure the following parameters: + +* **Concurrency**: + * `worker.rabbitmq.concurrency`: Set to `5`. Each worker processes up to 5 emails concurrently, which means each proxy IP address is perform maximum 5 email verifications at any give time. +* **Throttling**: + * `throttle.max_requests_per_minute`: Set to `60`. Limits request spikes to prevent SMTP server flags. + * `throttle.max_requests_per_day`: Set to `10,000`. Aligns with the recommended verifications per proxy IP. + +{% hint style="info" %} +The Dockerfile provided in the Commercial License Trial already has these parameters set up. +{% endhint %} + +#### Scaling Example + +For scaling, buy additional proxy IPs and spawn more workers accordingly. The "rule of thumb" is **10,000 verifications per IP per day**. + +For example, if your target is 10 million verifications per month: + +1. Daily volume = 10,000,000 emails / 30 days = 333,000 emails/day. +2. Number of required IPs = 333,000 emails/day / 10,000 emails/IP = **33 or 34 IPs**. + +We suggest deploying one Reacher instance per purchased IP. The settings outlined above ensure fair email verification usage as perceived by external email providers. For advanced applications, these values can be adjusted to optimize performance. + +## Understanding the architecture with Docker Compose + +We do not recommend using Docker Compose for a high-volume production setup. However, for understanding or learning the architecture, this [`docker_compose.yaml`](../../../docker-compose.yaml) file can be useful. + +## More questions? + +Contact [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention")if you have more questions about this architecture, such as: + +* deploying on Kubernetes (Ansible playbook, Pulumi) +* more specialized workers (e.g. some workers doing headless verification only, others doing SMTP only) From 9cbd54df0334e5d102f64d4b8d662f1631827715 Mon Sep 17 00:00:00 2001 From: "amaury@reacher.email" Date: Sun, 15 Dec 2024 18:53:10 +0000 Subject: [PATCH 2/4] GITBOOK-17: Proxies wording changes --- docs/self-hosting/proxies.md | 6 ++++-- docs/self-hosting/saas-vs-self-host.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/self-hosting/proxies.md b/docs/self-hosting/proxies.md index cd8333e0c..44b05fe66 100644 --- a/docs/self-hosting/proxies.md +++ b/docs/self-hosting/proxies.md @@ -20,16 +20,18 @@ Once you've purchased a proxy, run the Docker command (see how in [install.md](i * `-e RCH__PROXY__PORT=`: The corresponding port. * `-e RCH__PROXY__USERNAME=`: Optional. A username to authenticate the proxy. * `-e RCH__PROXY__PASSWORD=`: Optional. The corresponding password. +* `-e RCH__HELLO_NAME`: The identifier to use during the "HELO/EHLO" step. It should match a domain name owned by the proxy. Ask your proxy about this setting. +* `-e RCH__MAIL_FROM`: The email to use during the "MAIL FROM" step. It should be an email from the same domain as the HELLO\_NAME. Ask your proxy provider about this setting. {% hint style="info" %} -If you're using the [commercial-license-trial.md](licensing/commercial-license-trial.md "mention"), these fields are already populated with the built-in proxy. However, you can still pass these flags manually pointing to a proxy of your own choosing, and they will overwrite the built-in settings. +If you're using the [commercial-license-trial.md](licensing/commercial-license-trial.md "mention"), these fields are already populated with the built-in proxy. However, you can overwrite them by passing these flags again, pointing to a proxy of your own choosing. {% endhint %} ## Which 3rd-party proxies does Reacher recommend? Reacher has been working closely with [Proxy4Smtp](https://www.proxy4smtp.com) since early 2024. The service is run by Jon, an email verification expert who has had over 10 years experience in the industry. He understands the complications and challenges that arise from large scale SMTP connections, both with B2C and B2B emails. His proxies integrate seamlessly with Reacher. -Ask [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention")for an introduction to Jon. +Ask [amaury@reacher.email](https://app.gitbook.com/u/F1LnsqPFtfUEGlcILLswbbp5cgk2 "mention") for an introduction to Jon. ## Configure Reacher to respect Proxy limits diff --git a/docs/self-hosting/saas-vs-self-host.md b/docs/self-hosting/saas-vs-self-host.md index 7e9ee3a13..44a7a7c8d 100644 --- a/docs/self-hosting/saas-vs-self-host.md +++ b/docs/self-hosting/saas-vs-self-host.md @@ -11,7 +11,7 @@ Reacher SaaS is the cloud-hosted version of the service, accessible at [https:// Reacher is also available for self-hosting, enabling you to run the service on your own servers. This approach provides full control over the environment and data while leveraging the same verification engine as Reacher SaaS. Reacher is designed to make self-hosting straightforward and efficient. {% hint style="success" %} -Reacher's goal is to make Self-Hosting easy. You can [install.md](install.md "mention")as part of your **Commercial License Trial**. +Reacher's goal is to make Self-Hosting easy. You can [install.md](install.md "mention") as part of your **Commercial License Trial**. {% endhint %} ## Key Differences From abe10dea9a56a3d26f1b7752ed87917739f9000b Mon Sep 17 00:00:00 2001 From: "amaury@reacher.email" Date: Sun, 15 Dec 2024 18:55:51 +0000 Subject: [PATCH 3/4] GITBOOK-18: Proxies page typos --- docs/self-hosting/proxies.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/self-hosting/proxies.md b/docs/self-hosting/proxies.md index 44b05fe66..186663d17 100644 --- a/docs/self-hosting/proxies.md +++ b/docs/self-hosting/proxies.md @@ -20,8 +20,8 @@ Once you've purchased a proxy, run the Docker command (see how in [install.md](i * `-e RCH__PROXY__PORT=`: The corresponding port. * `-e RCH__PROXY__USERNAME=`: Optional. A username to authenticate the proxy. * `-e RCH__PROXY__PASSWORD=`: Optional. The corresponding password. -* `-e RCH__HELLO_NAME`: The identifier to use during the "HELO/EHLO" step. It should match a domain name owned by the proxy. Ask your proxy about this setting. -* `-e RCH__MAIL_FROM`: The email to use during the "MAIL FROM" step. It should be an email from the same domain as the HELLO\_NAME. Ask your proxy provider about this setting. +* `-e RCH__HELLO_NAME=`: The identifier to use during the "HELO/EHLO" step. It should match a domain name owned by the proxy. Ask your proxy provider about this setting. +* `-e` RCH\_\_FROM\_EMAIL`=`: The email to use during the "MAIL FROM" step. It should be an email from the same domain as the HELLO\_NAME. Ask your proxy provider about this setting. {% hint style="info" %} If you're using the [commercial-license-trial.md](licensing/commercial-license-trial.md "mention"), these fields are already populated with the built-in proxy. However, you can overwrite them by passing these flags again, pointing to a proxy of your own choosing. From eddd268fc5c37c14aee7938d924b73ce27c4d233 Mon Sep 17 00:00:00 2001 From: "amaury@reacher.email" Date: Sun, 15 Dec 2024 18:57:00 +0000 Subject: [PATCH 4/4] GITBOOK-19: Proxies formatting fix --- docs/self-hosting/proxies.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/self-hosting/proxies.md b/docs/self-hosting/proxies.md index 186663d17..4bbbc5983 100644 --- a/docs/self-hosting/proxies.md +++ b/docs/self-hosting/proxies.md @@ -21,7 +21,7 @@ Once you've purchased a proxy, run the Docker command (see how in [install.md](i * `-e RCH__PROXY__USERNAME=`: Optional. A username to authenticate the proxy. * `-e RCH__PROXY__PASSWORD=`: Optional. The corresponding password. * `-e RCH__HELLO_NAME=`: The identifier to use during the "HELO/EHLO" step. It should match a domain name owned by the proxy. Ask your proxy provider about this setting. -* `-e` RCH\_\_FROM\_EMAIL`=`: The email to use during the "MAIL FROM" step. It should be an email from the same domain as the HELLO\_NAME. Ask your proxy provider about this setting. +* `-e RCH__FROM_EMAIL=`: The email to use during the "MAIL FROM" step. It should be an email from the same domain as the HELLO\_NAME. Ask your proxy provider about this setting. {% hint style="info" %} If you're using the [commercial-license-trial.md](licensing/commercial-license-trial.md "mention"), these fields are already populated with the built-in proxy. However, you can overwrite them by passing these flags again, pointing to a proxy of your own choosing.