Gmail AutoResponder

Gmail AutoResponder is a full-stack web application for automated email processing.

Table of Contents

• 1. App Architecture
  • 1.1. Backend – Core: Google Apps Script
    • 1.1.1. App Settings
    • 1.1.2. Execution
    • 1.1.3. Logging
  • 1.2. Backend – Middleware: Django
  • 1.3. Frontend
• 2. Setup
  • 2.1. Provision
  • 2.2. Configure
  • 2.3. Deploy
• 3. Background
• 4. License

1. App Architecture

1.1. Backend – Core: Google Apps Script

The Core App is a Google Apps Script app deployed as an API executable, and managed through the Apps Script API using the Python client library.

1.1.1. App Settings

App settings are managed as Apps Script User Properties, and so are all stored as strings in key-value pairs.
Properties types here are only indicative, and often refer to their corresponding Django Forms Field classes in the Middleware App:

Property	Description
IS_GSUITE_USER	Boolean
enableApp	Boolean. (default: 'false')
coreAppEditUrl	String. Edit URL of the Apps Script project.
filters	JSON string; Message content filters. default: { "rawContent": [ 'report-type=disposition-notification', // Read receipts 'Content-Type: multipart/report', // Automatic delivery reports 'report-type=delivery-status', // Automatic delivery reports 'Content-Type: message/delivery-status' // Automatic delivery reports ], "from": [ '(^|<)((mailer-daemon|postmaster)@.)', 'noreply|no-reply|do-not-reply', '.+@.\bgoogle\.com', Session.getActiveUser().getEmail() ], "to": [ 'undisclosed-recipients' // Potential spams ] }
logger	JSON string. identifiers property of an AppLogger class (a child class of BaseLogger) instance. Example: { "id": "ABCDEF", "viewUri": "https://www.xxxx.yy/ABCDEF?view", "updateUri": "https://www.xxxx.yy/ABCDEF?update" }
timeinterval	Integer. (default: 10)
starthour	Integer. (default: 17)
finishhour	Integer. (default: 8)
utcoffset	Integer. (default: 0)
ccemailadr	String, one or a RFC-compliant comma-separated list of email addresses. default: ''
bccemailadr	String, one or a RFC-compliant comma-separated list of email addresses. default: ''
noreply	Boolean; whether or not to reply with a noreply@ email address. default: 0 if IS_GSUITE_USER === 'true', 2 otherwise.
msgbody	String; Response message body in HTML format. default: getDefaultMessageBody() function return value.
testEmail	String. (default: null or '') If set to a valid email address, enableApp, starthour and finishhour will be ignored, and a test message to that address will be sent in response to any received “non-filterable-out” email. Deleting the property or setting it to '' (or any other non-valid email value) will switch the application from its “Test Mode”.

1.1.2. Execution

When the Core App is enabled (i.e. enableApp === 'true'), main() function of the main script main.js will be continuously executed (triggered) on a recurring interval of timeinterval (+2) minutes from (starthour + utcoffset) to (finishhour + utcoffset).
On each execution, referred to as Execution n, the function issues a Gmail search query to fetch last received messages. The search query returns an array of Gmail threads that were updated in the last timeinterval (+2) minutes. The last message of each of these threads is extracted and processed, i.e. it would either be responded to or skipped if it matches one of the exclusion filters in the filters property.
A Session is a series of triggered executions within a 24 hours span (e.g. from 05-Sep-2018 @7:00pm to 06-Sep-2018 @7:00am).

In order to neither miss a message nor send an automated response more than once :

1. Although Execution (n-1) would normally have occurred timeinterval minutes ago, received messages are fetched from the last (timeinterval + 2) minutes, in order not to miss any messages in case of a delay.
2. IDs of processed messages from Execution (n-1) are cached with a (timeinterval + 6) minutes timeout. During Execution n, IDs of retrieved messages are checked against this cache to determine whether they were already processed or not.

1.1.3. Logging

On each execution, the following information are logged to the AppLogger class instance, which is constructed from an identifiers JSON object generated by parsing the logger Script property:

PROCESSED	EXECUTIONS
Metadata of both messages responded to and those skipped (filtered out) : Label (REP_SENT or SKIPPED), Date/time Sent (Original Message), Date/time Sent (Response) (when applicable), Message ID, Thread ID, From, Subject, Applied filter (if the message was skipped).	Gmail search query. Execution time. Number of threads returned (i.e. number of search results).

The default logger is a Google Spreadsheet, but it is fairly easy to extend the BaseLogger class to create other logging targets as long as the following points are considered :

• The database has a REST API.
• Possibility to connect with Read/Write permissions to an existing instance (database), or create a new one if required.
• Possibility to create two data collections: PROCESSED and EXECUTIONS (e.g.: GSpreadsheets: sheets, SQL: tables, DocumentDB: collections), each with the specified data fields (e.g.: GSpreadsheets: columns/header, SQL: columns, DocumentDB: fields).
• Logs would then be stored as single entries or 2D datasets (e.g.: GSpreadsheets: rows, SQL: records, DocumentDB: documents).

1.2. Backend – Middleware: Django

The Middleware backend component requires the Core app ID, and the Client ID credentials file (credentials.json) of the associated GCP project.

It is a Django app providing the following functionalities:

• Authentication: User sign-in through a full OAuth2 authentication flow.
• Sessions: Based on Django Sessions, allow users (identified by their parsed OIDC tokens) to access the webapp independently from any active Google account in the browser. No user data is kept after logout.
• API Gateway: Aggregates calls to the Core app by providing API endpoints to initialize, retrieve and update app settings. This enables the development of client-side apps, which typically use the OAuth2 consent flow for installed applications and store both access and refresh tokens. The Middleware app here acts as a proxy, by checking HTTP Authorization headers for bearer tokens and translating requests and responses between the custom client and the Core app.
• Data validation: Form data (App settings) validation and multi-level error handling: HTTP request, Django Forms API data validation, Apps Script API and Core App errors.
• Handling App URLs: mapping between features, URLs and views: Home page, Login, Authentication (OAuth2 authorized redirect URI), API Gateway endpoint URLs, Getting/Updating App settings, App reset and Logout.

1.3. Frontend

Django templates + {CSS framework}.

The Frontend part is basically a Django template providing access to all needed features: Logged-in user information on top of a form to view and update App Settings, along with Logout and App Enable/Disable/Reset commands.

2. Setup

• Notes:
  • This section is still being actively worked on. The whole process logic is currently drafted as a Bash script you can check here along with a sample output. You can also check this asciicast.
  • Had it not been for clasp and gcloud command line tools limitations/design choices, most (if not all) of the tasks would have been fully automatable.

2.1. Provision

Basically, all we need is a Google account and a Linux host (later referred to as the “Target Server”) with a well configured ‘deploy’ user, and optionally a fixed public IP address (if internet exposure is needed).

Install software requirements and configure development, testing and deployment environments:

• Development machine: Git, Python3, a modern and updated web browser, text editor/IDE, SSH client, virtualization software (optional, but recommended), and—when required—remote access (typically through a web interface) to a CI/CD server.

• Control Machine or Deployment Server: A dedicated host with SSH service installed and configured; typically with a configuration management software like Ansible, and—if need be—a CI/CD software or service, like Jenkins and GitHub Actions.

• Target Server: Install dependencies and make required configurations:

  • Install system-wide software requirements, generally: git, curl, jq, clasp, nginx, certbot, python3 and uwsgi.
  • Setup a Google Cloud Platform (GCP) Project:
    • Note the Project Number
    • Enable required Google APIs: Apps Script API, Google Drive API, Gmail API, Google Sheets API.
    • Configure OAuth consent screen:

      • Add scopes:

        openid
        https://www.googleapis.com/auth/script.scriptapp
        https://mail.google.com/
        https://www.googleapis.com/auth/drive
        https://www.googleapis.com/auth/userinfo.email
        https://www.googleapis.com/auth/spreadsheets
        https://www.googleapis.com/auth/userinfo.profile
        

      • Create a OAuth2 Client ID for a web application. Set redirect URIs, depending on the deployment stage. Get credentials JSON file.

  • Configure Google Apps Script:
    • Enable Google Apps Script API (from the Apps Script dashboard).
    • Authorize clasp command line tool.
    • Create a new Apps Script project to be deployed as an API Executable.
    • Switch Apps Script's Google Cloud Project association from the default project to the standard (user-managed) project already created.

2.2. Configure

Install app dependencies and make a base deployment of the backend:

• Push and deploy Core app code (/app/core content) to the newly created Apps Script project.
• Create and configure a Django app:
  • Install Python3 requirements (/app/middleware/requirements.txt) in a virtual environment.
  • Apply all migrations (python manage.py migrate)
• Configure NGINX, uWSGI and Certbot (for Testing and Production deployments).

2.3. Deploy

Continuous Deployment (CD): versioned for the Core and incremental for the Middleware.

There are three deployment stages:

Parameters	💻 Development	🧪 Testing	🏭 Production
User [1]	Apps Script project owner	Any allowed Google user	Any allowed Google user
devMode [2]	True	False	False
Versioned deployment [3]	No (use SCRIPT_ID)	Yes (use DEPLOYMENT_ID)	Yes (use DEPLOYMENT_ID)
HTTP Server [4]	Django Development Server	NGinx + uWSGI + certbot (specifically for public cloud deployments)	NGinx + uWSGI + certbot
Test Mode [5]	–	Yes	No
Hosts [6]	localhost, 127.0.0.1	- LAN: hostnames or private IP addresses of external hosts. - Internet: [sub]domain names or public IP addresses, ideally with HTTP Basic Auth.	[sub]domain name or public IP address
AppLogger [7]	Class name and identifiers. default: GSpreadsheetLogger	Class name and identifiers. default: GSpreadsheetLogger	Class name and identifiers. default: GSpreadsheetLogger
Debug & Log [8]	Yes	OK	No DEBUG mode

• ^[1] User: The Google account the Apps Script (Core) app is run as. This is set through the value of the access key of the API executable configuration field (executionApi) in Apps Script project manifest file appscript.json.

• ^[2] devMode: Boolean value of the HTTP Request body field devMode, of the Apps Script API method scripts.run. False implies a versioned deployment, while True lets the Core app run at the latest version of the Apps Script project code.
  (Defined in script_run_parameters.py)

• ^[3] Versioned deployment: whether to create a versioned deployment of the Apps Script (Core) app. In that case, a Deployment ID is used as the Core app ID (defined in script_run_parameters.py with the name CORE_APP_ID), instead of the Script ID.

• ^[4] HTTP Server: HTTP server used to run the Django project (Middleware app): either the Django built-in development server, or the {NGinx + uWSGI + certbot} software suite to provide HTTP server, Reverse proxy and HTTPS functionalities.

• ^[5] Test Mode: The app is set to “Test Mode” by calling the initSettings() function with a valid test email address parameter, e.g. initSettings(true, 'testadress@mydomain.com'), which would set the Apps Script user property testEmail to testadress@mydomain.com.

• ^[6] Hosts: All IP addresses or/and hostnames/FQDNs of the GCP project's OAuth2 authorized redirect URIs, that should also be added to Django app's (Backend Middlware) ALLOWED_HOSTS list in project/settings.py.

• ^[7] AppLogger: AppLogger class name along with its identifiers, to log Executions and Processed Messages.

• ^[8] Debug & Log: Whether to use Django's debugging and logging capabilities.

3. Background

I started Gmail AutoResponder back in 2017 as a script to manage automatic email responses outside the active hours of the company I worked for.
Although it was possible to set Gmail to individually send canned responses, I could neither make time-specific filters nor programmatically make Gmail trigger an event upon email reception. So, inspired by an answer on one of StackExchange forums, I had to figure out a way around and ultimately ended up with a basic Apps Script app, 6 instances of which have amazingly run for almost 3 years and processed more than 17k messages!

To see how the project progressed, check worklog.md.

4. License

This software is under the MIT license.