An easy, step-by-step guide to deploy Python code to AWS Lambda using Serverless Framework.
-
Make sure Node has been installed before installing Serverless. Otherwise, go to Node official page and follow the instructions on how to install Node.
-
Verify that both Node and npm have been installed on your computer.
node --version
Example Output
v16.15.0
npm --version
Example Output
8.5.5
-
Install Serverless globally
npm install -g serverless
-
Verify that Serverless has been successfully installed
serverless --version
Example Output
Framework Core: 3.0.1 (standalone) Plugin: 6.0.0 SDK: 4.3.0
-
Initialize Serverless Python project
serverless
-
Select any Python template that you are interested in. If you are new to AWS and Serverless, "AWS - Python - Starter" may be a good starting point.
-
Next, name your project (ideally no whitespace between words). Then, Serverless will automatically create a directory with given name and include necessary boilerplate and template files.
-
Select no when you are prompted to login or deploy the project.
-
Change to project directory and start developing your Serverless project.
cd <PROJECT_DIR>
Two of the most common ways to add third-party packages to Serverless Python project.
- pip + virtualenv + serverless-python-requirements
- pipenv + serverless-python-requirements (recommended)
I personally prefer the second option for the following reasons.
- pipenv allows developer to split development and production dependencies (similar to package.json in Node project)
- Similar to package.json, pipenv will store the project configuration in Pipfile. The file allows granular configuration and control, which is apparently not available in requirements.txt file.
- pipenv locks the version of the dependencies and store the details in Pipfile.lock to guarantee consistent, manageable dependencies.
- serverless-python-requirements supports pipenv and will only pick up the production dependencies upon packaging and deployment. Hence, it significantly reduces the size of Lambda functions and layers.
- The dependencies are installed in $HOME/.cache/pip/wheel/ directory, outside project directory. It means your project directory size remains small and is not bloated with dependencies.
However, feel free to choose any option that works best for you.
-
Create a new virtual environment inside project folder, e.g. venv
Command
python -m venv <VIRTUAL_ENVIRONMENT_NAME>
Example
Create a new virtual environment, venv inside project folder
python -m venv .\venv
-
Activate the virtual environment
Command
.\<VIRTUAL_ENVIRONMENT_NAME>\Scripts\activate
Activate virtual environment named venv
Example
.\venv\Scripts\activate
-
Check whether pip is installed
Command
pip -V
-
(Optional) Upgrade pip to the latest version
python -m pip install --upgrade pip
Note
If pip is accidentally removed, or not installed, it can be re-installed using the following command
python -m ensurepip --upgrade
-
Check the list of installed Python packages
pip list
Note
The list should only contain minimum installation, most likely pip and setuptools. If it contains any third-party packages, it indicates that the virtual environment has not been activated.
-
Create requirements.txt file and list the needed third-party packages
Example
requests pyyaml numpy
Note
The file may also contain requirement specifiers if you want to narrow down or select specific version to be installed in the virtual environment.
-
Install these packages by running the following command
pip install -r requirements.txt
Note
-r
flag tells pip to install recursively (in case of listing multi packages in the requirements) -
Check whether these packages are successfully installed
pip list
Example output
If requests package is successfully installed, it will be listed on the output
Package Version ------------------ --------- ... ... requests 2.26.0 ... ...
-
Initialize Node application
npm init
Note
package.json should exist in the project directory if the initialization is successful.
-
Install
serverless-python-requirements
plugin learn morenpm install --save serverless-python-requirements
-
Add these lines to our
serverless.yml
after installation is completeplugins: - serverless-python-requirements custom: pythonRequirements: dockerizePip: true
-
Install pipenv
pip install pipenv
-
Check for pipenv version to ensure the package is successfully installed.
python -m pipenv --version
Example Output
pipenv, version 2022.5.2
-
(Optional) Install pipenv inside virtual environment to enable pipenv command
python -m pipenv install pipenv --dev
Note
As this is the first third-party package installation, pipenv will automatically generate Pipfile and Pipfile.lock. The
--dev
option tells pipenv to install the local pipenv as development dependency. It will be excluded during deployment. -
Activate virtual environment
python -m pipenv shell
-
Check the packages installed inside the virtual environment
pip list
Alternatively, pipenv also comes with a feature to check dependency graph.
pipenv graph
If the virtual environment is configured properly, it should only contain pip, setuptools, pipenv and their dependencies.
-
Install all necessary dependencies.
Production dependency
pipenv install <PACKAGE_NAME>
Development dependency
pipenv install <PACKAGE_NAME> --dev
-
Initialize Node application
npm init
Note
package.json should exist in the project directory if the initialization is successful.
-
Install
serverless-python-requirements
plugin learn morenpm install --save serverless-python-requirements
-
Add the following config to serverless.yml after installation is complete
plugins: - serverless-python-requirements custom: pythonRequirements: dockerizePip: true
If the Lambda functions of Serverless application use the same dependencies, you can take full advantage of Lambda layer to further reduce the function size by packing it as centralized dependencies.
Before
┌─────────────────────┐ ┌─────────────────────┐
│ Function 1 │ │ Function n │
├─────────────────────┤ ├─────────────────────┤
│ Main function 5kB │ │ Main function 10kB │
├─────────────────────┤ ... ├─────────────────────┤
│ Dependencies 2MB │ │ Dependencies 2MB │
└─────────────────────┘ └─────────────────────┘
Total size 2.005MB Total size 2.010MB
After
┌─────────────────────┐ ┌─────────────────────┐
│ Function 1 │ │ Function n │
├─────────────────────┤ ├─────────────────────┤
│ Main function 5kB │ ... │ Main function 10kB │
└─────────────────────┘ └─────────────────────┘
Total size 5kB Total size 10kB
│ │
└─────────────────┬───────────────┘
│
┌─────────────────────┐
| Lambda Layer |
├─────────────────────┤
│ Dependencies 2MB │
└─────────────────────┘
Layer configuration in serverless.yml
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
dockerizePip: true
layer:
name: ${self:provider.stage}-pythonLambdaLayer
description: Python requirements lambda layer
#...
# explicitly filter and select necessary files and directories
package:
individually: true
excludeDevDependencies: true
patterns:
- "!./**" # exclude all files and directories
- "./src/**" # select only 'src' directory and its files and subdirectories
# link lambda function to dependency layer
functions:
myhandler:
handler: src/path/to/my.handler
#..
layers:
- Ref: PythonRequirementsLambdaLayer
Command
serverless deploy [--aws-profile <AWS_PROFILE_NAME>] [--stage <DEPLOYMENT_STAGE>]
Note
Both aws-profile and stage are optional, however, you are expected to have at least 1 profile (default profile) in your machine. Otherwise, Serverless will never know where to deploy nor gain access to your AWS account programmatically.
Below is how Serverless looks for the AWS profile, sorted by priority (high to low):
-
Command line arguments (
--aws-profile <AWS_PROFILE_NAME>
) -
Deployment script (typically serverless.yml)
-
Default AWS profile
- Linux & MacOS:
$HOME/.aws/{credentials, config}
- Windows:
C:\Users\<USER_NAME>\.aws\{credentials, config}
- Linux & MacOS:
To make sure the third-party package has been installed and deployed, check the .serverless
directory and it should have similar content as shown below.
your-function-name
├── .serverless
│ ├── requirements
│ │ ├── ...
│ │ └── ...
│ ├── cloudformation-template-create-stack.json
│ ├── cloudformation-template-update-stack.json
│ ├── pythonRequirements.zip
│ ├── function_1.zip
│ ├── ...
│ ├── function_n.zip
│ ├── requirements.txt
│ └── serverless-state.json
├── node_modules
│ ├── ...
│ └── ...
├── venv # virtual environment config + local dependencies
│ ├── ... # (not exist if configuration uses pipenv)
│ └── ...
├── ...
├── src
│ ├── function_1
│ │ ├── ...
│ │ └── handler.py
│ ├── ...
│ └── function_n
│ ├── ...
│ └── handler.py
├── handler.py
├── ...
├── package.json
├── Pipfile # (not exist if configuration uses pip + venv)
├── Pipfile.lock # (not exist if configuration uses pip + venv)
├── requirements.txt # (not exist if configuration uses pipenv)
└── serverless.yml