No module named 'docutils'

[tbbottle]

When trying to deploy a django application that depends on the docutils package, the following error is raised by the Django application:

Request Method: | GET
-- | --
https://....execute-api.us-east-1.amazonaws.com/dev/
3.0.1
ModuleNotFoundError
No module named 'docutils'
/tmp/.../readme_renderer/rst.py in <module>, line 18
/var/lang/bin/python3.8
3.8.6
['/tmp/expi',  '/var/task',  '/opt/python/lib/python3.8/site-packages',  '/opt/python',  '/var/runtime',  '/var/lang/lib/python38.zip',  '/var/lang/lib/python3.8',  '/var/lang/lib/python3.8/lib-dynload',  '/var/lang/lib/python3.8/site-packages',  '/opt/python/lib/python3.8/site-packages',  '/opt/python',  '/var/task']

Context

I did my best to figure out why: rebuilt my virtual environments from scratch, updated my package versions, moved from Python 3.6 to Python 3.8. The issue persisted: whenever I opened the zip file the package was missing.

I tried copying the docutils directory from my virtual environment to the root of my project, so it would be in the root of the zip. It still wouldn't show up.

Eventually I went looking in the Zappa source:

(.env) $ find .env/lib/python3.8/site-packages/zappa/ -type f -name "*.py" | xargs grep docu                                                          
.env/lib/python3.8/site-packages/zappa//core.py:    '*.hg', 'pip', 'docutils*', 'setuputils*', '__pycache__/*'

Which leads to core.py:

  197 # We never need to include these.
  198 # Related: https://github.com/Miserlou/Zappa/pull/56
  199 # Related: https://github.com/Miserlou/Zappa/pull/581
  200 ZIP_EXCLUDES = [
  201     '*.exe', '*.DS_Store', '*.Python', '*.git', '.git/*', '*.zip', '*.tar.gz',
  202     '*.hg', 'pip', 'docutils*', 'setuputils*', '__pycache__/*'
  203 ]

Thanks for providing context: reviewing the pull requests it looks like docutils should exist in the lambda environment without being included in the zappa package. However, it doesn't appear to be in my lambda environment.

I am not clear is this is me mis-configuring things, choosing bad versions, or that something has changed and the package is in fact no longer available.

All the environment setup has been handled by Zappa. Other than creating an execution policy and IAM user, zappa has done it all.

This is the third project I am deploying with zappa and it is awesome - it has saved me much time and many headaches while also helping me learn about AWS in a more structured way than I would have otherwise. Thank you.

Expected Behavior

Python applications should be able to depend on, and use, the docutils package in a zappa deployed lambda environment.

Actual Behavior

The docutils module cannot be found in the zappa created lambda environment.

Possible Fix

I have been trying to find a list from Amazon of the packages included by default in the lambda environment but haven't been successful. If docutils is not included (i.e. it isn't just me), removing the "docutils*" entry from ZIP_EXCLUDES solves the problem. I have confirmed this in my environment.

If it is just me, it would be nice if these excludes were documented somewhere other than the pull request (which wasn't hitting my web searches); I thought I was going crazy with docutils not appearing in the package.

Steps to Reproduce

1. Create a new function that import docutils and returns success.
2. Deploy the function using zappa deploy dev.
3. Visit the endpoint (or look at the touch results) - > result 500

Your Environment

• Zappa version used: zappa --version 0.52.0
• Operating System and Python version: Mac OSX 10.13.6 - Python 3.8.5
• The output of pip freeze:

alabaster==0.7.12
argcomplete==1.12.2
asgiref==3.2.3
Babel==2.8.0
bleach==3.1.0
boto3==1.16.63
botocore==1.19.63
ccy==1.1.0
certifi==2019.11.28
cffi==1.14.0
cfn-flip==1.2.3
chardet==3.0.4
click==7.1.2
cmarkgfm==0.4.2
Django==3.0.1
django-crispy-forms==1.8.1
django-extensions==2.2.5
django-storages==1.8
djangorestframework==3.11.0
docutils==0.16
durationpy==0.5
future==0.18.2
geoip2==3.0.0
gunicorn==20.0.4
hjson==3.0.2
idna==2.8
imagesize==1.2.0
importlib-metadata==3.4.0
Jinja2==2.10.3
jmespath==0.9.4
kappa==0.6.0
MarkupSafe==1.1.1
maxminddb==1.5.2
packaging==19.2
pip-tools==5.5.0
placebo==0.9.0
psycopg2-binary==2.8.6
pycparser==2.19
Pygments==2.5.2
pyparsing==2.4.6
python-dateutil==2.8.1
python-slugify==4.0.1
pytz==2019.3
PyYAML==5.4.1
readme-renderer==28.0
requests==2.22.0
s3transfer==0.3.4
six==1.13.0
snowballstemmer==2.0.0
Sphinx==2.3.1
sphinx-rtd-theme==0.4.3
sphinxcontrib-applehelp==1.0.1
sphinxcontrib-devhelp==1.0.1
sphinxcontrib-htmlhelp==1.0.2
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.2
sphinxcontrib-serializinghtml==1.1.3
sqlparse==0.3.0
stop-words==2018.7.23
text-unidecode==1.3
toml==0.10.2
tqdm==4.56.0
troposphere==2.6.3
typing-extensions==3.7.4.3
urllib3==1.25.7
webencodings==0.5.1
Werkzeug==0.16.1
wsgi-request-logger==0.4.6
zappa==0.52.0
zipp==3.4.0

• Link to your project (optional): if you need me to, I can create an empty project that depends on docutils and publish it - don't hesitate to ask.
• Your zappa_settings.json:

{
    "dev": {
        "aws_region": "us-east-1",
        "aws_environment_variables": {
            "RDS_HOST": "project.place.us-east-1.rds.amazonaws.com",
            "RDS_USER": "project",
            "RDS_NAME": "project",
        },
        "django_settings": "project.settings",
        "exclude": [".env", ".git", "__pycache__", "db.sqlite3", "*.zip", "*.tar.gz"],
        "include": [],
        "profile_name": "zappa-project",
        "project_name": "project",
        "runtime": "python3.8",
        "s3_bucket": "zappa-project",
        "slim_handler": false,
        "delete_local_zip": false,
        "vpc": {
            "SubnetIds": ["subnet-aaa"],
            "SecurityGroupIds": ["sg-aaa"]
        }
    }
}