Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: sam local cannot find Docker even when it is running #5646

Closed
V4G4X opened this issue Jul 29, 2023 · 9 comments
Closed

Bug: sam local cannot find Docker even when it is running #5646

V4G4X opened this issue Jul 29, 2023 · 9 comments
Labels
area/docker blocked/close-if-inactive Blocked for >14 days with no response, will be closed if still inactive after 7 days question

Comments

@V4G4X
Copy link

V4G4X commented Jul 29, 2023

Found this issue at many places but they were all closed and I was unable to open them.
Nor did the workaround there work for me, i.e sudo ln -sf "$HOME/.docker/run/docker.sock" /var/run/docker.sock

Description:

On sam local invoke/start-api SAM complains that Docker is not running. Where as it is.
docker ps works, and docker/getting-started is actually running on the side.

I am using Colima as opposed to Docker Desktop, but this issue existed even back then. I moved to colima for this reason.

Steps to reproduce:

  1. colima start --arch x86_64
  2. docker ps and verify that it's working.
  3. sam init and get a basic stack going.
  4. sam build
  5. sam local start-api
  6. Get the docker error: Error: Running AWS SAM projects locally requires Docker. Have you got it installed and running?

Observed result:

2023-07-29 16:19:21,052 | No config file found in this directory.
2023-07-29 16:19:21,054 | OSError occurred while reading TOML file: [Errno 2] No such file or directory: '/Users/varungawande/playground/goLambdaDebug/sunflowers/samconfig.toml'
2023-07-29 16:19:21,055 | Config file location: /Users/varungawande/playground/goLambdaDebug/sunflowers/samconfig.toml
2023-07-29 16:19:21,055 | Config file '/Users/varungawande/playground/goLambdaDebug/sunflowers/samconfig.toml' does not exist
2023-07-29 16:19:21,058 | Using SAM Template at /Users/varungawande/playground/goLambdaDebug/sunflowers/.aws-sam/build/template.yaml
2023-07-29 16:19:21,071 | Using config file: samconfig.toml, config environment: default
2023-07-29 16:19:21,072 | Expand command line arguments to:
2023-07-29 16:19:21,072 | --template_file=/Users/varungawande/playground/goLambdaDebug/sunflowers/.aws-sam/build/template.yaml --host=127.0.0.1 --port=3000 --static_dir=public --layer_cache_basedir=/Users/varungawande/.aws-sam/layers-pkg
--container_host=localhost --container_host_interface=127.0.0.1
2023-07-29 16:19:21,129 | local start-api command is called
2023-07-29 16:19:21,132 | No Parameters detected in the template
2023-07-29 16:19:21,143 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction is HelloWorldFunction
2023-07-29 16:19:21,143 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id
2023-07-29 16:19:21,144 | 0 stacks found in the template
2023-07-29 16:19:21,145 | No Parameters detected in the template
2023-07-29 16:19:21,153 | Sam customer defined id is more priority than other IDs. Customer defined id for resource HelloWorldFunction is HelloWorldFunction
2023-07-29 16:19:21,153 | There is no customer defined id or cdk path defined for resource ServerlessRestApi, so we will use the resource logical id as the resource id
2023-07-29 16:19:21,154 | 2 resources found in the stack
2023-07-29 16:19:21,154 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2023-07-29 16:19:21,155 | --base-dir is not presented, adjusting uri HelloWorldFunction relative to /Users/varungawande/playground/goLambdaDebug/sunflowers/.aws-sam/build/template.yaml
2023-07-29 16:19:21,157 | Docker is not reachable
Traceback (most recent call last):
  File "urllib3/connectionpool.py", line 703, in urlopen
  File "urllib3/connectionpool.py", line 398, in _make_request
  File "urllib3/connection.py", line 244, in request
  File "http/client.py", line 1256, in request
  File "http/client.py", line 1302, in _send_request
  File "http/client.py", line 1251, in endheaders
  File "http/client.py", line 1011, in _send_output
  File "http/client.py", line 951, in send
  File "docker/transport/unixconn.py", line 27, in connect
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "requests/adapters.py", line 486, in send
  File "urllib3/connectionpool.py", line 787, in urlopen
  File "urllib3/util/retry.py", line 550, in increment
  File "urllib3/packages/six.py", line 769, in reraise
  File "urllib3/connectionpool.py", line 703, in urlopen
  File "urllib3/connectionpool.py", line 398, in _make_request
  File "urllib3/connection.py", line 244, in request
  File "http/client.py", line 1256, in request
  File "http/client.py", line 1302, in _send_request
  File "http/client.py", line 1251, in endheaders
  File "http/client.py", line 1011, in _send_output
  File "http/client.py", line 951, in send
  File "docker/transport/unixconn.py", line 27, in connect
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "samcli/local/docker/utils.py", line 90, in is_docker_reachable
  File "docker/client.py", line 203, in ping
  File "docker/api/daemon.py", line 166, in ping
  File "docker/utils/decorators.py", line 46, in inner
  File "docker/api/client.py", line 237, in _get
  File "requests/sessions.py", line 602, in get
  File "requests/sessions.py", line 589, in request
  File "requests/sessions.py", line 703, in send
  File "requests/adapters.py", line 501, in send
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
2023-07-29 16:19:21,161 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-07-29 16:19:21,197 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2023-07-29 16:19:21,198 | Unable to find Click Context for getting session_id.
2023-07-29 16:19:21,198 | Sending Telemetry: {'metrics': [{'events': {'requestId': '24ab98b8-2213-401c-a963-973adbc1a16b', 'installationId': '2a43a88c-7a44-4ea7-8f6b-548d63c704d9', 'sessionId': 'eb864f46-993d-4189-9818-f763a229f59a',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.94.0', 'metricSpecificAttributes': {'events': [{'event_name': 'SamConfigFileExtension', 'event_value': '.toml', 'thread_id':
'6af42971e6b2425f886d0321d8c8e817', 'time_stamp': '2023-07-29 10:49:21.055', 'exception_name': None}]}}}]}
2023-07-29 16:19:21,200 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'c5f7dc9b-c250-4bce-baa7-e826793be477', 'installationId': '2a43a88c-7a44-4ea7-8f6b-548d63c704d9', 'sessionId': 'eb864f46-993d-4189-9818-f763a229f59a',
'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.13', 'samcliVersion': '1.94.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local start-api', 'metricSpecificAttributes':
{'projectType': 'CFN', 'gitOrigin': None, 'projectName': 'a9d90bce792e70da6e7c44cf8859e964b13e8e21dfedf17971e1a6ce77a105de', 'initialCommit': None}, 'duration': 90, 'exitReason': 'DockerIsNotReachableException', 'exitCode': 1}}]}
2023-07-29 16:19:21,944 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
2023-07-29 16:19:21,946 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: Running AWS SAM projects locally requires Docker. Have you got it installed and running?

Expected result:

Local Api is now active or some success like that.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

sam --info;

{
  "version": "1.94.0",
  "system": {
    "python": "3.8.13",
    "os": "macOS-13.4.1-arm64-arm-64bit"
  },
  "additional_dependencies": {
    "docker_engine": "Not available",
    "aws_cdk": "Not available",
    "terraform": "Not available"
  },
  "available_beta_feature_env_vars": [
    "SAM_CLI_BETA_FEATURES",
    "SAM_CLI_BETA_BUILD_PERFORMANCE",
    "SAM_CLI_BETA_TERRAFORM_SUPPORT",
    "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
  ]
}

Docker Version:
Docker version 24.0.5, build ced0996600
Docker Path:
/opt/homebrew/bin/docker

Colima Version:

colima version 0.5.5
git commit: 6251dc2c2c5d8197c356f0e402ad028945f0e830

runtime: docker
arch: x86_64
client: v24.0.5
server: v23.0.6

Colima Path:
/opt/homebrew/bin/colima

SAM Version:
SAM CLI, version 1.94.0
SAM Path:
/usr/local/bin/sam

OS Details:
Darwin Varuns-MacBook-Pro.local 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun 8 22:22:19 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8103 arm64

@V4G4X V4G4X added the stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. label Jul 29, 2023
@V4G4X
Copy link
Author

V4G4X commented Jul 29, 2023

And yes, I've tried restarting. 🙂

@mndeveci
Copy link
Contributor

Thanks for raising this issue.

I've installed Colima to see if I could get it working with SAM CLI. Apparently Colima sets a default context for Docker CLI so that you can communicate the Colima through that context configuration. But that is only available for Docker CLI, for any other clients (like SAM CLI which uses docker-py to communicate with Docker), you need to set DOCKER_HOST environment variable with the socket location.

So when I run docker context inspect I've got following output;

❯ docker context inspect
[
    {
        "Name": "default",
        "Metadata": {},
        "Endpoints": {
            "docker": {
                "Host": "unix:///Users/{myuser}/.colima/default/docker.sock",
            }
        },
    }
]

As you can see Colima sets socket file to a different location (default is /var/run/docker.sock). So we need to export DOCKER_HOST environment variable to the location above.

export DOCKER_HOST=unix:///Users/{myuser}/.colima/default/docker.sock

After setting this, SAM CLI should able to communicate through Docker APIs for its features.

I just want to remind that, this should work in theory, however the alternative that you are using (Colima for this instance) should support Docker APIs that SAM CLI uses with its minimum required Docker API version. If there are some discrepancies about how those APIs are implemented, then you might see some unexpected behavior.

@mndeveci mndeveci added area/docker question and removed stage/needs-triage Automatically applied to new issues and PRs, indicating they haven't been looked at. labels Jul 31, 2023
@moelasmar moelasmar added the blocked/close-if-inactive Blocked for >14 days with no response, will be closed if still inactive after 7 days label Jul 31, 2023
@V4G4X
Copy link
Author

V4G4X commented Aug 1, 2023

docker context ls game me:

NAME            DESCRIPTION                               DOCKER ENDPOINT                                          ERROR
colima *        colima                                    unix:///Users/varungawande/.colima/default/docker.sock
default         Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
desktop-linux   Docker Desktop                            unix:///Users/varungawande/.docker/run/docker.sock

I tried changing it here but it didn't work.

Okay but exporting that DOCKER_HOST worked.
I am curious though, if this is because Colima uses a different path than expected, why did this issue occur with Docker-Desktop aswell?

@justinmk3
Copy link

justinmk3 commented Aug 21, 2023

@mndeveci should the info in #5646 (comment) be added to the troubleshooting doc:
https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-troubleshooting.html

Currently that doc doesn't have an item for "Docker is not reachable. Mentioning docker context inspect and DOCKER_HOST are useful clues.

@mndeveci
Copy link
Contributor

mndeveci commented Sep 5, 2023

@justinmk3 good suggestion, started a thread with our documentation team about updating the troubleshooting section.

@V4G4X Colima uses a differet socket location and we need to set it up so that SAM CLI (or specifically docker-py) library can pick it up. You don't see this issue with Docker CLI since Colima automatically configures it through its context as I explained above.

@mndeveci
Copy link
Contributor

mndeveci commented Sep 5, 2023

Resolving this issue now, please let us know if you have further questions.

@mndeveci mndeveci closed this as completed Sep 5, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@pelagaggi
Copy link

This shell command worked for me
export DOCKER_HOST=$(docker context inspect | jq -r '.[0].Endpoints.docker.Host')

@avoidik
Copy link

avoidik commented Feb 5, 2024

# colima-intel-context
alias colima-intel-context='export DOCKER_HOST="$(docker context inspect colima-intel -f '\''{{.Endpoints.docker.Host}}'\'')"'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docker blocked/close-if-inactive Blocked for >14 days with no response, will be closed if still inactive after 7 days question
Projects
None yet
Development

No branches or pull requests

6 participants