Skip to content

Latest commit

 

History

History
305 lines (219 loc) · 9.23 KB

devctr-image.md

File metadata and controls

305 lines (219 loc) · 9.23 KB
Raw

Publishing a New Container Image

About the Container Image

Firecracker uses a Docker container to standardize the build process. This also fixes the build tools and dependencies to specific versions. Every once in a while, something needs to be updated. To do this, a new container image needs to be built locally, then published to the AWS ECR registry. The Firecracker CI suite must also be updated to use the new image.

Prerequisites

  1. Access to the fcuvm ECR repository.
  2. The docker package installed locally. You should already have this if you've ever built Firecracker from source.
  3. Access to both an x86_64 and aarch64 machines to build the container images.
  4. Ensure aws --version is >=1.17.10.

Steps

[optional] Update poetry.lock

This step is optional but recommended, to be on top of Python package changes.

./tools/devtool shell --privileged
poetry update --lock --directory tools/devctr/

This will change poetry.lock, which you can commit with your changes.

x86_64

  1. Login to the Docker organization in a shell. Make sure that your account has access to the repository:

    aws ecr-public get-login-password --region us-east-1 \
| docker login --username AWS --password-stdin public.ecr.aws

    For non-TTY devices, although not recommended a less secure approach can be used:

    docker login --username AWS --password \
$(aws ecr-public get-login-password --region us-east-1) public.ecr.aws

  2. Navigate to the Firecracker directory. Verify that you have the latest container image locally.

    docker images
REPOSITORY                         TAG     IMAGE ID        CREATED         SIZE
public.ecr.aws/firecracker/fcuvm   v26     8d00deb17f7a    2 weeks ago     2.41GB

  3. Make your necessary changes, if any, to the Dockerfile. There's one for all the architectures in the Firecracker source tree.

  4. Commit the changes, if any.

  5. Build a new container image with the updated Dockerfile.

     tools/devtool build_devctr

  6. Verify that the new image exists.

    docker images
REPOSITORY                         TAG       IMAGE ID         CREATED       SIZE
public.ecr.aws/firecracker/fcuvm   latest    1f9852368efb     2 weeks ago   2.36GB
public.ecr.aws/firecracker/fcuvm   v26       8d00deb17f7a     2 weeks ago   2.41GB

  7. Tag the new image with the next available version X and the architecture you're on. Note that this will not always be "current version in devtool + 1", as sometimes that version might already be used on feature branches. Always check the "Image Tags" on the fcuvm repository to make sure you do not accidentally overwrite an existing image.

    As a sanity check, run:

    docker pull public.ecr.aws/firecracker/fcuvm:vX

    and verify that you get an error message along the lines of

    Error response from daemon: manifest for public.ecr.aws/firecracker/fcuvm:vX not
found: manifest unknown: Requested image not found

    This means the version you've chosen does not exist yet, and you are good to go.

    docker tag 1f9852368efb public.ecr.aws/firecracker/fcuvm:v27_x86_64

docker images
REPOSITORY                         TAG          IMAGE ID       CREATED
public.ecr.aws/firecracker/fcuvm   latest       1f9852368efb   1 week ago
public.ecr.aws/firecracker/fcuvm   v27_x86_64   1f9852368efb   1 week ago
public.ecr.aws/firecracker/fcuvm   v26          8d00deb17f7a   2 weeks ago

  8. Push the image.

    docker push public.ecr.aws/firecracker/fcuvm:v27_x86_64

aarch64

Login to the aarch64 build machine.

Steps 1-4 are identical across architectures, change x86_64 to aarch64.

Then continue with the above steps:

  1. Build a new container image with the updated Dockerfile.

    tools/devtool build_devctr

  2. Verify that the new image exists.

    docker images
REPOSITORY                         TAG        IMAGE ID            CREATED
public.ecr.aws/firecracker/fcuvm   latest     1f9852368efb        2 minutes ago
public.ecr.aws/firecracker/fcuvm   v26        8d00deb17f7a        2 weeks ago

  3. Tag the new image with the next available version X and the architecture you're on. Note that this will not always be "current version in devtool + 1", as sometimes that version might already be used on feature branches. Always check the "Image Tags" on the fcuvm repository to make sure you do not accidentally overwrite an existing image.

    As a sanity check, run:

    docker pull public.ecr.aws/firecracker/fcuvm:vX

    and verify that you get an error message along the lines of

    Error response from daemon: manifest for public.ecr.aws/firecracker/fcuvm:vX not
found: manifest unknown: Requested image not found

    This means the version you've chosen does not exist yet, and you are good to go.

    docker tag 1f9852368efb public.ecr.aws/firecracker/fcuvm:v27_aarch64

docker images
REPOSITORY                         TAG            IMAGE ID
public.ecr.aws/firecracker/fcuvm   latest         1f9852368efb
public.ecr.aws/firecracker/fcuvm   v27_aarch64    1f9852368efb
public.ecr.aws/firecracker/fcuvm   v26            8d00deb17f7a

  4. Push the image.

    docker push public.ecr.aws/firecracker/fcuvm:v27_aarch64

  5. Create a manifest to point the latest container version to each specialized image, per architecture.

    docker manifest create public.ecr.aws/firecracker/fcuvm:v27 \
    public.ecr.aws/firecracker/fcuvm:v27_x86_64 public.ecr.aws/firecracker/fcuvm:v27_aarch64

docker manifest push public.ecr.aws/firecracker/fcuvm:v27

  6. Update the image tag in the devtool script. Commit and push the change.

    PREV_TAG=v26
CURR_TAG=v27
sed -i "s%DEVCTR_IMAGE_TAG=\"$PREV_TAG\"%DEVCTR_IMAGE_TAG=\"$CURR_TAG\"%" tools/devtool

Troubleshooting

Check out the rust-vmm-container readme for additional troubleshooting steps and guidelines.

I can't push the manifest

docker manifest is only supported when experimental cli features are enabled

See this article for explanations and fix.

How to test the image after pushing it to the Docker registry

Either fetch and run it locally on another machine than the one you used to build it, or clean up any artifacts from the build machine and fetch.

docker system prune -a

docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

tools/devtool shell
[Firecracker devtool] About to pull docker image public.ecr.aws/firecracker/fcuvm:v15
[Firecracker devtool] Continue?

I don't have access to the AWS ECR registry

docker push public.ecr.aws/firecracker/fcuvm:v27
The push refers to repository [public.ecr.aws/firecracker/fcuvm]
e2b5ee0c4e6b: Preparing
0fbb5fd5f156: Preparing
...
a1aa3da2a80a: Waiting
denied: requested access to the resource is denied

Only a Firecracker maintainer can update the container image. If you are one, ask a member of the team to add you to the AWS ECR repository and retry.

I pushed the wrong tag

Tags can be deleted from the AWS ECR interface.

Also, pushing the same tag twice will overwrite the initial content.

I did everything right and nothing works anymore

If you see unrelated Python errors, it's likely because the dev container pulls Python 3 at build time. Python 3 means different minor versions on different platforms, and is not backwards compatible. So it's entirely possible that docker build has pulled in unwanted Python dependencies.

To include only your changes, an alternative to the method described above is to make the changes inside the container, instead of in the Dockerfile.

Let's say you want to update cargo-audit (random example).

  1. Enter the container as root.

    tools/devtool shell -p

  2. Make the changes locally. Do not exit the container.

    cargo install cargo-audit --force

  3. Find your running container.

    docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED
e9f0487fdcb9        fcuvm:v14       "bash"              53 seconds ago

  4. Commit the modified container to a new image. Use the container ID.

    docker commit e9f0487fdcb9 fcuvm:v15_x86_64
    docker image ls
REPOSITORY      TAG                 IMAGE ID            CREATED
fcuvm           v15_x86_64          514581e654a6        18 seconds ago
fcuvm           v14                 c8581789ead3        2 months ago

  5. Repeat for aarch64.

  6. Create and push the manifest.