asset staging: do not recurse into the output directory #3899

jeshan · 2019-09-01T20:07:52Z

🐛 Bug Report

What is the problem?

I have hit a corner case with the code related to copy directories. It's going in recursively until Node can no longer handle such long file names.

aws-cdk/packages/@aws-cdk/assets/lib/fs/copy.ts

Lines 7 to 18 in 16eb658

    
           export function copyDirectory(srcDir: string, destDir: string, options: CopyOptions = { }, rootDir?: string) { 
        
             const follow = options.follow !== undefined ? options.follow : FollowMode.EXTERNAL; 
        
             const exclude = options.exclude || []; 
        
             rootDir = rootDir || srcDir; 
        
             if (!fs.statSync(srcDir).isDirectory()) { 
        
               throw new Error(`${srcDir} is not a directory`); 
        
             } 
        
             const files = fs.readdirSync(srcDir); 
        
             for (const file of files) {

The offending snippet is when I had a docker asset declared as follows:

        new DockerImageAsset(this, 'app-image-asset', {
            directory: '.',
            repositoryName: 'my-repo/app',
        });

Running cdk deploy showed me this error:

Error: ENAMETOOLONG: name too long, mkdir 'cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.7e4b3cc320bdf668daa4c0d065bb1a9841162c955cb702f7a01d7d3d7e88dbfe/cdk.out/asset.03918990104d8973acd2ad0abe022844192fcce618cf74f7734a7077f4111d1f'
    at Object.mkdirSync (fs.js:750:3)
    at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:39:16)
    at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
    at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
    at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
    at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
    at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
    at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
    at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)
    at copyDirectory (/mnt/sdb/ideaProjects/my-project/node_modules/@aws-cdk/assets/lib/fs/copy.js:40:13)

(Declaring the directory as its full path did not work neither.)
This is quite a serious issue for me, personally.

A possible solution is to exclude cdk.out. I think it's reasonable to assume that cdk.out name is reserved for this project.

As a temporary workaround for me, I've edited the code so that it looks like the following:

    const exclude = options.exclude || ['cdk.out'];

Another possible solution is to avoid recursion if we've already seen the directory in question.

Apparently, it's an issue where the docker build context is the same path as the cdk project.
That said, I'm not sure if the above is enough to reproduce the issue. When I comment the code to create the docker asset, the issue disappears (probably more of a symptom rather than the cause). I've checked if symlinks are the issue (apparently, it's not)

Environment

CDK CLI Version: 1.6.1
OS: all
Language: all

The text was updated successfully, but these errors were encountered:

rhboyd · 2019-09-01T23:08:16Z

This usually happens when you try to build an image from within a directory you want included in the image as well. It creates a recursive path that never ends.

joezhou888 · 2019-09-02T02:19:09Z

I had the same problem.

The only way to make it work is:
Project
-App Directory
-Env Directory

jeshan · 2019-09-02T08:14:35Z

Thanks Joe.
@rhboyd , we could make the exclude directory list an option in cdk.json. What do you think?

eladb · 2019-09-04T06:07:44Z

This is a bug. It makes sense that when we copy assets into the output directory, we won't descend into the same directory as we copy.

askulkarni2 · 2019-11-21T21:04:57Z

Saw this issue for a similar use case while using the aws-s3-assets module. I was trying to zip up my cdk project into s3 and use it to bootstrap CodeCommit. Excluding cdk.out seems to work.

const asset = new Asset(this, 'CodeAsset', {
      path: path.join(__dirname, '../'),
      exclude: [
        'node_modules',
        '.git',
        'cdk.out'
      ]
    });

BinaryShrub · 2020-01-30T18:40:29Z

I added cdk* to .dockerignore to get it working:

// Create Fargate Service
const fargateService = new ecs_patterns.NetworkLoadBalancedFargateService(
  this,
  'sample-app',
  {
    cluster,
    taskImageOptions: {
      image: ecs.ContainerImage.fromAsset('./') // blows up without cdk.out in .dockerignore
    }
  }
)

victorL91 · 2020-10-29T23:19:03Z

I bumped into this issue again. It may be stupid but I solved it by making sure that the version of aws-cdk on my machine was the same as on the package.json. I was not bootstrapping it the correct way.

wdolek · 2021-02-04T10:22:23Z

I added cdk* to .dockerignore to get it working:

Don't forget to check your cdk.json, it should contain:

{
  "context": {
    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true
  }
}

Otherwise .dockerignore is ignored 🤷

mikeshi80 · 2021-06-23T03:16:16Z

I added cdk* to .dockerignore to get it working:

Don't forget to check your cdk.json, it should contain:
{
  "context": {
    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true
  }
}
Otherwise .dockerignore is ignored

You saved me!!!

bashleigh · 2021-07-28T14:20:38Z

I added cdk* to .dockerignore to get it working:

Don't forget to check your cdk.json, it should contain:
{
  "context": {
    "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true
  }
}
Otherwise .dockerignore is ignored 🤷

Mine seems to ignore my .dockerignore file anyway. Could be because I've changed to context dir but I have also tried adding the .dockerignore file to the root of my context

const lith = new lambda.DockerImageFunction(this, `${name}lith`, {
      code: lambda.DockerImageCode.fromImageAsset(path.join(__dirname, '../../../../'), { // root of monolith 
        file: 'packages/deployment-service/Dockerfile', // location of Dockerfile
        ignoreMode: IgnoreMode.DOCKER,
        exclude: [
          'cdk.out',
        ],
      }),
      vpc,
    })

Tried both ~/.dockerfile and ~/packages/deployment-service/.dockerfile. Not sure what else to try!

Edit

Ok so after some investigation: .dockerignore needs to be at the root of the monorepo in my case, in other words: at ../../../../. However I still have a related issue which I'm debugging

another edit

Figured it out. Paths are specified from the asset path. So I needed to be more specific however I'd prefer it if my dockerignore file excluded the cdk dir globally as I specified but hey ho

exclude: [
  'packages/deployment-service/cdk',
],

stevemer · 2022-03-10T15:43:04Z

Re:
"@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,

It looks like this is no longer necessary.

jsii.errors.JSIIError: Unsupported feature flag '@aws-cdk/aws-ecr-assets:dockerIgnoreSupport'. This flag existed on CDKv1 but has been removed in CDKv2. CDK will now behave as the same as when the flag is enabled.

corymhall · 2022-05-25T14:39:59Z

It looks like this was fixed in #16034. Let us know if anyone is still seeing this issue.

github-actions · 2022-05-27T16:05:22Z

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

- aws/aws-cdk#3899 (comment)

wallacepreston · 2023-11-20T06:09:07Z

I'm still seeing this issue. I get this when adding the flag, alright. But I still get the ENAMETOOLONG: name too long error about the cdk.out dir

Re: "@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,

It looks like this is no longer necessary.
jsii.errors.JSIIError: Unsupported feature flag '@aws-cdk/aws-ecr-assets:dockerIgnoreSupport'. This flag existed on CDKv1 but has been removed in CDKv2. CDK will now behave as the same as when the flag is enabled.

abury · 2023-11-30T22:56:04Z

Seeing this issue pop up again in random repos :(

Figured it out thanks to @bashleigh 's detailed comment! I ignored the entire cdk folder at the root and all was well

austinschrader · 2024-01-04T19:37:44Z

Seeing this issue pop up again in random repos :(

Figured it out thanks to @bashleigh 's detailed comment! I ignored the entire cdk folder at the root and all was well

Could you share how you ignored the entire cdk folder at the root? .dockerignore doesn't seem to work as the flag @aws-cdk/aws-ecr-assets:dockerIgnoreSupport was deprecated in cdk v2

github-actions · 2024-02-18T00:29:03Z

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

robertIanClarkson · 2024-03-13T23:48:29Z

I am experimenting with polylith monorepo with components, bases, projects that looks similar to this...

├── .dockerignore << add */**/cdk.out
├── bases
│   └── ts
│       └── some-base-pkg
│           └── src
├── components
│   └── ts
│       └── some-component-pkg
│           └── src
└── projects
    └── ts
        └── some-project-pkg << running cdk deploy
            └── Dockerfile
            └── cdk.json
            └── src
            └── bin
                └── cdk-app.ts << cdk code

My cdk-app.ts includes some code like this

const asset = new cdk.aws_ecr_assets.DockerImageAsset(this, "some-id", {
  directory: "/my/project/root,
  file: "projects/ts/some-project-pkg/Dockerfile"
})

what worked for me was added */**/cdk.out to my root .dockerignore.
Hope this helps someone!

jeshan added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 1, 2019

SomayaB added the @aws-cdk/core Related to core CDK functionality label Sep 3, 2019

SomayaB assigned eladb Sep 3, 2019

eladb removed the needs-triage This issue or PR still needs to be triaged. label Sep 4, 2019

eladb changed the title ~~cdk crashes with "name too long" with Docker Asset~~ asset staging: do not recurse into the output directory Sep 4, 2019

eladb added the p1 label Sep 5, 2019

kuwabarahiroshi mentioned this issue Oct 3, 2019

DockerImageAsset: exclude option is ignored #4353

Closed

eladb added p2 and removed p1 labels Nov 11, 2019

eladb added p1 and removed p2 labels Mar 18, 2020

eladb added p2 and removed @aws-cdk/core Related to core CDK functionality p1 labels May 6, 2020

phstc mentioned this issue Jun 22, 2020

Getting ENAMETOOLONG: name too long when deploying awslabs/aws-rails-provisioner#12

Closed

eladb added the effort/small Small work item – less than a day of effort label Aug 4, 2020

eladb added the @aws-cdk/assets Related to the @aws-cdk/assets package label Aug 17, 2020

mullermp mentioned this issue Aug 26, 2020

ENAMETOOLONG Error on deploy awslabs/aws-rails-provisioner#17

Closed

MrArnoldPalmer mentioned this issue Oct 17, 2020

[aws-ecs] On MacOS the fromImage filled the trash with content recursively and lock those files in Trash #10858

Closed

l0b0 mentioned this issue Feb 11, 2021

feat: Validate checksum linz/geostore#64

Merged

eladb removed their assignment Feb 25, 2021

rix0rrr added p1 and removed p2 labels Mar 16, 2022

corymhall added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label May 25, 2022

github-actions bot closed this as completed Jun 1, 2022

phuwin added a commit to phuwin/cdk-fargate-docker-nextjs that referenced this issue Jul 1, 2022

chore: ignore cdk out

0339040

- aws/aws-cdk#3899 (comment)

mrgrain mentioned this issue Aug 23, 2022

Synthesis fails with the error ENAMETOOLONG cdklabs/decdk#164

Closed

alexjball mentioned this issue May 4, 2023

(aws-lambda-python-alpha): Bundling breaks if cdk.out is in the entry path #25449

Open

kellertk removed response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. closed-for-staleness This issue was automatically closed because it hadn't received any attention in a while. labels Nov 30, 2023

kellertk reopened this Nov 30, 2023

comcalvi added p2 and removed p1 labels Feb 13, 2024

github-actions bot added p1 and removed p2 labels Feb 18, 2024

pahud added p2 and removed p1 labels Jun 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

asset staging: do not recurse into the output directory #3899

asset staging: do not recurse into the output directory #3899

jeshan commented Sep 1, 2019

rhboyd commented Sep 1, 2019

joezhou888 commented Sep 2, 2019

jeshan commented Sep 2, 2019

eladb commented Sep 4, 2019

askulkarni2 commented Nov 21, 2019 •

edited

Loading

BinaryShrub commented Jan 30, 2020

victorL91 commented Oct 29, 2020

wdolek commented Feb 4, 2021 •

edited

Loading

mikeshi80 commented Jun 23, 2021

bashleigh commented Jul 28, 2021 •

edited

Loading

stevemer commented Mar 10, 2022

corymhall commented May 25, 2022

github-actions bot commented May 27, 2022

wallacepreston commented Nov 20, 2023

abury commented Nov 30, 2023 •

edited

Loading

austinschrader commented Jan 4, 2024

github-actions bot commented Feb 18, 2024

robertIanClarkson commented Mar 13, 2024

asset staging: do not recurse into the output directory #3899

asset staging: do not recurse into the output directory #3899

Comments

jeshan commented Sep 1, 2019

🐛 Bug Report

What is the problem?

Environment

rhboyd commented Sep 1, 2019

joezhou888 commented Sep 2, 2019

jeshan commented Sep 2, 2019

eladb commented Sep 4, 2019

askulkarni2 commented Nov 21, 2019 • edited Loading

BinaryShrub commented Jan 30, 2020

victorL91 commented Oct 29, 2020

wdolek commented Feb 4, 2021 • edited Loading

mikeshi80 commented Jun 23, 2021

bashleigh commented Jul 28, 2021 • edited Loading

Edit

another edit

stevemer commented Mar 10, 2022

corymhall commented May 25, 2022

github-actions bot commented May 27, 2022

wallacepreston commented Nov 20, 2023

abury commented Nov 30, 2023 • edited Loading

austinschrader commented Jan 4, 2024

github-actions bot commented Feb 18, 2024

robertIanClarkson commented Mar 13, 2024

askulkarni2 commented Nov 21, 2019 •

edited

Loading

wdolek commented Feb 4, 2021 •

edited

Loading

bashleigh commented Jul 28, 2021 •

edited

Loading

abury commented Nov 30, 2023 •

edited

Loading