fix: symlinks handling #3298

2ZeroSix · 2022-02-01T15:21:09Z

reverts to old behavior: symlinks to directories are handled as copy-paste Pub does not support publishing packages with directory symlinks #3143
fix of pubignore/gitignore:
- allow non-resolving symlinks if they are ignored
- don't warn on ignored directory symlinks

jonasfj · 2022-02-01T16:08:43Z

I'm a bit unsure if we want to support this, how does it handle cycles?
What is the need for symlink directories?

Might it not be better to just ask advanced users to copy files if they need to include them in multiple locations. They can wrap their dart pub publish command with a bash script living in tool/ -- it's ugly, but how many people actually need this?

2ZeroSix · 2022-02-01T16:27:25Z

It’s common use case for macOS/iOS plug-ins, third party libraries in ffi plugins etc

In our case it forces us to remove directory with symlink to common scripts before publish

pubignore also does not solve it since this check is done prior to application of ignore rules

2ZeroSix · 2022-02-01T17:24:50Z

added test for cycles
fixed wrongly appended / to path consisted of single delimiter

The only failing test should be one which ensures lish is failing on package with symlink to directory.
This seems to be a regression and was introduced somewhere between flutter 2.5+ and 2.8- (dart 2.12 and 2.15)

Previous behavior (pub bundled in dart 2.12 and behavior in current PR):

pub archives all filtered files in tar.gz
consumer receives them as separate files after unpacking of tar.gz
so, there is no OS-dependant problems for consumer and no headache for package developer

Current behavior:

pub throws on any (even ignored) directory symlink beneath package directory
package developers should apply workarounds to their pipelines
consumers doesn't see any difference

If you insist this is expected behavior: I could try to move the check after resolving of ignored files and allow only ignored directory symlinks

2ZeroSix · 2022-02-03T09:46:48Z

Hmm, it seems that in windows it fails to list dirrectory with loops in symlinks

It differs from what is done on posix-like systems

02:34 +32 -1: test/package_list_files_test.dart: throws on symlink cycles
02:34 +32 -2: test/package_list_files_test.dart: throws on symlink cycles [E]
  Expected: throws <Instance of 'DataException'> with `message`: contains 'Pub does not support publishing packages with non-resolving symlink:'
    Actual: <Closure: () => List<String>>
     Which: threw FileSystemException:<FileSystemException: Directory listing failed, path = 'C:\Users\runneradmin\AppData\Local\Temp\dart_test_5e4ddcd9\myapp\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\subdir\symlink\*' (OS Error: The system cannot find the path specified.
                  , errno = 3)>
            stack dart:io                                   _Directory.listSync
                  package:pub/src/package.dart 248:48       Package.listFiles.<fn>
                  package:pub/src/ignore.dart 291:28        Ignore.listFiles
                  package:pub/src/package.dart 245:19       Package.listFiles
                  test\package_list_files_test.dart 115:30  main.<fn>.<fn>
                  package:test_api                          expect
                  test\package_list_files_test.dart 114:5   main.<fn>
                  
            which is not an instance of 'DataException'
  
  package:test_api                         expect
  test\package_list_files_test.dart 114:5  main.<fn>

lib/src/package.dart

test/package_list_files_test.dart

lib/src/package.dart

sigurdm · 2022-02-10T14:29:20Z

Besides the cycle-handling this is looking quite good!

2ZeroSix · 2022-02-25T07:59:19Z

jfyi: I had no time to reslove windows-related bugs lately, will continue in a few days

(cherry picked from commit 25fd5bf)

…symlinked directories in published archive) (cherry picked from commit 163d23c)

This reverts commit db790fb

This reverts commit f371314

This reverts commit 47bb026

This reverts commit fbe23f1

This reverts commit aab94fb

sigurdm · 2024-10-22T10:46:05Z

I have done a bit of research, and it seems that both on windows, mac and linux resolveSymbolicLinksSync will fail with an exception if the nesting of a symlink to resolve is too deep.

By that logic we probably don't need to do anything explicit to prevent circularities. We just handle that exception.

I'll try to make a merge, and an update without the check for circularities.

sigurdm · 2024-10-22T13:04:09Z

Hmm - seems the CI doesn't agree quite with my research.

On mac it seems contents of sufficiently nested directories is not listed. And on windows sufficiently nested links exists as Link but not as a Directory...

spydon · 2024-10-22T19:52:46Z

Hmm - seems the CI doesn't agree quite with my research.

On mac it seems contents of sufficiently nested directories is not listed. And on windows sufficiently nested links exists as Link but not as a Directory...

Does that matter if it is documented? It would still work for all use cases where everything is set up correctly right?

sigurdm · 2024-10-24T07:14:43Z

Yeah - perhaps we don't need to care that much about symlink cycles. If you have them, you kind of set yourself up for trouble...

sigurdm · 2024-10-29T11:41:47Z

@jonasfj I reinstated some symlink detection. I think™ this works.

For every visited directory x, it resolves canonicalize(x) to x'.
Then for every parent of canonicalize(x), p:
- it resolves p to p'. If x' equals p' it reports a cycle.

Could you do a sanity check?

jonasfj

I think this will work.

jonasfj · 2024-10-29T11:53:55Z

lib/src/package.dart

+        } on FileSystemException catch (e) {
+          throw DataException(
+            'Could not resolve symbolic link $path. $e',
+          );
+        }


nit:

Suggested change

} on FileSystemException catch (e) {

throw DataException(

'Could not resolve symbolic link $path. $e',

);

}

} on FileSystemException catch (e) {

if (!link.existsSync()) {

return;

}

throw DataException(

'Could not resolve symbolic link $path. $e',

);

}

test/package_list_files_test.dart

jonasfj · 2024-10-29T12:10:31Z

test/package_list_files_test.dart

+        createDirectorySymlink(
+          p.join(d.sandbox, appPath, 'subdir', 'symlink'),
+          '..',
+        );


Consider making a d.link method

test/package_list_files_test.dart

lib/src/package.dart

Co-authored-by: Jonas Finnemann Jensen <jopsen@gmail.com>

jonasfj · 2024-10-31T13:43:43Z

test/lish/symlinks_test.dart

+    await dir(appPath, [
+      dir('b', [file('bb', 'bbb')]),
+    ]).create();
+    Link(p.join(sandbox, appPath, 'symlink_to_dir_outside_package'))
+        .createSync(p.join(sandbox, 'a'));
+    Link(p.join(sandbox, appPath, 'symlink_to_dir_outside_package_relative'))
+        .createSync(p.join('..', 'a'));
+    Link(p.join(sandbox, appPath, 'symlink_to_dir_inside_package'))
+        .createSync(p.join(sandbox, appPath, 'b'));
+    Link(p.join(sandbox, appPath, 'symlink_to_dir_inside_package_relative'))
+        .createSync('b');
+    Link(p.join(sandbox, appPath, 'b', 'l')).createSync(p.join(sandbox, 't'));


Why not use link?

jonasfj · 2024-10-31T13:46:17Z

test/link_descriptor.dart

Belongs in the test/descriptor/ folder:
https://github.com/dart-lang/pub/tree/master/test/descriptor

jonasfj · 2024-10-31T13:46:42Z

test/link_descriptor.dart

+d.Descriptor link(String name, String target, {bool forceDirectory = false}) {
+  return LinkDescriptor(name, target, forceDirectory: forceDirectory);
+}


Put this in test/descriptor.dart, I think..

test/lish/symlinks_test.dart

jonasfj · 2024-10-31T13:48:25Z

test/lish/symlinks_test.dart

+import 'package:tar/tar.dart';
+import 'package:test/test.dart';
+
+import '../descriptor.dart';


Suggested change

import '../descriptor.dart';

import '../descriptor.dart' as d;

To follow the pattern we have in other files.

This was referenced Feb 1, 2022

Pub does not support publishing packages with directory symlinks #3143

Open

fix: ignored directory symlinks #3300

Closed

sigurdm reviewed Feb 3, 2022

View reviewed changes

lib/src/package.dart Outdated Show resolved Hide resolved

jonasfj reviewed Feb 10, 2022

View reviewed changes

test/package_list_files_test.dart Outdated Show resolved Hide resolved

jonasfj reviewed Feb 10, 2022

View reviewed changes

test/package_list_files_test.dart Outdated Show resolved Hide resolved

jonasfj reviewed Feb 10, 2022

View reviewed changes

lib/src/package.dart Outdated Show resolved Hide resolved

2ZeroSix added 18 commits June 4, 2022 01:53

don't warn on ignored directory symlinks

0189750

(cherry picked from commit 25fd5bf)

Don't crash on directory symlinks (default behavior is copy-paste of …

7320081

…symlinked directories in published archive) (cherry picked from commit 163d23c)

add missing await

963b645

remove unnecessary brackets in string interpolation

b87e533

codestyle

c087295

add symlink cycle test

6e23dcc

use ignore rule without delimiter in checked-in but ignored warning test

b9b9314

probable fix for cycles on windows

4486f00

don't follow links

e49c45b

use appropriate error message (wrongly cherry-picked)

7e1cb20

double check for directory symlinks

05f6cd9

replace "throws" => "not throws on valid directory symlinks"

0a6ffaa

Revert "double check for directory symlinks"

10b8e3e

This reverts commit db790fb

Revert "don't follow links"

f6bd8d2

This reverts commit f371314

Revert "probable fix for cycles on windows"

bfc30e5

This reverts commit 47bb026

try to resolve symbolic link manually before listSync invocation

b1b69bc

Revert "Revert "probable fix for cycles on windows""

d97bfe2

This reverts commit fbe23f1

Revert "Revert "don't follow links""

add6e5c

This reverts commit aab94fb

publish integration test

599a42f

sigurdm added 3 commits October 22, 2024 12:21

Rely on system detecting cycles

4ed382a

fmt

0c56b15

Remove deprecated lint

f620e18

sigurdm added 2 commits October 25, 2024 10:35

Detect cycles

7679721

canonicalize before examining parents (gets rid of /.)

f8ee397

jonasfj reviewed Oct 29, 2024

View reviewed changes

lib/src/package.dart Show resolved Hide resolved

sigurdm and others added 5 commits October 29, 2024 13:26

Update test/package_list_files_test.dart

6d2ef92

Co-authored-by: Jonas Finnemann Jensen <jopsen@gmail.com>

Address review comments

3bd1765

Merge remote-tracking branch '2ZeroSix/fix/symlinks' into fix/symlinks

3837226

Add missing file

66b1ae3

attempt at windows fix

d4a89e0

sigurdm requested a review from jonasfj October 31, 2024 13:21

jonasfj reviewed Oct 31, 2024

View reviewed changes

test/lish/symlinks_test.dart Outdated Show resolved Hide resolved

jonasfj reviewed Oct 31, 2024

View reviewed changes

sigurdm added 4 commits October 31, 2024 14:05

Use link descriptors and prefix in lish/symlinks_test

8909acf

Move link() -> descriptor.dart

778e35c

move test/link_descriptor.dart -> test/descriptor/linkdescriptor.dart

b179ccf

attempt at windows fix2

2d8ec9b

sigurdm requested review from jonasfj and szakarias November 1, 2024 13:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: symlinks handling #3298

fix: symlinks handling #3298

2ZeroSix commented Feb 1, 2022 •

edited

Loading

jonasfj commented Feb 1, 2022

2ZeroSix commented Feb 1, 2022

2ZeroSix commented Feb 1, 2022 •

edited

Loading

2ZeroSix commented Feb 3, 2022

sigurdm commented Feb 10, 2022

2ZeroSix commented Feb 25, 2022

sigurdm commented Oct 22, 2024

sigurdm commented Oct 22, 2024

spydon commented Oct 22, 2024

sigurdm commented Oct 24, 2024

sigurdm commented Oct 29, 2024

jonasfj left a comment

jonasfj Oct 29, 2024

sigurdm Oct 29, 2024

jonasfj Oct 29, 2024

sigurdm Oct 29, 2024

jonasfj Oct 31, 2024 •

edited

Loading

sigurdm Oct 31, 2024

jonasfj Oct 31, 2024

sigurdm Oct 31, 2024

jonasfj Oct 31, 2024

sigurdm Oct 31, 2024

jonasfj Oct 31, 2024

sigurdm Oct 31, 2024

	import '../descriptor.dart';
	import '../descriptor.dart' as d;

fix: symlinks handling #3298

Are you sure you want to change the base?

fix: symlinks handling #3298

Conversation

2ZeroSix commented Feb 1, 2022 • edited Loading

jonasfj commented Feb 1, 2022

2ZeroSix commented Feb 1, 2022

2ZeroSix commented Feb 1, 2022 • edited Loading

2ZeroSix commented Feb 3, 2022

sigurdm commented Feb 10, 2022

2ZeroSix commented Feb 25, 2022

sigurdm commented Oct 22, 2024

sigurdm commented Oct 22, 2024

spydon commented Oct 22, 2024

sigurdm commented Oct 24, 2024

sigurdm commented Oct 29, 2024

jonasfj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonasfj Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

2ZeroSix commented Feb 1, 2022 •

edited

Loading

2ZeroSix commented Feb 1, 2022 •

edited

Loading

jonasfj Oct 31, 2024 •

edited

Loading