feat: allow dropping a folder into the dropzone #1066

gvdp · 2024-01-31T08:42:00Z

Recreated #834 after updating master with all the latest changes.

As for the question about how in-demand this is: we've needed it for one of our apps and it's been in production for over a year now. I made it so there is an opt-in boolean which needs to be enabled, so by default nothing should change for exisiting users.

Seems also like this is one of the oldest issues on this repo where some people have asked for it: #2

gilest

Hey @gvdp. Thank you for reopening this – appreciate your contributions as always ❤️

so by default nothing should change for exisiting users.

That's true.. But adding a feature will increase the bundle size for all addon users. When we add a feature we are committing as maintainers to support it for a long time, too

For testing purposes I have pushed this feature branch to docs/folder-drop which deploys a public docsite here:
https://docs-folder-drop.ember-file-upload.pages.dev/demo

I'll keep that updated so that we can test across different browsers and devices.

I did some light testing in Safari, Firefox, Chrome and Edge on Mac OS and Windows and found a few issues:

Dropping a directory of directories throws and leaves the dropzone in a broken state (dropzone.active stays true afterwards)
- Interested in your suggestion of how we might handle this
  - recursively open subdirectories and add their files, too
  - only add files one-level deep but do not throw an error when nested folders are present
The directory itself is being reported as an UploadFile in callbacks
- this might be useful programatically, it's a significant API change and something that's not modelled as part of this addon. For example there's no way to "upload" the directory from the queue
- maybe we can get notified about directory details at the dropzone-event level, but I'm not sure we want those items being added to the queue
In some browsers (Firefox on Mac OS) .DS_Store files are being added. Not sure there's ever a reason this would be desired. I guess we should consider how to handle dot files in general, too

In terms of the implementation, it would be great to reduce the usage of the TypeScript as keyword. That would give us more confidence about the robustness of this feature

Once we've resolved these issues, we'll also need to add documentation of this feature

API reference -> FileDropzone
Documentation -> Testing
(New page) Documentation -> Add directories via Drag and Drop

ember-file-upload/src/components/file-dropzone.ts

ember-file-upload/src/interfaces.ts

gilest · 2024-02-01T23:15:24Z

ember-file-upload/src/test-support.ts

+export async function dragAndDropDirectory(
+  selector: string,
+  folderName: string,
+  filesInDirectory: (File | Blob)[],
+  ...singleFiles: (File | Blob)[]
+) {


I wonder if we should use an options object here. From the call site the two types of files (in directory and single) are not very clear.

Fair call, it's not very clear indeed. Would you make 1 opts object? Or keep the selector and/or folderName and make an object for the rest? I also noticed there is no example / test really using the ...singleFiles argument.

I'm doubting if the folderName even needs to be here, but that'll probably be relevant for the bigger remark about the directory itself being reported as UploadFile

test-app/tests/integration/components/file-dropzone-test.gts

ember-file-upload/src/system/data-transfer-wrapper.ts

gvdp · 2024-02-05T13:38:50Z

Hi @gilest , thanks for the comments, appreciate the effort. I fixed some of the smaller code comments already. Not sure how you usually commit those? I made a few small extra commits now but they can be easily squashed.

The directory of directories is indeed something which does not work. In our own application this is not necessary and we give the end user an explicit notice of this. I don't think including everything recursively is the way to go, maybe we should just throw or show a specific, relevant error to the user?
The directory itself being reported as UploadFile is indeed not ideal, I seem to remember we did this for a reason but it's already some time ago and I don't know what it was :-) . As a File object it's not possible to see the difference between a file and a directory, but there is the isDirectory prop which we do already use. Maybe we should just not include the directoryFile in the array being sent here? I think this would not break our current implementation. That would probably also mean the directoryName prop in the test helper is no longer relevant.
The .DS_store files are also annoying. In our implementation we catch and filter those out in the consuming application. Not sure we should just ignore all dotfiles, sometimes it can be intentional that these are uploaded. Maybe we should just include a warning in the documentation that it is the responsibility of the consuming application to handle all hidden files?

I'll of course also add documentation and API reference stuff once we've resolved these open issues. Thanks again for the review!

gilest

As always, thanks for your continued work on this.

Sorry this review took so long – it's hard to find a chunk of time to dedicate to a complex feature like this.

For testing purposes I have pushed this feature branch to docs/folder-drop which deploys a public docsite here:
https://docs-folder-drop.ember-file-upload.pages.dev/demo

Have made some general suggestions, especially around improving resilience when a file fails to read.

Please don't be discouraged by my feedback 😅 This is library code so it's better that we consider failure modes and use-cases now than make breaking changes later

Will respond to remaining design questions in a separate comment.

gilest · 2024-02-23T22:58:03Z

ember-file-upload/src/system/data-transfer-wrapper.ts

+interface FutureProofDataTransferItem extends DataTransferItem {
+  getAsEntry?: () => FileSystemDirectoryEntry;
+}


According to the spec this may fail and return null. We probably want to have our types reflect that.

The method aborts and returns null if the dropped item isn't a file, or if the DataTransferItem object is not in read or read/write mode.

Suggested change

interface FutureProofDataTransferItem extends DataTransferItem {

getAsEntry?: () => FileSystemDirectoryEntry;

}

interface FutureProofDataTransferItem extends DataTransferItem {

getAsEntry?: () => FileSystemDirectoryEntry | null;

}

Added the null case

gilest · 2024-02-23T22:59:10Z

ember-file-upload/src/system/data-transfer-wrapper.ts

+      if (typeof item.getAsEntry === 'function') {
+        return item.getAsEntry();
+      }
+
+      return item.webkitGetAsEntry() as FileSystemDirectoryEntry;


Shorter way of doing this... Maybe the type signature of this function may needs to union null also

Suggested change

if (typeof item.getAsEntry === 'function') {

return item.getAsEntry();

}

return item.webkitGetAsEntry() as FileSystemDirectoryEntry;

return item.getAsEntry?.() ?? item.webkitGetAsEntry();

gilest · 2024-02-23T22:59:45Z

ember-file-upload/src/system/data-transfer-wrapper.ts

+        getEntry(item)
+          ?.createReader()
+          ?.readEntries(async (entries: FileSystemEntry[]) => {


We probably want to allow for the result of getEntry to return null. In which case we should probably reject this promise.

Added a Promise rejection when getEntry is falsy. Not sure it should maybe have a clearer message.

gilest · 2024-02-23T23:00:25Z

ember-file-upload/src/system/data-transfer-wrapper.ts

+            ).catch((err) => {
+              throw err;
+            });
+            resolve(readFiles.filter((file) => file !== undefined) as File[]);


Safer to filter out missing elements with a truthiness check

Suggested change

resolve(readFiles.filter((file) => file !== undefined) as File[]);

resolve(readFiles.filter(Boolean) as File[]);

Didn't know you could use the Boolean constructor like this. Updated.

gilest · 2024-02-23T23:01:41Z

ember-file-upload/src/system/data-transfer-wrapper.ts

+      files.push(...flattenedFileArray);
+    } else {
+      const droppedFiles: File[] = Array.from(this.dataTransfer?.files ?? []);
+      files.push(...droppedFiles);
+    }
+
+    return files;


Not sure we need to define a files variable and mutate it. Each of these branches could just be an outright return

Made it immediate returns

gilest · 2024-02-23T23:04:27Z

ember-file-upload/src/system/data-transfer-wrapper.ts

@@ -43,6 +47,81 @@ export default class DataTransferWrapper {
    return this.files.length ? this.files : this.items;
  }

+  async getFilesAndDirectories() {


[Perf] All of the child functions defined within getFilesAndDirectories do not rely on class state or local variable closure.

Because of this, they only need to be defined once in memory, rather than every time this methodn is called.

Let's move all of the child functions to module scope (you can put them in the top of this file).

I think extracting them also helps with readability of this feature.

Moved all the functions to the top.

gilest · 2024-02-23T23:06:25Z

ember-file-upload/src/system/data-transfer-wrapper.ts

+  async getFilesAndDirectories() {
+    const files: File[] = [];
+
+    const readEntry = async (entry: FileSystemEntry): Promise<File> => {


This function returns a promise and does not use the await keyword.

Suggested change

const readEntry = async (entry: FileSystemEntry): Promise<File> => {

const readEntry = (entry: FileSystemEntry): Promise<File> => {

Good catch. Removed the async keyword.
There is an eslint rule which can alert these cases, maybe you can add this to you configuration:
https://eslint.org/docs/latest/rules/require-await

gilest · 2024-02-23T23:12:33Z

ember-file-upload/src/test-support.ts

+}
+
+export async function dragAndDropDirectory(
+  selector: string,


Let's follow the same convention as the other test helpers and allow this to be a selector or an element.

Suggested change

selector: string,

target: string | HTMLElement,

gilest · 2024-02-23T23:22:39Z

Separate comment with "design feedback"

I fixed some of the smaller code comments already. Not sure how you usually commit those? I made a few small extra commits now but they can be easily squashed.

Extra commits is no problem. The whole feature will be squash-merged anyway 🙂

The directory of directories is indeed something which does not work. In our own application this is not necessary and we give the end user an explicit notice of this. I don't think including everything recursively is the way to go, maybe we should just throw or show a specific, relevant error to the user?

My suggestion: let's add the simplest implementation for now, which we can extend later (ideally without too much breaking change).

Given that, I think it would be ok include only files from the top-level of the dropped directory, and ignore subdirectories.

This is more flexible than any nested directories causing an error. For example, this avoids cause users to have to re-arrange their local directories just to upload a group of files which would be very frustrating.

The directory itself being reported as UploadFile is indeed not ideal, I seem to remember we did this for a reason but it's already some time ago and I don't know what it was :-) . As a File object it's not possible to see the difference between a file and a directory, but there is the isDirectory prop which we do already use. Maybe we should just not include the directoryFile in the array being sent here? I think this would not break our current implementation. That would probably also mean the directoryName prop in the test helper is no longer relevant.

Given the "simplest implementation" suggestion I would agree with this and prefer that we not report directories for now. If needed, we can model this later as an enhancement (along with recursion).

The .DS_store files are also annoying. In our implementation we catch and filter those out in the consuming application. Not sure we should just ignore all dotfiles, sometimes it can be intentional that these are uploaded. Maybe we should just include a warning in the documentation that it is the responsibility of the consuming application to handle all hidden files?

Agree with your suggestion – worth documenting. We shouldn't make this decision for users as we can't anticipate their use-cases.

gvdp · 2024-02-29T09:27:59Z

As always, thanks for your continued work on this.

Sorry this review took so long – it's hard to find a chunk of time to dedicate to a complex feature like this.

For testing purposes I have pushed this feature branch to docs/folder-drop which deploys a public docsite here: https://docs-folder-drop.ember-file-upload.pages.dev/demo

Have made some general suggestions, especially around improving resilience when a file fails to read.

Please don't be discouraged by my feedback 😅 This is library code so it's better that we consider failure modes and use-cases now than make breaking changes later

Will respond to remaining design questions in a separate comment.

No problem at all, I appreciate the feedback and I agree it's better to be thorough now. You're the one who probably has to maintain all this later :-).

…directories

gvdp · 2024-02-29T10:18:57Z

Separate comment with "design feedback"

I fixed some of the smaller code comments already. Not sure how you usually commit those? I made a few small extra commits now but they can be easily squashed.

Extra commits is no problem. The whole feature will be squash-merged anyway 🙂

The directory of directories is indeed something which does not work. In our own application this is not necessary and we give the end user an explicit notice of this. I don't think including everything recursively is the way to go, maybe we should just throw or show a specific, relevant error to the user?

My suggestion: let's add the simplest implementation for now, which we can extend later (ideally without too much breaking change).

Given that, I think it would be ok include only files from the top-level of the dropped directory, and ignore subdirectories.

This is more flexible than any nested directories causing an error. For example, this avoids cause users to have to re-arrange their local directories just to upload a group of files which would be very frustrating.

The directory itself being reported as UploadFile is indeed not ideal, I seem to remember we did this for a reason but it's already some time ago and I don't know what it was :-) . As a File object it's not possible to see the difference between a file and a directory, but there is the isDirectory prop which we do already use. Maybe we should just not include the directoryFile in the array being sent here? I think this would not break our current implementation. That would probably also mean the directoryName prop in the test helper is no longer relevant.

Given the "simplest implementation" suggestion I would agree with this and prefer that we not report directories for now. If needed, we can model this later as an enhancement (along with recursion).

The .DS_store files are also annoying. In our implementation we catch and filter those out in the consuming application. Not sure we should just ignore all dotfiles, sometimes it can be intentional that these are uploaded. Maybe we should just include a warning in the documentation that it is the responsibility of the consuming application to handle all hidden files?

Agree with your suggestion – worth documenting. We shouldn't make this decision for users as we can't anticipate their use-cases.

I removed the Promise rejection when there is a subdirectory so the user doesn't get an unnecessary error. I am wondering if we should log a warning or something in this case or just silently ignore this case? Or will just documenting this behavior be enough? I already added a small code comment as well.

Also removed the reporting of the dropped directory itself, now only the files included will be reported. I think I remember now that we used this to keep track of the name of the dropped folder, but that's not super important and we can solve on our end.
Everything still seems to work for me on the demo page :-)

Unrelated comment: I noticed while working on this some of the scripts in the root package.json don't quite work. When you run pnpm run test there is a prepare script being executed which doesn't exist, and when running pnpm test:watch:test-app the --server option also doesn't exist.

gilest · 2024-03-19T03:15:28Z

Hi @gvdp . Thank you for your comment and contribution. I had completely missed this notification in a sea of others. Apologies. I will try to review soon!

…n the first 100 entries

fix: recursively read folder entries because most browsers only return the first 100 entries

gvdp mentioned this pull request Jan 31, 2024

feat: add option to allow dropping a directory and reading the contents #834

Closed

gilest added the enhancement label Feb 1, 2024

gilest reviewed Feb 1, 2024

View reviewed changes

gvdp force-pushed the allow-folder-drop branch from 78b78fe to abb9f1b Compare February 5, 2024 13:22

gilest reviewed Feb 23, 2024

View reviewed changes

Glenn Van De Putte added 5 commits February 29, 2024 10:10

feat: allow dropping a folder into the dropzone

3cad399

wrap getWebkitAsEntry method to be future proof if API changes

7560c34

remove unnecessary getter

01007f9

fix typo in interface

8378fd2

implement code readability suggestions

3ef7bfd

gvdp force-pushed the allow-folder-drop branch from abb9f1b to 3ef7bfd Compare February 29, 2024 09:25

fix: dont report foldernames and dont throw errors when there are sub…

468e5bf

…directories

Glenn Van De Putte and others added 5 commits May 24, 2024 10:51

remove console log

3d58723

pass folderName to UploadFile object when dropping folders

8770ea8

fix: resolve and log warning when a subdirectory is dropped

a2faaac

fix: recursively read folder entries because most browsers only retur…

fa3d8a1

…n the first 100 entries

Merge pull request #1 from Bagaar/fix/recursively-read-large-folders

9119b21

fix: recursively read folder entries because most browsers only return the first 100 entries

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: allow dropping a folder into the dropzone #1066

feat: allow dropping a folder into the dropzone #1066

gvdp commented Jan 31, 2024 •

edited

Loading

gilest left a comment •

edited

Loading

gilest Feb 1, 2024

gvdp Feb 5, 2024 •

edited

Loading

gvdp commented Feb 5, 2024

gilest left a comment •

edited

Loading

gilest Feb 23, 2024 •

edited

Loading

gvdp Feb 29, 2024

gilest Feb 23, 2024 •

edited

Loading

gvdp Feb 29, 2024

gilest Feb 23, 2024 •

edited

Loading

gvdp Feb 29, 2024

gilest Feb 23, 2024

gvdp Feb 29, 2024

gilest Feb 23, 2024

gvdp Feb 29, 2024

gilest Feb 23, 2024 •

edited

Loading

gvdp Feb 29, 2024

gilest Feb 23, 2024

gvdp Feb 29, 2024

gilest Feb 23, 2024 •

edited

Loading

gvdp Feb 29, 2024

gilest commented Feb 23, 2024 •

edited

Loading

gvdp commented Feb 29, 2024

gvdp commented Feb 29, 2024

Separate comment with "design feedback"

gilest commented Mar 19, 2024

	resolve(readFiles.filter((file) => file !== undefined) as File[]);
	resolve(readFiles.filter(Boolean) as File[]);

	const readEntry = async (entry: FileSystemEntry): Promise<File> => {
	const readEntry = (entry: FileSystemEntry): Promise<File> => {

feat: allow dropping a folder into the dropzone #1066

Are you sure you want to change the base?

feat: allow dropping a folder into the dropzone #1066

Conversation

gvdp commented Jan 31, 2024 • edited Loading

gilest left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvdp Feb 5, 2024 • edited Loading

Choose a reason for hiding this comment

gvdp commented Feb 5, 2024

gilest left a comment • edited Loading

Choose a reason for hiding this comment

gilest Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gilest Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gilest Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gilest Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gilest Feb 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gilest commented Feb 23, 2024 • edited Loading

Separate comment with "design feedback"

gvdp commented Feb 29, 2024

gvdp commented Feb 29, 2024

Separate comment with "design feedback"

gilest commented Mar 19, 2024

gvdp commented Jan 31, 2024 •

edited

Loading

gilest left a comment •

edited

Loading

gvdp Feb 5, 2024 •

edited

Loading

gilest left a comment •

edited

Loading

gilest Feb 23, 2024 •

edited

Loading

gilest Feb 23, 2024 •

edited

Loading

gilest Feb 23, 2024 •

edited

Loading

gilest Feb 23, 2024 •

edited

Loading

gilest Feb 23, 2024 •

edited

Loading

gilest commented Feb 23, 2024 •

edited

Loading