[Question]: Best way to download the metadata of all the objects in a storageZone

[tacman]

Is there a way to download the metadata of all the objects in a storageZone? Obviously, a recursive function that calls listFiles() and/or listContents() would work, but generate many API calls if there are a lot of directories.

I have a million objects, each object has 0 or more photos and the photos and metadata for each object is in a directory keyed by the object id. S3 doesn't really have directories, so this approach is fine, but I'm wondering if I need to restructure if we migrate to Bunny.

            $list = $edgeStorageApi->listFiles(
                storageZoneName: $storageZoneName,
                path: '/'
            );

            foreach ($list->getContents() as $fileInfo) {
                $subList = $edgeStorageApi->listFiles(
                    $storageZoneName,
                    path: $fileInfo['ObjectName']
                );

               // @todo: add recursive call

            }

Bunny allows up to 50,000 files per directory, but recommends a max of 10,000. That makes sense, since there's no pagination so the JSON response will get pretty big.

[ToshY]

Hey @tacman 👋

I've converted it to a discussion. Thanks for the suggestion.

---

Is there a way to download the metadata of all the objects in a storageZone? Obviously, a recursive function that calls listFiles() and/or listContents() would work, but generate many API calls if there are a lot of directories.

I don't know if it's currently possible without doing it recursive like you've already suggested. The API specs denote that it just retrieves a list of files and directories located in the given directory.

If you want a definitive answer, I'd recommend creating a ticket at the bunny support desk and ask them directly. You can explain you're current situation, what you're trying to achieve, and ask if they have any suggestions on how to handle such a migration.

As you've mentioned S3 (I assume you're migration from S3 to Bunny storage zones) you might be interested in asking on where they stand regarding S3 compatibility for the storage zones: https://bunny.net/blog/whats-happening-with-s3-compatibility/.

[tacman]

Coincidentally, I posted a message to that blog post this morning. And I've reached out to their support. Thanks.

[ToshY]

If you figure it out I'd be interested to know on what they would suggest in your situation. 👌

[tacman]

Alas, their reply wasn't very helpful.

Are you able to provide examples of what your resume looks like currently? We only have a list endpoint that would return the files in a directory with basic information like size and a checksum. You would need to loop through each folder to get the contents. We also do not have any custom metadata that can be attached to an object.

My first idea is to organize by storageZone, each project in the system would have it's own storageZone, most of those would be under 10000 files. Those that have more could be further grouped.

But it sure don't feel very clean! Basically I'd be creating my own flat structure that mimics a directory, because there's no way to get objects except grouped by directory. Yuck. Flysystem has the ability to recurse, and someone wrote a Flysystem adapter, but I'm guessing they implemented it with recursive API calls.

 'images' => $defaultStorage->listContents('/', deep: true),

My S3 structure (simplifed for this example)

/projectCode/
    objectCode/
        meta.json
        image1.jpg
        image2.jpg

To use bunny, scanning the entire dataset would require a million API calls. I could convert this to

/projectCode (in it's own storageZone)
    objectCode.meta.json
    objectCode.image1.jpg
    objectCode.image2.jpg

With this structure, I could at least say "get me the files associated with this project".

This seems like a huge deficiency in working with large number of files. Videos have collections, why don't files? I keep thinking there must be a solution, but having read through the API docs I'm pretty sure there's not. :-(

[tacman]

Of course, I don't want to scan the entire dataset. Most of the time I'm scanning I already have the project and object codes. If I flatten the files, then I lose the ability to say "give me the files associated with this object", which I would have if I keep the original structure.

[tacman]

Indeed, the Flysystem adapter for bunny implements it as a recursive call.

    public function listContents(string $path, bool $deep): iterable
    {
        try {
            $entries = $this->client->list($path);
            // @codeCoverageIgnoreStart
        } catch (Exceptions\BunnyCDNException $e) {
            throw UnableToRetrieveMetadata::create($path, 'folder', $e->getMessage());
        }
        // @codeCoverageIgnoreEnd

        foreach ($entries as $item) {
            $content = $this->normalizeObject($item);
            yield $content;

            if ($deep && $content instanceof DirectoryAttributes) {
                foreach ($this->listContents($content->path(), $deep) as $deepItem) {
                    yield $deepItem;
                }
            }
        }
    }

https://github.com/PlatformCommunity/flysystem-bunnycdn/blob/7d93e74aa1af080a5e1fef558fddbf49fc74c8cc/src/BunnyCDNAdapter.php#L122

[tacman]

Another depressing response from bunny:

We appreciate your feedback and our team is working on implementing new API calls as well as improving the current ones we have.
You can also use FTP to browse the files, however, these are the only two available options at the moment, the API call and FTP connection.

[ToshY]

Okay, so in a nutshell:

• The API currently does not support it and the response you got from support doesn't really seem like it will be anytime soon.
• Other bunny implementations (like the flysystem adapter) do the exact same thing as you've already suggested yourself, which is a recursive call to get all objects.

I'm not sure if I can help you any further with this really. The only thing I can say right now is that if I were you I would accept the idea that it needs to be recursive at this point and start to think of a way on how to speed things up. If it's indeed over a million API calls that need to be made, maybe spliting into multiple messages (if you're familiar with messenger) and dispatching them through an async transport, then let it be handled by X amount of workers, might speed things up by X times.

---

If at some point in time the API will support it, let me know and I will add it to the library.

[tacman]

Indeed, I knew it wasn't possible with your library, since it's a wrapper for the bunny API calls, outside of adding the recursion as an option. But I enjoy brainstorming with you.

Yes, running the fetches in parallel would help, but realistically restructuring is the only way this will be possible.

I quite like the Flickr API. I have a prototype working, it allows all sorts of metadata and searching by metadata. I may end up using that as my archive, though that has its own challenges.

Thanks for brainstorming with me about this.