[Bug]: >100MB transfers with S3-compatible object storage fail because of UploadPartCopy activated

[xplosionmind]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

I configured Cubbit DS3 as my external storage endpoint, using Amazon S3 standard.

By default, Nextcloud activates the UploadPartCopy flag, which is not supported by Cubbit yet. I see no way to deactivate it in Nextcloud settings or in config.php.

Steps to reproduce

1. Add Cubbit S3 to external storage:
2. Add small files, no errors
3. Add or try to download (if already in ext storage) >100MB files

Expected behavior

Files should upload and download normally, regardless of their size.

Installation method

Other Community project: YunoHost

Nextcloud Server version

27.1.4

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Nginx

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "cloud.tommi.space"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "27.1.4.1",
        "overwrite.cli.url": "https:\/\/cloud.tommi.space",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "ldapIgnoreNamingRules": false,
        "updatechecker": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "integrity.check.disabled": true,
        "filelocking.enabled": true,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": "6379",
            "timeout": "0.0",
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "hashing_default_password": true,
        "logout_url": "https:\/\/server.tommi.space\/yunohost\/sso\/?action=logout",
        "maintenance": false,
        "loglevel": 2,
        "localstorage.allowsymlinks": true,
        "simpleSignUpLink.shown": false,
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "theme": ""
    }
}

List of activated Apps

Enabled:
  - activity: 2.19.0
  - calendar: 4.6.0
  - camerarawpreviews: 0.8.2
  - cloud_federation_api: 1.10.0
  - contacts: 5.4.2
  - contactsinteraction: 1.8.0
  - dav: 1.27.0
  - end_to_end_encryption: 1.13.1
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_external: 1.19.0
  - files_pdfviewer: 2.8.0
  - files_reminders: 1.0.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - fileslibreofficeedit: 1.1.0
  - imageconverter: 1.3.5
  - integration_openstreetmap: 1.0.6
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - maps: 1.1.1
  - metadata: 0.19.0
  - notes: 4.8.1
  - notifications: 2.15.0
  - oauth2: 1.15.1
  - password_policy: 1.17.0
  - polls: 5.4.2
  - provisioning_api: 1.17.0
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - stt_whisper: 1.0.7
  - tasks: 0.15.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - user_ldap: 1.17.0
  - viewer: 2.1.0
  - workflowengine: 2.9.0
Disabled:
  - admin_audit: 1.17.0
  - bruteforcesettings: 2.7.0
  - circles: 27.0.1 (installed 22.3.2)
  - comments: 1.17.0 (installed 1.10.0)
  - dashboard: 7.7.0 (installed 7.0.0)
  - encryption: 2.15.0 (installed 2.15.0)
  - files_downloadactivity: 1.16.0 (installed 1.16.0)
  - firstrunwizard: 2.16.0 (installed 2.16.0)
  - nextcloud_announcements: 1.16.0 (installed 1.14.0)
  - photos: 2.3.0 (installed 1.4.0)
  - privacy: 1.11.0 (installed 1.9.0)
  - support: 1.10.0 (installed 1.8.0)
  - survey_client: 1.15.0 (installed 1.13.0)
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0 (installed 1.10.0)
  - twofactor_totp: 9.0.0
  - updatenotification: 1.17.0 (installed 1.12.0)
  - user_status: 1.7.0 (installed 1.0.1)
  - weather_status: 1.7.0 (installed 1.0.0)

Nextcloud Signing status

No response

Nextcloud Logs

My nextcloud.log

Additional info

Maybe related to #6239

[solracsf]

This might help (or fix it): #41914

[xplosionmind]

This might help (or fix it): #41914

Nice! What this does to help this is essentially checking before using UploadPartCopy? Would this work both for upload and download?

[solracsf]

Would this work both for upload and download?

As the PR affects the connection, it should work in both ways.

[maandersson]

In our environment every users have ceph object storage mounted as external s3 storage.
We have applied #41914 on an image deployed in a test environment.
We still can not move/copy files of size >5GB.
In our production environment we had to reverted merged PR #40577 to make everything works as expected and it works with file sizes >5GB.
Is there a config option to completely disable multipart copy for s3 object storage?
We get a <Error><Code>NoSuchKey</Code> for every attempted part of the multipart upload in the nextcloud logs when a multipart copy fails.

[Ziyann]

In our environment every users have ceph object storage mounted as external s3 storage. We have applied #41914 on an image deployed in a test environment. We still can not move/copy files of size >5GB. In our production environment we had to reverted merged PR #40577 to make everything works as expected and it works with file sizes >5GB. Is there a config option to completely disable multipart copy for s3 object storage? We get a <Error><Code>NoSuchKey</Code> for every attempted part of the multipart upload in the nextcloud logs when a multipart copy fails.

Have you tried setting a very high value for copySizeLimit after applying #41914? Reading the code, that should prevent the multipart copy from triggering.
For reference, I'm also affected by this issue, using Google Cloud Storage (but haven't tested the patch yet). Some built-in solution would be greatly appreciated, either a configurable limit (which we could set to a very high value, thus never triggering multipart copy), or an enable/disable switch, because right now I'm manually reverting #40577 on production after upgrades.
There are many S3 providers with different features supported, so I think all these operations should be configurable.

[maandersson]

In our environment every users have ceph object storage mounted as external s3 storage. We have applied #41914 on an image deployed in a test environment. We still can not move/copy files of size >5GB. In our production environment we had to reverted merged PR #40577 to make everything works as expected and it works with file sizes >5GB. Is there a config option to completely disable multipart copy for s3 object storage? We get a <Error><Code>NoSuchKey</Code> for every attempted part of the multipart upload in the nextcloud logs when a multipart copy fails.

Have you tried setting a very high value for copySizeLimit after applying #41914? Reading the code, that should prevent the multipart copy from triggering. For reference, I'm also affected by this issue, using Google Cloud Storage (but haven't tested the patch yet). Some built-in solution would be greatly appreciated, either a configurable limit (which we could set to a very high value, thus never triggering multipart copy), or an enable/disable switch, because right now I'm manually reverting #40577 on production after upgrades. There are many S3 providers with different features supported, so I think all these operations should be configurable.

If i decrease this value i can not copy files of greater size than the set value.
Strangely enough the opposite is not true. If i increase this value i still can not copy/move files of size >5GB.

[Ziyann]

In our environment every users have ceph object storage mounted as external s3 storage. We have applied #41914 on an image deployed in a test environment. We still can not move/copy files of size >5GB. In our production environment we had to reverted merged PR #40577 to make everything works as expected and it works with file sizes >5GB. Is there a config option to completely disable multipart copy for s3 object storage? We get a <Error><Code>NoSuchKey</Code> for every attempted part of the multipart upload in the nextcloud logs when a multipart copy fails.

Have you tried setting a very high value for copySizeLimit after applying #41914? Reading the code, that should prevent the multipart copy from triggering. For reference, I'm also affected by this issue, using Google Cloud Storage (but haven't tested the patch yet). Some built-in solution would be greatly appreciated, either a configurable limit (which we could set to a very high value, thus never triggering multipart copy), or an enable/disable switch, because right now I'm manually reverting #40577 on production after upgrades. There are many S3 providers with different features supported, so I think all these operations should be configurable.

If i decrease this value i can not copy files of greater size than the set value. Strangely enough the opposite is not true. If i increase this value i still can not copy/move files of size >5GB.

Alright, I found another parameter causing issues: even with PR #40577 reverted, we call getConnection()->copy, which ends us up here: https://github.com/nextcloud/3rdparty/blob/v28.0.0/aws/aws-sdk-php/src/S3/ObjectCopier.php#L96-L103. mup_threshold is set to 5 GB by default; #41914 should probably override this default with copySizeLimit.
In my case, even with #40577 reverted, I needed to manually edit this file to avoid that code path, otherwise I couldn't upload >5GB files to shared folders (regular folders worked fine).

[solracsf]

With #41914 MultiPartCopy can also be fully disabled.

'useMultipartCopy' => false,

[Ziyann]

In our environment every users have ceph object storage mounted as external s3 storage. We have applied #41914 on an image deployed in a test environment. We still can not move/copy files of size >5GB. In our production environment we had to reverted merged PR #40577 to make everything works as expected and it works with file sizes >5GB. Is there a config option to completely disable multipart copy for s3 object storage? We get a <Error><Code>NoSuchKey</Code> for every attempted part of the multipart upload in the nextcloud logs when a multipart copy fails.

Have you tried setting a very high value for copySizeLimit after applying #41914? Reading the code, that should prevent the multipart copy from triggering. For reference, I'm also affected by this issue, using Google Cloud Storage (but haven't tested the patch yet). Some built-in solution would be greatly appreciated, either a configurable limit (which we could set to a very high value, thus never triggering multipart copy), or an enable/disable switch, because right now I'm manually reverting #40577 on production after upgrades. There are many S3 providers with different features supported, so I think all these operations should be configurable.

If i decrease this value i can not copy files of greater size than the set value. Strangely enough the opposite is not true. If i increase this value i still can not copy/move files of size >5GB.

Alright, I found another parameter causing issues: even with PR #40577 reverted, we call getConnection()->copy, which ends us up here: https://github.com/nextcloud/3rdparty/blob/v28.0.0/aws/aws-sdk-php/src/S3/ObjectCopier.php#L96-L103. mup_threshold is set to 5 GB by default; #41914 should probably override this default with copySizeLimit. In my case, even with #40577 reverted, I needed to manually edit this file to avoid that code path, otherwise I couldn't upload >5GB files to shared folders (regular folders worked fine).

This is now fixed with #43086.

Now the only remaining pending fixes are owncloud/TarStreamer#32 and DeepDiver1975/PHPZipStreamer#20.
I've been running a production server with all these applied, on GCE with GCS primary storage for 3 months now, and things are working flawlessly :)

[xplosionmind]

Sorry to bother you all with a stupid question, but I cannot find any hint in the documentation on where should I go and what should I do to disable UploadPartCopy?

[Ziyann]

Sorry to bother you all with a stupid question, but I cannot find any hint in the documentation on where should I go and what should I do to disable UploadPartCopy?

Set 'useMultipartCopy' => false like so, and make sure you're on Nextcloud 28.0.4 or newer:

  array (
    'class' => '\\OC\\Files\\ObjectStore\\S3',
    'arguments' =>
    array (
      'bucket' => '***',
      'region' => 'europe-west1',
      'hostname' => 'storage.googleapis.com',
      'port' => '80',
      'useMultipartCopy' => false,
      'objectPrefix' => 'urn:oid:',
      'autocreate' => false,
      'use_ssl' => false,
      'use_path_style' => false,
      'legacy_auth' => false,
      'key' => '***',
      'secret' => '***',
    ),
  ),

[xplosionmind]

Thank you, @Ziyann!

But what if I am using it as external storage and not as primary storage?

[maandersson]

You should be able to set it with occ like this.
occ files_external:config [MountID] useMultipartCopy 0

[xplosionmind]

Thanks, @maandersson! It is not necessary to use occ, since it is possible to toggle this option in the web interface.