Server really slow on folders with many files

[Zegorax]

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Steps to reproduce

1. Create a groupfolder
2. Create a folder in the groupfolder
3. Add a lot of files inside of it (7000 files for example)
4. Access the folder through the WebUI

Expected behaviour

The folder should load in an acceptable time

Actual behaviour

The folder takes more than 10 minutes to load. It is really disturbing for users, and cause timeouts on Desktop Clients trying to sync.

In this folder, there is 7357 photos which are approximately 2Mb each.
Doing a ls -al instantly gives every file :

Loading the same folder through the WebUI takes 9.5 minutes (!) to load. The request is only 7.5 Mb in size :

It is also doing a heavy database load :

I checked and the index in nextcloud/server#8962 is already present.

Server configuration

Operating system:
Nextcloud Docker image (Latest tag)

Web server: See Docker image

Database: MySQL 8.0

PHP version: See Docker image

Nextcloud version: 20.0.1

Updated from an older Nextcloud/ownCloud or fresh install: N/A

Where did you install Nextcloud from: Docker

List of activated apps:

App list

Enabled: - accessibility: 1.6.0 - activity: 2.13.2 - audioplayer: 3.0.0 - bruteforcesettings: 2.0.1 - calendar: 2.1.2 - cloud_federation_api: 1.3.0 - comments: 1.10.0 - contacts: 3.4.2 - contactsinteraction: 1.1.0 - dav: 1.16.0 - deck: 1.2.2 - external: 3.7.1 - federatedfilesharing: 1.10.1 - federation: 1.10.1 - files: 1.15.0 - files_downloadactivity: 1.9.0 - files_external: 1.11.1 - files_fulltextsearch: 20.0.0 - files_pdfviewer: 2.0.1 - files_rightclick: 0.17.0 - files_sharing: 1.12.0 - files_trashbin: 1.10.1 - files_versions: 1.13.0 - files_videoplayer: 1.9.0 - firstrunwizard: 2.9.0 - fulltextsearch: 20.0.0 - fulltextsearch_elasticsearch: 20.0.0 - groupfolders: 8.2.0 - impersonate: 1.7.0 - logreader: 2.5.0 - lookup_server_connector: 1.8.0 - maps: 0.1.8 - nextcloud_announcements: 1.9.0 - notifications: 2.8.0 - oauth2: 1.8.0 - onlyoffice: 6.2.0 - password_policy: 1.10.1 - photos: 1.2.0 - previewgenerator: 3.1.0 - privacy: 1.4.0 - provisioning_api: 1.10.0 - ransomware_protection: 1.8.0 - recommendations: 0.8.0 - serverinfo: 1.10.0 - settings: 1.2.0 - sharebymail: 1.10.0 - side_menu: 1.21.0 - support: 1.3.0 - survey_client: 1.8.0 - systemtags: 1.10.0 - tasks: 0.13.6 - text: 3.1.0 - theming: 1.11.0 - twofactor_backupcodes: 1.9.0 - updatenotification: 1.10.0 - user_ldap: 1.10.2 - user_status: 1.0.0 - viewer: 1.4.0 - weather_status: 1.0.0 - workflowengine: 2.2.0 Disabled: - admin_audit - camerarawpreviews - dashboard - encryption - mail

Are you using external storage, if yes which one: No

Are you using encryption: No

Are you using an external user-backend, if yes which one: LDAP

LDAP configuration (delete this part if not used)

Irrelevant

Client configuration

Irrelevant, tried with multiple

[ShadyF]

Experiencing the same issue. Opening a folder containing large amounts of files causes CPU to spike up in the same manner described here (although with PostgreSQL).

Using the docker image
Nextcloud version: 20.0.6
Database: PostgreSQL 10

Any updates on this?

[patricksebastien]

Also experiencing this performance issue. Here's more info:
#1506 (comment)

[feizhoulv]

The same issue .
About 9000 files .
When opening a folder with a browser it takes 4 seconds to open。

[Dennis1993]

The same problem here.
We use Group Folders sind 4 weeks and many users says, that the cloud is very slow. In den group folder we have 15.000 files (8 GB) and the web interface loads minimum 11 seconds.
How can we fix that?

If I disable the plugin, the web page is loaded in 1 second.

NextCliud: 27.0.2 with PHP 8.2 on Ubuntu 20.04

Thanks!

[DySprozin]

same problem: 3000 files, over 10 minutes (!), cpu 100%
query from pg_stat_activity:

SELECT "f"."fileid", "mapping_type", "mapping_id", "mask", "a"."permissions", "path" FROM "oc_group_folders_acl" "a" INNER JOIN "oc_filecache" "f" ON "f"."fileid" = "a"."fileid" WHERE ("path_hash" IN ($1)) AND ("storage" = $2) AND ((("mapping_type" = $3) AND ("mapping_id" = $4)) OR (("mapping_type" = $5) AND ("mapping_id" = $6)) OR (("mapping_type" = $7) AND ("mapping_id" = $8)) OR (("mapping_type" = $9) AND ("mapping_id" = $10)) OR (("mapping_type" = $11) AND ("mapping_id" = $12)) OR (("mapping_type" = $13) AND ("mapping_id" = $14)) OR (("mapping_type" = $15) AND ("mapping_id" = $16)) OR (("mapping_type" = $17) AND ("mapping_id" = $18)) OR (("mapping_type" = $19) AND ("mapping_id" = $20)) OR (("mapping_type" = $21) AND ("mapping_id" = $22)) OR (("mapping_type" = $23) AND ("mapping_id" = $24)) OR (("mapping_type" = $25) AND ("mapping_id" = $26)) OR (("mapping_type" = $27) AND ("mapping_id" = $28)) OR (("mapping_type" = $29) AND ("mapping_id" = $30)) OR (("mapping_type" = $31) AND ("mapping_id" = $32)) OR (("

[icewind1991]

Please try #2560 and see if it makes a difference

[joshtrichards]

FYI to those on this thread: the mentioned patch is now realized in all current versions of GF for >=NC25.

Please provide feedback whether updating addresses this issue. Thanks!

[joshtrichards]

Well, it's been a year with no feedback or comments (positive or negative) after the last change. Closing...