Loading folders paginated

[archigo]

I have a "outer" folder, inside of which there are 100.000+ sub folders.

I need to go through all of these folders and set their permissions.

I can do this one by one, but I can not figure out how to load these folders in a batched way.
Anything i try gives me a Microsoft.SharePoint.SPQueryThrottledException when i touch the folders.

I think it is because skip/take is not supported on .GetFolderByServerRelativeUrlAsync(archiveFolderPath).Folders, but I am not sure what else I can do to load these folders paginated.

Any suggestions would be appreciated.

using var spContext = await pnpContextFactory.CreateAsync("");

var archiveFolderPath =  $"{spContext.Uri.PathAndQuery}/Delte dokumenter/Archive";;

var containingFolder = await spContext.Web.GetFolderByServerRelativeUrlAsync(archiveFolderPath);

var roleDefinitions = (await spContext.Web.GetAsync(p => p.RoleDefinitions)).RoleDefinitions;
var role = roleDefinitions.AsRequested().FirstOrDefault(p => p.Id == 123);

var index = 0;
var pageSize = 100;

while (index < 100000)
{
    // also tried
    //await foreach(var folder in containingFolder.Folders.AsAsyncEnumerable())
    // but this also throws throttling error
    var folders = await containingFolder.Folders.Skip(index * pageSize).Take(pageSize).QueryProperties(x => x.ListItemAllFields, x => x.Name).ToListAsync();

    foreach (var folder in folders)
    {
        var item = folder.ListItemAllFields;

        await item.ResetRoleInheritanceBatchAsync();

        await item.BreakRoleInheritanceBatchAsync(copyRoleAssignments: false, clearSubscopes: true);

       // In the real case the principalId is different depending on the folder
        var principalId= 123;

        await item.AddRoleDefinitionBatchAsync(principalId: principalId, roleDefinition: role);
    }
    await spContext.ExecuteAsync();
}

[patrikhellgren]

Not a direct answer to your question but just so that you are aware of possible limitations you might (or already have) run into. There are some limitations when you are getting into that large document libraries, see SharePoint Online Limits. As you don't specify the exact structure of your library it's hard to say if that is the case here but these limitations tend to come into play in different ways so it could be what you are seeing here.
As an example I have seen strange things happen (i.e. no success) when copying files into document libraries with just over 130000 files and folders in it. As this was at a client they ended up with a MS support case and I don't know if and how it was resolved.

[archigo]

@patrikhellgren Thanks
I do not think we are hitting these limits yet, as sharepoint works otherwise fine.
I can do what I need to, by calling GetFolderByServerRelativeUrlAsync on each individual folder.
I just can not find any place in the documentation that mentions how to load multiple folders in a paginated / batched way

The structure is like this:
/sistes/{SystemName}/Shared documents/Archive/ManyManyFolders

[jansenbe]

@archigo : checkout below code, I just enumerated 11K+ folders with it:

                string listTitle = "PNP_SDK_TEST_LargeFolderLibrary";
                var myList = context.Web.Lists.GetByTitle(listTitle);

                string viewXml = @"<View Scope='RecursiveAll'>
                        <ViewFields>
                          <FieldRef Name='Title' />
                          <FieldRef Name='FileLeafRef' />
                          <FieldRef Name='FSObjType'/>
                          <FieldRef Name='FileDirRef'/>
                        </ViewFields>
                         <Query>
                            <And>
                                <Eq><FieldRef Name=""FSObjType"" /><Value Type=""Integer"">1</Value></Eq>
                            </And>
                        </Query>
                        <OrderBy Override='TRUE'><FieldRef Name='ID' Ascending='FALSE' /></OrderBy>
                        <RowLimit Paged='TRUE'>5000</RowLimit>
                       </View>";

                // Load all the needed data using paged requests
                bool paging = true;
                string nextPage = null;
                int pages = 0;
                while (paging)
                {
                    // Execute the query
                    await myList.LoadItemsByCamlQueryAsync(new CamlQueryOptions()
                    {
                        ViewXml = viewXml,
                        DatesInUtc = true,
                        PagingInfo = nextPage
                    },
                    // Load list item collections (e.g. RoleAssignments)
                    p => p.RoleAssignments.QueryProperties(p => p.PrincipalId,
                          p => p.RoleDefinitions.QueryProperties(rd => rd.Id, rd => rd.Name)),
                    // Load FieldValuesAsText to get access to 'system' properties
                    p => p.FieldValuesAsText,
                    // Load the HasUniqueRoleAssignments property
                    p => p.HasUniqueRoleAssignments);
                    pages++;

                    if (myList.Items.Length == pages * 5000)
                    {
                        nextPage = $"Paged=TRUE&p_ID={myList.Items.AsRequested().Last().Id}";
                    }
                    else
                    {
                        paging = false;
                    }
                }

[archigo]

I am not a fan of Caml, but it works :)