Not able to use get_directory_contents

[jbd]

Hello,

I'm trying to list a directory with 718 directories in it using get_directory_content API function, but I can't make it work.

If I don't use a limit option, or if I use a limit option higher than the number of directories I've got, I can't
get anything from the function. AM I misusing something ?

api_response = ns_api.get_directory_contents(options.path) # does not work
print(api_response) # {'children': None, 'resume': None}

api_response = ns_api.get_directory_contents(options.path, limit=8192) # does not work
print(api_response) # {'children': None, 'resume': None}

api_response = ns_api.get_directory_contents(options.path, limit=1) # does work
print(api_response) # print {'children': [{'access_time': None,
                             #  'atime_val': None,
                             # 'block_size': None,
                             # 'blocks': None,

And even with limit=1, I'm not able to resume the listing after with something like that:

api_response = api.get_directory_contents(directory, detail='name,gid', type='container', limit=1)
            
while api_response.resume:
                api_response = api.get_directory_contents(directory, resume=api_response.resume)
                for container in api_response.children:
                        yield container

Here is a usable gist: https://gist.github.com/jbd/8a5f691e75509dc5397243a105dc61fa

[Atomicology]

@jbd: The options.path needs to be prefixed with ifs, so if you are looking to query the directory /ifs/data, then it must be specified as follows:

api_response = ns_api.get_directory_contents('ifs/data')

This method is unfortunately not flexible enough to support either an absolute path with / or a relative path under /ifs/ because these language bindings are auto-generated. Thus, if your path is absolute, then you will need to pass it to get_directory_contents as options.path[1:]. Otherwise, if it's relative to /ifs, then you will need to prepend it as 'ifs/' + options.path. Hopefully that helps.

[jbd]

Hi,

thank you for your answer. I guess my initial message was not clear enough. I've got no problem with the directory_path subtelties but I don't seem to be able to use get_directory_contents correctly.

Here is a full example, I hope it will be easier to follow.

1. Create a thousand directories on the Isilon cluster

ISILON-1# mkdir -p /ifs/data/apitest/{0000..1023}
ISILON-1# find  /ifs/data/apitest -type d |wc -l
    1025

I've got no problem to make get_directory_contents to work with this code :

if limit:
    api_response = api_instance.get_directory_contents(directory_path, limit=limit)
else:
    api_response = api_instance.get_directory_contents(directory_path)

if api_response.children:
    pprint(len(api_response.children))
else:
    pprint(api_response)

See https://gist.github.com/jbd/3504a32fdc3be782d774198e55711ea4 for the full usable program

$ python issue_simple.py ifs/data/apitest
1000

This corresponds to default value of the "limit" parameter.

The rest of the tests looks OK:

128
$ python issue_simple.py ifs/data/apitest 256
256
$ python issue_simple.py ifs/data/apitest 512
512
$ python issue_simple.py ifs/data/apitest 4096
1024

But if I use another directory with some contents, api_response.children is None :

ISILON-1# find /ifs/projets/p01 -maxdepth 1 | wc -l
     719

ISILON-1# ls -led /ifs/projets/p01
drwxr-xr-x    719 root  wheel  19782 Oct 23 12:15 /ifs/projets/p01
 OWNER: user:root
 GROUP: group:wheel
 SYNTHETIC ACL
 0: user:root allow dir_gen_read,dir_gen_write,dir_gen_execute,std_write_dac,delete_child 
 1: group:wheel allow dir_gen_read,dir_gen_execute 
 2: everyone allow dir_gen_read,dir_gen_execute

$ python issue_simple.py ifs/projets/p01
{'children': None, 'resume': None}

If I specify an arbitrary limit in the get_directory_contents call, it works until a certain size:

$ python issue_simple.py ifs/projets/p01 128
128
$ python issue_simple.py ifs/projets/p01 256
256
$ python issue_simple.py ifs/projets/p01 512
{'children': None, 'resume': None}

The sweet spot is limit=318:

$ python issue_simple.py ifs/projets/p01 318
318
$ python issue_simple.py ifs/projets/p01 319
{'children': None, 'resume': None}

What could explain this behavior ? The equivalent curl commands with the same user are working fine.

$ curl --referer XXXX -H 'Accept: application/json; indent=4' -H 'X-CSRF-Token: 14829322-bcbd-4889-89f7-XXXXXXXXX' -s --insecure -b @cookie https://isiloncluster:8080/namespace/ifs/projets/p01 | wc -l
718

[jbd]

I've got a directory with a '\303' C escape code in it. This is a problem, but I guess this should not make get_directory_contents working incorrectly (or it should raise an exception) and silently fails.

Here is a reproducer:

ISILON-1# mkdir -p /ifs/data/apitest/$(printf 'weird\303_directory')
ISILON-1# ls /ifs/data/apitest 
weird_directory
ISILON-1# ls -b /ifs/data/apitest
weird\303_directory

$ python issue_simple.py ifs/data/apitest 
{'children': None, 'resume': None}

The curl command is working:

$ curl --referer XXXXX -H 'Accept: application/json; indent=4' -H 'X-CSRF-Token: 8655b8e1-cc68-42dc-8b15-XXXXX' -s --insecure -b @cookie https://isiloncluster:8080/namespace/ifs/data/apitest
{"children":[{
   "name" : "weird_directory"
}

[jbd]

In the case of the "weird_directory" with the embedded non-printable char, the response.data object is not considered as a valid json here:

isilon_sdk_python/isi_sdk/api_client.py

Line 253 in 1f0fb83

data = json.loads(response.data)

The fact that it cannot be transformed to json seems like the kwargs cannot be build in deserialize_object:

isilon_sdk_python/isi_sdk/api_client.py

Line 669 in 1f0fb83

isinstance(data, (list, dict))):