File path is only filename after ingestion using the REST Gateway

[Bradfordio]

Hello,

I'm relatively new to fscrawler so apologies if this is a very basic mistake I'm making on my end.

We are currently attempting to ingest millions of files of all different filetypes stored on an S3 bucket into ES. To accomplish this we fscrawler running as an REST gateway on a RH Ec2 instance. We also have a python script which does the following:

• Reads from an SQS queue where the messages are an object key.
  

• Reads the object key and if there are any prefixes (folders in S3) it then makes the same directory structure on the ec2 instance:
  

• Downloads the file with the correct filename to that location
  

• Uploads the file to ES index via curl command

cmd = os.system("curl -F file=@'" + directory + "/" + filename + "' -F id=\"" + uid + "\" http://127.0.0.1:8080/fscrawler/_upload")

This process does work for us and we have successfully ingested many files. However one thing we have noticed which is vital for us is that the path fields for both virtual and real are only returning with just the filename:

"path" : {
            "virtual" : "sample-data-audio-wav.wav",
            "real" : "sample-data-audio-wav.wav"
          }

Can anyone please help me to understand why the linux file path is missing?

Config:
fscrawler mappings are the default:

https://github.com/dadoonet/fscrawler/blob/master/settings/src/main/resources/fr/pilato/elasticsearch/crawler/fs/_default/7/_settings.json

fscrawler settings:

---
name: "path_test"
fs:
  url: "/tmp/es"
  update_rate: "15m"
  excludes:
  - "*/~*"
  json_support: false
  filename_as_id: true
  add_filesize: true
  remove_deleted: true
  add_as_inner_object: false
  store_source: true
  index_content: true
  attributes_support: true
  raw_metadata: true
  xml_support: false
  index_folders: false
  lang_detect: false
  continue_on_error: false
  ocr:
    language: "eng"
    enabled: true
    pdf_strategy: "ocr_and_text"
  follow_symlinks: true
elasticsearch:
  nodes:
  - url: "***"
  bulk_size: 100
  flush_interval: "5s"
  index: "path_test"
rest:
  url: "http://127.0.0.1:8080/fscrawler"

[dadoonet]

That's interesting.

I did not test it in debug mode but I'm wondering what FSCrawler is actually getting when we call it with @'" + directory + "/" + filename.

If we are able to get the full information in FSCrawler, then that's a bug.
Otherwise, the only workaround would be to provide the fullpath as a field using this https://fscrawler.readthedocs.io/en/latest/admin/fs/rest.html#additional-tags.

Could you confirm that the workaround could work for you?