-
Notifications
You must be signed in to change notification settings - Fork 0
/
README
30 lines (21 loc) · 1.21 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
1) logon to hcp website and register to receive data: http://www.humanconnectomeproject.org/data/
-get Username, Access Key ID, and Secret Access Key ID
2) installation
-install anaconda
-create new anaconda environment
-activate new anaconda environment
-install aws command line tools: conda install -c conda-forge awscli
3) Configure AWS
- run: aws configure
-put in access key id, secret access key id, ‘us-east-1’, ‘text’
4) Customizing batch_download script
- 3 places to change
1) header - variables for paths
2) files_to_download - list of dictionaries of files to download
3) end of download_files() loop/function - printing for each loop and final printing
5) Directory structure
- batch_download.py assumes that there is a directory brain_data (specify path in header of script) that exists. Create if one does not exist. The script will create a directoy for each patient if one does not already exist (name of dir is patient_i)
=============
To recursively search for files on aws:
aws s3 ls --recursive hcp-openaccess/HCP_1200/ | grep ‘pattern-to-search’
-https://stackoverflow.com/questions/41179573/how-to-grep-into-files-stored-in-s3