Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mpi on candide #661

Open
wants to merge 284 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
284 commits
Select commit Hold shift + click to select a range
07cafb4
Testing dockerfile with only conda env installed
martinkilbinger Nov 15, 2023
92a999c
Fixing numpy and astropy versions
martinkilbinger Nov 15, 2023
2b6600d
installation on canfar working: removed most versions from yml file; …
martinkilbinger Nov 17, 2023
b6e5c97
Dockerfile for image 0:7, 0:8
martinkilbinger Nov 17, 2023
d3aeccb
removed pysap check for example runner
martinkilbinger Nov 18, 2023
bc66565
Installation of sextractor and psfex with conda
martinkilbinger Nov 19, 2023
0b3aac6
script to call curl with canfar contained updated, adding NCORE as ar…
martinkilbinger Nov 19, 2023
69317cb
added init scripts for canfar; job_sp added n_smp for further jobs
martinkilbinger Nov 19, 2023
d35bd69
script to call curl for canfar container: added resources
martinkilbinger Nov 19, 2023
d93a60e
testing input numbers
martinkilbinger Nov 19, 2023
fb83a27
removed debug msg
martinkilbinger Nov 20, 2023
e33972b
numpy error fixed with version upgrade
martinkilbinger Nov 20, 2023
edbec76
Merge branch 'science_portal_run' of github.com:martinkilbinger/shape…
martinkilbinger Nov 20, 2023
9aa16be
Getting ready for exclusive-one-tile processing
martinkilbinger Nov 22, 2023
e5d0335
Added command line option to specify exclusive ID for processing
martinkilbinger Nov 22, 2023
49d8880
Merge remote-tracking branch 'origin/exclusive' into science_portal_run
martinkilbinger Nov 22, 2023
d02347e
curl script exclusive ID
martinkilbinger Nov 24, 2023
12e6b63
Dockerfile conda -> source activate
martinkilbinger Nov 24, 2023
7d20699
Merge remote-tracking branch 'origin/docker_image' into docker_image
martinkilbinger Nov 24, 2023
1d3df1a
Merge branch 'docker_image' of github.com:martinkilbinger/shapepipe-1…
martinkilbinger Nov 24, 2023
b05f2a5
comment added
martinkilbinger Nov 24, 2023
45942d5
Updated Dockerfile
martinkilbinger Nov 24, 2023
7d1891e
running with exclusive ID
martinkilbinger Nov 24, 2023
02affa8
exp runs
martinkilbinger Nov 25, 2023
d8ad328
local curl script NCORES -> 1
martinkilbinger Nov 25, 2023
60c3cd1
Renamed science-portal scripts (local/remote)
martinkilbinger Nov 28, 2023
d331171
added BigMac depdendency for macOS
sfarrens Nov 30, 2023
e335922
activate dev environment
sfarrens Dec 1, 2023
0737c0f
trying to run Pi
martinkilbinger Dec 1, 2023
7dba136
Merge remote-tracking branch 'origin/science_portal_run' into exclusive
martinkilbinger Dec 1, 2023
c65d75a
numpy -> 1.22 to avoid asscalar bug
martinkilbinger Dec 1, 2023
2a7fab4
added cs_util to dev env
sfarrens Dec 1, 2023
1e7b633
aux script to create links for exposure output runs for tile
martinkilbinger Dec 3, 2023
b006368
Merge remote-tracking branch 'origin/science_portal_run' into exclusive
martinkilbinger Dec 3, 2023
85f0c99
Added aux script to update runs log file
martinkilbinger Dec 3, 2023
d0c3d9f
update runs log file script: deal with multiple runs of same module
martinkilbinger Dec 3, 2023
323d44a
run_log: added function get_all_dirs
martinkilbinger Dec 4, 2023
dedbc3f
Fixed new function get_all_dirs
martinkilbinger Dec 4, 2023
e67f549
Fixed (as for MCCD) FITS key bug
martinkilbinger Dec 4, 2023
55677e7
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 4, 2023
b6cb27d
vignet makers: can use last and all in additional input inage directo…
martinkilbinger Dec 4, 2023
096af60
Dockerfile + jupyter, activate
martinkilbinger Dec 8, 2023
3c37250
file handler raises error if no process
martinkilbinger Dec 8, 2023
667eb7b
combine mask outputs
martinkilbinger Dec 8, 2023
bc20ab2
canfar curl command: added kind (tile, exp) as option
martinkilbinger Dec 13, 2023
1b4b3bf
curl remote job script init_run_exclusive_canfar: command line option…
martinkilbinger Dec 13, 2023
bd82f95
curl local command: added -k kind
martinkilbinger Dec 14, 2023
33ea4a1
summary missing ID 32 fixed (?)
martinkilbinger Dec 15, 2023
50df9a7
canfar scripts command line options
martinkilbinger Dec 15, 2023
ba17ebc
curl canfar local script added job, kind
martinkilbinger Dec 15, 2023
394e464
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 15, 2023
94428b4
Merge pull request #4 from martinkilbinger/science_portal_run
martinkilbinger Dec 15, 2023
9ac9dc1
SP ngmxix (job 128) running on canfar
martinkilbinger Dec 17, 2023
1d25a8b
curl canfar local script updated
martinkilbinger Dec 17, 2023
3d2b127
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 17, 2023
ca6b602
curl scripts updated
martinkilbinger Dec 19, 2023
cf69907
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 19, 2023
c643d0c
run summary more OO
martinkilbinger Dec 19, 2023
523b10f
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 19, 2023
53d9d35
update_runs_log_file script: fixed bug when run dir is empty
martinkilbinger Dec 21, 2023
ce1179e
improved canfar job scripts; fixed some make cat bugs
martinkilbinger Dec 21, 2023
a005119
added curl to Dockerimage
martinkilbinger Dec 21, 2023
64f04cf
Merge remote-tracking branch 'origin/exclusive' into exclusive
martinkilbinger Dec 21, 2023
22ee0f3
Remove temp hack from mask
martinkilbinger Dec 21, 2023
c62861d
Merge branch 'exclusive' of github.com:martinkilbinger/shapepipe-1 in…
martinkilbinger Dec 21, 2023
b682e7c
removed unused code from mask
martinkilbinger Dec 21, 2023
9b9dbf1
Merge pull request #3 from martinkilbinger/exclusive
martinkilbinger Dec 21, 2023
dd57fd8
Merge remote-tracking branch 'origin/develop' into develop
martinkilbinger Dec 21, 2023
7fc3645
curl scripts updated
martinkilbinger Dec 23, 2023
15a4b7b
combine psf validation files: preles now with prepare_tiles_for_final…
martinkilbinger Dec 25, 2023
0d56e23
Update post_processing.md
martinkilbinger Dec 25, 2023
fde5982
prepare tiles script loop tests
martinkilbinger Dec 25, 2023
9c5cad4
Merge branch 'p3' of github.com:martinkilbinger/shapepipe-1 into p3
martinkilbinger Dec 25, 2023
ac0a339
curl canfar local script minor change
martinkilbinger Dec 25, 2023
4048bf1
added vos doc md file
martinkilbinger Dec 25, 2023
0a3f80e
Update post_processing.md
martinkilbinger Dec 25, 2023
d81bdf2
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
672001d
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
cc96a0c
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
87bb2b7
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
76c08a6
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
faa0836
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
85f6d91
Update vos_retrieve.md
martinkilbinger Dec 25, 2023
4f81635
combine runs script renamed
martinkilbinger Dec 25, 2023
5574393
Merge branch 'p3' of github.com:martinkilbinger/shapepipe-1 into p3
martinkilbinger Dec 25, 2023
d4cb47c
Update post_processing.md
martinkilbinger Dec 25, 2023
2ffba3b
P3 proceesing to final cat
martinkilbinger Dec 26, 2023
67c966c
Merge branch 'p3' of github.com:martinkilbinger/shapepipe-1 into p3
martinkilbinger Dec 26, 2023
efb2799
Merge branch 'develop' into p3
martinkilbinger Dec 26, 2023
1f36bf5
Merge pull request #5 from martinkilbinger/p3
martinkilbinger Dec 26, 2023
9f81502
config files updated
martinkilbinger Jan 6, 2024
6eb11cf
PSFEx interp runner: allowing all: for ME_DOT_PSF_PDIRS
martinkilbinger Jan 15, 2024
485da58
init run exc script: added -d option; updates
martinkilbinger Jan 15, 2024
ef3a2c9
job sp canfar script: remove old vos upload code
martinkilbinger Jan 15, 2024
a20cb4f
summary create library and param files
martinkilbinger Jan 15, 2024
29b463b
minor changes
martinkilbinger Jan 15, 2024
cb67e8f
curl local script: now working with -e ID and -f file_IDs, in dry and…
martinkilbinger Jan 15, 2024
f0acd65
summary: fixed main path, verbose
martinkilbinger Jan 16, 2024
1bc602b
Started canfar howto
martinkilbinger Jan 16, 2024
d6ff437
Added summary run notebook
martinkilbinger Jan 16, 2024
1a81ae3
psfex_interp: continue instead of error if one of the .psf files not …
martinkilbinger Jan 17, 2024
93d6c6e
fixed symlink config files
martinkilbinger Jan 17, 2024
9c0b9a5
Updated summary run
martinkilbinger Jan 17, 2024
b19ad03
revert to main psfex in link exp for tiles script
martinkilbinger Jan 17, 2024
af38556
curl job script: remoging session logs
martinkilbinger Jan 17, 2024
8c9921f
comments
martinkilbinger Jan 17, 2024
18d104c
Merge pull request #6 from martinkilbinger/P7
martinkilbinger Jan 17, 2024
4276fea
Update canfar.md
martinkilbinger Jan 19, 2024
3f15533
Update canfar.md
martinkilbinger Jan 19, 2024
cfefcfb
Update canfar.md
martinkilbinger Jan 19, 2024
24480e2
Update canfar.md
martinkilbinger Jan 19, 2024
48e6114
Merge pull request #8 from martinkilbinger/martinkilbinger-patch-1
martinkilbinger Jan 19, 2024
cd7c61b
Merge pull request #7 from martinkilbinger/martinkilbinger-canfar-doc
martinkilbinger Jan 19, 2024
9a964c5
updated canfar doc
martinkilbinger Jan 26, 2024
e0100ca
script with akaha lib to count headlerss jobs
martinkilbinger Jan 26, 2024
984ec90
cleaned up curl submit script
martinkilbinger Jan 26, 2024
842e382
minor modifs to summary
martinkilbinger Jan 26, 2024
d9846e9
Removed VM_HOME; jon sp canfar cleaned up
martinkilbinger Jan 26, 2024
c5deb34
Dockerfile cleand up
martinkilbinger Jan 30, 2024
d507780
summary params minor bug fixed
martinkilbinger Feb 1, 2024
b4e0aa0
major bugx fixed: N_SMP was n_SMP, not propagated
martinkilbinger Feb 1, 2024
f8a07d7
major bugx fixed: N_SMP was n_SMP, not propagated
martinkilbinger Feb 1, 2024
30116b8
summary nb
martinkilbinger Feb 1, 2024
9e44fa6
sumamry updated
martinkilbinger Feb 11, 2024
3ab923f
terminal title from within curl script
martinkilbinger Feb 11, 2024
ee5b3ad
Update tiles_P7.txt
martinkilbinger Feb 11, 2024
79cb00a
Merge pull request #9 from martinkilbinger/P7
martinkilbinger Feb 11, 2024
92e4308
merged Dockerfile from docker_image
martinkilbinger Feb 11, 2024
155f2fe
added COSMOS tile list
martinkilbinger Feb 23, 2024
a86b6fd
Merge remote-tracking branch 'origin/develop' into develop
martinkilbinger Feb 23, 2024
1d928d4
monitoring and job handling scripts; small modifs, output; canfar pip…
martinkilbinger Feb 23, 2024
403e8c5
Merge pull request #10 from martinkilbinger/curl
martinkilbinger Feb 23, 2024
c56dae0
Added (back) from v1 exp mccd ini file
martinkilbinger Feb 23, 2024
a6d8b52
combine run script: flag_tile and flag_exp as diff options
martinkilbinger Mar 5, 2024
1a2d47d
mask runer excluding processed files working
martinkilbinger Mar 6, 2024
de82961
combine runs for masks: changed output target dir name
martinkilbinger Mar 6, 2024
33b5b75
Added COSMOS tile list
martinkilbinger Mar 6, 2024
8928c86
summary bug fixed
martinkilbinger Mar 6, 2024
fc5a7df
comment removed
martinkilbinger Mar 6, 2024
e882df3
style
martinkilbinger Mar 6, 2024
7cdcd73
Merge pull request #11 from martinkilbinger/mask
martinkilbinger Mar 6, 2024
fde8fe9
Added DR5 tile list file
martinkilbinger Mar 6, 2024
802a8cf
t branMerge remote-tracking branch 'origin/develop' into develop
martinkilbinger Mar 6, 2024
2fb58eb
Merge remote-tracking branch 'upstream/develop' into develop
martinkilbinger Mar 6, 2024
160c1b4
Merge remote-tracking branch 'origin/develop' into docker_image
martinkilbinger Mar 6, 2024
93e86e9
Delete scripts/sh/init_canfar.sh
martinkilbinger Mar 6, 2024
8b92dbf
Merge pull request #13 from martinkilbinger/docker_image
martinkilbinger Mar 6, 2024
fe1eb3a
Merge remote-tracking branch 'origin/develop' into develop
martinkilbinger Mar 6, 2024
720afe9
Merge remote-tracking branch 'origin/develop' into plot_spectro
martinkilbinger Mar 6, 2024
07a5924
Merge pull request #14 from martinkilbinger/plot_spectro
martinkilbinger Mar 6, 2024
4ce26ba
Merge remote-tracking branch 'origin/develop' into develop
martinkilbinger Mar 6, 2024
a6bf8a5
remove centos ci
martinkilbinger Mar 6, 2024
339f13f
Merge branch 'develop' of github.com:martinkilbinger/shapepipe-1 into…
martinkilbinger Mar 6, 2024
bddc491
repaired plot spectro notebook
martinkilbinger Mar 6, 2024
4dffd47
added doc string
martinkilbinger Mar 6, 2024
41cfd97
Merge branch 'develop' of github.com:martinkilbinger/shapepipe-1 into…
martinkilbinger Mar 6, 2024
5675c06
python example runner: added back mccd and pysap
martinkilbinger Mar 6, 2024
14569b2
corrected typo in doc string
martinkilbinger Mar 6, 2024
b406438
updated canfar pipeline doc
martinkilbinger Mar 13, 2024
fe66831
fixed combined mask input dir
martinkilbinger Mar 13, 2024
282d76b
added parameters to final cat
martinkilbinger Mar 13, 2024
9ac6bb4
summary
martinkilbinger Mar 13, 2024
25ef4e2
Removed rarely used ngmix njob/ngal options; splits computed automati…
martinkilbinger Mar 13, 2024
75fb975
canfar init + run job: removed prev Ms and Mc dirs for job 256
martinkilbinger Mar 13, 2024
376bbf1
merge sep cats: error if ngmix #HDUs < 6
martinkilbinger Mar 13, 2024
06564a1
Merge pull request #15 from martinkilbinger/cleanup_canfar
martinkilbinger Mar 13, 2024
21deebc
env-dev updated to min versions on canfar (docker image SP 1.1)
martinkilbinger Apr 3, 2024
bb15d08
Merge remote-tracking branch 'upstream' into 651-conda-activate-fix-MK
martinkilbinger Apr 4, 2024
1ea5d4a
Merge remote-tracking branch 'origin/develop' into 651-conda-activate…
martinkilbinger Apr 4, 2024
f2a5a06
minor changes
Apr 10, 2024
df90225
Improved error msg for empty process list and -e flag
martinkilbinger May 14, 2024
b7cf7ad
canfar job descr and summary updated
martinkilbinger May 20, 2024
df983c4
fixed err msg
martinkilbinger May 20, 2024
cca5694
config files updated
martinkilbinger May 20, 2024
6b2f498
summary updated
martinkilbinger May 20, 2024
86816d6
spread model: added check on non-positive object size
martinkilbinger May 20, 2024
1e5f83d
no change
martinkilbinger May 20, 2024
182ff22
added checks
martinkilbinger May 20, 2024
9827d29
scripts updated
martinkilbinger May 20, 2024
a872dfd
split exp runner: added checks and err msg for data integrity
martinkilbinger May 21, 2024
d445932
added Mh cfg file
martinkilbinger May 21, 2024
f91cc10
summary v1.2: split_exp using exposures, no longer shdu
martinkilbinger May 21, 2024
4c9f938
Added script to convert pix to world coordinates of PSF measurements
May 28, 2024
66253b1
Merge branch 'v1.4' of github.com:martinkilbinger/shapepipe-1 into v1.4
martinkilbinger May 28, 2024
89622db
merge sep module: fixed cat handling if not sufficient HDUs on input
martinkilbinger May 30, 2024
badca69
summary: split Ms and Mh; writing/reading list_exp to file for sped-up
martinkilbinger May 30, 2024
fd6bb21
improved log and error msg
martinkilbinger May 30, 2024
760c351
tile processing: using all split exp inputs, not only last
martinkilbinger May 30, 2024
79abdc8
convert psf script err msg
martinkilbinger May 30, 2024
03e8150
local Mh handling; 256 512 split
martinkilbinger May 30, 2024
01fea37
psf converting from val input to merge stars working (not yet plotting)
martinkilbinger Jun 3, 2024
1af8b49
summary script fixed bitwise & type
martinkilbinger Jun 3, 2024
5468a5c
init canfar script: added DISPLAY to avoid setools qt bug
martinkilbinger Jun 3, 2024
6bf14e0
trying job name
martinkilbinger Jun 3, 2024
da379f1
merge starcat mod: added input HDU number
martinkilbinger Jun 3, 2024
6c4029b
get IDs of running jobs
martinkilbinger Jun 11, 2024
38719f5
merge star cats w/o plotting; psf conv stars
martinkilbinger Jun 11, 2024
e9e76af
curl&init scrips: remove -k kind options, determined automatically ac…
martinkilbinger Jun 11, 2024
af5f1db
minor bug fixes, messages
martinkilbinger Jun 11, 2024
b2ceed2
combine run for psf_conv updated input dir
martinkilbinger Jun 11, 2024
d61e4c5
merge star cat: set SExcat to False, so output in HDU 1
martinkilbinger Jun 11, 2024
e679935
added gfortran to Docker; pytest to env
martinkilbinger Jun 11, 2024
65d4a68
rho stats using shear_psf_leakage
Jun 14, 2024
69d78fb
canfar doc updated
martinkilbinger Jun 14, 2024
2f11260
cleaned up rho stats code
martinkilbinger Jun 14, 2024
a92236b
rho stats testing
martinkilbinger Jun 14, 2024
67a5d44
added -p psf to curl and init scripts (to test\!); -m MH
martinkilbinger Jun 14, 2024
8436d16
Merge branch 'v1.4' of github.com:martinkilbinger/shapepipe-1 into v1.4
martinkilbinger Jun 14, 2024
5109bed
cgf file INPUT updated
martinkilbinger Jun 14, 2024
5226a0a
rho stats plots added UNIONS option (shear_psf_leakage)
martinkilbinger Jun 15, 2024
5a3e89a
plot rho stats for converted star cat
martinkilbinger Jun 16, 2024
f5468d7
ran black; pipeline canfar minor update
Jun 19, 2024
a64ff7f
small modifs to make thing smoother
martinkilbinger Jun 25, 2024
124decc
canfar scripts: -m 1 creates local exp headers indep of job
martinkilbinger Jun 26, 2024
2ae3d35
summary check special working
martinkilbinger Jun 26, 2024
9f4e7fb
ngmix: added last check that all 8 HDUs are present
martinkilbinger Jun 26, 2024
fe69e4d
debug messages removed
martinkilbinger Jun 26, 2024
93eb86a
Merge branch 'v1.4' of github.com:martinkilbinger/shapepipe-1 into v1.4
martinkilbinger Jun 26, 2024
1c75779
Added GaAP photometry to SEx param file
martinkilbinger Jul 1, 2024
26c1b78
summary specials
martinkilbinger Jul 1, 2024
c0e5435
print msg if empty vignets
martinkilbinger Jul 1, 2024
a7203a3
doc
martinkilbinger Jul 1, 2024
7f842a3
cleaning up and adding doc strings to vignet maker
martinkilbinger Jul 1, 2024
ee4ef4f
added clarifying comments to vignet maker
martinkilbinger Jul 1, 2024
1c8b4cd
cleaned up curl script
martinkilbinger Jul 12, 2024
3210b88
canfar scripts adopted to new env; added debug
martinkilbinger Jul 12, 2024
ae97110
shapepipe_run script adopted to new canfar env
martinkilbinger Jul 12, 2024
ebb40b1
setools matplotlib Agg to avoid display error
martinkilbinger Jul 12, 2024
ca026ef
ngmix: catching missing keys in postage stamp sqlite database files
martinkilbinger Aug 29, 2024
126be96
combin runs script: added tile_detection
martinkilbinger Aug 29, 2024
f60a5f0
Merge branch 'v1.4' of github.com:martinkilbinger/shapepipe-1 into v1.4
martinkilbinger Aug 29, 2024
08d7a7c
cleaned up convery pix script
martinkilbinger Oct 8, 2024
8b4619c
summary added special, n_sh as variable
martinkilbinger Oct 8, 2024
e5c8046
debugging curl canfar
martinkilbinger Oct 8, 2024
c8f657a
clarified doc on merging star cat
martinkilbinger Oct 8, 2024
e0eb0f4
Added checks and improved messages for keys in FITS files
martinkilbinger Oct 8, 2024
ba3a0d8
debugging curl canfar
martinkilbinger Oct 8, 2024
ca7f3e5
Pi psfex HDU number changed
martinkilbinger Oct 8, 2024
e219400
cosmetic changes
martinkilbinger Oct 8, 2024
34edaaf
doc canfar: options sp_local=0,1
martinkilbinger Oct 21, 2024
96ec184
split image config more flexible
martinkilbinger Oct 21, 2024
45384c9
init_curl script: allowing for individual -j 2, 8 (sp_local=1)
martinkilbinger Oct 21, 2024
925ae5b
Merge remote-tracking branch 'upstream/develop' into 651-conda-activa…
martinkilbinger Oct 21, 2024
c6be9cc
Merge branch '651-conda-activate-fix-MK' into v1.4
martinkilbinger Oct 21, 2024
f1ed5a8
added vos
martinkilbinger Oct 21, 2024
f79e0bf
Merge remote-tracking branch 'origin/v1.4' into tiles_202402
martinkilbinger Oct 21, 2024
e2ec9fa
Merge remote-tracking branch 'origin/tiles_202402' into v1.4
martinkilbinger Oct 21, 2024
161d3ca
Merge remote-tracking branch 'origin/v1.4' into cosmos
martinkilbinger Oct 21, 2024
27dc9d2
Merge remote-tracking branch 'origin/v1.4' into mpi
martinkilbinger Oct 21, 2024
805fe03
file_handler updated
martinkilbinger Oct 21, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions .github/workflows/ci-dev.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: CI

on:
pull_request:
branches:
- develop

jobs:

test-full:
name: Full Test Suite
runs-on: ${{ matrix.os }}

strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest]
python-version: [3.8]

steps:
- uses: actions/checkout@v2

- name: Install Linux dependencies
if: runner.os == 'Linux'
run: sudo apt-get install libopenblas-dev

- name: Install macOS Dependencies
shell: bash -l {0}
if: runner.os == 'macOS'
run: |
brew tap sfarrens/sf
brew install bigmac libomp

- name: Set up conda
uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
python-version: ${{ matrix.python-version }}
auto-activate-base: true

- name: Install package
shell: bash -l {0}
run: |
./install_shapepipe --env-dev --develop

- name: Run tests
shell: bash -l {0}
run: |
conda activate shapepipe-dev
python setup.py test
shapepipe_run -c example/config.ini
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ on:
branches:
- main
- master
- develop

jobs:

Expand Down
41 changes: 41 additions & 0 deletions .github/workflows/deploy-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: Create and publish a Docker image

on:
push:
branches: ['master']

env:
REGISTRY: ghcr.io
IMAGE_NAME: ${{ github.repository }}

jobs:
build-and-push-image:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Log in to the Container registry
uses: docker/login-action@f054a8b539a109f9f41c372932f1ae047eff08c9
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@98669ae865ea3cffbcbaa878cf57c20bbf1c6c38
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}

- name: Build and push Docker image
uses: docker/build-push-action@ad44023a93711e3deb337508980b4b5e9bcdc5dc
with:
context: .
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
57 changes: 57 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
FROM continuumio/miniconda3

LABEL Description="ShapePipe Docker Image"
ENV SHELL /bin/bash

ARG CC=gcc-9
ARG CXX=g++-9

# gcc < 10 is required to compile ww
ENV CC=gcc-9
ENV CXX=g++-9

RUN apt-get update --allow-releaseinfo-change && \
apt-get update && \
apt-get upgrade -y && \
apt-get install apt-utils -y && \
apt-get install make -y && \
apt-get install automake -y && \
apt-get install autoconf -y && \
apt-get install gcc-9 g++-9 -y && \
apt-get install gfortran -y && \
apt-get install locales -y && \
apt-get install libgl1-mesa-glx -y && \
apt-get install xterm -y && \
apt-get install cmake protobuf-compiler -y && \
apt-get install libtool libtool-bin libtool-doc -y && \
apt-get install libfftw3-bin libfftw3-dev -y && \
apt-get install libatlas-base-dev liblapack-dev libblas-dev -y && \
apt-get install vim -y && \
apt-get install locate -y && \
apt-get install curl -y && \
apt-get install acl -y && \
apt-get install sssd -y && \
apt-get clean

ADD nsswitch.conf /etc/

RUN sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && \
locale-gen
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

SHELL ["/bin/bash", "--login", "-c"]

COPY ./environment.yml ./
COPY install_shapepipe README.rst setup.py setup.cfg ./
RUN touch ./README.md

RUN conda update -n base -c defaults conda -c defaults
RUN conda env create --file environment.yml

COPY shapepipe ./shapepipe
COPY scripts ./scripts

RUN source activate shapepipe
#RUN pip install jupyter
9 changes: 9 additions & 0 deletions auxdir/CFIS/tiles_202402/tiles_COSMOS.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
299.183
300.183
301.183
299.184
300.184
301.184
299.185
300.185
301.185
Loading
Loading