Skip to content

Commit

Permalink
fix crontab in ckan_run_harvester (#229)
Browse files Browse the repository at this point in the history
* update config so command line tools can load extensions. fix crontab loading in ckan_run_harvester

* add redis config back in

* disabled plugins list in production_root_url.ini (.env values work fine and cause conflicts if they are not disabled)
Updated Dockerfile to lock pyopenssl and cryptography to the highest versions that still support x509 in the manner that the ckan harvester expects

* add ckan_home volume to run harvester so the config file is in sync with other containers

* mounted crontab file in ckan_run_harvester container as a volume
 - commented out crontab setup line in entrypoint
 - crontab file in contrib/docker needs to be owned by root:root in order to execute

* add entrypoint config to docker-compose for run harvester as the file in /docker-entrypoint.d/ckan-run-harvester-entrypoint.sh is only run once on the first container start and never again until attached volumes are cleared

---------

Co-authored-by: Scott Bruce <scott.bruce@mi.mun.ca>
  • Loading branch information
fostermh and sjbruce authored Jul 24, 2024
1 parent eb77bc1 commit bdce540
Showing 7 changed files with 38 additions and 11 deletions.
5 changes: 3 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -13,8 +13,9 @@ RUN --mount=type=cache,target=/root/.cache/pip apk add --no-cache .build-deps li
linux-headers curl-dev tiff-dev zlib-dev zstd-dev lz4-dev libarchive-dev \
libjpeg-turbo-dev libpng-dev libwebp-dev expat-dev postgresql-dev openjpeg-dev \
# For cryptography
gcc musl-dev python3-dev libffi-dev openssl-dev cargo pkgconfig py3-urllib3 py3-cryptography \
&& pip install cryptography \
gcc musl-dev python3-dev libffi-dev cargo pkgconfig py3-urllib3 \
&& pip install pyopenssl==23.2.0 \
&& pip install cryptography==38.0.4 \
# Build PROJ
&& mkdir proj \
&& apk add --no-cache cmake \
3 changes: 2 additions & 1 deletion contrib/docker/ckan-run-harvester-entrypoint.sh
100644 → 100755
Original file line number Diff line number Diff line change
@@ -17,7 +17,8 @@ abort () {
}

# setup crontab jobs
cat ./ckan/contrib/docker/crontab | crontab -
cat /srv/app/src/ckan/contrib/docker/crontab | crontab -
echo -e "crontab set to: \n$(crontab -l)"

set_environment () {
export CKAN_SITE_ID=${CKAN_SITE_ID}
1 change: 1 addition & 0 deletions contrib/docker/docker-compose.cpu_limited.yml
Original file line number Diff line number Diff line change
@@ -126,6 +126,7 @@ services:
user: root
command: /bin/bash -c "echo 'ckan_run_harvester started' && crond -f 2>&1 "
ports: []
entrypoint: /docker-entrypoint.d/ckan-run-harvester-entrypoint.sh
volumes:
- ./ckan-run-harvester-entrypoint.sh:/docker-entrypoint.d/ckan-run-harvester-entrypoint.sh
deploy:
5 changes: 4 additions & 1 deletion contrib/docker/docker-compose.local.yml
Original file line number Diff line number Diff line change
@@ -90,10 +90,13 @@ services:
ckan_run_harvester:
<<: *ckan_app
container_name: ckan_run_harvester
entrypoint: /ckan-run-harvester-entrypoint.sh
entrypoint: /docker-entrypoint.d/ckan-run-harvester-entrypoint.sh
user: root
command: /bin/bash -c "echo 'ckan_run_harvester started' && cron -f 2>&1 "
ports: []
volumes:
- ckan_home:/srv/app
- ./ckan-run-harvester-entrypoint.sh:/docker-entrypoint.d/ckan-run-harvester-entrypoint.sh
depends_on:
- ckan
- ckan_gather_harvester
3 changes: 3 additions & 0 deletions contrib/docker/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -112,8 +112,11 @@ services:
user: root
command: /bin/bash -c "echo 'ckan_run_harvester started' && crond -f 2>&1 "
ports: []
entrypoint: /docker-entrypoint.d/ckan-run-harvester-entrypoint.sh
volumes:
- ckan_home:/srv/app
- ./ckan-run-harvester-entrypoint.sh:/docker-entrypoint.d/ckan-run-harvester-entrypoint.sh
- ./crontab:/etc/crontabs/root
healthcheck:
test: ["CMD-SHELL", "ps -o args | grep -q '^crond -f'"]
start_period: 5s
22 changes: 15 additions & 7 deletions contrib/docker/production_root_url.ini
Original file line number Diff line number Diff line change
@@ -125,18 +125,26 @@ ckan.plugins = envvars
# wmts_view
# ckan_spatial_harvester
# datastream_harvester
#geonetwork_harvester
# geonetwork_harvester

# module-path:file to schemas being used
# scheming.dataset_schemas = ckanext.scheming:cioos_siooc_schema.json
# scheming.presets = ckanext.scheming:presets.json
# ckanext.fluent:presets.json
# scheming.dataset_fallback = true
# scheming.organization_schemas = ckanext.scheming:organization.json
# scheming.group_schemas = ckanext.scheming:group.json
scheming.dataset_schemas = ckanext.scheming:cioos_siooc_schema.json
scheming.presets = ckanext.scheming:presets.json
ckanext.fluent:presets.json
scheming.dataset_fallback = true
scheming.organization_schemas = ckanext.scheming:organization.json
scheming.group_schemas = ckanext.scheming:group.json


## Redis Settings

# URL to your Redis instance, including the database to be used.
ckan.redis.url = redis://redis:6379/1

ckan.harvest.mq.type = redis
ckan.harvest.mq.hostname = redis
ckan.harvest.mq.port = 6379
ckan.harvest.mq.redis_db = 1
ckan.harvest.log_scope = 0
ckan.harvest.log_timeframe = 30
ckan.harvest.log_level = info
10 changes: 10 additions & 0 deletions contrib/docker/release_notes/cioos_1.6.0.md
Original file line number Diff line number Diff line change
@@ -63,6 +63,16 @@ sudo chmod +x /usr/local/bin/docker-compose
sudo docker-compose --version
```


You will need to allow execution of the entrypoint files so that docker can run them on container start

```
cd ~/ckan/contrib/docker
chmod 775 ./ckan-entrypoint.sh
chmod 775 ./ckan-harvester-entrypoint.sh
chmod 775 ./ckan-run-harvester-entrypoint.sh
```

## Recreate CKAN container
```
sudo docker kill datapusher

0 comments on commit bdce540

Please sign in to comment.