S3, Raster Crop Optim, Vector Crop Optim (#11)

* S3 (#10) * WIP - s3 integrated & working, testing * refactored integ tests for awss3 and localfs switch * WIP - API tests to awss3 switch * updated api job tests for s3 * phantom fiona bbox bugfix * refactored image build to remove source builds for Arropw * vector gpkg processors to use new fiona method * test refactoring for S3 testing under docker * refactored vector file assertion for lower memory * better test cleanup for awss3 * redis as non root * refactored raster_crop for large file support * refactored raster_crop for large file support * updated tests for large raster support * image version bump for CI * boundary dict in execution env fix * concurrent tasks overwriting data fix * env parse bugfix awss3 instance * env parse bugfix awss3 instance * standardise output filenaming * improved processing info collection * bigtiff check * standardise output filenaming * improved processing info collection * rc image bump 0.2.6 * Add ext $ref for GeoJSON / DataPackage schemas (#13) * Revert "Add ext $ref for GeoJSON / DataPackage schemas (#13)" (#14) This reverts commit 757c989. * backend switch for tree test * updated with GeoJSON and Datapackage external schemas (#15) * updated with GeoJSON and Datapackage external schemas * formatting * added pixel-validation, refactored crop for offet bug (#17) * Add processor metadata fields (#16) - data_title: short title for data listing - data_title_long: full title for data listing - data_summary: longer description - data_citation: suggested citation string * Processor api hierarchy et al (#18) * processor endpoint hierarchy, remove nulls/unset * add data_format to Processor meta * progress step for hash generation * boundary search bugfix 0.0 and name func * ignore empty dirs in test teardown * version bump images * Set Boundary geom, envel schemas to (Multi)Polygon (#19) Previous GeoJSON schema was incorrect as the Boundary fields are instances of GeoJSON Geometry rather than GeoJSON. Setting the schemas for Boundary.geometry to MultiPolygon and Boundary.envelope to Polygon allow for the most precise and correct typing. * Isimp drought (#20) * isimp drought procesor * comment cleanup * intermittent 500 on job status bugfix * removed long-running test, which added little * boundary search targets name_long (#21) * boundary search targets name_long * boundary search targets name_long * update missing task details message * switch for hiding test processors * switch for hiding test processors * switch for hiding test processors * improve state reporting in API * sort boundaries * refactor exists flow * improve output for job state * improve output for job state * Remove indent in meta to avoid markdown code fmt (#22) When formatting Markdown, blocks of indented text are treated as code. Make sure that multi-line metadata fields inside python files don't contain leading indentation, to avoid code-block formatting in UI * refactored s3 tree list * order by name_long * docs * gitaction test workflow * resync ignore * gitaction testing * gitaction testing * gitaction testing * gitaction testing * gitaction testing * gitaction testing * gitaction testing * gitaction testing * gitaction testing * debug gitaction * debug gitaction * add processor tests to action * add path omission for gitaction * add path omission for gitaction --------- Co-authored-by: Maciej Ziarkowski <maciej.ziarkowski@gmail.com> Co-authored-by: Tom Russell <tomalrussell@gmail.com>
nismod · Mar 28, 2023 · 8c65bc6 · 8c65bc6
1 parent 0aca320
commit 8c65bc6
Show file tree

Hide file tree

Showing 75 changed files with 4,242 additions and 920 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -3,6 +3,5 @@ tests/data/packages
 tests/data/processing
 tests/data/tmp
 tests/data/global
-tests/data/gridfinder
 tests/data/countries_all.geojson
 .git
diff --git a/.github/workflows/build_dev.yaml b/.github/workflows/build_dev.yaml
@@ -6,14 +6,15 @@ on:
   push:
     branches:
       - develop
+    paths: ['api/**', 'dataproc/**', 'tests/**', 'config.py', 'Dockerfile', 'requirements.txt', '.github/**']
 
 env:
-  VERSION: 0.2.4
   REGISTRY: ghcr.io
   IMAGE_NAME: ${{ github.repository }}
+  TEST_IMAGE_TAG: test
 
 jobs:
-  build-and-push-image:
+  build-test-push-dev-image:
     runs-on: ubuntu-latest
     permissions:
       contents: read
@@ -23,6 +24,11 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v3
 
+      - name: Set variables
+        run: |
+          VER=$(cat ./VERSION)
+          echo "VERSION=$VER" >> $GITHUB_ENV
+
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v2
 
@@ -33,11 +39,24 @@ jobs:
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
-      - name: Build and push Docker image
+      - name: Build Test Docker image
+        uses: docker/build-push-action@v3
+        with:
+          context: .
+          load: true
+          tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.VERSION }}-dev-${{ env.TEST_IMAGE_TAG }}
+
+      - name: Run Test Suite
+        env:
+          AUTOPKG_VERSION: ${{ env.VERSION }}-dev-${{ env.TEST_IMAGE_TAG }}
+        run: docker-compose -f .github/workflows/test/docker-compose-ci-test.yaml run test || docker-compose -f .github/workflows/test/docker-compose-ci-test.yaml logs dataproc
+
+      - name: Build and Push Docker image
         uses: docker/build-push-action@v3
         with:
           context: .
           cache-from: type=gha
           cache-to: type=gha,mode=max
           push: true
           tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.VERSION }}-dev
+
diff --git a/.github/workflows/build_prod_and_release.yaml b/.github/workflows/build_prod_and_release.yaml
@@ -6,31 +6,16 @@ on:
   push:
     branches:
       - master
+    paths: ['api/**', 'dataproc/**', 'tests/**', 'config.py', 'Dockerfile', 'requirements.txt', '.github/**']
 
 env:
-  VERSION: 0.2.4
   REGISTRY: ghcr.io
   IMAGE_NAME: ${{ github.repository }}
+  TEST_IMAGE_TAG: test
 
 jobs:
-  release:
-    name: Create Release
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout code
-        uses: actions/checkout@v2
-      - name: Create Release
-        id: create_release
-        uses: actions/create-release@v1
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        with:
-          tag_name: v${{ env.VERSION }}
-          release_name: Release v${{ env.VERSION }}
-          draft: false
-          prerelease: false
 
-  build-and-push-image:
+  build-and-push-image-release:
     runs-on: ubuntu-latest
     permissions:
       contents: read
@@ -40,6 +25,11 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@v3
 
+      - name: Set variables
+        run: |
+          VER=$(cat VERSION)
+          echo "VERSION=$VER" >> $GITHUB_ENV
+
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v2
 
@@ -50,6 +40,18 @@ jobs:
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
 
+      - name: Build Test Docker image
+        uses: docker/build-push-action@v3
+        with:
+          context: .
+          load: true
+          tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.VERSION }}-${{ env.TEST_IMAGE_TAG }}
+
+      - name: Run Test Suite
+        env:
+          AUTOPKG_VERSION: ${{ env.VERSION }}-${{ env.TEST_IMAGE_TAG }}
+        run: docker-compose -f .github/workflows/test/docker-compose-ci-test.yaml run test || docker-compose -f .github/workflows/test/docker-compose-ci-test.yaml logs dataproc
+
       - name: Build and push Docker image
         uses: docker/build-push-action@v3
         with:
@@ -58,3 +60,14 @@ jobs:
           cache-to: type=gha,mode=max
           push: true
           tags: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ env.VERSION }}
+
+      - name: Create Release
+        id: create_release
+        uses: actions/create-release@v1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          tag_name: v${{ env.VERSION }}
+          release_name: Release v${{ env.VERSION }}
+          draft: false
+          prerelease: false
diff --git a/.github/workflows/test/.citest.env b/.github/workflows/test/.citest.env
@@ -0,0 +1,36 @@
+# DB Vars
+POSTGRES_USER=postgres
+POSTGRES_DB=ccgautopkg
+POSTGRES_HOST=db
+POSTGRES_PASSWORD=postgres
+
+# Autopkg Vars
+PYTHONPATH="${PYTHONPATH}:/usr/src/app"
+AUTOPKG_LOG_LEVEL=DEBUG
+AUTOPKG_DEPLOYMENT_ENV=test
+AUTOPKG_INTEGRATION_TEST_ENDPOINT="http://api:8000"
+AUTOPKG_INCLUDE_TEST_PROCESSORS=True
+
+# Localfs Backend
+AUTOPKG_STORAGE_BACKEND=localfs
+AUTOPKG_LOCALFS_STORAGE_BACKEND_ROOT_TEST=/usr/src/app/tests/data/packages
+AUTOPKG_LOCALFS_PROCESSING_BACKEND_ROOT_TEST=/usr/src/app/tests/data/processing
+AUTOPKG_POSTGRES_USER=postgres
+AUTOPKG_POSTGRES_PASSWORD=postgres
+AUTOPKG_POSTGRES_HOST=db
+AUTOPKG_POSTGRES_PORT=5432
+AUTOPKG_POSTGRES_DB=ccgautopkg
+
+AUTOPKG_CELERY_BROKER=redis://redis:6379
+AUTOPKG_CELERY_BACKEND=redis://redis:6379
+AUTOPKG_CELERY_CONCURRENCY=1
+AUTOPKG_TASK_LOCK_TIMEOUT=600
+AUTOPKG_TASK_EXPIRY_SECS=3600
+AUTOPKG_REDIS_HOST=redis
+AUTOPKG_PACKAGES_HOST_URL=http://localhost/packages
+
+# Raster Cropping Memory Allocation
+GDAL_CACHEMAX=64
+
+# Test Flags
+AUTOPKG_TEST_GRI_OSM="False"
diff --git a/.github/workflows/test/docker-compose-ci-test.yaml b/.github/workflows/test/docker-compose-ci-test.yaml
@@ -0,0 +1,56 @@
+version: '3'
+services:
+  db:
+    image: postgis/postgis:13-3.1-alpine
+    ports:
+      - 5432:5432
+    env_file:
+      - .citest.env
+
+  redis:
+    image: redis:6.2-alpine
+    command: redis-server --save 20 1 --loglevel debug
+
+  dataproc:
+    image: ghcr.io/nismod/irv-autopkg:${AUTOPKG_VERSION}
+    restart: always
+    depends_on:
+      - redis
+    env_file:
+      - .citest.env
+    volumes:
+      - packages:/usr/src/app/tests/data/packages
+      - processing:/usr/src/app/tests/data/processing
+    command: celery --app dataproc.tasks worker
+
+  api:
+    image: ghcr.io/nismod/irv-autopkg:${AUTOPKG_VERSION}
+    restart: always
+    depends_on:
+      - db
+      - redis
+    env_file:
+      - .citest.env
+    volumes:
+      - ./wait-for-it.sh:/opt/wait-for-it.sh
+      - packages:/usr/src/app/tests/data/packages
+      - processing:/usr/src/app/tests/data/processing
+    command: /opt/wait-for-it.sh db:5432 --timeout=20 -- uvicorn api.main:app --host 0.0.0.0 --port 8000
+
+  test:
+    image: ghcr.io/nismod/irv-autopkg:${AUTOPKG_VERSION}
+    depends_on:
+      - api
+      - dataproc
+    env_file:
+      - .citest.env
+    volumes:
+      - ./wait-for-it.sh:/opt/wait-for-it.sh
+      - ./run_tests.sh:/opt/run_tests.sh
+      - packages:/usr/src/app/tests/data/packages
+      - processing:/usr/src/app/tests/data/processing
+    command: /opt/wait-for-it.sh api:8000 --timeout=20 -- /opt/wait-for-it.sh redis:6379 --timeout=20 -- sh /opt/run_tests.sh
+
+volumes:
+  packages:
+  processing:
diff --git a/.github/workflows/test/run_tests.sh b/.github/workflows/test/run_tests.sh
@@ -0,0 +1,7 @@
+#!/bin/sh
+pip install pytest && \
+echo Executing API Tests... && \
+ls -lrt /usr/src/app/tests/data && \
+pytest -p no:warnings -v /usr/src/app/tests/api && \
+echo Executing Processor Tests... && \
+pytest -p no:warnings -v /usr/src/app/tests/dataproc
diff --git a/.github/workflows/test/test_results/.gitkeep b/.github/workflows/test/test_results/.gitkeep