Merge pull request #17 from brazil-data-cube/dev

docs: updating replication example and adding data references

.github/workflows/ci.yml

@@ -12,5 +12,5 @@ jobs:
       - uses: actions/setup-python@v2
         with:
           python-version: 3.x
-      - run: pip install mkdocs
+      - run: pip install ./docs
       - run: cd docs && mkdocs gh-deploy --force

composes/replication/docker-compose.notebook.yaml

@@ -55,6 +55,13 @@ services:
         source: ${PWD}/analysis/
         target: /home/jovyan/analysis
 
+      #
+      # 5. Volume with the mapping for the input and output directories. 
+      #
+      - type: bind
+        source: ${PWD}/analysis/data/examples/replication_example
+        target: ${PWD}/analysis/data/examples/replication_example # Must be the same path set in the `DATA_DIRECTORY` environment variable.
+
     ports:
 
       #

docs/pages/__init__.py

@@ -0,0 +1,17 @@
+#
+# This file is part of Brazil Data Cube compendium-harmonization.
+# Copyright (C) 2022 INPE.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <https://www.gnu.org/licenses/gpl-3.0.html>.
+#

docs/pages/reproducible-research/replication-example/index.en.md

@@ -112,10 +112,49 @@ For each approach, automatic commands can be used through the `Makefile`. You ca
 make replication_notebook
 ```
 
+!!! attention "Changes required"
+
+    As the replication example uses other data from that presented in the minimal example, you should modify some Jupyter Notebook parameters — specifically, the `day_diff` parameter. This parameter defines the number of days difference between images. This change should only be made for tests where only Sentinel-2 is used.
+
+    So, you must change the following sections of the Jupyter Notebook:
+
+    - `4.3.1: Searching for image pairs`
+    - `4.4.1: Searching for image pairs`
+    - `4.5.1: Searching for image pairs`
+    - `4.6.1: Searching for image pairs`
+
+    In all of them, you must change `day_diff` to `15 days`:
+
+    ```python
+    validation_funcs.search_pairs_s2(sentinel2_sceneid_list, day_diff=15)
+    ```
+
 *Dagster*
 
 ``` sh
 make replication_pipeline
 ```
 
+!!! attention "Changes required"
+
+    As the replication example uses other data from that presented in the minimal example, you should modify some Jupyter Notebook parameters — specifically, the `day_diff` parameter. This parameter defines the number of days difference between images. This change should only be made for tests where only Sentinel-2 is used.
+
+    So, you must add the following changes to the `solids` section of your [Dagster configuration file](/en/tools/utilitary):
+
+    ```yaml
+    solids:
+        validation_sr_s2_sen2cor:
+            config:
+            day_difference: 15
+        validation_sr_s2_lasrc:
+            config:
+            day_difference: 15
+        validation_nbar_s2_sen2cor:
+            config:
+            day_difference: 15
+        validation_nbar_s2_lasrc:
+            config:
+            day_difference: 15
+    ```
+
 Once the environment is chosen, the processing and data analysis can be performed.

docs/pages/reproducible-research/replication-example/index.pt.md

@@ -112,10 +112,49 @@ Para ambas abordagens adotadas, tem-se comandos de automação no `Makefile`. Se
 make replication_notebook
 ```
 
+!!! attention "Modificação necessária"
+
+    Como o exemplo de replicação utiliza dados diferentes dos apresentados no exemplo mínimo, faz-se necessário pequenas modificações nos parâmetros utilizados no Jupyter Notebook. Especificamente, o parâmetro `day_diff`. Esse parâmetro define a quantidade de dias de diferença entre imagens. Essa mudança deve ser feita apenas para testes onde faz-se o uso apenas do Sentinel-2.
+
+    Com isso, você deve alterar as seguintes seções do Jupyter Notebook:
+
+    - `4.3.1: Searching for image pairs`
+    - `4.4.1: Searching for image pairs`
+    - `4.5.1: Searching for image pairs`
+    - `4.6.1: Searching for image pairs`
+
+    Em todas elas, deve-se alterar o `day_diff` para 15 dias:
+
+    ```python
+    validation_funcs.search_pairs_s2(sentinel2_sceneid_list, day_diff=15)
+    ```
+
 *Dagster*
 
 ``` sh
 make replication_pipeline
 ```
 
-Com o ambiente escolhido, o processamento e análise dos dados pode ser realizado.
+!!! attention "Modificação necessária"
+
+    Como o exemplo de replicação utiliza dados diferentes dos apresentados no exemplo mínimo, faz-se necessário pequenas modificações nos parâmetros do pipeline. Especificamente, o parâmetro `day_diff` deve ser alterado para `15 dias`. Esse parâmetro define a quantidade de dias de diferença entre imagens. Essa mudança deve ser feita apenas para testes onde faz-se o uso apenas do Sentinel-2.
+
+    Com isso, você deve adicionar na seção `solids` de seu [arquivo de configuração Dagster](/tools/utilitary/) as seguintes mudanças:
+
+    ```yaml
+    solids:
+        validation_sr_s2_sen2cor:
+            config:
+            day_difference: 15
+        validation_sr_s2_lasrc:
+            config:
+            day_difference: 15
+        validation_nbar_s2_sen2cor:
+            config:
+            day_difference: 15
+        validation_nbar_s2_lasrc:
+            config:
+            day_difference: 15
+    ```
+
+Com o ambiente escolhido, o processamento e análise dos dados podem ser realizados.

docs/pyproject.toml

@@ -4,6 +4,7 @@ version = "0.1.0"
 description = ""
 authors = ["Your Name <you@example.com>"]
 license = "GPLv3"
+packages = [{include = "pages"}]
 
 [tool.poetry.dependencies]
 python = "^3.8"

tools/example-toolkit/config/example_files.json

@@ -1,9 +1,9 @@
 {
-  "repositoryTag": "v5.0",
+  "repositoryTag": "v0.1-data",
   "dataFiles": [
-    "https://github.com/M3nin0/reproducible-large-data/releases/download/v5.0/minimal-example_landsat8_data.zip",
-    "https://github.com/M3nin0/reproducible-large-data/releases/download/v5.0/minimal-example_lasrc_auxiliary_data.zip",
-    "https://github.com/M3nin0/reproducible-large-data/releases/download/v5.0/minimal-example_scene_id_list.zip",
-    "https://github.com/M3nin0/reproducible-large-data/releases/download/v5.0/minimal-example_sentinel2_data.zip"
+    "https://github.com/brazil-data-cube/compendium-harmonization/releases/download/v0.1-data/minimal-example_landsat8_data.zip",
+    "https://github.com/brazil-data-cube/compendium-harmonization/releases/download/v0.1-data/minimal-example_lasrc_auxiliary_data.zip",
+    "https://github.com/brazil-data-cube/compendium-harmonization/releases/download/v0.1-data/minimal-example_scene_id_list.zip",
+    "https://github.com/brazil-data-cube/compendium-harmonization/releases/download/v0.1-data/minimal-example_sentinel2_data.zip"
   ]
 }

tools/example-toolkit/config/replication_files.json

@@ -1,6 +1,9 @@
 {
-  "repositoryTag": "",
+  "repositoryTag": "v0.1-data",
   "dataFiles": [
-    ""
+    "https://github.com/brazil-data-cube/compendium-harmonization/releases/download/v0.1-data/replication-example_landsat8_data.zip",
+    "https://github.com/brazil-data-cube/compendium-harmonization/releases/download/v0.1-data/replication-example_lasrc_auxiliary_data.zip",
+    "https://github.com/brazil-data-cube/compendium-harmonization/releases/download/v0.1-data/replication-example_scene_id_list.zip",
+    "https://github.com/brazil-data-cube/compendium-harmonization/releases/download/v0.1-data/replication-example_sentinel2_data.zip"
   ]
-}
+}