Skip to content

Commit

Permalink
commit
Browse files Browse the repository at this point in the history
  • Loading branch information
@sirbastiano
sirbastiano authored Mar 23, 2024
1 parent 4ad3af2 commit 436e4a6
Show file tree
Hide file tree
Showing 3 changed files with 115 additions and 130 deletions.
147 changes: 17 additions & 130 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,25 @@
![GitHub issues](https://img.shields.io/github/issues/ESA-PhiLab/PyRawS?style=flat-square)
![GitHub pull requests](https://img.shields.io/github/issues-pr/ESA-PhiLab/PyRawS?style=flat-square)
[![Tests](https://github.com/ESA-PhiLab/PyRawS/actions/workflows/run_tests.yml/badge.svg)](https://github.com/ESA-PhiLab/PyRawS/actions/workflows/run_tests.yml)
![Python 3.8](https://img.shields.io/badge/python-3.8-blue.svg)
![Python 3.9](https://img.shields.io/badge/python-3.9-blue.svg)
![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)
[![PyPI](https://img.shields.io/pypi/v/pyraws.svg)](https://pypi.org/project/pyraws/)
[![Docs](https://img.shields.io/badge/Docs-GitHub%20Pages-blue)](https://esa-philab.github.io/PyRawS/)

*(Disclaimer: This project is currently under development.)*

# PyRawS
![Alt Text](resources/images/PyRawS_logo.png)
<p align="center">
<img src="resources/images/PyRawS_logo.png" alt="Alt Text" width="450"/>
</p>

## News & Updates

- [ ] [#23](https://github.com/ESA-PhiLab/PyRawS/issues/23)
- [x] [#28](https://github.com/ESA-PhiLab/PyRawS/issues/28)
- [ ] New Readme :tada: (23/03/2024)

## About the project
`Python for RAW Sentinel-2 data (PyRawS)` is a powerful open-source Python package that provides a comprehensive set of tools for working with [Sentinel-2 Raw data](#sentinel-2-raw-data)🔬.
<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>
Expand Down Expand Up @@ -56,139 +70,12 @@ The PyRawS repository includes the following directories:


## Installation
### Pre-requirements
Before all, clone this repository. We suggest using git from CLI, execute:

``` git clone https://github.com/ESA-PhiLab/PyRawS ```


### Create the PyRawS environment
#### - On Linux
```
# For Linux, the installation is straightforward.
# You just need to run the following command from the main directory:
\bin\bash\ source pyraws_install.sh
# NB: Your data should be placed in the data directory in the main.
```

#### - On Other Os
```
# To install the environment, we suggest to use anaconda.
# You can create a dedicated conda environment by using the `environment.yml` file by running the following command from the main directory:
conda env create -f environment.yml
# To activate your environment, please execute:
conda activate PyRawS
```

Create a file `sys_cfg.py` in the directory `pyraws\pyraws`, and add two variables as follows:

``` PYRAWS_HOME_PATH="Absolute path to the main pyraws directory.```

``` DATA_PATH="Absolute path to the data directory. " ```

By default the data directory is located in PyRawS main directory.

<!-- You can also install `pyraws` [PyPy](https://www.pypy.org/) by running the following command from the main:
``` pip install pyraws ``` -->
### Docker

To use PyRawS with docker, use one of the following methods.


#### Method 1: Pull the docker image from Docker Hub

``` docker pull sirbastiano94/pyraws:latest ```

#### Method 2: Build docker image

Follow these steps:

1. Clone the repository and build the docker image by running the following command from the main directory:

``` docker build -t pyraws:latest --build-arg CACHEBUST=$(date +%s) -f dockerfile . ```

2. Run the docker image by executing:

``` docker run -it --rm -p 8888:8888 pyraws:latest ```

Install pyraws referring to the guide in [here](quickstart/INSTALLATION.md).

### Set-up for coregistration study

If you want to perform the coregistration study, you need to:

* clone the repository [SuperGlue Inference and Evaluation Demo Script](https://github.com/magicleap/SuperGluePretrainedNetwork);
* rename the subdirectory `models` to `superglue_models`;
* move `superglue_models` into `coregistration_study`.

Coregistration study results can be generated by using the `pyraws_coregistration_study.ipynb` notebook. The database `coregistration_study_db.csv` containing info on the discarded eruption events is used to generate results in the notebook.

Coregistration notebook results can be generated by using the `pyraws_coregistration_profiling.ipynb` notebook.

### Data directory

All the data used by PyRawS need to be located in a directory called `data`, containing all the different datasets. You can place the `data` directory where you want and update the `DATA_PATH` variable in the `sys_cfg.py` file with its path (please, refer to: [Set-up for coregistration study](#set-up-for-coregistration-study)).
By default the data directory is located in PyRawS main directory.
<br>
To create a specific dataset, you need to create subfolder with the dataset name (e.g., `THRAWS`) in the `data` directory specific for your dataset. Such directory shall contain the following subdirectories:

* `raw`: it will contain [decompressed raw](#sentinel-2-raw-data).
* `l1c`: it will contain [L1C](#sentinel-2-l1c-data).

> [!NOTE]
> Every [raw data](#sentinel-2-decompressed-raw-data) and [L1C](#sentinel-2-l1c-data) data shall be contained in a specific subdirectory --placed in `raw`or `l1c`-- whose name shall match one of the **ID_event** entries of the **.csv** file, located in `database` directory.
## Working with Database

This section of the respository contains an example of database (`THRAWS`) that is used by PyRawS to read and process [Sentinel-2 Raw data](#sentinel-2-raw-data) and [Sentinel-2 L1 data](#sentinel-2-l1-data) correctly. The minimal fields of the database include:

> [!WARNING]
> Additionally, make sure that all required dependencies and packages are installed before running the code in the notebook. Please note that it is important to carefully follow the instructions in the notebook to ensure that a database is created correctly and without errors.




| ID_event | Start | End | Sat | Coords (Lat, Lon) | class | Raw_useful_granules | Raw_complementary_granules | Polygon_square | Raw_files | l1c_files | bbox_list | bbox_list_merged | Polygon |
|-----------------------|-------------|-------------------|-----|----------------------|----------|---------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------|--------------------------------------------|-------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Barren_Island_00 | 28/09/2018 | 28/09/2018 23:59 | S2A | (12.28474241, 93.86212046) | eruption | [2] | [4] | POLYGON ((93.82076157486179 12.244061101857305, 93.8207488665122 12.325417348895849, 93.9034920534878 12.325417348895849, 93.90347934513821 12.244061101857305, 93.82076157486179 12.244061101857305)) | ['S2A_OPER_MSI_L0__GR_EPAE_20180929T132957_S20180928T042450_D10_N02.06', 'S2A_OPER_MSI_L0__GR_EPAE_20180929T132957_S20180928T042453_D09_N02.06', 'S2A_OPER_MSI_L0__GR_EPAE_20180929T132957_S20180928T042453_D10_N02.06', 'S2A_OPER_MSI_L0__GR_EPAE_20180929T132957_S20180928T042457_D09_N02.06', 'S2A_OPER_MSI_L0__GR_EPAE_20180929T132957_S20180928T042457_D10_N02.06'] | ['S2A_MSIL1C_20180928T040541_N0206_R047_T46PEU_20180929T135308'] | {2: [[[403, 290], [427, 284], [435, 316], [410, 322]]]} | {2: [[[403, 290], [427, 284], [435, 316], [410, 322], [403, 290]]]} | POLYGON((93.81618097217948 12.158183811523221, 93.81613704194822 12.411292126632908, 93.90810387805179 12.41129212663291, 93.90805994782052 12.158183811523221, 93.81618097217948 12.158183811523221)) |
| Barren_Island_01 | 28/10/2018 | 28/10/2018 23:59 | S2A | (12.28474241, 93.86212046) | eruption | [2] | [4] | POLYGON ((93.82076157486179 12.244061101857305, 93.8207488665122 12.325417348895849, 93.9034920534878 12.325417348895849, 93.90347934513821 12.244061101857305, 93.82076157486179 12.244061101857305)) | ['S2A_OPER_MSI_L0__GR_EPAE_20181028T064658_S20181028T042451_D10_N02.06', 'S2A_OPER_MSI_L0__GR_EPAE_20181028T064658_S20181028T042455_D09_N02.06', 'S2A_OPER_MSI_L0__GR_EPAE_20181028T064658_S20181028T042455_D10_N02.06', 'S2A_OPER_MSI_L0__GR_EPAE_20181028T064658_S20181028T042458_D09_N02.06', 'S2A_OPER_MSI_L0__GR_EPAE_20181028T064658_S20181028T042458_D10_N02.06'] | ['S2A_MSIL1C_20181028T040851_N0206_R047_T46PEU_20181028T070450'] | {2: [[[391, 59], [412, 54], [418,





* **ID_event**: ID of the event (e.g., volcanic-eruption, wildfire, not-event). All the other fields of the row are referred to that `Sentinel-2` acquisition.
* **class**: class of the event (e.g., eruption, fire, not-event). Leave it **empty**
* **Raw_useful_granules**: list of [Raw useful granules](#raw-useful-granule). Set to `None` or leave it empty if you do not know what are the [Raw useful granules](#raw-useful-granule).
* **Raw_complementary_granules**: list of [Raw complementry granules](#raw-complementary-granule). Set to `None` or leave it empty if you do not know what are the [Raw complementry granules](#raw-complementary-granule).
* **Raw_files**: list of [Raw granules](#sentinel-2-raw-granule) (**mandatory**).
* **l1c_files**: list of [L1 tiles](#sentinel-2-l1c-tile) (mandatory if you need L1C data).
* **bbox_list**: dictionary {[Raw useful granules](#raw-useful-granule) : [bounding box list for that granule]}. Set to `None` or leave it **empty** if you do not know the bounding box location.

To create a new database (e.g., `my_database_name`), please, proceed as follows:

1. Create a ".csv" file with the structure shown above and place it into the `database`subfloders (e.g., `my_db.csv`). You can use start from the [thraws_db.csv](https://github.com/ESA-PhiLab/PyRawS/-/blob/main/PyRawS/database/thraws_db.csv) database and edit it accordingly to your specification.
2. Create subdirectory `my_database_name` in the `data` subdirectory and populate it with the corresponding [Sentinel-2 Raw data](#sentinel-2-raw-data) and [Sentinel-2 L1 data](#sentinel-2-l1-data) as described in [Data directory](#data-directory).
3. Update the `DATABASE_FILE_DICTIONARY` in `PyRawS/utils/constants.py` as follows:

```DATABASE_FILE_DICTIONARY={"THRAWS" : "thraws_db.csv", "my_database_name" : "my_db.csv"}```

> [!IMPORTANT]
> In case you want to create your own database of event for your target aplication, the user should refer to the notebook called "database_creation.ipynb". This notebook contains the necessary code and instructions for creating the database. Simply follow the steps outlined in the notebook to successfully create something similar to the "THRAWS" database.
> [!TIP]
> **N.B** The creation of a database is not mandatory. However, it is strongly advisable. Indeed, without creating a database you can still open `Raw data` as described in [Open a Raw event from path](#open-a-raw-event-from-path). However, some pieces of information such as the [Raw useful granules](#raw-useful-granule) associated to a specific event, the event bounding boxes or the image class can be retrieved only when the database is set-up.

## Sentinel-2 Raw granules and events
## Sidenote: Sentinel-2 Raw granules and events

## Raw events and Raw granules
![Alt Text](resources/images/etna_00_granules.png)

Downloading [Sentinel-2 Raw data](#sentinel-2-raw-data) requires to specify a polygon surrounding the area of interest and a date. Given the pushbroom nature of the [Sentinel-2](https://sentinel.esa.int/documents/247904/685211/sentinel-2-products-specification-document) sensor, bands of data at [Raw](#sentinel-2-raw-data) level do not look at the same area (i.e., they are not registered). Therefore,to be sure to collect all the band around an event (i.e., volcanic eruptions, wildfires) rectangular polygons centered on the events of area 28x10 $km^2$ are used (white rectangular in the image above). This leads to download all the [Raw granules](#sentinel-2-raw-granule) whose reference band (`B02`) interesects the polygon area. <br> The image above shows the footprint of the all the [Sentinel-2 Raw granules](#sentinel-2-raw-granule) that are downloaded for the eruption named "`Etna_00`" in our database by using the white rectangular polygon. We define the collection of [Raw granules](#sentinel-2-raw-granule) that are downloaded for each of the rows of our database "[Sentinel-2 Raw event](#Sentinel-2-RAW-event)". <br> However, as you can see in the image above, most of the [Sentinel-2 Raw granules](#sentinel-2-raw-granule) in `Etna_00` [Sentinel-2 Raw event](#Sentinel-2-RAW-event) do not contain the volcanic eruption (big red spot) of interest (red rectangulars). Indeed, only the yellow and the pink rectangulars intersects or include part of the volcanic eruption. <br> In addition, the fact that one [Raw granule](#sentinel-2-raw-granule) intersects or include one event, this does not mean that the latter interesects or is included in all the bands of that [Raw granule](#sentinel-2-raw-granule). In particular,
Expand Down
Loading

0 comments on commit 436e4a6

Please sign in to comment.