diff --git a/README.md b/README.md index 40d1cdf9..f457fbd8 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ The `environment` directory provides documentation and materials used to run the The wis2box training is designed to be run on a local network containing all data, images and configurations. -A set of hardware (WiFi router and 3 mini PCs) is brought along to local training sessions. The hardware setup will provide a dedicate student VM with Ubuntu and docker to each participant. +A set of hardware (Wi-Fi router and 3 mini PCs) is brought along to local training sessions. The hardware setup will provide a dedicate student VM with Ubuntu and docker to each participant. A local registry mirroring Docker Hub is setup on the local hardware to reduce the time needed to download large Docker images in a low bandwidth environment. diff --git a/documentation/docs/practical-sessions/accessing-your-student-vm.md b/documentation/docs/practical-sessions/accessing-your-student-vm.md index a875c4da..3ae299bc 100644 --- a/documentation/docs/practical-sessions/accessing-your-student-vm.md +++ b/documentation/docs/practical-sessions/accessing-your-student-vm.md @@ -64,7 +64,7 @@ Your student VM has the following software pre-installed: ## Connect to your student VM on the local training network -Use the following configuration to connect your PC on the local WiFi broadcasted in the room during WIS2 training: +Use the following configuration to connect your PC on the local Wi-Fi broadcasted in the room during WIS2 training: - **SSID: WIS2-training** - **password: dataismagic!** diff --git a/documentation/docs/practical-sessions/bufr-command-line-tools.md b/documentation/docs/practical-sessions/bufr-command-line-tools.md index 3df46767..003e16bc 100644 --- a/documentation/docs/practical-sessions/bufr-command-line-tools.md +++ b/documentation/docs/practical-sessions/bufr-command-line-tools.md @@ -570,7 +570,7 @@ Inspect the output file using `bufr_ls` and confirm that the originating centre echo "export CSV2BUFR_TEMPLATES=/data/wis2box/bufr-templates" >> wis2box.env ``` - And restart the wis2box-stack: + And restart the wis2box stack: ```{.copy} python3 wis2box-ctl.py stop diff --git a/documentation/docs/practical-sessions/configuring-station-metadata.md b/documentation/docs/practical-sessions/configuring-station-metadata.md index 8ce8e568..1a1c2388 100644 --- a/documentation/docs/practical-sessions/configuring-station-metadata.md +++ b/documentation/docs/practical-sessions/configuring-station-metadata.md @@ -71,7 +71,7 @@ Token successfully created The **wis2box-webapp** provides a graphical user interface to edit station metadata. -Open the **wis2box-webapp** in your browser by navigating to `http:///wis2box-app`: +Open the **wis2box-webapp** in your browser by navigating to `http:///wis2box-webapp`: wis2box-webapp @@ -79,7 +79,7 @@ And select stations: wis2box-webapp-select-stations -When you click add 'add new station' you are asked to provide the WIGOS-station-identifier for the station you want to add: +When you click add 'add new station' you are asked to provide the WIGOS station identifier for the station you want to add: wis2box-webapp-import-station-from-oscar @@ -101,7 +101,7 @@ Go back to the station list and you will see the station you added: Please use stations from your country if possible, especially if you brought your own data. - Otherwise, you can use the following WIGOS-station-identifiers for testing purposes: + Otherwise, you can use the following WIGOS station identifiers for testing purposes: - 0-20000-0-91334 - 0-20000-0-96323 (note missing station elevation in OSCAR) @@ -157,6 +157,10 @@ You also have the option to view/update/delete the station in the **wis2box-weba Please update/delete the station metadata for one of the stations you added using the **wis2box-webapp**. +## Bulk station metadata upload + +wis2box also has the ability to perform "bulk" loading of station metadata from a CSV file. See the official [wis2box documentation](https://docs.wis2box.wis.wmo.int/en/1.0b5/reference/running/station-metadata.html) for more information. + ## Conclusion !!! success "Congratulations!" diff --git a/documentation/docs/practical-sessions/converting-synop-data-to-bufr.md b/documentation/docs/practical-sessions/converting-synop-data-to-bufr.md index 4e5cacbb..10cea36e 100644 --- a/documentation/docs/practical-sessions/converting-synop-data-to-bufr.md +++ b/documentation/docs/practical-sessions/converting-synop-data-to-bufr.md @@ -18,7 +18,10 @@ Surface weather reports from land surface stations have historically been report (00, 06, 12 and 18 UTC) and intermediate (03, 09, 15, 21 UTC) synoptic hours. Prior to the migration to BUFR these reports were encoded in the plain text FM-12 SYNOP code form. Whilst the migration to BUFR was scheduled to be complete by 2012 a large number of reports are still exchanged in the legacy -FM-12 SYNOP format. +FM-12 SYNOP format. Further information on the FM-12 SYNOP format can be found in the WMO Manual on Codes, +Volume I.1 (WMO-No. 306, Volume I.1). + +[WMO Manual on Codes, Volume I.1](https://library.wmo.int/records/item/35713-manual-on-codes-international-codes-volume-i-1) To aid with completing migration to BUFR some tools have been developed for encoding FM-12 SYNOP reports to BUFR, in this session you will learn how to use these tools as well @@ -196,7 +199,7 @@ AAXX 21121 identifier is given by the traditional station identifier with ``0-20000-0`` prepended, e.g. ``15015`` has become ``0-20000-0-15015``. -Using the station list page from the web application import the missing stations from OSCAR/Surface +Using the station list page from the web application import the missing stations from [OSCAR/Surface](https://oscar.wmo.int/surface/) into the wis2box and repeat the exercise. Three BUFR files should be generated and there should be no warnings or errors listed in the web application. diff --git a/documentation/docs/practical-sessions/downloading-and-finding-data-from-wis2.md b/documentation/docs/practical-sessions/downloading-and-finding-data-from-wis2.md index a79cb3c3..24ed46c5 100644 --- a/documentation/docs/practical-sessions/downloading-and-finding-data-from-wis2.md +++ b/documentation/docs/practical-sessions/downloading-and-finding-data-from-wis2.md @@ -31,7 +31,7 @@ pywis-pubsub subscribe --help ``` !!! note - pywis-pubsub is pre-installed on the local training environment, but can be installed from anywhere with `pip3 pywis-pubsub` + pywis-pubsub is pre-installed on the local training environment, but can be installed from anywhere with `pip3 install pywis-pubsub` Update the sample configuration (see the sections marked **TBD**) to connect to the Météo-France Global Broker: @@ -39,12 +39,10 @@ Update the sample configuration (see the sections marked **TBD**) to connect to vi ~/exercise-materials/pywis-pubsub-exercises/config.yml ``` -Open MQTT Explorer and connect to the Météo-France Global Broker. - Update the following values in the configuration: - **broker**: `mqtts://everyone:everyone@globalbroker.meteo.fr:8883` -- **subscribe_topics**: fill in one to many topics `origin/#` and `cache/#` (on separate lines) +- **subscribe_topics**: fill in one to many topics `cache/#` (on separate lines) - **storage.option.path**: add a directory to where data should be downloaded Run the `pywis-pubsub` command: @@ -91,6 +89,11 @@ pywis-pubsub subscribe -c ~/exercise-materials/pywis-pubsub-exercises/config.yml Let's use [pywiscat](https://github.com/wmo-im/pywiscat) to query the GDC +At the moment, the available GDCs are as follows: + +- Environment and Climate Change Canada, Meteorological Service of Canada: https://api.weather.gc.ca/collections/wis2-discovery-metadata +- China Meteorological Administration: https://api.weather.gc.ca/api/collections/wis2-discovery-metadata + ```bash pywiscat wis2 search --help ``` @@ -98,6 +101,10 @@ pywiscat wis2 search --help !!! note pywiscat is pre-installed on the local training environment, but can be installed from anywhere with `pip3 install pywiscat` +!!! note + by default, pywiscat connects to Canada's Global Discovery Catalogue. To set to a different catalogue, set the `PYWISCAT_GDC_URL` + environment variable + ```bash pywiscat wis2 search ``` diff --git a/documentation/docs/practical-sessions/initializing-wis2box.md b/documentation/docs/practical-sessions/initializing-wis2box.md index acf0752e..4506e4c5 100644 --- a/documentation/docs/practical-sessions/initializing-wis2box.md +++ b/documentation/docs/practical-sessions/initializing-wis2box.md @@ -173,7 +173,7 @@ Or check the content of the data-mappings.yml via WinSCP by browsing to the new !!! question - How many different keys are defined for 'data' in the data-mappings.yml file? + How many different keys are defined for `data` in the `data-mappings.yml` file? ??? success "Click to reveal answer" @@ -189,7 +189,7 @@ Or check the content of the data-mappings.yml via WinSCP by browsing to the new !!! note - The `data-mappings.yml` define the plugins used to transform your data. For more information see [data pipeline plugins in the wis2box-documentation](https://docs.wis2box.wis.wmo.int/en/latest/reference/running/data-pipeline-plugins.html) + The `data-mappings.yml` file defines the plugins used to transform your data. For more information see [data pipeline plugins in the wis2box-documentation](https://docs.wis2box.wis.wmo.int/en/latest/reference/running/data-pipeline-plugins.html) ## wis2box start and status diff --git a/environment/README.md b/environment/README.md index 739fa2f7..4734f395 100644 --- a/environment/README.md +++ b/environment/README.md @@ -1,7 +1,7 @@ # Training environment The wis2box portable training environment consists of the following components: -- One (1) Nighthawk WiFi router +- One (1) Nighthawk Wi-Fi router - Three (3) ZOTAX ZBOX Mini PCs (M Series Edge MI623) Each ZOTAX ZBOX Mini PC has been fitted with: @@ -12,30 +12,28 @@ The ZOTAX ZBOX Mini PCs are running virtualization software: ProxMox version 7.3 The 3 PCs together are setup in a ProxMox cluster to provide VMs for participants during the training. -The Nighthawk is setup to broadcast a password-protected WiFi network with SSID **WIS2-training**. +The Nighthawk is setup to broadcast a password-protected Wi-Fi network with SSID **WIS2-training**. The local network is defined by 10.0.0.1/22 Adminstrators can access training environment from the following locations: -- Nighthawk WiFi router interface at https://10.0.0.1 +- Nighthawk Wi-Fi router interface at https://10.0.0.1 - ProxMox cluster interface at https://10.0.1.1:8006 -Each VM host has a 'vm-clone-base'-template from which new VMs can be created. +Each VM host has a 'vm-clone-base' template from which new VMs can be created. The 'vm-clone-base' template consists of: - 2 vCPUs - 48 GB local storage - 4 GB RAM -- Ubuntu 20.0.4 with the following tools installed - - Docker CE - - mosquitto-clients - - python3 - - python3-pip - - unzip - - docker-compose - - minio pywis-pubsub (installed via `pip3`) -- `/etc/docker/daemon.json` with a Docker registry mirror pointing to http://10.0.2.222:5000 +- Ubuntu 22.0.4 LTS with the following system packages installed: + - Docker CE v2.21.0 + - docker compose 24.0.6 + - packages installed via `pip3`: + - minio + - pywis-pubsub + - pywiscat ### VM naming convention @@ -51,18 +49,27 @@ Student VMs: - on vm-host-2: 10.0.2.11, 10.0.2.12, 10.0.2.13 etc. - on vm-host-2: 10.0.3.11, 10.0.3.12, 10.0.3.13 etc. -local-repo-vm: 10.0.2.222 +### student VM setup -### MinIO setup +Student VMs can be setup by cloning the a base template. -MinIO is used to provide the exercise materials and documentation. The `local-repo-vm-222` VM must be started in -order to create MinIO storage. The following buckets must be created manually with `readonly` permissions: +The script `setup_student_vm.sh` can be used to by the `wmo_admin` account to create a new user account on the student VM and add the latest wis2box release to the home directory of the new user along with the exercise materials. -- `exercise-materials` -- `documentation` +The WiFi router has pre-configured mac-to-IP settings. After cloning the base template set the MAC address to match the IP using the following logic: -### student VM setup +- `00:00:00:00:01:11` -> `10.0.1.11` +- `00:00:00:00:02:12` -> `10.0.2.12` +- etc. -Student VMs can be setup by cloning the a base template. +### local DNS -The script `setup_student_vm.sh` can be used to by the `wmo_admin` account to create a new user account on the student VM and add the latest wis2box release to the home directory of the new user along with the exercise materials. +A local DNS-server is setup on a VM with IP 10.0.2.111. + +DNS server UI +http://dns-server.wis2.training:5380/ +or +http://10.0.2.111:5380/ + +The DNS server UI can be used to check the status of the DNS-server and/or to add additional A-records. The DNS server is preconfigured with A-records naming each host for each local training participants. + +The DNS-server is hosted on a VM on vm-host-2. If vm-host-2 is lost during transit or broken a backup is available on vm-host-1. If the DNS-server for some reason does not work at all, participants can just use the IP as hostname. diff --git a/environment/instructor/README.md b/environment/instructor/README.md new file mode 100644 index 00000000..704f78ce --- /dev/null +++ b/environment/instructor/README.md @@ -0,0 +1,41 @@ +# Instructor helpers + +This directory contains resources that may be valuable for instructors as part of +delivery of the training. Note that the below need to be run while connected to +the **WIS2-training** Wi-Fi network. + +# Local global services + +Local global services provide a test environment to simulate WIS2 workflows using +the classroom as a network of WIS2 nodes. + +## Global Broker + +TBD + +### Live map (`live.html`) + +Serve this webpage via HTTP on your VM. Ensure that the following line: + +```javascript +const host = 'ws://tbd.wis2.training:8884/ws'; +``` + +...is updated to point to the Global Broker installed above. + +## Global Cache + +[wis2-gc](https://github.com/wmo-im/wis2-gc) is a Reference Implememtation of +a WIS2 Global Cache (GC). Follow the setup instructions, pointing +to the Global Broker installed above. + +## Global Discovery Catalogue + +[wis2-gdc](https://github.com/wmo-im/wis2-gdc) is a Reference Implememtation of +a WIS2 Global Discovery Catalogue (GDC). Follow the setup instructions, pointing +to the Global Broker installed above. + +# Other + +* `instructor_bashrc.sh`: sample bash prompt to help with screen readability while + displaying a termnial during a practical session. diff --git a/environment/instructor_bashrc.sh b/environment/instructor/instructor_bashrc.sh similarity index 100% rename from environment/instructor_bashrc.sh rename to environment/instructor/instructor_bashrc.sh diff --git a/environment/instructor/live.html b/environment/instructor/live.html new file mode 100644 index 00000000..0651d838 --- /dev/null +++ b/environment/instructor/live.html @@ -0,0 +1,75 @@ +!DOCTYPE html> + + + wis2box training live data and metadata notifications + + + + + + + +
+

wis2box training live data and metadata notifications

+
+
+
+ Powered by WIS2 +
+ + + diff --git a/environment/local-repo-vm-222/README.md b/environment/local-repo-vm-222/README.md index 4556d952..02ef4d17 100644 --- a/environment/local-repo-vm-222/README.md +++ b/environment/local-repo-vm-222/README.md @@ -2,9 +2,9 @@ This is the stack to be run on the VM with IP 10.0.2.222 -This stack is used to make a local set of services available within the WIS2-training network: +This stack is used to make a local set of services available within the **WIS2-training** network: -- docker-registry-mirror: to cache docker-images and reduce build time +- docker-registry-mirror: to cache Docker images and reduce build time - MinIO bucket: to make training materials locally available over HTTP - mosquitto broker: to demonstrate MQTT and/or to act as potential local GB