Skip to content

Commit

Permalink
Jakarta editorial updates (#102)
Browse files Browse the repository at this point in the history
* Update initializing-wis2box.md

* Update configuring-station-metadata.md

* add section on bulk station metadata loading

* add section for instructor helpers

* Addresses #103, links to FM-12 definition and OSCAR/Surface added.

* editorial

* subscribe only to cache

* enumerate GDCs, fix pip3 text

* enumerate GDCs, fix pip3 text

* Update README.md

update student VM info, add information about DNS

---------

Co-authored-by: david-i-berry <dberry@wmo.int>
Co-authored-by: Maaike <maaike.limper@gmail.com>
  • Loading branch information
3 people authored Oct 18, 2023
1 parent 8bc5d48 commit 7aef174
Show file tree
Hide file tree
Showing 12 changed files with 175 additions and 38 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ The `environment` directory provides documentation and materials used to run the

The wis2box training is designed to be run on a local network containing all data, images and configurations.

A set of hardware (WiFi router and 3 mini PCs) is brought along to local training sessions. The hardware setup will provide a dedicate student VM with Ubuntu and docker to each participant.
A set of hardware (Wi-Fi router and 3 mini PCs) is brought along to local training sessions. The hardware setup will provide a dedicate student VM with Ubuntu and docker to each participant.

A local registry mirroring Docker Hub is setup on the local hardware to reduce the time needed to download large Docker images in a low bandwidth environment.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ Your student VM has the following software pre-installed:

## Connect to your student VM on the local training network

Use the following configuration to connect your PC on the local WiFi broadcasted in the room during WIS2 training:
Use the following configuration to connect your PC on the local Wi-Fi broadcasted in the room during WIS2 training:

- **SSID: WIS2-training**
- **password: dataismagic!**
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -570,7 +570,7 @@ Inspect the output file using `bufr_ls` and confirm that the originating centre
echo "export CSV2BUFR_TEMPLATES=/data/wis2box/bufr-templates" >> wis2box.env
```

And restart the wis2box-stack:
And restart the wis2box stack:

```{.copy}
python3 wis2box-ctl.py stop
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,15 +71,15 @@ Token successfully created

The **wis2box-webapp** provides a graphical user interface to edit station metadata.

Open the **wis2box-webapp** in your browser by navigating to `http://<your-host>/wis2box-app`:
Open the **wis2box-webapp** in your browser by navigating to `http://<your-host>/wis2box-webapp`:

<img alt="wis2box-webapp" src="../../assets/img/wis2box-webapp.png" width="800">

And select stations:

<img alt="wis2box-webapp-select-stations" src="../../assets/img/wis2box-webapp-select-stations.png" width="250">

When you click add 'add new station' you are asked to provide the WIGOS-station-identifier for the station you want to add:
When you click add 'add new station' you are asked to provide the WIGOS station identifier for the station you want to add:

<img alt="wis2box-webapp-import-station-from-oscar" src="../../assets/img/wis2box-webapp-import-station-from-oscar.png" width="800">

Expand All @@ -101,7 +101,7 @@ Go back to the station list and you will see the station you added:

Please use stations from your country if possible, especially if you brought your own data.

Otherwise, you can use the following WIGOS-station-identifiers for testing purposes:
Otherwise, you can use the following WIGOS station identifiers for testing purposes:

- 0-20000-0-91334
- 0-20000-0-96323 (note missing station elevation in OSCAR)
Expand Down Expand Up @@ -157,6 +157,10 @@ You also have the option to view/update/delete the station in the **wis2box-weba

Please update/delete the station metadata for one of the stations you added using the **wis2box-webapp**.

## Bulk station metadata upload

wis2box also has the ability to perform "bulk" loading of station metadata from a CSV file. See the official [wis2box documentation](https://docs.wis2box.wis.wmo.int/en/1.0b5/reference/running/station-metadata.html) for more information.

## Conclusion

!!! success "Congratulations!"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,10 @@ Surface weather reports from land surface stations have historically been report
(00, 06, 12 and 18 UTC) and intermediate (03, 09, 15, 21 UTC) synoptic hours. Prior to the migration
to BUFR these reports were encoded in the plain text FM-12 SYNOP code form. Whilst the migration to BUFR
was scheduled to be complete by 2012 a large number of reports are still exchanged in the legacy
FM-12 SYNOP format.
FM-12 SYNOP format. Further information on the FM-12 SYNOP format can be found in the WMO Manual on Codes,
Volume I.1 (WMO-No. 306, Volume I.1).

[WMO Manual on Codes, Volume I.1](https://library.wmo.int/records/item/35713-manual-on-codes-international-codes-volume-i-1)

To aid with completing migration to BUFR some tools have been developed for
encoding FM-12 SYNOP reports to BUFR, in this session you will learn how to use these tools as well
Expand Down Expand Up @@ -196,7 +199,7 @@ AAXX 21121
identifier is given by the traditional station identifier with ``0-20000-0`` prepended,
e.g. ``15015`` has become ``0-20000-0-15015``.

Using the station list page from the web application import the missing stations from OSCAR/Surface
Using the station list page from the web application import the missing stations from [OSCAR/Surface](https://oscar.wmo.int/surface/)
into the wis2box and repeat the exercise.

Three BUFR files should be generated and there should be no warnings or errors listed in the web application.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,20 +31,18 @@ pywis-pubsub subscribe --help
```

!!! note
pywis-pubsub is pre-installed on the local training environment, but can be installed from anywhere with `pip3 pywis-pubsub`
pywis-pubsub is pre-installed on the local training environment, but can be installed from anywhere with `pip3 install pywis-pubsub`

Update the sample configuration (see the sections marked **TBD**) to connect to the Météo-France Global Broker:

```bash
vi ~/exercise-materials/pywis-pubsub-exercises/config.yml
```

Open MQTT Explorer and connect to the Météo-France Global Broker.

Update the following values in the configuration:

- **broker**: `mqtts://everyone:everyone@globalbroker.meteo.fr:8883`
- **subscribe_topics**: fill in one to many topics `origin/#` and `cache/#` (on separate lines)
- **subscribe_topics**: fill in one to many topics `cache/#` (on separate lines)
- **storage.option.path**: add a directory to where data should be downloaded

Run the `pywis-pubsub` command:
Expand Down Expand Up @@ -91,13 +89,22 @@ pywis-pubsub subscribe -c ~/exercise-materials/pywis-pubsub-exercises/config.yml

Let's use [pywiscat](https://github.com/wmo-im/pywiscat) to query the GDC

At the moment, the available GDCs are as follows:

- Environment and Climate Change Canada, Meteorological Service of Canada: https://api.weather.gc.ca/collections/wis2-discovery-metadata
- China Meteorological Administration: https://api.weather.gc.ca/api/collections/wis2-discovery-metadata

```bash
pywiscat wis2 search --help
```

!!! note
pywiscat is pre-installed on the local training environment, but can be installed from anywhere with `pip3 install pywiscat`

!!! note
by default, pywiscat connects to Canada's Global Discovery Catalogue. To set to a different catalogue, set the `PYWISCAT_GDC_URL`
environment variable

```bash
pywiscat wis2 search
```
Expand Down
4 changes: 2 additions & 2 deletions documentation/docs/practical-sessions/initializing-wis2box.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,7 +173,7 @@ Or check the content of the data-mappings.yml via WinSCP by browsing to the new

!!! question

How many different keys are defined for 'data' in the data-mappings.yml file?
How many different keys are defined for `data` in the `data-mappings.yml` file?

??? success "Click to reveal answer"

Expand All @@ -189,7 +189,7 @@ Or check the content of the data-mappings.yml via WinSCP by browsing to the new

!!! note

The `data-mappings.yml` define the plugins used to transform your data. For more information see [data pipeline plugins in the wis2box-documentation](https://docs.wis2box.wis.wmo.int/en/latest/reference/running/data-pipeline-plugins.html)
The `data-mappings.yml` file defines the plugins used to transform your data. For more information see [data pipeline plugins in the wis2box-documentation](https://docs.wis2box.wis.wmo.int/en/latest/reference/running/data-pipeline-plugins.html)


## wis2box start and status
Expand Down
51 changes: 29 additions & 22 deletions environment/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Training environment

The wis2box portable training environment consists of the following components:
- One (1) Nighthawk WiFi router
- One (1) Nighthawk Wi-Fi router
- Three (3) ZOTAX ZBOX Mini PCs (M Series Edge MI623)

Each ZOTAX ZBOX Mini PC has been fitted with:
Expand All @@ -12,30 +12,28 @@ The ZOTAX ZBOX Mini PCs are running virtualization software: ProxMox version 7.3

The 3 PCs together are setup in a ProxMox cluster to provide VMs for participants during the training.

The Nighthawk is setup to broadcast a password-protected WiFi network with SSID **WIS2-training**.
The Nighthawk is setup to broadcast a password-protected Wi-Fi network with SSID **WIS2-training**.
The local network is defined by 10.0.0.1/22

Adminstrators can access training environment from the following locations:

- Nighthawk WiFi router interface at https://10.0.0.1
- Nighthawk Wi-Fi router interface at https://10.0.0.1
- ProxMox cluster interface at https://10.0.1.1:8006

Each VM host has a 'vm-clone-base'-template from which new VMs can be created.
Each VM host has a 'vm-clone-base' template from which new VMs can be created.

The 'vm-clone-base' template consists of:

- 2 vCPUs
- 48 GB local storage
- 4 GB RAM
- Ubuntu 20.0.4 with the following tools installed
- Docker CE
- mosquitto-clients
- python3
- python3-pip
- unzip
- docker-compose
- minio pywis-pubsub (installed via `pip3`)
- `/etc/docker/daemon.json` with a Docker registry mirror pointing to http://10.0.2.222:5000
- Ubuntu 22.0.4 LTS with the following system packages installed:
- Docker CE v2.21.0
- docker compose 24.0.6
- packages installed via `pip3`:
- minio
- pywis-pubsub
- pywiscat

### VM naming convention

Expand All @@ -51,18 +49,27 @@ Student VMs:
- on vm-host-2: 10.0.2.11, 10.0.2.12, 10.0.2.13 etc.
- on vm-host-2: 10.0.3.11, 10.0.3.12, 10.0.3.13 etc.

local-repo-vm: 10.0.2.222
### student VM setup

### MinIO setup
Student VMs can be setup by cloning the a base template.

MinIO is used to provide the exercise materials and documentation. The `local-repo-vm-222` VM must be started in
order to create MinIO storage. The following buckets must be created manually with `readonly` permissions:
The script `setup_student_vm.sh` can be used to by the `wmo_admin` account to create a new user account on the student VM and add the latest wis2box release to the home directory of the new user along with the exercise materials.

- `exercise-materials`
- `documentation`
The WiFi router has pre-configured mac-to-IP settings. After cloning the base template set the MAC address to match the IP using the following logic:

### student VM setup
- `00:00:00:00:01:11` -> `10.0.1.11`
- `00:00:00:00:02:12` -> `10.0.2.12`
- etc.

Student VMs can be setup by cloning the a base template.
### local DNS

The script `setup_student_vm.sh` can be used to by the `wmo_admin` account to create a new user account on the student VM and add the latest wis2box release to the home directory of the new user along with the exercise materials.
A local DNS-server is setup on a VM with IP 10.0.2.111.

DNS server UI
http://dns-server.wis2.training:5380/
or
http://10.0.2.111:5380/

The DNS server UI can be used to check the status of the DNS-server and/or to add additional A-records. The DNS server is preconfigured with A-records naming each host for each local training participants.

The DNS-server is hosted on a VM on vm-host-2. If vm-host-2 is lost during transit or broken a backup is available on vm-host-1. If the DNS-server for some reason does not work at all, participants can just use the IP as hostname.
41 changes: 41 additions & 0 deletions environment/instructor/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Instructor helpers

This directory contains resources that may be valuable for instructors as part of
delivery of the training. Note that the below need to be run while connected to
the **WIS2-training** Wi-Fi network.

# Local global services

Local global services provide a test environment to simulate WIS2 workflows using
the classroom as a network of WIS2 nodes.

## Global Broker

TBD

### Live map (`live.html`)

Serve this webpage via HTTP on your VM. Ensure that the following line:

```javascript
const host = 'ws://tbd.wis2.training:8884/ws';
```

...is updated to point to the Global Broker installed above.

## Global Cache

[wis2-gc](https://github.com/wmo-im/wis2-gc) is a Reference Implememtation of
a WIS2 Global Cache (GC). Follow the setup instructions, pointing
to the Global Broker installed above.

## Global Discovery Catalogue

[wis2-gdc](https://github.com/wmo-im/wis2-gdc) is a Reference Implememtation of
a WIS2 Global Discovery Catalogue (GDC). Follow the setup instructions, pointing
to the Global Broker installed above.

# Other

* `instructor_bashrc.sh`: sample bash prompt to help with screen readability while
displaying a termnial during a practical session.
File renamed without changes.
75 changes: 75 additions & 0 deletions environment/instructor/live.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
!DOCTYPE html>
<html>
<head>
<title>wis2box training live data and metadata notifications</title>
<link rel="stylesheet" href="https://unpkg.com/leaflet/dist/leaflet.css" />
<script src="https://unpkg.com/leaflet/dist/leaflet.js"></script>
<script src="https://unpkg.com/mqtt@4.0.1/dist/mqtt.min.js"></script>
<script>

const host = 'ws://tbd.wis2.training:8884/ws';
const options = {
username: 'everyone',
password: 'everyone',
keepalive: 60,
protocolVersion: 5,
reconnectPeriod: 1000,
connectTimeout: 30 * 1000,
};

console.log('Connecting mqtt client');
const client = mqtt.connect(host, options);

client.on('connect', function () {
client.subscribe('origin/#', function (err) {
if (!err) {
console.log('Connected!')
}
})
})

client.on('error', (err) => {
console.log('Connection error: ', err)
client.end()
});

client.on('reconnect', () => {
console.log('Reconnecting...')
});

</script>
</head>

<body>
<header>
<h1>wis2box training live data and metadata notifications</h1>
</header>
<div id="map" style="height: 600px;"></div>
<footer>
Powered by WIS2
</footer>
</body>
<script>
var map = L.map('map').setView([0, 0], 2);
var tiles = L.tileLayer('https://server.arcgisonline.com/ArcGIS/rest/services/World_Shaded_Relief/MapServer/tile/{z}/{y}/{x}', {
maxZoom: 19,
attributionControl: false
}).addTo(map);
map.attributionControl.setPrefix('');

client.on('message', function (topic, message) {
const content = JSON.parse(message.toString())
addMarker(content);
console.log(content);
})

function addMarker(feature) {
var marker = L.geoJSON(feature).addTo(map);
marker.bindPopup('<h1><a target="_blank" href="' + feature.links[0].href + '">' + feature.id + '</a>');
setTimeout(removeMarker, 5000, marker);
}
function removeMarker(feature) {
feature.removeFrom(map);
}
</script>
</html>
4 changes: 2 additions & 2 deletions environment/local-repo-vm-222/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

This is the stack to be run on the VM with IP 10.0.2.222

This stack is used to make a local set of services available within the WIS2-training network:
This stack is used to make a local set of services available within the **WIS2-training** network:

- docker-registry-mirror: to cache docker-images and reduce build time
- docker-registry-mirror: to cache Docker images and reduce build time
- MinIO bucket: to make training materials locally available over HTTP
- mosquitto broker: to demonstrate MQTT and/or to act as potential local GB

Expand Down

0 comments on commit 7aef174

Please sign in to comment.