“...the bullshit piled up so fast... you needed wings to stay above it.”
– Apocalypse Now, 1979

Towards Reliable IoT

Introduction

DOIT DEvKit V1 ESP32-WROOM-32 is an inexpensive (15€) microcontroller board with Wi-Fi, Bluetooth LE, and ESP-NOW. One can connect it to a lot of sensors with ready-made drivers. The challenge is to control such a board globally, via the internet.

ESP32 and IoT

There are several ways to make the ESP32 visible globally:

1. Cloud services: ESP RainMaker, Firebase, Blynk, Arduino IoT Cloud, Yaler.net, Amazon API Gateway with Websocket API, Amazon API Gateway with RESTful API, Husarnet, CloudMQTT, HiveMQ, RemoteXY, Google Cloud IoT, Viper/Zerynth: 1, 2, 3, Tuya IoT Development Platform, Thingschain: 1, 2, Azure IoT: 1, 2, ZeroTier?, Notehub: 1, 2, ngrok...

   Google IoT Core was discontinued on August 16, 2023. Free plans come and go. Husarnet is the only one from the listed above which provides its open source code.

2. An HTTP web app: 1, 2. Notice that the board needs to know the URL or the IP address of the app to send GET/POST requests, but the app does not need to know the address of the board.

3. A similar MQTT intermediary which could just be the SSH tunnel from your local MQTT broker to a rented VPS. The MQTT has its pros and cons. Subjectively, (i) the MQTT is more reliable than the HTTP in the ESP world, and (ii) there is no need to write a web app to send/receive data (although the same may hold for the HTTP). One can simply run the tunnelled Mosquitto broker as an "MQTT app" on the VPS, and issue "mosquitto_pub/sub" commands w.r.t. the MQTT topics that the board will pub/sub to.

4. Wireguard on the ESP32: 0, 1, 2, 3, 4, 5, 6. This is for some overly optimistic dubious uses where the ESP32 becomes a VPN node on a par with Linux boards. No MicroPython support, no ssh, a tiny fractured LwIP networking user base. Wireguard still needs a public static IP.

5. Connecting the ESP32 to the Linux PC over Wi-Fi that runs the MQTT broker within its LAN, thus delegating the problem of global connectivity effectively to the PC space.

The last option is our choice here. It is the most reliable as the ESP32 communicates only within its LAN. We do not need to hunt for the reviews of all these clouds and make tough choices. When things go wrong, we can locate the problem easier since the global connectivity happens only in the Linux space. We do not need to deal with opaque evolving 3rd party services, vendor lock-in, opaque firmware/software on the ESP32, we can add as many devices as we want, run MicroPython, there are no subscription payments. The downside is that this demands running 24/7 an extra PC/Linux board (Ubuntu1 shown in the figure above). Many out of the box Android conveniences present in, say, the ESP RainMaker cloud, will not be available, like a simple push button/dash board Android app, Wi-Fi provisioning, device discoverability, OTA updates. However, we can control the ESP32 globally from Android via the Linux intermediary and go-libp2p.

One can run an MQTT broker on a router, e.g. with OpenWrt Linux: 1, 2 or RutOS, but these router OSes (6-8MB .bin image size) are too limiting. More importantly, one cannot use go-libp2p with them.

In order to establish remote PC connections, we have tested Hyprspace, EdgeVPN, and anywherelan (awl). All of them are built on top of go-libp2p which is both, the FOSS code, and the running network with NAT traversal aka hole punching. This software allows to ssh into a remote computer without a public static IP.

According to Max Inden, 2022, the libp2p network "powers the IPFS, Ethereum 2, Filecoin and Polkadot network and there are ~100K libp2p based nodes online at any given time".

Do these tools always work though, are they equally good?

hyprspace: 895 LOC of Go. Minimal, but weaker hole punching.

awl: 6.5 KLOC of Go. Just about right.

EdgeVPN: 7.5 KLOC of Go. Solid punching, but problems with 24/7 runs. No Android support.

go-libp2p: 67 KLOC of Go. The base layer for the three above.

EdgeVPN may have an edge over Hyprspace, but there is a problem with longer runs. awl is more reliable. It is also more convenient (desktop browser GUI for one-click handshakes, runs on Android), but not always. If for some reason one has to reinstall the Android app, the latter generates a new peer id which then needs to be confirmed again on the other end. EdgeVPN simply shares the same secret file and has no handshakes.

Some Photos

This hobby/demo hardware has been assembled and soldered by Saulius Rakauskas (InfoVega).

Circuit Diagram

• esp32-30pin (the 30-pin variant of DOIT DEVIT V1 ESP32-WROOM-32, not 36).

• DHT22, Multiple LEDs, Capacitive Soil Moisture Sensor v1.2, see boot.py.

• SSD1306 with in-software I2C. Dropped it, rewind to "the last before revamp" commit if interested.

Commands

• Ubuntu PC:

  sudo apt-get install python3-pip
  sudo pip3 install esptool
  sudo pip3 install rshell

• USB connection:

  ls /dev/ttyUSB*
  dmesg | grep ttyUSB

• Flashing/reflashing MicroPython firmware via USB:

  sudo esptool.py --port /dev/ttyUSB0 flash_id
  sudo esptool.py --port /dev/ttyUSB0 erase_flash
  sudo esptool.py --chip esp32 --port /dev/ttyUSB0 --baud 460800 write_flash -z 0x1000 esp32-ota-20220618-v1.19.1.bin

• Release:

  rshell --buffer-size=30 -p /dev/ttyUSB0 -a
  boards
  ls /pyboard
  cp main.py /pyboard
  cp umqttsimple.py /pyboard
  cp boot.py /pyboard

  Disconnect the device from USB and use only a power supply.

• MQTT:

  sudo apt install mosquitto mosquitto-clients
  hostname -I
  ifconfig -a | grep inet

  The hostname/ifconfig command will provide the local IP address assigned by the router to a computer (PC-1) which runs the MQTT broker (Moscquitto), such as '192.168.1.107'. It will have to be entered in boot.py manually/explicitly.

  The Mosquitto broker always runs on Ubuntu by default once the OS starts. However, depending on the exact Ubuntu and Mosquitto versions one may need some additional minimal configuration. Since Mosquitto version 2.0, one needs to create a custom config file and place it somewhere, say at /home/sara/esp32/custom_mosquitto.conf, with this minimal content:

  listener 1883
  allow_anonymous true

  Then start the MQTT broker with this config:

  mosquitto -c /home/sara/esp32/custom_mosquitto.conf

  Sometimes you may get "Error: Address already in use". That means you are starting another Mosquitto broker on the same machine. Kill one or the other with

  ps -ef | grep mosquitto
  sudo kill 1234

  One should get the console message indicating the connected MicroPython MQTT client with its local IP address such as 192.168.1.108, the latter may change with each device reboot, it is assigned by a router.

  Read the sensor data from the device:

  mosquitto_sub -d -h 192.168.1.107 -t "testincr"

  Publish the messages "on" or "off" to control the LED output:

  mosquitto_pub -d -h 192.168.1.107 -t "output" -m "on" -q 1
  mosquitto_pub -d -h 192.168.1.107 -t "output" -m "off" -q 1

• EdgeVPN:

  Download the latest edgevpn executable for your OS, e.g. edgevpn-v0.23.1-Linux-x86_64.tar.gz.

  Use the same release version on all the computers that will be connected into a VPN, otherwise there might be conflicts as some versions have changed the way the keys are stored and decoded.

  cd into extracted folder and generate config.yaml:

  ./edgevpn -g > config.yaml

  Distribute it on all the machines that will be connected to EdgeVPN (manually by carrying a USB stick, via email, etc.).

  Assuming the machine runs Ubuntu, place the executable with config.yaml in the same folder and run

  sudo IFACE=edgevpn0 ADDRESS=10.1.0.7/24 EDGEVPNCONFIG=config.yaml ./edgevpn --log-level debug

  The command assigns a virtual IP address 10.1.0.7 to the machine on which it is executed. Assign a different address to each machine that you connect to EdgeVPN, i.e. 10.1.0.1.. 10.1.0.254. but use the same config.yaml in each case.

  It may take several minutes for the node to join the EdgeVPN network which is the libp2p network under the hood. The Linux desktop GUI is totally unnecessary.

  If everything is fine, you should be able to connect to any Linux computer of your VPN, e.g.

  ssh username@10.1.0.8
  username@10.1.0.8's password: 
  Last login: Mon Jul 10 15:13:34 2023 from 10.1.0.4
  
  ...
  
  exit
  logout
  Connection to 10.1.0.8 closed.

  When you are able to connect from A to B, A's ./edgevpn --log-level debug will show a lot of information including the following lines:

  {"level":"DEBUG","time":"2023-07-11T14:44:39.455+0300","caller":"discovery/dht.go:204","message":" Announcing ourselves..."}
  {"level":"DEBUG","time":"2023-07-11T14:44:56.542+0300","caller":"discovery/dht.go:207","message":" Successfully announced!"}
  {"level":"DEBUG","time":"2023-07-11T14:44:56.542+0300","caller":"discovery/dht.go:210","message":" Searching for other peers..."}
  {"level":"DEBUG","time":"2023-07-11T14:44:56.583+0300","caller":"discovery/dht.go:230","message":" Known peer (already connected):
  

  followed by the list of so called multiaddresses of known/connected peers which will include global or local IPv4. While being on the A side, you should be able to see the local LAN IPv4 address of B assigned to B by its encompassing router on the B side!

• awl:

  There is one unsolved issue with edgevpn that occurs on the 24/7 runs, so we switched to awl which seems to be more reliable.

  Download the awl-tray binary and run it, see 1 and 2 for more details.

  awl is nice in that once you start it on Android, you can then run an SSH app such as JuiceSSH and get the remote access to your Linux terminal. Make sure the awl VPN has no overlapping/duplicated addresses and delete all the previous connections on JuiceSSH before connecting. awl is not very polished yet so one can sometimes mess up virtual addresses with many-to-one connections and loops.

ESP32 and MicroPython

• Hardware errors. When the DHT sensor is detached from the chip's pin, executing the line "dht_sensor.measure()" or "dht_sensor.start()" in the MicroPython REPL will reboot the device with a "useful" error message:

  >>> import main
  ets Jun  8 2016 00:22:57
  
  rst:0x8 (TG1WDT_SYS_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
  configsip: 0, SPIWP:0xee
  clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
  mode:DIO, clock div:2
  load:0x3fff0030,len:4540
  ho 0 tail 12 room 4
  load:0x40078000,len:12448
  load:0x40080400,len:4124
  entry 0x40080680
  W (57) boot.esp32: PRO CPU has been reset by WDT.
  W (58) boot.esp32: WDT reset info: PRO CPU PC=0x400803c0
  W (59) boot.esp32: WDT reset info: APP CPU PC=0x40093cb2
  MicroPython v1.17 on 2022-01-10; 4MB/OTA module with ESP32
  Type "help()" for more information.
  >>> 

• Too little RAM. ESP12e/ESP8266 has a pathetic amount of RAM, but ESP32 is no cake either. Importing the font arial35 from Peter Hinch's ssd1306 lib along with freesans20 is still possible when running the DHT measurement with the display without the networking stack. Adding the async networking and the MQTT libs exposes insufficient RAM:

  "MemoryError: memory allocation failed, allocating 6632 bytes" (breaks at "#import gui.fonts.arial35 as arial35").

• Low quality sensors, see e.g. this discussion of Capacitive Soil Moisture Sensor v1.2. The solutions based on the electrical resistance are worse due to a rapid corrosion of the electrodes. We have tried submerging them into a cheap industrial gypsum, but that did not work at all.

• Monitoring soil moisture is a harder problem than sensing air humidity. Perhaps the best way is to set up a camera and observe the ground surface, but that is out of the scope of the low RAM devices, unless one is content with 320x240@25FPS.

• Despite all the amazing async MQTT lib work by Peter Hinch, the device could only send the MQTT messages, the receiving did not work. So we went back to umqttsimple.

• After a long search and disappointment the resilience w.r.t. the Wi-Fi loss was reached thanks to this code by Rui and Sara Santos.

Notes

• Tiny ESP32 RAM = very limited software, esp. limited global connectivity options. With some acrobatics one may run "Wireguard" on the ESP32, but the go-libp2p apps are beyond the reach.

• All this gigantic Web2 VPN service activity exists mostly because A and B do not have proper addresses. We cannot use MAC, we do not have the IPv6. So how does one send a message to a board/PC? One must deal with the OSI model, overlay mesh networks, proxies and reverse proxies, tunneling and self-hosting, TUN/TAP, STUN/TURN/ICE, TCP meltdown, CGNAT, SOCKS5, ARP, ICMP, subnet masks (CIDR), gateways, stream multiplexing, vsock/socat, ports aka socket numbers, port forwarding, host names, DNS and mDNS, UPnP, DHCP, virtual interfaces, iptables/firewalls, Linux kernel routes, routers and routing... B.A.T.M.A.N.

• The p2p tools such as Hyprspace, EdgeVPN, awl, Syncthing: 1, 2 solve the global connectivity problem without imposing paid services.

• Connect a webcam to a remote Ubuntu PC, install ffmpeg. Run awl-tray followed by this one liner with mplayer. It worked in the year 2010, it still works in 2023.

• 24/7 concerns: Idle mode with a monitor shut off consumes about 0.036A electric current at 220V, which amounts to 7.92W power. 720-hour monthly run will demand 5.7 kWh of energy. The rate of 0.30€ per 1kWh will induce a monthly electricity fee of 1.71€. Non-Idle mode: Consider the worst case upper bound which is running perpetually youtube in a browser with the laptop monitor on. It may increase the power consumption 5x.

• KVM1 on Hostinger costs 4.99€ and one gets a public static IP and 1TB of bandwidth with it, but we do not need that in the IoT. There are ways to push down power consumption. Raspberry Pi Zero W 2 may demand only 0.7W power, and notice that awl should run on both, ARM64 and ARM32 (hence Raspberry Pi Zero 2 W and older models too). The problem is that these platforms are little tested with go-libp2p. awl may run on ARM32, but delve, the Go debugger, may not.

• There exist inexpensive (sub-30€) one-time payment-based IoT devices designed for direct remote control and monitoring with the Android apps, see e.g. Clas Ohlson for a smart plug, and YCC365 for a remote 180° IP camera. The latter could even be hackable to some extent, but these are limited closed source proprietary products where one relies on their respective IT clouds for communication. However, if this fits your needs, you get the simplest time-saving solution.

• The most basic services are tricky. 70 KLOC of go-libp2p to give your computer a proper VPN address.

• Some go-libp2p-based FOSS efforts worth mentioning, and the lines of code (LOC):

  Syncthing: p2p Dropbox. 110 KLOC of Go, 38 KLOC of Js, 11 KLOC of CSS.

  Berty: p2p messenger built with React Native, blocked in Iran. 80 KLOC of Go, 50 KLOC of TypeScript/Js, 5 KLOC of Java.

  zkvote?: anonymous p2p voting based on zero-knowledge protocols. 4 KLOC of Go, 6 KLOC of Js.

  peerchat (MIT), cryptogram (GPL3): Minimal (sub 1 KLOC of Go) working p2p Linux chat terminals.

  ...

Some ESP32 References

Essential:

• MicroPython firmware
• micropython-Rui-Santos
• umqtt.simple
• esp32-30pin

Optional (problems to be aware of):

• micropython-mqtt-async
• micropython-nano-gui
• umqtt.robust
• umqtt.robust dies when MQTT broker gets restarted #102
• umqtt.simple socket behaviour when WiFi is degraded #103
• umqtt.robust: Resubscribe to topics after doing reconnect. #186
• can't await mqtt.simple publish method #357
• unstable MQTT on ESP8266 (4+ days) #2568
• umqtt cannot import MQTTClient #250

Appendix: Some Non-Toy Applications of Low RAM Devices

1. Waste bin level detectors based on ultrasonic distance sensors?

2. A bus card reader? We used to have some early low RAM devices here in Vilnius for about 5-10 years. This year (2023) the bus card readers got replaced with Estonian Ridango devices which, I suspect, run Linux.

3. Mapping out minefields? See 1 and 2 for two completely different systems. The first one presents a radar mounted on a drone, while the second one is a Colpitts oscillator-based metal detector on a four-wheel robot.

4. A GPS Tracker.

5. AirTag: 1, 2, 3.

6. Non-invasive blood glucose and hemoglobin monitoring: 1, 2, 3. Ketones, alcohol: 1. eCO2.

7. A smart walking cane for the blind.

8. Motor control.