NVS support in bootloader (IDFGH-1184)

[avrahamshukron]

Environment

• Development Kit: [ESP32-PICO-Kit]
• Kit version (for WroverKit/PicoKit/DevKitC): [v4]
• Module or chip used: [ESP32-PICO-D4]
• IDF version (run git describe --tags to find it): v3.2
• Build System: [Make]
• Compiler version (run xtensa-esp32-elf-gcc --version to find it): 1.22.0-80-g6c4433a
• Operating System: [macOS]
• Power Supply: [USB]

Problem Description

I have a custom bootloader with some recovery mechanism implemented for my product.
The bootloader needs some persistent data to perform its operation (E.g boot counter, external WD time to configure etc.).

The easiest way to save such data in a reliable matter is NVS.
Unfortunately, in is not available in the bootloader binary and it is not trivial to compile it into the bootloader.

It would be really helpful to have NVS support in the bootloader, especially for mass-production products.

[negativekelvin]

I don't think it is entirely practical, have you considered using the same strategy as otadata using 2 sectors to store raw binary data?

[avrahamshukron]

That is exactly what am I currently doing... 2 sectors containing the same data with CRC checksum on each.
Upon boot I read them and fix the corrupted one if something bad happened.

But - NVS is far more generic and capable, and I feel like I'm re-inventing the wheel.

It can't be that I'm the only one who finds NVS useful in the context of the bootloader...

[shlomos]

Hi,
I am having the same problem, I implement my own bootloader state machine and had to implement my own i2c driver to use in bootloader context (which was a pain).
Currently having a hard time managing flash IO correctly when unexpected PoRs are possible at boot time.
My whole project uses NVS, and using the same code base I already have in the app makes perfect sense.
An NVS support could greatly ease the pain of flash storage access in bootloader.

[negativekelvin]

Not arguing against it's usefulness, but bootloader runs from ram and can't conflict with the app sections it is going to then load to ram so minimal footprint is desirable.

[projectgus]

Hi @shlomos @avrahamshukron ,

Thanks for sending this request and giving some information about your requirements. Is there any more details you can give about exactly what conditions you are checking for during the early boots? We have some current plans for adding optional bootloader support for detecting cases like the device getting stuck in boot/WDT loops.

Adding NVS support in the bootloader is a non-trivial amount of work, and it's relatively risky because we don't support bootloader OTA updates - the NVS partition format would need to remain compatible with the bootloader for the life of the product, and any bugs in the bootloader NVS implementation will remain there for the life of the product.

However I totally appreciate that the alternative is fairly primitive and fiddly by comparison.

The more details you can give us about exactly what case(s) you are checking for, what persistent information you need (and which parts of this information are written by the app vs written by the bootloader), the more chance there is that we can figure out a way to accommodate these use cases.

[avrahamshukron]

Our product is basically an IoT device that will be deployed by the thousands in field.

Once deployed, the cost of retrieving it back for maintenance is too high so our software and firmware needs to be reliable enough and contain recovery mechanisms.

But there's a catch 22 here:
We must be able to upgrade the firmware in order to keep communication with our servers.

So we came up with the following scheme that revolves around 2 software images:

1. Recovery image - basic image that can connect to our servers and can flah other OTA partitions.
2. Normal image with the complete functionality.

There bootloader job is to load the correct image based on some logic that we devised. This logic uses some non-volatile data, such as boot counter (SOFT_BOOT_COUNTER), external WD configuration (EXT_WD_TIME) and weather to load the recovery or normal image (IMAGE_SELECT).

On power-on reset, the bootloader will load the recovery image.

On soft reset:

1. If 'BOOT_COUNTER > 10', it'll perform hard reset using external WD (which will be experienced as POR) it'll load whatever is stored in IMAGE_SELECT.

2. If the counter was less or equal 10, it'll increment it by 1.

3. Arm external WD to EXT_WD_TIME

4. It'll look at IMAGE_SELECT and load the appropriate image and pass control to it.

This is the basic flow, I left a few details out for brevity and because some of it is confidential.

Hope that helps...