partition aware custom bootloader with release 5.2 (IDFGH-12214)

[tobozo]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

Hi,

I'm trying to create a bootloader hook that acts as follows:

if (reset_reason == power_on_reset && factory_partition_exists() ) 
{ 
  load factory()
}

The bootloader hooks example provided with esp-idf it too poor to be practical so I searched for a real world implementation and found this but it's based on an older version of esp-idf, so I tried to apply the esp-idf 5.2 migration guide :

Original dependency setup:

idf_component_register(...
                       REQUIRES spi_flash)

Updated dependency setup:

idf_component_register(...
                       REQUIRES spi_flash esp_partition)

However adding esp_partition to PRIV_REQUIRES in CMakeLists.txt produces compilation errors (see below).

I probably missed something, but this feels like a bug.

/espressif/esp-idf-v5.2/components/app_update/esp_ota_ops.c:28:10: fatal error: esp_system.h: No such file or directory
   28 | #include "esp_system.h"

/espressif/esp-idf-v5.2/components/esp_partition/partition.c:80:5: error: unknown type name 'spi_flash_mmap_handle_t'
   80 |     spi_flash_mmap_handle_t handle;

Full log

$ idf.py build
Executing action: all (aliases: build)
Running ninja in directory /projects/blah/build
Executing "ninja all"...
[1/4] cd /projects/blah/build/esp-idf/esptool_py && ~/.espressif/python_env/idf5.2_py3.11...blah/build/partition_table/partition-table.bin /projects/blah/build/main.bin
main.bin binary size 0x30fd0 bytes. Smallest app partition is 0x100000 bytes. 0xcf030 bytes (81%) free.
[2/4] Performing build step for 'bootloader'
FAILED: bootloader-prefix/src/bootloader-stamp/bootloader-build bootloader/bootloader.elf bootloader/bootloader.bin bootloader/bootloader.map /projects/blah/build/bootloader-prefix/src/bootloader-stamp/bootloader-build /projects/blah/build/bootloader/bootloader.elf /projects/blah/build/bootloader/bootloader.bin /projects/blah/build/bootloader/bootloader.map 
cd /projects/blah/build/bootloader && /usr/bin/cmake --build .
[1/9] Building C object esp-idf/app_update/CMakeFiles/__idf_app_update.dir/esp_ota_ops.c.obj
FAILED: esp-idf/app_update/CMakeFiles/__idf_app_update.dir/esp_ota_ops.c.obj 
~/.espressif/tools/xtensa-esp-elf/esp-13.2.0_20230928/xtensa-esp-elf/bin/xtensa-esp32-elf-gcc -DBOOTLOADER_BUILD=1 -DESP_PLATFORM -DIDF_VER=\"v5.2-dirty\" -DSOC_MMU_PAGE_SIZE=CONFIG_MMU_PAGE_SIZE -DSOC_XTAL_FREQ_MHZ=CONFIG_XTAL_FREQ -D_GLIBCXX_HAVE_POSIX_SEMAPHORE -D_GLIBCXX_USE_POSIX_SEMAPHORE -D_GNU_SOURCE -I/projects/blah/build/bootloader/config -I/espressif/esp-idf-v5.2/components/app_update/include -I/espressif/esp-idf-v5.2/components/log/include -I/espressif/esp-idf-v5.2/components/esp_rom/include -I/espressif/esp-idf-v5.2/components/esp_rom/include/esp32 -I/espressif/esp-idf-v5.2/components/esp_rom/esp32 -I/espressif/esp-idf-v5.2/components/esp_common/include -I/espressif/esp-idf-v5.2/components/esp_hw_support/include -I/espressif/esp-idf-v5.2/components/esp_hw_support/include/soc -I/espressif/esp-idf-v5.2/components/esp_hw_support/include/soc/esp32 -I/espressif/esp-idf-v5.2/components/esp_hw_support/port/esp32/. -I/espressif/esp-idf-v5.2/components/newlib/platform_include -I/espressif/esp-idf-v5.2/components/xtensa/esp32/include -I/espressif/esp-idf-v5.2/components/xtensa/include -I/espressif/esp-idf-v5.2/components/xtensa/deprecated_include -I/espressif/esp-idf-v5.2/components/soc/include -I/espressif/esp-idf-v5.2/components/soc/esp32 -I/espressif/esp-idf-v5.2/components/soc/esp32/include -I/espressif/esp-idf-v5.2/components/bootloader_support/include -I/espressif/esp-idf-v5.2/components/bootloader_support/bootloader_flash/include -I/espressif/esp-idf-v5.2/components/bootloader_support/private_include -I/espressif/esp-idf-v5.2/components/esp_app_format/include -I/espressif/esp-idf-v5.2/components/esp_bootloader_format/include -I/espressif/esp-idf-v5.2/components/esp_partition/include -I/espressif/esp-idf-v5.2/components/efuse/include -I/espressif/esp-idf-v5.2/components/efuse/esp32/include -I/espressif/esp-idf-v5.2/components/spi_flash/include -I/espressif/esp-idf-v5.2/components/hal/platform_port/include -I/espressif/esp-idf-v5.2/components/hal/esp32/include -I/espressif/esp-idf-v5.2/components/hal/include -mlongcalls -Wno-frame-address  -fdiagnostics-color=always -ffunction-sections -fdata-sections -Wall -Werror=all -Wno-error=unused-function -Wno-error=unused-variable -Wno-error=unused-but-set-variable -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -Wno-enum-conversion -gdwarf-4 -ggdb -Os -freorder-blocks -fmacro-prefix-map=/espressif/esp-idf-v5.2/components/bootloader/subproject=. -fmacro-prefix-map=/espressif/esp-idf-v5.2=/IDF -fstrict-volatile-bitfields -fno-jump-tables -fno-tree-switch-conversion -fno-stack-protector -std=gnu17 -Wno-old-style-declaration -MD -MT esp-idf/app_update/CMakeFiles/__idf_app_update.dir/esp_ota_ops.c.obj -MF esp-idf/app_update/CMakeFiles/__idf_app_update.dir/esp_ota_ops.c.obj.d -o esp-idf/app_update/CMakeFiles/__idf_app_update.dir/esp_ota_ops.c.obj -c /espressif/esp-idf-v5.2/components/app_update/esp_ota_ops.c
/espressif/esp-idf-v5.2/components/app_update/esp_ota_ops.c:28:10: fatal error: esp_system.h: No such file or directory
  28 | #include "esp_system.h"
      |          ^~~~~~~~~~~~~~
compilation terminated.
[2/9] Building C object esp-idf/esp_partition/CMakeFiles/__idf_esp_partition.dir/partition.c.obj
FAILED: esp-idf/esp_partition/CMakeFiles/__idf_esp_partition.dir/partition.c.obj 
~/.espressif/tools/xtensa-esp-elf/esp-13.2.0_20230928/xtensa-esp-elf/bin/xtensa-esp32-elf-gcc -DBOOTLOADER_BUILD=1 -DESP_PLATFORM -DIDF_VER=\"v5.2-dirty\" -DSOC_MMU_PAGE_SIZE=CONFIG_MMU_PAGE_SIZE -DSOC_XTAL_FREQ_MHZ=CONFIG_XTAL_FREQ -D_GLIBCXX_HAVE_POSIX_SEMAPHORE -D_GLIBCXX_USE_POSIX_SEMAPHORE -D_GNU_SOURCE -I/projects/blah/build/bootloader/config -I/espressif/esp-idf-v5.2/components/esp_partition/include -I/espressif/esp-idf-v5.2/components/log/include -I/espressif/esp-idf-v5.2/components/esp_rom/include -I/espressif/esp-idf-v5.2/components/esp_rom/include/esp32 -I/espressif/esp-idf-v5.2/components/esp_rom/esp32 -I/espressif/esp-idf-v5.2/components/esp_common/include -I/espressif/esp-idf-v5.2/components/esp_hw_support/include -I/espressif/esp-idf-v5.2/components/esp_hw_support/include/soc -I/espressif/esp-idf-v5.2/components/esp_hw_support/include/soc/esp32 -I/espressif/esp-idf-v5.2/components/esp_hw_support/port/esp32/. -I/espressif/esp-idf-v5.2/components/newlib/platform_include -I/espressif/esp-idf-v5.2/components/xtensa/esp32/include -I/espressif/esp-idf-v5.2/components/xtensa/include -I/espressif/esp-idf-v5.2/components/xtensa/deprecated_include -I/espressif/esp-idf-v5.2/components/soc/include -I/espressif/esp-idf-v5.2/components/soc/esp32 -I/espressif/esp-idf-v5.2/components/soc/esp32/include -I/espressif/esp-idf-v5.2/components/spi_flash/include -I/espressif/esp-idf-v5.2/components/hal/platform_port/include -I/espressif/esp-idf-v5.2/components/hal/esp32/include -I/espressif/esp-idf-v5.2/components/hal/include -I/espressif/esp-idf-v5.2/components/bootloader_support/include -I/espressif/esp-idf-v5.2/components/bootloader_support/bootloader_flash/include -I/espressif/esp-idf-v5.2/components/bootloader_support/private_include -I/espressif/esp-idf-v5.2/components/app_update/include -I/espressif/esp-idf-v5.2/components/esp_app_format/include -I/espressif/esp-idf-v5.2/components/esp_bootloader_format/include -mlongcalls -Wno-frame-address  -fdiagnostics-color=always -ffunction-sections -fdata-sections -Wall -Werror=all -Wno-error=unused-function -Wno-error=unused-variable -Wno-error=unused-but-set-variable -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -Wno-enum-conversion -gdwarf-4 -ggdb -Os -freorder-blocks -fmacro-prefix-map=/espressif/esp-idf-v5.2/components/bootloader/subproject=. -fmacro-prefix-map=/espressif/esp-idf-v5.2=/IDF -fstrict-volatile-bitfields -fno-jump-tables -fno-tree-switch-conversion -fno-stack-protector -std=gnu17 -Wno-old-style-declaration -MD -MT esp-idf/esp_partition/CMakeFiles/__idf_esp_partition.dir/partition.c.obj -MF esp-idf/esp_partition/CMakeFiles/__idf_esp_partition.dir/partition.c.obj.d -o esp-idf/esp_partition/CMakeFiles/__idf_esp_partition.dir/partition.c.obj -c /espressif/esp-idf-v5.2/components/esp_partition/partition.c
/espressif/esp-idf-v5.2/components/esp_partition/partition.c: In function 'load_partitions':
/espressif/esp-idf-v5.2/components/esp_partition/partition.c:80:5: error: unknown type name 'spi_flash_mmap_handle_t'
  80 |     spi_flash_mmap_handle_t handle;
      |     ^~~~~~~~~~~~~~~~~~~~~~~
/espressif/esp-idf-v5.2/components/esp_partition/partition.c:103:21: error: implicit declaration of function 'spi_flash_mmap' [-Werror=implicit-function-declaration]
  103 |     esp_err_t err = spi_flash_mmap(partition_align_pg_size,
      |                     ^~~~~~~~~~~~~~
/espressif/esp-idf-v5.2/components/esp_partition/partition.c:104:36: error: 'SPI_FLASH_SEC_SIZE' undeclared (first use in this function); did you mean 'SPI_FLASH_OPI_STR'?
  104 |                                    SPI_FLASH_SEC_SIZE, SPI_FLASH_MMAP_DATA, (const void **)&p_start, &handle);
      |                                    ^~~~~~~~~~~~~~~~~~
      |                                    SPI_FLASH_OPI_STR
/espressif/esp-idf-v5.2/components/esp_partition/partition.c:104:36: note: each undeclared identifier is reported only once for each function it appears in
/espressif/esp-idf-v5.2/components/esp_partition/partition.c:104:56: error: 'SPI_FLASH_MMAP_DATA' undeclared (first use in this function); did you mean 'SPI_FLASH_OPI_DTR'?
  104 |                                    SPI_FLASH_SEC_SIZE, SPI_FLASH_MMAP_DATA, (const void **)&p_start, &handle);
      |                                                        ^~~~~~~~~~~~~~~~~~~
      |                                                        SPI_FLASH_OPI_DTR
/espressif/esp-idf-v5.2/components/esp_partition/partition.c:231:5: error: implicit declaration of function 'spi_flash_munmap' [-Werror=implicit-function-declaration]
  231 |     spi_flash_munmap(handle);
      |     ^~~~~~~~~~~~~~~~
cc1: some warnings being treated as errors
ninja: build stopped: subcommand failed.
ninja: build stopped: subcommand failed.
Compilation failed because esp_ota_ops.c (in "app_update" component) includes esp_system.h, provided by esp_system component(s).
However, esp_system component(s) is not in the requirements list of "app_update".
To fix this, add esp_system to PRIV_REQUIRES list of idf_component_register call in /espressif/esp-idf-v5.2/components/app_update/CMakeLists.txt.
ninja failed with exit code 1, output of the command is in the /projects/blah/build/log/idf_py_stderr_output_440355 and /projects/blah/build/log/idf_py_stdout_output_440355

[filzek]

@tobozo Hello there,

simple solve this by add to CMakelists.txt

INCLUDE_DIRS "C:/Espressif/frameworks/esp-idf-v5.2/components/bootloader_support/private_include/"
PRIV_REQUIRES bootloader_support

also replace the sys_delay_ms(1000000);

[filzek]

One ver interesting thing that could be done over the changes past the old bootloaders SDK is if we could be able to deploy a OTA bootloader update (to secure bootloader v1) to new SDK V5.2, if the sign pem file is avaliable to sign the new bootloader, the dev team could prepare one special application to enable update the bootloader on the fly, it could solve lots and tons of old problems.

[tobozo]

@filzek thanks for your reply 👍

I went with something like this:

idf_component_register(
  SRCS "hooks.c"
  INCLUDE_DIRS ${IDF_PATH}/components/bootloader_support/private_include/
  PRIV_REQUIRES bootloader_support
)

But then it means I have to use the bootloader_support component instead of esp_partitions and app_update, right?

also replace the sys_delay_ms(1000000);

I'm not sure where that comes from or where I should replace that., is it encryption related? I'm not using any of it in that project anyway.

[ESP-Marius]

Hi, @tobozo

Just double checking: have you seen CONFIG_BOOTLOADER_FACTORY_RESET? Not sure if this could somehow fulfill your use-case or if you still need to modify the bootloader.

What you see here is due to that not all components are intended/designed to be pulled into the bootloader build. E.g. app_update or esp_partition are not built or used for the default esp-idf bootloader, so when you try to pull them in you run into issues.

In general I'd recommend you to stick with the API IDF use in the bootloader (mostly bootloader_support) where possible.

[tobozo]

@ESP-Marius thanks for your reply 👍

M5CoreS3 doesn't have any gpio button to trigger features from CONFIG_BOOTLOADER_FACTORY_RESET, so I can't use it in this scenario.

But I think the issue is because I'm trying to implement that in bootloader_hooks instead of bootloader_override.

Indeed the bootloader_override example seems a more appropriate place to change the selected partition number, so I'll experiment with that and keep this thread updated with the results.

[tobozo]

bootloader_override was the way to go, now I'll be hanging on #3495 as all my attempts to distinguish poweron reset from software reset in the bootloader failed, and I'll probably go with some sort of reboot counter instead.

I'll close this issue as no longer relevant since I was just looking in the wrong place.

Thanks for your support! 👍