CPU: CPUs started in inconsistent modes

[SikkiLadho]

We have been noticing a backtrace in UART log, indicated that CPUs are started in inconsistent modes. Here's what might be causing it.

After all cores are up, linux checks the mode they are in, if they are in different modes, it writes to UART about the inconsistency of the nodes.

Looking here we can read a comment,

A correctly-implemented bootloader must start all CPUs in the same mode:
In this case, both 32bit halves of __boot_cpu_mode will contain the
same value (either 0 if booted in EL1, BOOT_CPU_MODE_EL2 if booted in EL2).
Should the bootloader fail to do this, the two values will be different.
This allows the kernel to flag an error when the secondaries have come up.

Trusted Firmware only allows primary cpu(cpu0) to pass through kernel8.img,(which in this case is Leo's code). We never brought up the secondary cores at Leo and TF-A gave all those cores to Linux kernel in HYP/EL2 mode. Here is the CPU node BCM2711 device tree.

When linux brings up the secondary cores, CPU0 is in EL1(becasue we eret it to EL1 here. However, secondary cores(cpu1/2/3) are still in HYP/EL2 mode. Thus the inconsistency of modes between primary and secondary cores.

What do we need to do?

1. Modify the device tree blob with libfdt to use PSCI instead of spin_table to bring up the secondary cores. Here's the CPU node in DTB to be modified.
2. Trapping all asynchronous exceptions at Leo(EL2) and looking for smc with smc_fid: PSCI_CPU_ON_AARCH64

As we research more into this, I will update this issue very soon.