How can LiteX find boot.bin but then be unable to mount the filesystem when running catalog in LiteOs on Arty-100T?

[JamesTimothyMeech]

I built a LiteX generated SoC with: python3 -m litex_boards.targets.digilent_arty --cpu-type femtorv --cpu-variant gracilis --variant a7-100 --toolchain vivado --with-spi-sdcard --sdcard-adapter digilent --build --load

I then ran make for LiteOS, copied it to the SD card and ran litex_term /dev/ttyUSB1 I got a message "Could not mount filesystem"

james@james-Laptop-13th-Gen-Intel-Core:~/Desktop/Casino/FPGA-System-on-Chip-Firmware/LiteX$ litex_term /dev/ttyUSB1

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2023 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Jul  6 2023 15:28:50
 BIOS CRC passed (f4e26a1b)

 LiteX git sha1: 648c70de

--=============== SoC ==================--
CPU:		FemtoRV-GRACILIS @ 100MHz
BUS:		WISHBONE 32-bit @ 4GiB
CSR:		32-bit data
ROM:		128.0KiB
SRAM:		8.0KiB
L2:		8.0KiB
SDRAM:		256.0MiB 16-bit @ 800MT/s (CL-7 CWL-5)
MAIN-RAM:	256.0MiB

--========== Initialization ============--
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Read leveling:
  m0, b00: |00000000000000000000000000000000| delays: -
  m0, b01: |00000000000000000000000000000000| delays: -
  m0, b02: |11111111111100000000000000000000| delays: 05+-05
  m0, b03: |00000000000000111111111111110000| delays: 20+-06
  m0, b04: |00000000000000000000000000000001| delays: -
  m0, b05: |00000000000000000000000000000000| delays: -
  m0, b06: |00000000000000000000000000000000| delays: -
  m0, b07: |00000000000000000000000000000000| delays: -
  best: m0, b03 delays: 20+-06
  m1, b00: |00000000000000000000000000000000| delays: -
  m1, b01: |00000000000000000000000000000000| delays: -
  m1, b02: |11111111111100000000000000000000| delays: 05+-05
  m1, b03: |00000000000000011111111111111000| delays: 21+-06
  m1, b04: |00000000000000000000000000000000| delays: -
  m1, b05: |00000000000000000000000000000000| delays: -
  m1, b06: |00000000000000000000000000000000| delays: -
  m1, b07: |00000000000000000000000000000000| delays: -
  best: m1, b03 delays: 21+-06
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
  Write: 0x40000000-0x40200000 2.0MiB     
   Read: 0x40000000-0x40200000 2.0MiB     
Memtest OK
Memspeed at 0x40000000 (Sequential, 2.0MiB)...
  Write speed: 25.3MiB/s
   Read speed: 35.9MiB/s

--============== Boot ==================--
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Booting from SDCard in SPI-Mode...
Booting from boot.json...
boot.json file not found.
Booting from boot.bin...
Copying boot.bin to 0x40000000 (28504 bytes)...
[########################################]
Executing booted program at 0x40000000

--============= Liftoff! ===============--

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2021 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs


 LiteX git sha1: 648c70de

--=============== SoC ==================--
CPU:		FemtoRV-GRACILIS @ 48MHz
BUS:		WISHBONE 32-bit @ 4GiB
CSR:		32-bit data
ROM:		128KiB
SRAM:		8KiB
L2:		8KiB
SDRAM:		131072KiB 16-bit @ 192MT/s (CL-6 CWL-5)

--============= Console ================--

liteOS> catalog
switching ESP32 off
Could not mount filesystem

liteOS> 

I am confused as to how it can find and mount boot.bin but then cannot find and run other files present on the same SD card. Confusingly the same SD card works fine on the orangecrab and has not trouble mounting. Could this be something to do with the fact that LiteX forces me to build the SoC with a 100MHz clock frequency? See : enjoy-digital/litex#1726
Image of hardware setup here:

[BrunoLevy]

The only possible explanation that I have is that there are some differences in the way the LiteX BIOS initializes the SDCard and the way I'm doing it in LiteOS. The best way to figure out what happens is digging into the sources of the LiteX BIOS and see whether there are differences with LiteOS.

• LiteX BIOS is here: LiteX/litex/litex/soc/software/bios/, start from main.c and see where it starts the SDCard, mounts the filesystem and loads boot.bin
• LiteOS is there: learn-fpga/LiteX/software/LiteOS/, commands are in builtins.c, you will see that it calls fatfs_set_ops_spisdcard() then f_mount(). It seems that f_mount() returns an error, there may be other things to initialize.

I took a quick look and for now I do not see a difference. Here is what the LiteX BIOS does:

• in boot.c, things happen in sdcardboot(), that first calls fatfs_set_ops_spisdcard(), then sdcardboot_from_bin() that calls copy_file_from_sdcard_to_ram(), that calls f_mount(),f_open() etc... For now I do not see a difference with what I'm doing, but you can dig in the same area, maybe you'll see something. Sometimes LiteX evolves and there may be another thing to initialize (for instance, fatfs_set_ops_spisdcard() did not exist long time ago, and when it was introduced I had to figure out why it was not working...).

You have another possibility: graft the catalog and the run commands directly into LiteX BIOS. It is dirty, but it will tell you whether some initialization is missing in LiteOS.

Another possibility: examine what LiteX BIOS's main() function is doing (in main.c), and if there is something to initialize that I'm not doing in LiteOS (I took a quick look but did not see anything).

[BrunoLevy]

I think I got it !

A hint: from your logs, the BIOS thinks the CPU runs at 100 MHz, whereas LiteOS thinks it runs at 48MHz, so I bet you have on the SDCard a LiteOS image that does not correspond to your SOC (I'd bet this is the boot.bin image that corresponds to your orange crab).

Although the instruction set is the same (because you used femtorv-gracilis for both), the amount of RAM, devices configurations, mapped CSR addresses for the SPI controller may be different, then maybe LiteOS tries to access the SDCard through addresses that are not the right ones. LiteX has a (super smart) mechanism that generates the .h files with the constants and macros for accessing the CSRs, but these CSR can change when you change the hardware (then you need to regenerate boot.bin and all your programs).

You need to recompile LiteOS according to your SOC (make clean; make in LiteOS).

[JamesTimothyMeech]

Thanks! I noticed the difference in frequency but I rebuilt LiteOS after building the SoC and the problem persisted. I will now very carefully clear the SD card, recompile LiteOS and all my .elf programs after building the SoC for the arty to see if the frequencies for LiteX and LiteOS will match

[BrunoLevy]

Something else: make sure you point to the right subdirectory of LiteX/build, that is, the definition of LITEX_PLATFORM in makefile.inc, this would explain a lot ! (still pointing to your orangecrab build and pulling header files from there).

[JamesTimothyMeech]

Thanks everything works now! I had the orangecrab hardcoded in makefile.inc and also I was also not setting BUILD_DIR in the LiteOS makefile. Two pretty stupid mistakes, sorry about that. I have learnt my lesson not to hard code these things in the makefile. I should now have everything I need to run my experiments, thanks for bearing with me and my many questions!

[BrunoLevy]

Cool !
I'll try to add a mechanism to display as much information as possible about the target SOC in LiteOS and in the binaries, so that this type of problem is more easily detected (it also happened to me !!)