Cleaned up SBI start process (a0 contains hart id), test and verify t…

…he bpif3 native uart rx, another pass on README documentation, add sbif3 single hart jtag config file, default make is now vf2 board but v0.31 release will have all board binaries precompiled.

Makefile

@@ -10,8 +10,8 @@ runtime_config := sbi
 #
 
 #board_config := unmatched
-#board_config := vf2
-board_config := bpif3
+board_config := vf2
+#board_config := bpif3
 
 K=kernel
 U=user

README-opensbi-BananaPi-F3.md

@@ -32,7 +32,7 @@ In summary, the observed differences beween bpif3 and other boards:
 - DDR, Uboot and OpenSBI exist at different addresses. Linux appeared to be loading around 0x10000000, so xv6 is linked
 to locate starting at this address.
 - It has 8 cores, but I have been unable to bring them all online (only 4 appear to start when requesting them from SBI using the
-published API).
+published API, with 4, 5 and 6 seemingly missing in action :).
 - As mentioned, Jtag is not available for cores 1-7 until the SOC is executing somewhere in Linux.
 - The SPL boot section appears to be cryptographically signed.
 - The realtime core runs some ".elf" file when booting Linux. I can not find the source to this .elf file.  
@@ -73,7 +73,35 @@ interconnection can damage the board, interface or (unlikely) the host PC.
 
 Fascinatingly, this board fails the usertests program frequently, but sometimes at a new place - rmdot.
 
-It may also fail at sbrkmuch (!) just like unmatched. Curious. And sometimes it passes all tests.  
+```
+...
+test subdir: OK
+test bigwrite: OK
+test bigfile: OK
+test fourteen: OK
+test rmdot: usertrap(): unexpected scause 0x2 pid=19216
+            sepc=0x6e26 stval=0x0
+FAILED
+SOME TESTS FAILED
+$
+```
+
+It may also fail at sbrkmuch (!) just like unmatched. Curious. And sometimes it passes all tests: 
+
+```
+...
+test bigdir: OK
+test manywrites: OK
+test badwrite: OK
+test execout: OK
+test diskfull: balloc: out of blocks
+balloc: out of blocks
+OK
+test outofinodes: ialloc: no inodes
+OK
+ALL TESTS PASSED
+$
+```
 
 I did not do as much detailed debugging with this board as was done on unmatched to look beyond the failure string.
 
@@ -82,23 +110,20 @@ I did not do as much detailed debugging with this board as was done on unmatched
 Its possible to run xv6 with a single core. In main.c, comment out this:
 
 ```
-    // cant start hart 1 for now since 0 was mapped to 1
-    //for (int i = 2; i < NCPU; i++)
+    // harts 0-7 (inclusive)
+    //for (int i = 0; i < NCPU; i++)
     //{
     //  if (i != boothartid)
     //  {
-    //    sbi_hart_start(i, harts_entry[i]);
-    //
+    //    sbi_hart_start(i, (unsigned long) &_entry_sbi);
     //  }
     //}
 ```
 That single core will boot xv6 with a single core and jtag works with core 0 and all is good.
 
 The problem is with cores 1-7, which appear inaccessible pre-Linux. Attempts to start them with the above code is partially successful.
 
-I end up with exactly 4 cores running, and those are visible to Jtag once they are running. The others silently fail for some unknown reason.
-
-Note: I presently have a wart in the code having to do with core 1 that is the reason for starting i at 2.  This should get resolved in a next revision.
+I end up with exactly 5 cores running. Those are visible to Jtag once they are running. The others silently fail Jtag for some unknown reason.
 
 # Jtag Workaround Attempt
 
@@ -110,24 +135,32 @@ For all cores,
 ```
 reg satp 0x0
 reg sie 0x0
-reg tp = N (where N ranges from 2..7)
+reg tp = N (where N ranges from 1..7)
 reg pc = (address of) spin
 ```
 
 After loading the code with core 0 (thread 1 in gdb), xv6 runs with core 0 and all the rest are waiting at wfi at spin (label in entry_sbi.S).
 
-Next, halt the code and one by one set a core pc to ```_entry_hartN``` (where N ranges from 2..7).
+Next, halt the code and one by one initialize a hart with the hart ID in a0, and set pc to ```_entry_sbi```  
+
+For example, hart 1 can enter xv6 this way (assuming it was previously at the label ```spin```):
+
+```
+reg a0=1
+reg pc=_entry_sbi
+resume
+```
 
-All revectored cores are now running XV6.
+After doing this for all harts, they were confirmed running xv6. So xv6 is ok with all 8 cores, but something stuck in SBI to launch them.
 
-But attempts to get this to occur correctly (via SBI and also from uboot) have failed. I even tried to tuck all of the Linux cores back to SBI by first setting their PC to the new function 'park' which calles SBI to "stop" them and then try to run unmodified xv6 that asks cores 2..7 to start at the appropriate time.
+I tried to tuck all of the Linux cores back into SBI by first setting their PC to the new function 'park' which calles SBI to "stop" them and then try to run unmodified xv6 that calls SBI to start cores 1..7 at the appropriate time.
 
-That plan failed, although park seems to power down the core (and Jtag cant talk to it once that happens).
+That plan failed, although park seems to power down the core (and Jtag cant talk to it once that happens). So thats a clue.
 
 The real solution to this is to get Jtag to work while the code is running Uboot to try and figure out why they cant emerge when requested. Unresponsive cores appear to be "powered off" and the code to turn them on (so maybe they could wait at a wfi like the U74 cores do) is not
 documented (I specifically saw some undocumented register addresses getting written to in the SBI source code that power them up).
 
-This code snapshot calls SBI for cores 2-7 and ends up with 4 running cores with no error returned for the cores that dont start (yes, the return value was checked offline).
+This code snapshot calls SBI for cores 0-7 (skipping the boot core) and ends up with 5 running cores with no error returned for the cores that dont start (yes, the return value was checked offline).
 
 # Closing Thought
 

README-opensbi-overview.md

@@ -257,8 +257,9 @@ If the sequence is successful, the startup should be similar to this:
 
 xv6 kernel is booting
 
-hart 3 starting
 hart 2 starting
+hart 1 starting
+hart 3 starting
 hart 7 starting
 init: starting sh
 $ ls
@@ -286,7 +287,7 @@ console        3 20 0
 $
 ```
 
-Note that only 4 of 8 cores are running on this board. This is the subject of some discussion in the README-opensbi-BananaPi-F3.md file.
+Note that only 5 of 8 cores are running on this board. This is the subject of some discussion in the README-opensbi-BananaPi-F3.md file.
 
 # Code compilation
 
@@ -313,9 +314,7 @@ param.h must be used to select direct UART polling:
 The location and nature of UART Rx polling is different across the boards, so the configuration changes
 parameters necessary to perform direct uart polling (still within the timer interrupt/trap) instead of SBI API.
 
-Note that at the present time, this option is not available for the bpif3 board (mainly because I didnt code or
-test it yet). So only use this option on unmatched (optional) or vf2 (mandatory). Refer to the param.h file for
-a table of valid options depending on the board.
+This option works on all 3 boards in the 3.1 release. So only use this option on unmatched and bpif3 (optional) or vf2 (mandatory). Refer to the param.h file for a table of valid options depending on the board.
 
 With options chosen, compile the code using riscv64-linux-gnu compiler tools that are available in your path. On debian, this command both installs the required tools and inserts it into your PATH:
 

README-opensbi-unmatched.md

@@ -40,10 +40,36 @@ If the default USB Jtag interface works for you, a slightly different Jtag .cfg
 One executable in the ramdisk is called usertests. It samples the xv6 kernel API and prints dialog on its progress. The tests
 are cleverly written with expected results (including for faults). 
 
-Unfortunately, I've met my match with the intermittent failure of one particular test on unmatched: sbrkmuch() within usertests.c. This test runs fine:
+Unfortunately, I've met my match with an intermittent failure of one particular test on unmatched: sbrkmuch() within usertests.c:
+
+```
+...
+test iref: OK
+test forktest: OK
+test sbrkbasic: OK
+test sbrkmuch: usertrap(): unexpected scause 0xd pid=6475
+            sepc=0x6a3a stval=0x11068
+FAILED
+SOME TESTS FAILED
+$
+```
+
+This test runs fine:
 
 - on the VF2 board
 - occasionally (unmodified). Run the test a few times in a row from the command prompt.
+```
+...
+test badwrite: OK
+test execout: OK
+test diskfull: balloc: out of blocks
+balloc: out of blocks
+OK
+test outofinodes: ialloc: no inodes
+OK
+ALL TESTS PASSED
+$
+```
 - when run standalone (via command line parameter to just run sbrkmuch).
 - when its moved or called differently from its original version. Please try included "usertests2" which simply reverses the call order of sbrkbasic and sbrkmuch with no other changes. Both versions are on the supplied ramdisk.
 - of course, in the original MIT QEMU emulation.

jtag/spacemit_k1_ftdi_core0.cfg

@@ -0,0 +1,86 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+set _SPEED 8000
+#
+# FTDI MiniModule
+#
+# http://www.ftdichip.com/Support/Documents/DataSheets/Modules/DS_FT2232H_Mini_Module.pdf
+#
+adapter speed  4000
+adapter driver ftdi
+ftdi_device_desc "FT2232H-56Q MiniModule"
+ftdi_vid_pid 0x0403 0x6010
+reset_config trst_only
+transport select jtag
+
+# Every pin set as high impedance except TCK, TDI, TDO and TMS
+ftdi_layout_init 0x0008 0x000b
+
+# nTRST defined on pin CN2-14 of the MiniModule (pin ADBUS5 [AD5] on the FT2232H chip)
+# This choice is arbitrary. Use other GPIO pin if desired.
+#ftdi_layout_signal nSRST -data 0x0020 -oe 0x0020
+ftdi_layout_signal nTRST -data 0x0020 -oe 0x0020
+
+set _CHIPNAME riscv
+
+jtag newtap pre unknown -irlen 1 -expected-id 0x00000000 -disable
+jtag configure pre.unknown -event tap-enable ""
+
+jtag newtap $_CHIPNAME cpu -irlen 5 -expected-id 0x10000E21 -disable
+jtag configure $_CHIPNAME.cpu -event tap-enable ""
+
+jtag newtap post unknown -irlen 9 -expected-id 0x08501C0D
+
+set _TARGETNAME $_CHIPNAME.cpu
+
+# cluster 0
+target create $_TARGETNAME.0 riscv -chain-position $_TARGETNAME -coreid 0 -rtos hwthread
+#target create $_TARGETNAME.1 riscv -chain-position $_TARGETNAME -coreid 1
+#target create $_TARGETNAME.2 riscv -chain-position $_TARGETNAME -coreid 2
+#target create $_TARGETNAME.3 riscv -chain-position $_TARGETNAME -coreid 3
+
+# cluster 1
+#target create $_TARGETNAME.4 riscv -chain-position $_TARGETNAME -coreid 0 -dbgbase 0x400 -rtos hwthread
+#target create $_TARGETNAME.5 riscv -chain-position $_TARGETNAME -coreid 1 -dbgbase 0x400
+#target create $_TARGETNAME.6 riscv -chain-position $_TARGETNAME -coreid 2 -dbgbase 0x400
+#target create $_TARGETNAME.7 riscv -chain-position $_TARGETNAME -coreid 3 -dbgbase 0x400
+
+# cerate smp then we can use one gdb to debug all harts
+# harts on same smp will halt/resume together, if don't want it, just comment out all "target smp" lines,
+# then each hart needs one gdb to debug, means we need 8 gdb for 8 harts.
+#target smp $_TARGETNAME.0 $_TARGETNAME.1 $_TARGETNAME.2 $_TARGETNAME.3 $_TARGETNAME.4 $_TARGETNAME.5 $_TARGETNAME.6 $_TARGETNAME.7
+
+$_TARGETNAME.0 configure -event examine-end {
+    puts [ adapter speed ]
+    puts [ scan_chain ]
+    puts [ targets ]
+}
+
+proc default_mem_access {} {
+    foreach t [target names] {
+        # $t riscv set_mem_access sysbus progbuf
+        $t riscv set_mem_access progbuf
+    }
+}
+
+jtag configure post.unknown -event setup {
+    global _CHIPNAME
+    global _SPEED
+
+    irscan post.unknown 0x98
+    drscan post.unknown 16 0xa
+
+    jtag tapenable pre.unknown
+    jtag tapenable $_CHIPNAME.cpu
+
+    adapter speed $_SPEED
+}
+
+# TAP riscv.cpu is disabled, can't be the current target, use $t configure..
+default_mem_access
+
+foreach t [target names] {
+    # read in gdb: i r csr1498
+    $t riscv expose_csrs 1498-1499,1520-1521,1984-2003,2032-2039,3008-3023,3064-3071,4040-4041,4048-4051
+}
+
+

kernel/entry_sbi.S

@@ -8,61 +8,26 @@
 .section .text
 
 .global _entry
-.globl _entry_hart1
-.globl _entry_hart2
-.globl _entry_hart3
-.globl _entry_hart4
-#if defined(BOARD_BPIF3)
-.globl _entry_hart5
-.globl _entry_hart6
-.globl _entry_hart7
-.globl _entry_hart8
-#endif
+.globl _entry_sbi
+
 .globl _entry_mistake
 .extern __bss_start
 .extern __bss_end
 
 _entry:
-        # for openboot initiated launch, tp has the hart id in tp already
-        # and in the general case can be any hart
-        # 0 is reserved for siFive RT cores (unmatched, VF2) and will not enter here in tp
-#if defined(BOARD_BPIF3)
-        # processor 0 is usable on BPIF3 board (spacemit k1) and code expects tp from 1..NCPU for stack
-        # we observe processor 0 always selected for this board but presumably can range from 0-7
-        addi tp, tp, 1
-#endif
+        # for uboot initiated launch, the hart id is in tp already
+        # and is chosen randomly in in the general case
+
+        # 0 is reserved for siFive RT cores (unmatched, VF2) and wont be selected
+        # 0 is used on bpif3, however
+
         call clear_bss
         j entry_run
-        # xv6/opensbi initiated harts do not self identify hart id so when we
-        # start them in main() we direct their start address appropriately
-        # logic there prevents a request to send in the same hart as uboot did
-        # and already range from 1..max
-_entry_hart1:
-        li tp,1
-        j entry_run
-_entry_hart2:
-        li tp,2
-        j entry_run
-_entry_hart3:
-        li tp,3
-        j entry_run
-_entry_hart4:
-        li tp,4
-#if defined(BOARD_BPIF3)
-        # BPI F3 has spacemit chip with 8 usable cores
-        j entry_run
-_entry_hart5:
-        li tp,5
-        j entry_run
-_entry_hart6:
-        li tp,6
-        j entry_run
-_entry_hart7:
-        li tp,7
-        j entry_run
-_entry_hart8:
-        li tp,8
-#endif
+
+        # opensbi initiated harts identify their hart in a0
+        # xv6 needs it in tp
+_entry_sbi:
+        mv tp,a0
 entry_run:
 
         # set up a stack for C.
@@ -73,7 +38,11 @@ entry_run:
         li a0, 1024*4
         # csrr a1, mhartid
         mv a1,tp
-        # addi a1, a1, 1
+#if defined(BOARD_BPIF3)
+        # processor 0 is usable on BPIF3 board (spacemit k1) so we need to adjust for the stack calculation
+        # unmatched and SV2 enables 1..4 and uboot+main() leaves 0 inside SBI (since its not a RV64G processor)
+        addi a1, a1, 1
+#endif
         mul a0, a0, a1
         add sp, sp, a0
         # jump to start() in start.c