kernel NULL pointer dereference when loading ONIC.ko

[wnew]

Good day all.

We recently upgraded our Alveo AU50s with xbmgmt and flashed the latest shell. Since then I have been unable to load the onic drivers.

I have built the drivers against 5.19.0-50-generic and 6.2.0-26-generic and tried the shell with no user application build in both Vivado 2023.1 and 2022.1 but all combinations fail with insmod outputting "killed" and with the follow dmesg output.

Any suggestions would be much appreciated.

[ 1552.797006] OpenNIC Linux Kernel Driver 0.21
[ 1552.797061] onic 0000:c1:00.0: enabling device (0000 -> 0002)
[ 1552.797190] onic 0000:c1:00.0 onic193s0f0 (uninitialized): Set MAC address to 00:0a:35:c8:b0:46
[ 1552.797192] onic 0000:c1:00.0: device is a master PF
[ 1552.797366] onic 0000:c1:00.0: Allocated 8 queue vectors
[ 1553.297756] here3
[ 1553.297759] here4 0000000000000000
[ 1553.297764] BUG: kernel NULL pointer dereference, address: 0000000000000008
[ 1553.297765] #PF: supervisor read access in kernel mode
[ 1553.297767] #PF: error_code(0x0000) - not-present page
[ 1553.297769] PGD 0 P4D 0
[ 1553.297771] Oops: 0000 [#1] PREEMPT SMP NOPTI
[ 1553.297773] CPU: 15 PID: 5331 Comm: insmod Tainted: G OE 6.2.0-26-generic #26~22.04.1-Ubuntu
[ 1553.297775] Hardware name: Supermicro Super Server/H12SSL-i, BIOS 2.3 10/20/2021
[ 1553.297776] RIP: 0010:qdma_invalidate_fmap_ctxt+0x20/0x70 [onic]
[ 1553.297787] Code: 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 31 c9 31 d2 48 89 e5 48 83 ec 10 65 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 <0f> b7 47 08 c7 47 0a 00 00 00 00 48 8d 75 f4 25 ff 07 00 00 c1 e0
[ 1553.297789] RSP: 0018:ffffa1fca1cef8b8 EFLAGS: 00010246
[ 1553.297790] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 1553.297791] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1553.297792] RBP: ffffa1fca1cef8c8 R08: 0000000000000000 R09: 0000000000000000
[ 1553.297793] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8e6721cba9c0
[ 1553.297794] R13: 0000000000000000 R14: ffff8e665b04e000 R15: 0000000000000000
[ 1553.297795] FS: 00007fa2ce43dc40(0000) GS:ffff8e74debc0000(0000) knlGS:0000000000000000
[ 1553.297796] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1553.297797] CR2: 0000000000000008 CR3: 0000000261ca6006 CR4: 0000000000770ee0
[ 1553.297798] PKRU: 55555554
[ 1553.297799] Call Trace:
[ 1553.297800]
[ 1553.297803] onic_clear_hardware+0x65/0xf0 [onic]
[ 1553.297810] onic_init_hardware+0x1c9/0x7e0 [onic]
[ 1553.297816] onic_probe+0x22b/0x3e0 [onic]
[ 1553.297825] local_pci_probe+0x4b/0xb0
[ 1553.297828] pci_call_probe+0x55/0x190
[ 1553.297830] pci_device_probe+0x84/0x120
[ 1553.297832] really_probe+0x1ed/0x450
[ 1553.297835] __driver_probe_device+0x8a/0x1a0
[ 1553.297837] driver_probe_device+0x23/0xd0
[ 1553.297839] __driver_attach+0x10f/0x220
[ 1553.297841] ? __pfx___driver_attach+0x10/0x10
[ 1553.297843] bus_for_each_dev+0x83/0xe0
[ 1553.297845] driver_attach+0x1e/0x30
[ 1553.297847] bus_add_driver+0x152/0x250
[ 1553.297849] driver_register+0x83/0x160
[ 1553.297851] ? __pfx_init_module+0x10/0x10 [onic]
[ 1553.297857] __pci_register_driver+0x68/0x80
[ 1553.297859] onic_init_module+0x3d/0xff0 [onic]
[ 1553.297864] do_one_initcall+0x49/0x240
[ 1553.297868] ? kmalloc_trace+0x2a/0xb0
[ 1553.297871] do_init_module+0x52/0x240
[ 1553.297874] load_module+0xb96/0xd60
[ 1553.297876] ? security_kernel_post_read_file+0x5c/0x80
[ 1553.297879] ? kernel_read_file+0x25c/0x2b0
[ 1553.297882] __do_sys_finit_module+0xcc/0x150
[ 1553.297884] ? __do_sys_finit_module+0xcc/0x150
[ 1553.297886] __x64_sys_finit_module+0x18/0x30
[ 1553.297888] do_syscall_64+0x5c/0x90
[ 1553.297891] ? ksys_mmap_pgoff+0x123/0x270
[ 1553.297893] ? exit_to_user_mode_prepare+0x3b/0xd0
[ 1553.297896] ? syscall_exit_to_user_mode+0x2a/0x50
[ 1553.297898] ? do_syscall_64+0x69/0x90
[ 1553.297899] ? exit_to_user_mode_prepare+0x3b/0xd0
[ 1553.297901] ? syscall_exit_to_user_mode+0x2a/0x50
[ 1553.297903] ? do_syscall_64+0x69/0x90
[ 1553.297904] ? exc_page_fault+0x92/0x1b0
[ 1553.297906] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 1553.297909] RIP: 0033:0x7fa2cdb1ea3d
[ 1553.297911] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c3 a3 0f 00 f7 d8 64 89 01 48
[ 1553.297912] RSP: 002b:00007fffab80e038 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 1553.297914] RAX: ffffffffffffffda RBX: 000055da3d208780 RCX: 00007fa2cdb1ea3d
[ 1553.297915] RDX: 0000000000000000 RSI: 000055da3bb46cd2 RDI: 0000000000000003
[ 1553.297916] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 1553.297917] R10: 0000000000000003 R11: 0000000000000246 R12: 000055da3bb46cd2
[ 1553.297917] R13: 000055da3d20bd60 R14: 000055da3bb45888 R15: 000055da3d208890
[ 1553.297919]
[ 1553.297920] Modules linked in: onic(OE+) ipmi_ssif binfmt_misc intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd nls_iso8859_1 kvm irqbypass crct10dif_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha512_ssse3 aesni_intel crypto_simd cryptd rapl ast ftdi_sio wmi_bmof xocl(OE) xclmgmt(OE) drm_shmem_helper joydev usbserial input_leds drm_kms_helper i2c_algo_bit libcrc32c syscopyarea sysfillrect sysimgblt ccp ptdma k10temp acpi_ipmi ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel msr parport_pc ppdev lp parport drm ramoops pstore_blk reed_solomon pstore_zone efi_pstore ip_tables x_tables autofs4 mlx5_ib ib_uverbs ib_core hid_generic rndis_host usbhid cdc_ether usbnet hid mii mlx5_core mlxfw psample crc32_pclmul tls ahci pci_hyperv_intf tg3 libahci xhci_pci i2c_piix4 xhci_pci_renesas wmi
[ 1553.297952] CR2: 0000000000000008
[ 1553.297953] ---[ end trace 0000000000000000 ]---
[ 1554.216545] RIP: 0010:qdma_invalidate_fmap_ctxt+0x20/0x70 [onic]
[ 1554.216558] Code: 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 31 c9 31 d2 48 89 e5 48 83 ec 10 65 48 8b 04 25 28 00 00 00 48 89 45 f8 31 c0 <0f> b7 47 08 c7 47 0a 00 00 00 00 48 8d 75 f4 25 ff 07 00 00 c1 e0
[ 1554.216559] RSP: 0018:ffffa1fca1cef8b8 EFLAGS: 00010246
[ 1554.216561] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 1554.216562] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ 1554.216563] RBP: ffffa1fca1cef8c8 R08: 0000000000000000 R09: 0000000000000000
[ 1554.216564] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8e6721cba9c0
[ 1554.216565] R13: 0000000000000000 R14: ffff8e665b04e000 R15: 0000000000000000
[ 1554.216565] FS: 00007fa2ce43dc40(0000) GS:ffff8e74debc0000(0000) knlGS:0000000000000000
[ 1554.216567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1554.216568] CR2: 0000000000000008 CR3: 0000000261ca6006 CR4: 0000000000770ee0
[ 1554.216569] PKRU: 55555554
[ 1554.216570] note: insmod[5331] exited with irqs disabled

[cneely-amd]

This might have similarities to:
Xilinx/open-nic-driver#28 or
Xilinx/open-nic-driver#25

Issue #28 has a long discussion with some suggestions, but it would be helpful to understand more clearly what the correct solution is.

[wnew]

Great thanks. Ill take a read and see what I can figure out.