Adding support for persistent storage and retrieval of DPU reboot-cause #169

rameshraghupathy · 2024-10-11T03:18:33Z

Adding support for persistent storage and retrieval of DPU reboot-cause

oleksandrivantsiv · 2024-10-24T18:54:10Z

scripts/process-reboot-cause

@@ -69,6 +70,45 @@ def read_reboot_cause_files_and_save_state_db():
                x = TIME_SORTED_FULL_REBOOT_FILE_LIST[i]
                os.remove(x)

+def read_dpu_reboot_cause_files_and_save_chassis_state_db():
+    # Get platform using device_info.get_platform()
+    platform_info = device_info.get_platform_info()


The code to query DPUs from the platform.json should be part of src/sonic-py-common/sonic_py_common/device_info.py. There is an implementation for getting the get_num_dpus, so we can put something like get_dpus_info. This will prevent a code duplication in all places that need to access DPUs information

@oleksandrivantsiv will do

@oleksandrivantsiv done

oleksandrivantsiv · 2024-10-24T19:15:11Z

tests/determine-reboot-cause_test.py

+
+        # Assert that makedirs was called for the DPU directories
+        mock_makedirs.assert_any_call(os.path.join('/host/reboot-cause/module', 'dpu0'))
+        mock_makedirs.assert_any_call(os.path.join('/host/reboot-cause/module', 'dpu1'))


read_dpu_reboot_cause_files_and_save_chassis_state_d is not covered by the tests. Please new test for it.

@oleksandrivantsiv will do but can we defer it as the entire files doe not have a test yet.

hardware implementation requirements

prgeor · 2024-11-25T05:11:28Z

scripts/process-reboot-cause

@@ -69,6 +71,84 @@ def read_reboot_cause_files_and_save_state_db():
                x = TIME_SORTED_FULL_REBOOT_FILE_LIST[i]
                os.remove(x)

+def get_dpus():


@rameshraghupathy can we make use of existing APIs? @vvolam ?

@prgeor sure, will do, there is a dependency with @vvolam's commit and will do it once that is committed

@prgeor done

prgeor · 2024-11-25T05:14:03Z

scripts/process-reboot-cause

+    try:
+        # Assuming you have a way to list the files in the directory
+        files = os.listdir(dpu_history_path)
+        # Filter and sort the files based on your criteria (e.g., by modification time)


@rameshraghupathy why eg? Are we not sure?

@prgeor cleaned

@rameshraghupathy I mean why eg? You are indeed sorting by modification time so just put "sorted by modification time"

@prgeor cleaned

prgeor · 2024-11-25T05:15:24Z

scripts/process-reboot-cause

+                            _hash = f"{REBOOT_CAUSE_TABLE_NAME}|{data['gen_time']}"
+                            state_db.set(state_db.STATE_DB, _hash, 'cause', data.get('cause', ''))
+                            state_db.set(state_db.STATE_DB, _hash, 'time', data.get('time', ''))
+                            state_db.set(state_db.STATE_DB, _hash, 'user', data.get('user', ''))
+                            state_db.set(state_db.STATE_DB, _hash, 'comment', data.get('comment', ''))


@rameshraghupathy function says chassis STATE_DB but here its STATE_DB?

@prgeor Fixing the DB

prgeor · 2024-11-25T05:16:31Z

scripts/process-reboot-cause

+            # Get sorted reboot cause files for the DPU
+            reboot_files = get_sorted_reboot_cause_files(os.path.join(dpu_history_path, "history"))
+
+            for reboot_file in reboot_files:


@rameshraghupathy How do we handle a case where NPU comes late so that DPU to NPU mid plane is not UP by the time this process starts?

@prgeor As shown in the HLD the NPU-chassisd will fetch the reboot-cause from the DPU and persist it.

kperumalbfn · 2024-12-05T19:53:02Z

@oleksandrivantsiv could you please review and approve

prgeor · 2024-12-06T06:14:58Z

scripts/process-reboot-cause

+        return []
+
+
+def read_dpu_reboot_cause_files_and_save_chassis_state_db():


@rameshraghupathy can we break this into two functions?

@prgeor It is already done in two functions get_sorted_reboot_cause_files and saving to chassisStateDB using this function. The name of the function is misleading, so renamed it to save_dpu_reboot_cause_files_to_chassis_state_db()

mssonicbld · 2024-12-13T01:40:33Z

/azp run

azure-pipelines · 2024-12-13T01:40:42Z