Telemetry2 thread info #9084

jsarha · 2024-04-26T11:11:23Z

No description provided.

lgirdwood

Good stuff, will help with debug, I expect we will also be able to call these APIs via shell too.

src/debug/telemetry/telemetry2_slot.c

src/debug/telemetry/telemetry2_thread_info.c

lgirdwood · 2024-04-26T12:17:58Z

src/debug/telemetry/telemetry2_thread_info.c

+	return 0;
+}
+
+static uint8_t thread_info_cpu_utilization(struct k_thread *thread,


It looks like some of this file is RTOS specific and should go in Zephyr as Zephyr APIs.

Its just refining the Zephyr thread stats cycle-counter values to produce load information. It requires cached values from previous round, and a generic Zephyr size implementation would become quite complex, requiring caller specific context allocation etc. If there is time for it, I can do it of course, but AFAIK we do not have it ATM.

tools/telemetry2/thread-analyzer.py

jsarha · 2024-04-29T21:02:13Z

One major change idea for this would be making each core data to have its own chunk. In the end it would same couple of bytes and maybe look a bit cleaner. A down side would be the core specific chunks be there in the telemetry2 slot in 'random' order depending on which cores are started up first.

plbossart

lots of unclear parts.

If this is a Zephyr-based telemetry, then why not call it telemetry-zephyr instead of a telemetry2 which means someone will suggest a telemetry3 at some point...

plbossart · 2024-04-29T22:00:59Z

src/debug/telemetry/Kconfig

@@ -9,3 +9,8 @@ config SOF_TELEMETRY
 	  systick_info measurement which measures scheduler task performance and may
 	  slightly affect overall performance.

+config SOF_TELEMETRY2_SLOT
+	bool "Enable SOF telemetry2 slot in debug window"


what on earth is telemetry2?

I think @lgirdwood 's original plan was to move the current telemetry node under a chunk in this new system, e.g. not to waste full 4k just for the current telemetry structure. That of course would need agreement from Windows side and some coordination, so for the time being I just decided to call it telemetry2.

The precisions are even less clear than the original, sorry. you're drowning me in details I didn't even know I didn't need :-)

src/debug/telemetry/telemetry2_slot.c

src/debug/telemetry/telemetry2_thread_info.c

plbossart · 2024-04-29T22:10:09Z

src/debug/telemetry/telemetry2_thread_info.c

+
+struct thread_info_chunk {
+	struct telemetry2_chunk_hdr hdr;
+	uint16_t core_count;


a thread can run on multiple cores?

This is the header of the whole thread info chunk, and the number indicates how many cores we have. The code keeps all the core separate all the time. Nothing else seems to work (DSP hangs). So there is a thread running on each core reporting the threads on that core. This header is just written once in the beginning and is used by the user-space client to find the core specific sections. I'll add some more comments.

tools/telemetry2/thread-info.py

jsarha · 2024-05-02T22:24:23Z

Applied all pending comments, but the big question about the new telemetry slots name is still unanswered. What about telemetry_v2, that would indicate that telemetry_v2 is trying to be a super-set of telemetry (v1) and while it can coexist with it, it can also make it obsolete. There is probably also still more comments to be written.

plbossart · 2024-05-03T12:44:49Z

telemetry_v2 is trying to be a super-set of telemetry (v1) and while it can coexist with it, it can also make it obsolete.

That really requires explanations !

lgirdwood · 2024-05-06T16:06:49Z

telemetry_v2 is trying to be a super-set of telemetry (v1) and while it can coexist with it, it can also make it obsolete.

That really requires explanations !

Intention is to be able to see Zephyr RTOS resources in realtime, not break backwards compatibility with existing telemetry and grow alongside Zephyr. Today only 1 backend, the mem Window, but should also work with probes in future.

V2 naming is really about the ABI, i.e. the feature supports existing "v1" telemetry for compatibility (do not break and can embed) but v2 is to exploit all the data that is available via Zephyr RTOS APIs around scheduling, memory, IRQs, locking, MMU, PM etc.

Expectation is that Zephyr data will grow/change over time hence an IPC based data format is used that will allow ABI growth and wont break tooling.

jsarha · 2024-05-07T21:19:58Z

Made some necessary updated due to changes to my PR on Zephyr side (zephyrproject-rtos/zephyr#71408) and while at it, added some text to telemetry2 (I did not change the name just yet) commit message and associated Kconfig help section and header. Also added a commit to increase debug window size from 12k to 16k.

kv2019i

I don't regularly review drafts, but some drive-by comments given we have some shared problems with dw-slot usage. No blockers in the PR really. If we want to enable this by default, some review/scrutiny on the perf impact might be needed.

src/debug/telemetry/Kconfig

src/debug/telemetry/telemetry2_thread_info.c

kv2019i · 2024-05-14T13:08:03Z

app/boards/intel_adsp_ace15_mtpm.conf

@@ -79,7 +79,7 @@ CONFIG_INTEL_ADSP_IPC=y
 CONFIG_WATCHDOG=y
 CONFIG_LL_WATCHDOG=y

-CONFIG_MEMORY_WIN_2_SIZE=12288
+CONFIG_MEMORY_WIN_2_SIZE=16384


I wonder how we can make this scale better. Did @marcinszkudlinski you have ideas on this, I think you hit this with tracing backend as well.

I now pushed first PR for shell support zephyrproject-rtos/zephyr#72738 . I was contemplating on adding it to a new slot (and increase WIN_2 size), but:

shell won't be enable in all builds (just like tracing and probably this telemetry2 as well)

having to touch the CONFIG_MEMORY_WIN_2_SIZE requires touching multiple board specific files, so enabling the feature with a simpler overlay no longer works

No better solution yet. For shell, I now reuse the dw-slot for mtrace and force exclusivity between shell and mtrace.

Lets hard code today and we cant print it in teh tooling "I am using LNL so using hard codec value of X", in the future we can have a more dynamic approach.

jsarha · 2024-05-22T20:14:41Z

Update the implementation to use the latest API update for this PR: zephyrproject-rtos/zephyr#71408

Also some updates to code comments. Got rid of C++ //-comments.

TELEMETRY2 is a generic memory structure for passing real-time debug information from the SOF system. It only provides the framework for passing pieces of abstract data from DSP side to host side debugging tools. This commit implements telemetry2 data passing from DSP to host side using debug memory window. The commit defines telemetry2 payload header and telemetry2 chunk header. Adds function to search, or if not found reserve, a chunk of specified size and id from telemetry2 slot. Each chunk is forced to star from 64-byte cache line boundary. A pointer to the chunk is returned. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

Adds code for collecting thread statistic for cpu and stack usage, and for writing that information to telemetry2 memory chuck. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

Simple code to open telemetry2 debugfs file, check telemetry2 payload header, look for thread info chunk, decode it periodically, and print contents to stdout. Without any arguments this the script opens and polls telemetry2 debugfs file contents. The DSP should be up and running For there to be any data to be decoded. When DSP up, the telemetry2 thread info support is compiled into the FW, and the SOF Linux driver supports telemetry2 slot mapping to debugfs, the output should look like this: core_count: 3 Core 0 load 1% 0x401184a8 cpu 0.0% stack 2.0% 0x401183e0 cpu 0.0% stack 2.0% 1 thread info cpu 0.0% stack 8.2% 0 thread info cpu 0.0% stack 47.1% edf_workq cpu 0.0% stack 10.2% sysworkq cpu 0.0% stack 31.8% logging cpu 0.8% stack 19.6% idle 00 cpu 98.0% stack 33.7% Core 2 load 6% ll_thread2 cpu 0.0% stack 18.8% idle 02 cpu 93.3% stack 27.1% 0x40118570 cpu 0.4% stack 47.1% 2 thread info cpu 0.0% stack 47.1% Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

Increase Debug Window (Memory window 2) size by one slot (4096 bytes). The extra space is needed for telemetry2 slot. See SOF_TELEMETRY2_SLOT option for details. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

jsarha · 2024-05-24T14:20:04Z

Relax locking for telemetry2 chunk reservation. Fix some minor style issues.

lgirdwood

Getting better.

lgirdwood · 2024-05-29T14:51:24Z

src/debug/telemetry/telemetry2_slot.h

+	uint32_t hdr_size;	// size of this header only in bytes
+	uint32_t total_size;	// total size of the whole payload in bytes
+	uint32_t abi;		// ABI version, can be used by tool to warn users if too old.
+	uint64_t tstamp;	// aligned with host epoch.


We should same epoch as logging.

lgirdwood · 2024-05-29T14:52:10Z

src/debug/telemetry/telemetry2_thread_info.c

+#define THREAD_INFO_MAX_THREADS 16
+
+struct thread_info {
+	char name[14];


is there a Zephyr define for this ? - can we reuse it ?

lgirdwood · 2024-05-29T14:52:42Z

src/debug/telemetry/telemetry2_thread_info.c

+	char name[14];
+	uint8_t stack_usage;	/* precentage */
+	uint8_t cpu_usage;	/* precentage */
+} __packed;


ok, but wont that need cache line alignment ?

lgirdwood · 2024-05-29T14:55:18Z

src/debug/telemetry/telemetry2_thread_info.c

@@ -65,6 +65,7 @@ static struct previous_counters { // Cached data from previous round
 * k_thread_foreach_current_cpu().
 */
 struct user_data {
+	int core;


Data in commit message looks good, but I would put the busy/idle % on the top row (ranking on % busy stack can come later).

lgirdwood · 2024-05-29T14:56:56Z

tools/telemetry2/thread-info.py

+import ctypes
+import time
+import sys
+


Can we comment this file more, i.e. describe what the macros and code blocks do.

lgirdwood · 2024-05-29T14:58:59Z

app/boards/intel_adsp_ace15_mtpm.conf

@@ -79,7 +79,7 @@ CONFIG_INTEL_ADSP_IPC=y
 CONFIG_WATCHDOG=y
 CONFIG_LL_WATCHDOG=y

-CONFIG_MEMORY_WIN_2_SIZE=12288
+CONFIG_MEMORY_WIN_2_SIZE=16384


Lets hard code today and we cant print it in teh tooling "I am using LNL so using hard codec value of X", in the future we can have a more dynamic approach.

lgirdwood · 2024-05-29T15:04:26Z

src/debug/telemetry/telemetry2_slot.h

@@ -54,6 +54,9 @@

 #include <stdint.h>

+// HACK: This should be in zephyr/soc/intel/intel_adsp/common/include/adsp_debug_window.h


So if the payload is application specific we should be in SOF header, but RTOS specific in Zephyr header - if it s both then we cant keep in in application header.

lgirdwood · 2024-07-05T15:44:30Z

@jsarha any updates ? I think we should have all Zephyr commits upstream now ? Can you rebase too.

jsarha · 2024-07-08T14:36:20Z

@lgirdwood I am currently doing (or actually re-spinning after two week bug-hunt break) the major rework to change to debug window protocol to core specific circular buffers, you requested. I do not expect it to be ready before end of the week, when I start my two week summer vacation, but I hope it does not take more than another week after that.

kv2019i · 2024-09-09T08:08:16Z

@jsarha This can be closed now with #9398 merged ?

jsarha · 2024-09-13T09:35:58Z

Yes. I'll close it now.

jsarha mentioned this pull request Apr 26, 2024

[FEATURE][ChromeOS] Expose DSP load tracker info for the visibility to ChromeOS #5521

Open

lgirdwood reviewed Apr 26, 2024

View reviewed changes

jsarha force-pushed the telemetry2-thread-info branch from abe86e9 to 54fe421 Compare April 29, 2024 21:36

plbossart reviewed Apr 29, 2024

View reviewed changes

jsarha mentioned this pull request May 2, 2024

Telemetry2 debugfs thesofproject/linux#4956

Closed

jsarha force-pushed the telemetry2-thread-info branch from 6439323 to e61a976 Compare May 2, 2024 22:18

jsarha force-pushed the telemetry2-thread-info branch from e61a976 to b3a1d59 Compare May 3, 2024 09:48

jsarha mentioned this pull request May 7, 2024

Thread analyzer adsp fix zephyrproject-rtos/zephyr#71408

Merged

jsarha force-pushed the telemetry2-thread-info branch from b3a1d59 to 4285f67 Compare May 7, 2024 21:17

kv2019i reviewed May 14, 2024

View reviewed changes

jsarha force-pushed the telemetry2-thread-info branch from 4285f67 to fa7f51d Compare May 22, 2024 20:12

Jyri Sarha added 6 commits May 24, 2024 16:54

telemetry2: thread_info: Add telemetry2 thread info module

0acff86

Adds code for collecting thread statistic for cpu and stack usage, and for writing that information to telemetry2 memory chuck. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

memory_window: Increase Debug Window size from 12k to 16k

aad8ab4

Increase Debug Window (Memory window 2) size by one slot (4096 bytes). The extra space is needed for telemetry2 slot. See SOF_TELEMETRY2_SLOT option for details. Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>

src/debug/telemetry/telemetry2_slot.h: HACK add ADSP_DW_SLOT_TELEMETRY2

ab028a0

app/prj.conf: HACK enable Thread Info

7747c68

jsarha force-pushed the telemetry2-thread-info branch from fa7f51d to 7747c68 Compare May 24, 2024 14:19

lgirdwood reviewed May 29, 2024

View reviewed changes

lgirdwood added this to the v2.11 milestone Jun 18, 2024

kv2019i removed this from the v2.11 milestone Sep 10, 2024

jsarha closed this Sep 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Telemetry2 thread info #9084

Telemetry2 thread info #9084

jsarha commented Apr 26, 2024

lgirdwood left a comment

lgirdwood Apr 26, 2024

jsarha Apr 29, 2024

jsarha commented Apr 29, 2024 •

edited

Loading

plbossart left a comment

plbossart Apr 29, 2024

jsarha May 2, 2024

plbossart May 2, 2024

plbossart Apr 29, 2024

jsarha May 2, 2024

jsarha commented May 2, 2024

plbossart commented May 3, 2024

lgirdwood commented May 6, 2024

jsarha commented May 7, 2024

kv2019i left a comment

kv2019i May 14, 2024

lgirdwood May 29, 2024

jsarha commented May 22, 2024

jsarha commented May 24, 2024 •

edited

Loading

lgirdwood left a comment

lgirdwood May 29, 2024

lgirdwood May 29, 2024

lgirdwood May 29, 2024

lgirdwood May 29, 2024

lgirdwood May 29, 2024

lgirdwood May 29, 2024

lgirdwood May 29, 2024

lgirdwood commented Jul 5, 2024 •

edited

Loading

jsarha commented Jul 8, 2024 •

edited

Loading

kv2019i commented Sep 9, 2024

jsarha commented Sep 13, 2024

		@@ -54,6 +54,9 @@

		#include <stdint.h>

		// HACK: This should be in zephyr/soc/intel/intel_adsp/common/include/adsp_debug_window.h

Telemetry2 thread info #9084

Telemetry2 thread info #9084

Conversation

jsarha commented Apr 26, 2024

lgirdwood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsarha commented Apr 29, 2024 • edited Loading

plbossart left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsarha commented May 2, 2024

plbossart commented May 3, 2024

lgirdwood commented May 6, 2024

jsarha commented May 7, 2024

kv2019i left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsarha commented May 22, 2024

jsarha commented May 24, 2024 • edited Loading

lgirdwood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lgirdwood commented Jul 5, 2024 • edited Loading

jsarha commented Jul 8, 2024 • edited Loading

kv2019i commented Sep 9, 2024

jsarha commented Sep 13, 2024

jsarha commented Apr 29, 2024 •

edited

Loading

jsarha commented May 24, 2024 •

edited

Loading

lgirdwood commented Jul 5, 2024 •

edited

Loading

jsarha commented Jul 8, 2024 •

edited

Loading