-
Notifications
You must be signed in to change notification settings - Fork 667
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Dev support oneflow insight (#10370)
本Pr实现了OneFlow Insight模块,相关issue:Oneflow-Inc/OneTeam#2162 当我们需要profiling cuda kernel执行时间/瓶颈分析时,通常会基于nvidia提供的nsys指令,生成对应的profile文件(早期的.qdrep以及现在的.nsys-rep)并用Nvidia的GUI软件Nsight Systems来可视化分析、查看。 在nsys生成profile文件的同时,还会生成平台无关的数据信息,记录在.sqlite文件中,OneFlow Insight模块就可以通过解析.sqlite,来生成符合Google Chrome Trace Event格式的JSON文件,使得可以直接通过Chrome或者Edge浏览器,通过`chrome://tracing/` 或 `edge://tracing/`来解析和渲染此JSON文件,从而进行可视化分析、查看,效果如下: <img width="1320" alt="image" src="https://github.com/Oneflow-Inc/oneflow/assets/28823622/cbfab9bc-47bd-474c-8f39-e145348db17d"> --------- Co-authored-by: oneflow-ci-bot <ci-bot@oneflow.org>
- Loading branch information
1 parent
a0f2122
commit 82c965b
Showing
4 changed files
with
557 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
# OneFlow Insight | ||
|
||
## Overview | ||
|
||
OneFlow Insight is a module designed for profiling CUDA kernel execution time and bottleneck analysis. Typically, this is done using the nsys command provided by Nvidia, which generates corresponding profile files (formerly .qdrep and now .nsys-rep). These files can be visualized and analyzed using Nvidia's GUI software, Nsight Systems. | ||
|
||
In addition to generating profile files, nsys also produces platform-independent data information recorded in a .sqlite file. The OneFlow Insight module can parse this .sqlite file to generate a JSON file formatted according to the Google Chrome Trace Event standard. This allows for direct visualization and analysis through Chrome or Edge browsers using chrome://tracing/ or edge://tracing/ (supported by trace-event-profiling-tool, see:https://www.chromium.org/developers/how-tos/trace-event-profiling-tool/). | ||
|
||
|
||
## Usage | ||
|
||
1. Generate profile files using the following nsys command: | ||
|
||
```bash | ||
nsys profile --export=sqlite -o profile_data | ||
``` | ||
|
||
This will produce .nsys-rep files along with a .sqlite file. | ||
|
||
2. Use OneFlow Insight to parse the .sqlite file and generate a JSON file: | ||
|
||
```bash | ||
python3 sqlite_to_google_trace_event.py --input 'profile_data.sqlite' -o trace.json | ||
``` | ||
|
||
3. Open Chrome or Edge browser and navigate to chrome://tracing/ or edge://tracing/. | ||
|
||
4. Load the generated trace.json file for visualizing and analyzing the profiling data. | ||
|
||
## Visualization Example | ||
|
||
![OneFlow Insight Visualization](trace.json.png) | ||
|
||
The above image demonstrates the visualization capabilities using Chrome or Edge browser with the generated JSON file. | ||
|
||
Feel free to explore and gain insights into your CUDA kernel execution performance! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
sqlite3 | ||
argparse | ||
traceback |
Oops, something went wrong.