Network-On-Chip Capability

[Soilorian]

Hello Mr.Sun,

I have been working with the simulator a lot and wanted to know if there is anything related to Network-On-Chip already implemented in mgpu-sim? More specifically, I am looking for how the CUs, caches, and DRAM are connected to each other and also how the GPUs communicate with one another.

another question: I tried the debugging ISA option but the files it generated with .debug extension were all empty. I was wondering, am I missing something? I used the following command:

./fir -timing -debug-isa -gpus=1,2 -length=40960

and also can you guide me to where the instructions are decoded and where can I find the instructions themselves after the benchmark is compiled?

Thank you in advance!

[syifan]

Akita and MGPUSim provides some NOC support. See here https://github.com/sarchlab/akita/tree/v3/noc.

Within a GPU, everything is pretty much just connected with wires (direct connections), while you can configure them to use more complex network.

For the question regarding the debug files are empty, it is very likely to because of this line

https://github.com/sarchlab/mgpusim/blob/v3/emu/isadebugger.go#L40
https://github.com/sarchlab/mgpusim/blob/v3/timing/cu/isa_debugger.go#L72

Remove the condition but keep the content will get all the instructions printed.

[syifan]

Could you please let me know what you want to build? Perhaps I can help you configure such systems.

[Soilorian]

I am aiming to implement the SAC(share-aware cache) cache system for LLC of the gpus. in order to do so, I need to be able to decide that the LLC caches data going to the SM from local or remote sources or it will cache the data coming from the gpu's memory for remote or local accesses.

this image shows exactly what I'm trying to achieve.

and this is what my final project should look like.

also I think you have missed my question regarding how can I find the instructions being executed for the program. I need them for the pre-process needed for SAC.

thanks in advance!

[syifan]

OK. I would say this is not really an "NOC" problem but a memory system configuration problem since the NOC latency is not the most critical thing you care about, but the topology, right? The default configuration is pretty much A. Have you checked how the current L1 cache is connected with the L2 cache? Here is the code.
https://github.com/sarchlab/mgpusim/blob/v3/samples/runner/r9nanobuilder.go#L297.

If you understand this code, it may help you configure (b) and (c). For the NOC in the figure, we pretty much just use a direct connection, which serves as a crossbar.

[Soilorian]

yes that's true, the latency of NOC is not of high importance to me at the moment.

I ,in fact, did take a look at this code but I didn't realize how each GPU can access other GPU's memories. But now that you have pointed out this line, I think that's handled by lowModuleFinder and rdmaEngine, right? so by changing their ordering and a little bit of how they function I can implement what I'm aiming for.

[syifan]

Yes. The LowModuleFinder is where the magic happens. RDMA is like a network interface that tunnels GPUs. To model your architecture, you definitely need to work on the LowModuleFindera and RDMAs. It may take some effort but should be doable.