Skip to content

Commit 91d6315

Browse files
cirosantillicirosantilli
committed
gem5 stats: more info, start looking into HDF5
1 parent f42c525 commit 91d6315

File tree

1 file changed

+137
-62
lines changed

1 file changed

+137
-62
lines changed

‎README.adoc

Lines changed: 137 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -11741,6 +11741,10 @@ End the simulation.
1174111741

1174211742
Sane Python scripts will exit gem5 with status 0, which is what `fs.py` does.
1174311743

11744+
===== m5 dumpstats
11745+
11746+
Makes gem5 dump one more statistics entry to the <<gem5-m5out-stats-txt-file>>.
11747+
1174411748
===== m5 fail
1174511749

1174611750
End the simulation with a failure exit event:
@@ -12092,7 +12096,7 @@ This file contains important statistics about the run:
1209212096
cat "$(./getvar --arch aarch64 m5out_dir)/stats.txt"
1209312097
....
1209412098

12095-
Whenever we run `m5 dumpstats` or `m5 exit`, a section with the following format is added to that file:
12099+
Whenever we run `m5 dumpstats` or when fs.py and se.py are exiting (TODO other scripts?), a section with the following format is added to that file:
1209612100

1209712101
....
1209812102
---------- Begin Simulation Statistics ----------
@@ -12110,6 +12114,74 @@ system.cpu.dtb.inst_hits
1211012114

1211112115
For x86, it is interesting to try and correlate `numCycles` with:
1211212116

12117+
In LKMC f42c525d7973d70f4c836d2169cc2bd2893b4197 gem5 5af26353b532d7b5988cf0f6f3d0fbc5087dd1df, the stat file for a <<c>> hello world:
12118+
12119+
....
12120+
./run --arch aarch64 --emulator gem5 --userland userland/c/hello.c
12121+
....
12122+
12123+
which has a single dump done at the exit, has size 59KB and stat lines of form:
12124+
12125+
....
12126+
final_tick 91432000 # Number of ticks from beginning of simulation (restored from checkpoints and never reset)
12127+
....
12128+
12129+
We can reduce the file size by adding the `?desc=False` magic suffix to the stat flie name:
12130+
12131+
....
12132+
--stats-file stats.txt?desc=false
12133+
....
12134+
12135+
as explained in:
12136+
12137+
....
12138+
gem5.opt --stats-help
12139+
....
12140+
12141+
and this reduces the file size to 39KB by removing those excessive comments:
12142+
12143+
....
12144+
final_tick 91432000
12145+
....
12146+
12147+
although trailing spaces are still prse
12148+
12149+
We can further reduce this size by removing spaces from the dumps with this hack:
12150+
12151+
....
12152+
ccprintf(stream, " |%12s %10s %10s",
12153+
ValueToString(value, precision), pdfstr.str(), cdfstr.str());
12154+
} else {
12155+
- ccprintf(stream, "%-40s %12s %10s %10s", name,
12156+
- ValueToString(value, precision), pdfstr.str(), cdfstr.str());
12157+
+ ccprintf(stream, "%s %s", name, ValueToString(value, precision));
12158+
+ if (pdfstr.rdbuf()->in_avail())
12159+
+ stream << " " << pdfstr.str();
12160+
+ if (cdfstr.rdbuf()->in_avail())
12161+
+ stream << " " << cdfstr.str();
12162+
12163+
if (descriptions) {
12164+
if (!desc.empty())
12165+
....
12166+
12167+
and after that the file size went down to 21KB.
12168+
12169+
===== gem5 HDF5 statistics
12170+
12171+
We can make gem5 dump statistics in the <<hdf5>> format by adding the magic `h5://` prefix to the file name as in:
12172+
12173+
....
12174+
gem5.opt --stats-file h5://stats.h5
12175+
....
12176+
12177+
as explained in:
12178+
12179+
....
12180+
gem5.opt --stats-help
12181+
....
12182+
12183+
TODO what is the advantage? The generated file for `--stats-file h5://stats.h5?desc=False` in LKMC f42c525d7973d70f4c836d2169cc2bd2893b4197 gem5 5af26353b532d7b5988cf0f6f3d0fbc5087dd1df was 946K, so much larger than the text version!
12184+
1211312185
===== gem5 only dump selected stats
1211412186

1211512187
TODO
@@ -12118,6 +12190,66 @@ https://stackoverflow.com/questions/52014953/how-to-dump-only-a-single-or-certai
1211812190

1211912191
To prevent the stats file from becoming humongous.
1212012192

12193+
===== gem5 stats internals
12194+
12195+
This describes the internals of the <<gem5-m5out-stats-txt-file>>.
12196+
12197+
GDB call stack to `dumpstats`:
12198+
12199+
....
12200+
Stats::pythonDump () at build/ARM/python/pybind11/stats.cc:58
12201+
Stats::StatEvent::process() ()
12202+
GlobalEvent::BarrierEvent::process (this=0x555559fa6a80) at build/ARM/sim/global_event.cc:131
12203+
EventQueue::serviceOne (this=this@entry=0x555558c36080) at build/ARM/sim/eventq.cc:228
12204+
doSimLoop (eventq=0x555558c36080) at build/ARM/sim/simulate.cc:219
12205+
simulate (num_cycles=<optimized out>) at build/ARM/sim/simulate.cc:132
12206+
....
12207+
12208+
`Stats::pythonDump` does:
12209+
12210+
....
12211+
void
12212+
pythonDump()
12213+
{
12214+
py::module m = py::module::import("m5.stats");
12215+
m.attr("dump")();
12216+
}
12217+
....
12218+
12219+
This calls `src/python/m5/stats/__init__.py` in `def dump` does the main dumping
12220+
12221+
That function does notably:
12222+
12223+
....
12224+
for output in outputList:
12225+
if output.valid():
12226+
output.begin()
12227+
for stat in stats_list:
12228+
stat.visit(output)
12229+
output.end()
12230+
....
12231+
12232+
`begin` and `end` are defined in C++ and output the header and tail respectively
12233+
12234+
....
12235+
void
12236+
Text::begin()
12237+
{
12238+
ccprintf(*stream, "\n---------- Begin Simulation Statistics ----------\n");
12239+
}
12240+
12241+
void
12242+
Text::end()
12243+
{
12244+
ccprintf(*stream, "\n---------- End Simulation Statistics ----------\n");
12245+
stream->flush();
12246+
}
12247+
....
12248+
12249+
`stats_list` contains the stats, and `stat.visit` prints them, `outputList` contains by default just the text output. I don't see any other types of output in gem5, but likely JSON / binary formats could be envisioned.
12250+
12251+
Tested in gem5 b4879ae5b0b6644e6836b0881e4da05c64a6550d.
12252+
1212112253
==== gem5 config.ini
1212212254

1212312255
The `m5out/config.ini` file, contains a very good high level description of the system:
@@ -14457,66 +14589,6 @@ TODO: analyze the trace for:
1445714589

1445814590
TODO: like <<gem5-event-queue-minorcpu-syscall-emulation-freestanding-example-analysis>> but even more complex!
1445914591

14460-
==== gem5 stats internals
14461-
14462-
This describes the internals of the <<gem5-m5out-stats-txt-file>>.
14463-
14464-
GDB call stack to `dumpstats`:
14465-
14466-
....
14467-
Stats::pythonDump () at build/ARM/python/pybind11/stats.cc:58
14468-
Stats::StatEvent::process() ()
14469-
GlobalEvent::BarrierEvent::process (this=0x555559fa6a80) at build/ARM/sim/global_event.cc:131
14470-
EventQueue::serviceOne (this=this@entry=0x555558c36080) at build/ARM/sim/eventq.cc:228
14471-
doSimLoop (eventq=0x555558c36080) at build/ARM/sim/simulate.cc:219
14472-
simulate (num_cycles=<optimized out>) at build/ARM/sim/simulate.cc:132
14473-
....
14474-
14475-
`Stats::pythonDump` does:
14476-
14477-
....
14478-
void
14479-
pythonDump()
14480-
{
14481-
py::module m = py::module::import("m5.stats");
14482-
m.attr("dump")();
14483-
}
14484-
....
14485-
14486-
This calls `src/python/m5/stats/__init__.py` in `def dump` does the main dumping
14487-
14488-
That function does notably:
14489-
14490-
....
14491-
for output in outputList:
14492-
if output.valid():
14493-
output.begin()
14494-
for stat in stats_list:
14495-
stat.visit(output)
14496-
output.end()
14497-
....
14498-
14499-
`begin` and `end` are defined in C++ and output the header and tail respectively
14500-
14501-
....
14502-
void
14503-
Text::begin()
14504-
{
14505-
ccprintf(*stream, "\n---------- Begin Simulation Statistics ----------\n");
14506-
}
14507-
14508-
void
14509-
Text::end()
14510-
{
14511-
ccprintf(*stream, "\n---------- End Simulation Statistics ----------\n");
14512-
stream->flush();
14513-
}
14514-
....
14515-
14516-
`stats_list` contains the stats, and `stat.visit` prints them, `outputList` contains by default just the text output. I don't see any other types of output in gem5, but likely JSON / binary formats could be envisioned.
14517-
14518-
Tested in gem5 b4879ae5b0b6644e6836b0881e4da05c64a6550d.
14519-
1452014592
==== gem5 code generation
1452114593

1452214594
gem5 uses a ton of code generation, which makes the project horrendous:
@@ -16545,7 +16617,10 @@ https://en.wikipedia.org/wiki/Hierarchical_Data_Format
1654516617

1654616618
Binary format to store data. TODO vs databases, notably SQLite: https://datascience.stackexchange.com/questions/262/hierarchical-data-format-what-are-the-advantages-compared-to-alternative-format
1654716619

16548-
Examples: link:userland/libs/hdf5[]
16620+
Examples:
16621+
16622+
* link:userland/libs/hdf5[]
16623+
* gem5 can dump statistics as HDF5: <<gem5-hdf5-statistics>>
1654916624

1655016625
=== Userland content filename conventions
1655116626

0 commit comments

Comments
 (0)