Improve DWARF5 support and refactor (new history) #3703

imbillow · 2023-07-31T07:11:36Z

DO NOT SQUASH ME!

Your checklist for this pull request

I've read the guidelines for contributing to this repository
I made sure to follow the project's coding style
I've documented or updated the documentation of every function and struct this PR changes. If not so I've explained why.
I've added tests that prove my fix is effective or that my feature works (if possible)
I've updated the rizin book with the relevant information (if needed)

Detailed description

Existing DWARF loaders have incomplete support for DWARF5, and the support for DWARF location expressions is too simplified. It also uses a lot of SDBs to store DWARF information.

This PR completely refactors the DWARF loader, fixes a lot of bugs, and makes a lot of improvements to these issues.

TODO (after merge)

add fuzztests

Test plan

CI is green

Closing issues

closes #3548
closes #3535
closes #1004
closes #3541
closes #3700
partially addresses #3581
...

See also https://gcc.gnu.org/wiki/DebugFission

Second version of #3565

librz/analysis/dwarf_process.c

librz/bin/dwarf/lists.c

librz/bin/dwarf/loclists.c

librz/core/disasm.c

librz/include/rz_analysis.h

librz/include/rz_bin_dwarf.h

meson.build

librz/include/rz_analysis.h

XVilka · 2023-08-24T04:02:59Z

librz/bin/dwarf/serialize.c

+	TRY_LOAD_SECTION("debug_line", dw->line,
+		rz_bin_dwarf_line_from_buf(buf, &dw->encoding, dw->info, RZ_BIN_DWARF_LINE_INFO_MASK_LINES_ALL));
+
+	RzBuffer *loc = load_section("debug_loc", sdb);


@thestr4ng3r what do you think about this approach of loading/saving the debug sections? cc @wargio @ret2libc

Are these sections actually info that has to be saved/loaded at all? If it is something that is parsed directly out of the binary and will never be edited during the session, it does not have to be saved as it can be restored from the binary again on the next load (might need tests so this property is actually guaranteed).

For the RzAnalysisDwarfVariable on the other hand, we have no choice other than to serialize it, or at least some kind of reference into the binary where the same info can be restored from, since variables can be renamed, etc.

@thestr4ng3r Rizin uses only a small part of that information, in fact, especially at this stage. Thus do not storing whole sections should reduce the resulting size of the project file. Moreover, as some binaries have huge debug information sections, storing them serialized might become a problem, for example for files like this one: #676

What I mean is that RzAnalysisDwarfVariable should not be edited and it can be restored via sections of DWARF. then we can reference it via DIE offset.

@imbillow but then you don't need to store the whole section content in the SDB, only pointers to it?

@imbillow but then you don't need to store the whole section content in the SDB, only pointers to it?

Yes.

Exactly. If a DIE offset for a variable is enough to reconstruct the info, then it is enough to store that as part of the variable storage info and then reconstruct the info from the bin itself.

Ok, lets stick to this approach then - saving pointers to the information. Then, if it works - the only missing thing is project migration tests, also test for loading this kind of storage from the project file. Then the PR will be ready to merge

imbillow · 2023-08-24T04:13:08Z

There are more functions detected now. Is this intended?

What new were added?


[XX] db/abi/compilers/clang_64 ELF_ABI : Clang Of flagspaces

RZ_NOPLUGINS=1 /private/var/folders/bc/yvvw2g2s4jlg9r8x7tf61xv800009d/T/woodpecker-local-1287038090/rizinorg/rizin/prefix/bin/rizin -escr.utf8=0 -escr.color=0 -escr.interactive=0 -eflirt.sigdb.load.system=false -eflirt.sigdb.load.home=false -N -A -Qc 'fsl~functions

' bins/abi_bins/elf/compilers/clang/echo_clang_Of

-- stdout

--- expected

+++ actual

@@ -1,1 +1,1 @@

-   51 * functions

+   68 * functions



-- stderr

[x] Analyze all flags starting with sym. and entry0 (aa)

[x] Analyze function calls

[x] Analyze len bytes of instructions for references

[x] Check for classes

[x] Analyze local variables and arguments

[x] Type matching analysis for all functions

[x] Applied 0 FLIRT signatures via sigdb

[x] Propagate noreturn information

[x] Integrate dwarf function information.

[x] Resolve pointers to data sections

[x] Use -AA or aaaa to perform additional experimental analysis.

See https://ci.rizin.re/rizinorg/rizin/build/3026/6

Some functions that had no name are now have auto-named

XVilka · 2023-08-24T06:03:21Z

@imbillow could you please add a new test for this kind of anonymous function too?

imbillow · 2023-08-24T07:47:35Z

@imbillow could you please add a new test for this kind of anonymous function too?

My bad. It's actually just a mistake to rename an existing function with the generated name, should has been fixed.

test/integration/test_project_migrate.c

.reuse/dep5

- rz_base_type_clone_into - rz_base_type_clone

- rz_bits_leading_zeros - rz_json_eq - rz_json_string_eq - RZ_STR_EQ - RZ_STR_NE - rz_vector_clone_into - sdb_diff_eq

- `RZ_PROJECT_VERSION` 14 - Add `RzCallable`.has_unspecified_parameters - Add `RzAnalysis`.debug_info - Add `RzAnalysisVarStorageType` composite and `eval` `pending` - Support for parse DWARF section "debug_loclists", "debug_ranges", "debug_rnglists" - Partial support for eval DWARF expr_loc - Support for anonymous Type, function variable, struct member - Cache all DWARF information in `RzAnalysisDebugInfo` and remove `SDB` based caching. - Add arm32, arm64, TriCore DWARF register name - Fix same name basetype

imbillow requested review from XVilka, ret2libc, wargio, thestr4ng3r and kazarmy as code owners July 31, 2023 07:11

github-actions bot added rz-test RzBin RzAnalysis API RzCore RzType DWARF labels Jul 31, 2023

XVilka added this to the 0.6.0 milestone Jul 31, 2023