Mach-O Universal Binaries

[jeremy-rifkin]

Does libdwarf support Mach-O fat/universal objects? I'm currently resolving dwarf symbols from a mach-o object. My code works for single-objects binaries but not multi-architecture objects.

If libdwarf doesn't support these yet, would you accept a patch to add support?
Update: Looking into the code a little, I think it'll take me a while to understand how the mach-o reading works. but support in theory should be simple. There's a design decision to be made in finding a way to allow the user to specify which architecture they want to look at.

[davea42]

libdwarf reads Mach-O and has for a year or two. Uses no MacOS libraries, just does
it itself (as for Elf and PE).

If you correctly pass an executable path (say "x') to dwarf_init_path()
libdwarf will look in x.dSYM and find the debug object in the directory tree the compiler creates.

Read the doc about how to let that happen (let libdwarf return the path string to you , you provide
space for the string).

Also note
DW_PATHSOURCE*
as that tells what found the object with DWARF.

If you are curious, look for DSYM_SUFFIX in libdwarf source for details.

[davea42]

Of course all I said means libdwarf reads Mach-O objects placed into Win/Mingw64 or placed in Linux
as libdwarf reads all sorts anywhere it works at all, and deals with endianness as required.

So, for example, on Big-Endian Ubuntu, reads all those little endian objects correctly. And vice-versa,
though there are just a few bit-endian objects in regression tests.

[jeremy-rifkin]

Thank you. I have been successful reading dwarf symbols with objects that look like the following:

$ file demo
demo: Mach-O 64-bit executable x86_64

$ file demo.dSYM/Contents/Resources/DWARF/demo
demo.dSYM/Contents/Resources/DWARF/demo: Mach-O 64-bit dSYM companion file x86_64

I don't think I observed libdwarf automatically searching for a corresponding dSYM file and ended up manually checking and appending the dSYM path. I will try looking again tomorrow.

However, when I produce universal binaries that look like the following I'm unable to load dwarf symbols with either method:

$ file demo
demo: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64Mach-O 64-bit executable arm64]
demo (for architecture x86_64): Mach-O 64-bit executable x86_64
demo (for architecture arm64):  Mach-O 64-bit executable arm64

$ file demo.dSYM/Contents/Resources/DWARF/demo
demo.dSYM/Contents/Resources/DWARF/demo: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dSYM companion file x86_64Mach-O 64-bit dSYM companion file x86_64] [arm64:Mach-O 64-bit dSYM companion file arm64Mach-O 64-bit dSYM companion file arm64]
demo.dSYM/Contents/Resources/DWARF/demo (for architecture x86_64):      Mach-O 64-bit dSYM companion file x86_64
demo.dSYM/Contents/Resources/DWARF/demo (for architecture arm64):       Mach-O 64-bit dSYM companion file arm64

I've done a cursory look through the libdwarf code and I haven't found any code that seems to deal with these universal mach-o binaries, but of course I'm not familiar with the codebase so I could have missed something. I didn't see the magic numbers for fat mach-o files in the codebase either, 0xcafebabe or 0xbebafeca.

[davea42]

I've not had an object files with multiple architectures in hand (that I noticed)
so I have no idea what the 'file' commands are showing me.
So it's no surprise to me that libdwarf does not understand.

An 'ls' command in demo.dSYM/Contents/Resources/DWARF/demo
might help.

Or if you can just make a tar file with the contents of demo and all its pieces that would be great.

The source in libdwarf that does the finding is src/lib/libdwarf/dwarf_object_detector.c

An alternative, maybe: tell me how, with MacOS command line tools (C compiler) to
produce such a universal object here (I have a Mac Mini with latest software to work with).
I could write a hello-world and try to build a multi-part universal ,
with some hints from you (I build and test libdwarf
and dwarfdump on the Mac Mini). Since you know how to do it...

[davea42]

In the libdwarf source I mentioned notice the function
dwarf_object_detector_fd(). Does the heavy lifting.

[davea42]

MH_MAGIC in dwarf_macho_loader.h and so on have the magic numbers.

[davea42]

Also note DSYM_SUFFIX in dwarf_object_detector.c , the reason libdwarf works
on non-universal binaries.

[jeremy-rifkin]

Hello, here is a tar with the demo universal executable and it's .dSYM: demo.tar.gz. I built it with -DCMAKE_C_FLAGS='-arch arm64 -arch x86_64' -DCMAKE_CXX_FLAGS='-arch arm64 -arch x86_64'. It seems sometimes these multi-architecture binaries may be automatically created, maybe on newer versions of macos / newer versions of clang.

The fat mach-o format should be fairly easy to add support for in theory:

Offset	Bytes	Description
0	4	Magic number
4	4	Number of binaries
8	N	Mach-O file
8 + N	M	Mach-O file
...	...	...

Each mach-o then has a cpu type and subtype that's checked during loading.

Thank you for the pointers to the code. I will look at seeing if this easy to add support for in libdwarf.

[jeremy-rifkin]

Regarding libdwarf automatically looking for .dSYM files, I stepped through things in a debugger and realized dwarf_object_detector_path_dSYM wasn't running since I wasn't passing a true_path_out_buffer. When I pass a buffer it now automatically finds the .dSYM.

[davea42]

"The fat mach-o format should be fairly easy to add support for in theory:"

Ah. I found fat.h in the command line tools. Never had a reason to notice it.
Seems straightforward to get me started.

That implies opening a file one needs to indicate which part of the fat format applies
to the current request. Perhaps a new argument for that when opening libdwarf
access? Meaining in the demo app, either 0 or 1?

Oh my.
Will think about it.

[davea42]

I've added the FAT structs into dwarf_macho_loader..h
and will think more about this.

[jeremy-rifkin]

Thank you so much for your willingness to look into this and to support this!

That implies opening a file one needs to indicate which part of the fat format applies
to the current request. Perhaps a new argument for that when opening libdwarf
access? Meaning in the demo app, either 0 or 1?

I have no idea what the right design decision is here 😄 Figuring out the appropriate index might require the end user to parse basic mach-o information themselves which isn't the end of the world but it could be nice to benefit from the work libdwarf already odes. Maybe a callback to let the user choose based on cpu type and cpu subtype, but that'd only be for mach-o. It would be nice if all fat binary files used similar techniques to indicate their platform but it seems they don't. An index definitely would be easier and might be the right choice.

[davea42]

For an index 0 would work just fine for non universal and universal.

Given the magic numbers are well defined this seems simple to implement.
I'll build some into a local branch for testing. Given other responsibilities
it might take a couple days to get this right.

I'll keep existing interfaces and when index not specified will assume zero.
So little new code and no code duplication.

Even given the simple functions you mention the threads
issue needs more thought. But for some functions
an alternate interface could work with no locking around or inside
those function. Maybe.

[jeremy-rifkin]

Thank you so much! I have been playing around with a patch locally but there's enough involved in the mach-o loading that I think I'll have to leave it to you. I look forward to support being added for this, please let me know if there's anything I can do to help!

[davea42]

Yes, there is a strong assumption that the beginning of a particular object is a file offset zero. oops.

First I will update my other project, readelfobj-code,which reads object files and prints their details.
It will be a simpler case than libdwarf and it will be easy to see reading of the universal
macho-case is working. For all the object file types this was the training ground.
So I can be sure I actually understand. I'll get back to you in a couple days.

[jeremy-rifkin]

Thank you! I was noticing the assumption as well when I was looking through the code.

I realized my table above was an incorrect representation of the fat mach-o structure, my apologies. I figure you may be looking at another reference but just for the sake of correctness, below is more accurate:

Offset	Bytes	Description
0	4	Magic number
4	4	Number of binaries
8	20	Fat arch entry
28	20	Fat arch entry
...	...	...

With fat arch entries being:

Offset	Bytes	Description
0	4	CPU type
4	4	CPU subtype
8	4	File offset
12	4	Size
16	4	Section alignment

[davea42]

Yes, I have the correct table layout in hand. So far I'm correctly reading
the small header., and planning another smallish bit of code to
read the arch entry headers.

[davea42]

readelfobj-code's object detector now correctly reads Mach-O universal binaries (the report
is not brilliant...yet).

In addition, small changes to the reader (made already) means reading and processing a selected object in that
binary to report its DWARF should work just fine. The two parts are not connected up yet. That's next.

[jeremy-rifkin]

Thanks so much for the update and for working on this!

[davea42]

object_detector and readobjmacho now work correctly on ordinary and universal-binary Mach-O object files.
I need to remove debug printf before pushing to the readelfob-code project.

[jeremy-rifkin]

Thank you for the update and the quick work!

[davea42]

In spite of the inadequate output readobjmacho and object_detector print
about universal binaries, I have moved on to libdwarf/dwarfdump.
I added a new option --format-universalnumber=<n> to dwarfdump
and we now call a new libdwarf function that allows specifying
a value indicating which (in a universal binary) libdwarf should
read and therefore which dwarfdump should print.

The value zero for the universalnumber is the default and is always
valid everywhere.

New libdwarf functions dwarf_init_path_a() and dwarf_init_path_dl_a()
allow specifying a universalnumber. (the older versions are
still supported but cannot access other than the initial binary in a
universal binary).

[jeremy-rifkin]

That sounds fantastic. Are these changes currently in libdwarf or not quite yet?

[davea42]

Libdwarf now deals with universal binaries properly.
There is lots of debug printf present. Will be committing to local branch,
then cleaning debug, then fixing indent/whitespace issues.
To celebrate, I'm taking a coffee break!

I expect to merge into main and push to github today.
After that I'll add a test to libdwarf-regressiontests using your demo objects.
More later.

[jeremy-rifkin]

Awesome news, thank you again so much. I look forward to integrating the changes into what I'm working on.

[davea42]

Documentation has been updated and is on prevanders
and github. -fsanitize finds no errors, but I need to
run coverityscan.

Happy Friday!

[jeremy-rifkin]

I'm giving the tries a test locally. I was trying to load a dsym object directly (file reports it's a mach-o dSYM companion file), which previously worked but now I'm getting a DW_DLE_MACHO_CORRUPT_COMMAND. I'm looking into it to double check that it is in fact a valid mach-o.

Something minor that caught my eye while looking through the changes: Should this comment be updated or moved down?

libdwarf-code/src/lib/libdwarf/dwarf_generic_init.c

Lines 127 to 137 in 7b3fdd2

	/* New in December 2018. */
	int dwarf_init_path_a(const char *path,
	char * true_path_out_buffer,
	unsigned true_path_bufferlen,
	unsigned groupnumber,
	unsigned universalnumber,
	Dwarf_Handler errhand,
	Dwarf_Ptr errarg,
	Dwarf_Debug * ret_dbg,
	Dwarf_Error * error)
	{

[jeremy-rifkin]

Ok, the DW_DLE_MACHO_CORRUPT_COMMAND was correctly raised I think. I was doing something on my end resulting in an incorrect load command.

[jeremy-rifkin]

Ok, continued testing locally and everything seems to be working great

[davea42]

Sometimes it can be useful to download and build the readelfobj-code project
and try object_detector and readobmacho on the object because there
it reports issues instantly in some detail. If there are any actual issues
in the object file. Sounds now like that's not necessary.

I am altering that comment to read 2023. The old comment is irrelevant and misplaced.

[jeremy-rifkin]

I'll go ahead and mark this discussion as resolved, thank you again for the super fast response and willingness to add support for this!