bug: generated pdb cannot be read by sentry.io (wrong machine type, missing section headers)

[mcdurdin]

Thanks for your work in creating map2pdb!

We are trying to use map2pdb to integrate our app Keyman more cleanly with Sentry -- currently we use .dbg but those are somewhat limited. However, Sentry reported that the .pdb files generated by map2pdb are not parseable by them. They noted a few issues:

So far it looks like your Delphi application is built as Win32 (32-bit/x86), but the PDB is being detected as x86_64 (64-bit). This is a significant mismatch. The map2pdb tool is likely generating a PDB file with an incorrect MachineType. In the symbolic crate's PDB handling (pdb.rs), the architecture is determined from the PDB's DBI stream's machine type. For a 32-bit Delphi build, this should be MachineType::X86 (value 0x014c), not Amd64 (value 0x8664).

Another thing I noticed is that even though Sentry reports the PDB was found with "has_debug_info": true and "has_symbols": true, the actual symbolication fails with "symbolicator_status": "missing_symbol" for all your application frames. The symbolic library converts internal PDB section offsets to RVAs using an address map (pdb.rs). If to_rva() returns None, the function is skipped entirely. This happens when:

• The PDB lacks proper section headers
• The section offset doesn't map to a valid RVA
• The address map (OMAP stream) is missing or invalid

What's most likely happening here is that the map2pdb tool converts Delphi's .map file format to PDB format, but it may not be generating a fully compatible PDB because of:

• Wrong machine type in the DBI stream header
• Missing or incorrect section headers that the address map needs to translate section:offset pairs to RVAs
• Address mapping issues: Delphi .map` files use segment:offset notation that needs proper conversion

I tried to examine the generated .pdb with cvdump but found that it could not be read due to a missing dbgtypeSectionHdr stream. However, I did see in the code that IMAGE_FILE_MACHINE_AMD64 is always written, so that part of the analysis at least is correct:

map2pdb/Source/debug.info.writer.pdb.pas

Line 842 in c629a0f

DBIStreamHeader.Machine := IMAGE_FILE_MACHINE_AMD64;

dbgtypeSectionHdr appears to be a list of IMAGE_SECTION_HEADERs:

https://github.com/microsoft/microsoft-pdb/blob/805655a28bd8198004be2ac27e6e0290121a5e89/cvdump/dumppdb.cpp#L2340

I have not at this point attempted to create a patch for this but would be happy to work with you to address these issues.

Keyman issue: keymanapp/keyman#15182

[andersmelander]

First of all, map2pdb was primarily created to produce pdb files that can be consumed by Microsoft's pdb reader msdia and through that, used by VTune. And since this was a significant effort, given that it's an undocumented format, I stopped as soon as I got it working.

That said, I'm definitely not opposed to enhancing map2pdb so it can be used by other readers and tools - within reason of course :-) It appears though that the Sentry pdb reader is doing a bit more input validation than it needs to. I can understand that they want to ensure that they aren't reading junk but I can't really see the purpose of validating the MachineType; They can (and I'm sure they do) get that information from the associated executable, and the information doesn't affect how the remaining information in the pdb is interpreted.

Anyway, I have now made a small modification where I guess the architecture from the map file; It isn't something that is explicitly specified anywhere.

Sentry reports the PDB was found with "has_debug_info": true and "has_symbols": true

Yes, of course it does :-)

https://github.com/getsentry/symbolic/blob/b75c1c9e5186f554862ae07f4ab11481badc5734/symbolic-debuginfo/src/pdb.rs#L241-L248
https://github.com/getsentry/symbolic/blob/b75c1c9e5186f554862ae07f4ab11481badc5734/symbolic-debuginfo/src/pdb.rs#L221-L225

Apart from that I do set the PDBRawFeatureSig feature flags specified in the Info stream to indicate the presence of symbols and type data.

map2pdb/Source/debug.info.writer.pdb.pas

Lines 438 to 443 in c629a0f

	{$ifdef PDB_Minimal_Debug}
	Result.Writer.Write(Cardinal(Ord(PDBRawFeatureSig.MinimalDebugInfo)));
	Result.Writer.Write(Cardinal(Ord(PDBRawFeatureSig.VC140)));
	{$else PDB_Minimal_Debug}
	Result.Writer.Write(Cardinal(Ord(PDBRawFeatureSig.VC110)));
	{$endif PDB_Minimal_Debug}

map2pdb/Source/debug.info.pdb.pas

Lines 224 to 237 in c629a0f

	PDBRawFeatureSig = (
	VC110 = 20091201, // No other features flags are present
	// PDB contains an IPI Stream
	VC140 = 20140508, // Other feature flags may be present
	// PDB contains an IPI Stream
	NoTypeMerge = $4D544F4E, // Presumably duplicate types can appear in the TPI Stream, although it’s
	// unclear why this might happen.
	MinimalDebugInfo = $494E494D // Program was linked with /DEBUG:FASTLINK
	// There is no TPI / IPI stream, all type info is contained in the original
	// object files.
	);

As far as I remember, I had to cheat and specify a value (VC110 in this case) that indicated the presence of symbol/type information even though I can't supply any (the map file doesn't contain it) in order to get either the MS pdb reader or VTune to accept the pdb. I do write the empty type (TPI) and symbol (IPI) streams though, so that should be okay.

Missing or incorrect section headers that the address map needs to translate section:offset pairs to RVAs

I will need some more details on this one: Missing or incorrect? If missing, which? If incorrect, how?
Keep in mind that that VTune, WinDbg, and other tools, are perfectly able to translate the offsets in the PDB to RVAs.

Address mapping issues: Delphi .map` files use segment:offset notation that needs proper conversion

And that conversion is done:

map2pdb/Source/debug.info.writer.pdb.pas

Line 1476 in c629a0f

Symbol.Rva := Section.Offset;

map2pdb/Source/debug.info.writer.pdb.pas

Line 1373 in c629a0f

Publics[Index].PublicSym32.Offset := Module.Offset + Symbol.Offset; // Make offset relative to segment

Note that the term segments is a bit misleading; This isn't segments as in 16-bit, but more sections as in PE file format.

I tried to examine the generated .pdb with cvdump but found that it could not be read due to a missing dbgtypeSectionHdr stream.

Works for me - Or did you mean that dump of the SECTION HEADERS section failed? If so, yes, I can reproduce that:

*** SECTION HEADERS
DBIOpenDbg(, dbgtypeSectionHdr,) failed.

*** ORIGINAL SECTION HEADERS
DBIOpenDbg(, dbgtypeSectionHdrOrig,) failed.

I think this might be the missing part:

map2pdb/Source/debug.info.writer.pdb.pas

Lines 790 to 808 in c629a0f

	function TDebugInfoPdbWriter.EmitDBISubstreamDebugHeader(Writer: TBinaryBlockWriter): Cardinal;
	begin
	Result := Writer.Position;
	// We don't provide any of the optional debug streams yet
	for var HeaderType := Low(PDBDbgHeaderType) to High(PDBDbgHeaderType) do
	Writer.Write(Word(TMSFStream.NullStreamIndex));
	// TODO : I believe the SectionHdr stream contains the segment names
	(*
	for var Segment in FDebugInfo.Segments do
	begin
	xxx.Add(Segment.SegClassName);
	xxx.Add(Segment.Name);
	end;
	*)
	Result := Writer.Position - Result;
	end;

but again, the information in them, or the OMAP sections for that matter, shouldn't really be required.

This Sentry issue is likely related: getsentry/pdb#17

dbgtypeSectionHdr appears to be a list of IMAGE_SECTION_HEADERs

Yes but that is just the in-memory structure used to interface with msdia. The on-disk structure is likely much more complicated; After all, this is pdb :-/

Although I would love to give these problems a go, I honestly doubt that I will have much time to do so anytime soon but if you are up for doing it yourself, I'll be happy to assist as much as I can.

[mcdurdin]

First of all, map2pdb was primarily created to produce pdb files that can be consumed by Microsoft's pdb reader msdia and through that, used by VTune. And since this was a significant effort, given that it's an undocumented format, I stopped as soon as I got it working.

Indeed, I am very grateful to you for the work here. I have been using map2pdb with VTune with great success. Quite a few years ago, I went down the road of writing a 'map2pdb' of my own (after getting frustrated with the instability of Delphi's debugger) and soon realized that I just didn't have the time to wade into the undocumented swamp. So discovering your work on map2pdb made me very happy, and I applaud the results you have already achieved. My bug report here is really to document the gap with Sentry -- no expectations on your availability to chase this 😁.

In terms of the feedback from Sentry, I focused on what seemed to be the two actionable steps:

1. the mismatching machine type, which as you noted is not explicitly found in the .map, so would ideally be taken from the corresponding PE. I built map2pdb locally and tweaked the values to what I believe are the correct x86 equivalents, but Sentry's systems still did not successfully read the new .pdb.

2. So, then I wanted to look at dbgtypeSectionHdr. And that's when I realized I just don't have time to learn enough about .pdb or even map2pdb itself to submit a PR, at this point.

So I wrote up this issue as a way of at least documenting it. For now, our Keyman project will go back to using .dbg files for integration into Sentry, and if we want to do profiling or deep debugging, we'll do a local build with map2pd; it does seem like getting both .pdb and .dbg for the same build is tricky, because of the binding to the PE.

I am not going to be able to pick this up until January at the earliest, I expect, but will reach out if/when I get some time to dig deeper!

(I wonder if Sentry get interns who like to take on "little" projects?)

[andersmelander]

I got bored during an online meeting so I had a quick look at it again and...

dbgtypeSectionHdr appears to be a list of IMAGE_SECTION_HEADERs
Yes but that is just the in-memory structure used to interface with msdia. The on-disk structure is likely much more complicated;

It turned out that the ODS was indeed IMAGE_SECTION_HEADER. I now output the optional section header stream with a list of these structures populated from the sections. I'm unsure if the values are 100% correct but at least cvdump can now parse it.

[andersmelander]

I disabled the section header output again as it broke VTune :-(

After reading through all the information I have been able to find about the section headers and the OMAP stuff, I'm convinced that Sentry should be able to survive without this info (which, as I've mentioned, is optional) in the pdb - and it seems that they at one point didn't even support reading it.

[mcdurdin]

I disabled the section header output again as it broke VTune :-(

That's a shame! Thanks for having a go.

After reading through all the information I have been able to find about the section headers and the OMAP stuff, I'm convinced that Sentry should be able to survive without this info (which, as I've mentioned, is optional) in the pdb - and it seems that they at one point didn't even support reading it.

Perhaps so, but I can't control what Sentry does with the pdb.