Possible to populate Win64 COFF objects with CodeView dbginfo?

[TurkeyMan]

Since LLVM doesn't look like it will support PDB any time soon, and DWARF isn't particularly useful to most windows dev environments, is it possible to populate the COFF objects with CodeView debug info?

MSVC has the (legacy) /Z7 option, which populates the objects with CV debuginfo, which the MS link.exe is still capable if parsing and inserting into the final PDB while linking...

If LLVM/LDC were able to populate the windows objects with CV debuginfo, they may be much more useful when used in typical MSVC dev environments.

[dnadlinger]

This is a question for @redstar – these sorts of issues are exactly where my Windows dev knowledge is lacking…

[8l]

http://dsource.org/projects/cv2pdb

http://www.dsource.org/projects/visuald

HTH

[TurkeyMan]

This has a key limitation though; you can't link VC and LDC objects together, since VC produces a PDB for the stuff it compiles, and then cv2pdb overwrites it. The resulting debug info is mutually exclusive.

It would certainly be much better if the objects had CV debuginfo, and let link.exe do what it's supposed to do.

[redstar]

As already discussed on the LLVM list the way to go is to convert the dwarf debugging information to CV or PDB format while writing the COFF file.
That's a point on my TODO list but with lower priority then the exception handling.

[TurkeyMan]

Okay cool. Yeah that's where the LLVM guys ended up too. It seems very indirect.
I wonder if code from cv2pdb can be integrated into LDC directly so it can be done in one pass...

[redstar]

Some links:
CodeView:
CV4 documentation
CV5 documentation
Notes on CV8
Matching debug info
ming-w64 on CodeView
PDB:
PDB files internals
Cracking PDB symbol files
pdbparser
PDB File Format
Chapter 1 of Windows 2000 Secrets
The RSDS pdb format

[TurkeyMan]

Perhaps worth mentioning that Walter has preliminary CV8 support written in DMD-Win64 already, and is continuing to work on it. Perhaps this is a good reference for the debuginfo format.
I'm using the MS linker to build the PDB file, and can do the same when using LDC. So PDB format is not necessary, just the CV8 data in the objects.

Nice documents! I hope Walter knows about all of those... or have the 2 of you been talking? :)

[redstar]

I started to implement this. It seems to be easier then expected. The construction of DWARF debug symbols is encapsulated in class DwarfDebug, which is used in the AsmPrinter. I defined the interface in a new abstract base class and introduced a new CodeViewDebug class. Looks like I can output the first bytes into a .debug$S section :-)

If you are curious then you can look at this diff. But there is still a lot to do - the debug symbol encoding is anything than simple.

[TurkeyMan]

Awesome, nice work.
I recommend checking out Walter's recent work too in DMD, he has CV8 debuginfo in COFF objects from DMD working pretty nicely now. Might be of use, since it's all so poorly documented.

[redstar]

I still fight with the results of a hard disk failure... Nevertheless I can report some progress here. I started to write the first bits to `.debug$T´ section.

And yes, I am aware of the code produced by Walter and I am studying it in case of questions.

[TurkeyMan]

Awesome! I really think this will do a lot of good for Windows users!
DMD is great, I'm really happy to have usable access to the reference compiler on Windows now, but LDC will offer a backend with far better optimisation, maturity, and portability, which will be of unquestionable value to Windows users.

[redstar]

As promise I posted first code to the LLVM commit list. I can already generate the right type information for simple signatures like long add(long a, long b). The generated .debug$T section is identical to the one emitted by DMD.

I now try to emit line information. Then it should be possible to really 'see' something in a debugger.

[pravic]

@redstar Great and really needed work!

[TurkeyMan]

You sir, are rocking a full jar of awesome sauce! :P .. Walter had some trouble with the call stack too, how's that looking?

[Trass3r]

What's the current state of this?

[redstar]

There is only one improvement. The last missing piece (a special relocation) has been added to LLVM 2 month ago by the Google people. I'll restart work on this as soon as I have some more time.

BTW: LLVM has now COFF line number support. I don't know if this helps much...

[TurkeyMan]

That motion from Clang to output objects that were fully compatible with MSVC, how is that going?
I thought they'd have to deal with this problem in Clang to meet that goal of theirs?

Will we see MSVC compatible objects from LDC any time soon? :)

[Trass3r]

http://clang.llvm.org/docs/MSVCCompatibility.html
Only line tables. But yeah eventually they will have to solve that problem too.

[TurkeyMan]

Any recent motion on this issue?
I think this is the last significant issue standing between D and many many windows users.
I think it's safe to say a lot of potential Windows users will be turned away by DMD's inferior codegen. People don't often use native languages if weak performance is an option.

[Trass3r]

That and EH. No news on dbginfo.

[TurkeyMan]

I don't use EH, but some libraries (including phobos) use it. Personally, I can work-around EH. Although I'm sure it'll be a massive win for Windows too, I think the debuginfo is the biggest blocker. Nobody I've ever met would dev on Windows without the debugger. People are utterly accustomed to visual studio :)
Most people in my industry would equally avoid EH as me...

[Trass3r]

Sure, users can avoid EH, but it is a big development issue cause it results in most druntime/phobos unittests crashing which masks potential real ldc problems.

Well naturally, the VS debugger kicks ass. I don't know any other one that comes even close.
A good thing you can do is push for it on cfe-dev. Their priorities are a bit off, working on 100% msvc ABI compatibility for features that could easily be worked around rather than fundamental blockers. But I already got them to pull EH up in the list.

[Trass3r]

Ok now that EH is working this is the number 1 roadblock.

[Trass3r]

lists.cs.uiuc.edu/pipermail/cfe-dev/2014-January/034747.html
-.-

[Ilod]

Hello,
FYI, since this thread was my principal source of information on this subject, I recently started to look in debug information generation for Visual Studio with LLVM, as I need it for C++ development. Not sure I will be able to commit some useful stuff, as I'm not really familiar with this, but with the resource listed in this thread I was finally able today to add functionnal breakpoints to the line table generation in really simple programs. I will probably start a thread on LLVM ML/start working on some patch next week.

[Trass3r]

Are you able to get source code interleaved with disassembly?
Some of the links above are dead by now.

https://android.googlesource.com/toolchain/yasm/+/master/modules/dbgfmts/codeview/cv8.txt

[Ilod]

Yeah, I may have some small problems as I still have to setup function prolog/epilog correctly, but I can display disassembly and it seems to correspond to the code. There was not much missing to the existing line table generation to do this in fact, they were juste missing the symbol function declaration.

[Trass3r]

The function symbols have just landed in llvm: http://reviews.llvm.org/D5772
This enables basic source-level debugging (no variables of course)!

[redstar]

There is still work on this topic in LLVM: http://reviews.llvm.org/D7153

[redstar]

Some more news: Microsoft is going to provide CodeView symbols to LLVM.
See the announcement: http://permalink.gmane.org/gmane.comp.compilers.clang.devel/45575
LLVM: http://reviews.llvm.org/D14209
clang: http://reviews.llvm.org/D14210

[TurkeyMan]

Oh, god almighty, thank you! :P
I see that their patch comes with a 'Clang portion' along with an 'LLVM portion'. I guess this implies an LDC portion. How complex is that?

[redstar]

I just discover the announcement. It says that they want to introduce new metadata nodes. This could mean that we have to create a 2nd DIBuilder. But I hope that a more generic design emerges. Debug info generation is centralized in the DIBuilder class which should therefore the only class which needs to be enhanced.

[TurkeyMan]

Reading those articles, it looks like they generate CV metadata in Clang independently from DWARF. Kinda looks like the Clang patch is significant, and also requires full duplication in LDC... :/

[redstar]

Yes, that's the approach of the RFC. But the first comments in the discussion about the RFC points out that it is better to have an uniform solution.

[TurkeyMan]

That's nice, I didn't read that far.
I saw he made a comment where he made a change to the Microsoft mangler to support some stuff; this means he's making changes to the MS codebase (Visual Studio?) to complete this work. If you can get in on details required to nicely support D; ie, things like local functions, which MS might not support, then perhaps you can influence him to add support for these cases to MS's CodeView description while he's at it...
Ie, being involved with this guy while he's on this task might lead to influence in the MS toolchain, and that means we might be able to slip technology in there required by D (local functions? D closures?) that he would have otherwise ignored ;)

Do you know what this means for de-mangling? Does VS use the prettyName field he refers to to display function names, which we can populate correctly, or would some work be required by MS to have good support?

[redstar]

I have a different understanding here: he changed the Microsoft Mangler in clang, see http://reviews.llvm.org/D14210#e4b48f2a.

[redstar]

See #1716 for current state.

[kinke]

Let's close this then and continue with #1716, as (incomplete) debuginfos are already generated.