What's the proper way to debug D programs with GDB?

[Vladimir Panteleev]

Hi all,

I'm trying to debug some random crashes of some D services running on a
Linux headless server.
I have compiled/linked my programs and Phobos with -g and without -O
(using DMD), and applied the gdb patches from
http://dsource.org/projects/gdb-patches/ .
I'm running a script based on
https://wiki.ubuntu.com/Backtrace#Summary%20in%20script%20form to
automatically record the backtrace and restart the program.
However, gdb still doesn't print very nice backtraces. Here's an example
output:


[Thread debugging using libthread_db enabled]
[New Thread 0xf75506d0 (LWP 29591)]
Die: DW_TAG_<unknown> (abbrev = 9, offset = 449149)
has children: FALSE
attributes:
DW_AT_byte_size (DW_FORM_data1) constant: 8
DW_AT_type (DW_FORM_ref4) constant ref: 449143 (adjusted)
Dwarf Error: Cannot find type of die [in module
/home/wnbot/wahostbot/HostingBuddy]
#0 0x080a72aa in _aaIn (aa={a = 0xf7461410}, keyti=0x8136969) at
internal/aaA.d:253
pkey = (void *) 0xff9439d8
len = 97
key_hash = 145249
i = 40
e = (struct aaA.aaA *) 0x180002
c = 50Die: DW_TAG_<unknown> (abbrev = 4, offset = 52531)
has children: FALSE
attributes:
DW_AT_byte_size (DW_FORM_data1) constant: 8
DW_AT_type (DW_FORM_ref4) constant ref: 52523 (adjusted)

Dwarf Error: Cannot find type of die [in module
/home/wnbot/wahostbot/HostingBuddy]


So, what's the proper way to set up D debugging on Linux? Should I just
look into gdc?

[Robert Clipsham]

Post by Vladimir Panteleev
Hi all,
I'm trying to debug some random crashes of some D services running on a
Linux headless server.
I have compiled/linked my programs and Phobos with -g and without -O
(using DMD), and applied the gdb patches from
http://dsource.org/projects/gdb-patches/ .
I'm running a script based on
https://wiki.ubuntu.com/Backtrace#Summary%20in%20script%20form to
automatically record the backtrace and restart the program.
However, gdb still doesn't print very nice backtraces. Here's an example
[Thread debugging using libthread_db enabled]
[New Thread 0xf75506d0 (LWP 29591)]
Die: DW_TAG_<unknown> (abbrev = 9, offset = 449149)
has children: FALSE
DW_AT_byte_size (DW_FORM_data1) constant: 8
DW_AT_type (DW_FORM_ref4) constant ref: 449143 (adjusted)
Dwarf Error: Cannot find type of die [in module
/home/wnbot/wahostbot/HostingBuddy]
#0 0x080a72aa in _aaIn (aa={a = 0xf7461410}, keyti=0x8136969) at
internal/aaA.d:253
pkey = (void *) 0xff9439d8
len = 97
key_hash = 145249
i = 40
e = (struct aaA.aaA *) 0x180002
c = 50Die: DW_TAG_<unknown> (abbrev = 4, offset = 52531)
has children: FALSE
DW_AT_byte_size (DW_FORM_data1) constant: 8
DW_AT_type (DW_FORM_ref4) constant ref: 52523 (adjusted)
Dwarf Error: Cannot find type of die [in module
/home/wnbot/wahostbot/HostingBuddy]
So, what's the proper way to set up D debugging on Linux? Should I just
look into gdc?

It seems dmd is producing bad debug info here, you should report a bug
if you can narrow it down to a test case which still does that. I've
never had a problem using ldc with gdb and gdb-patches, so you could try
that if you're using D1/Tango. Otherwise I don't know what to suggest,
until the dmd bug is fixed... Other people have had similar problems, so
if you manage to get a test case that you can put in bugzilla it might
catch Walter's attention, getting gdb playing nicely with D seems to be
important to him :)

[grauzone]

Post by Robert Clipsham

Post by Vladimir Panteleev
Hi all,
I'm trying to debug some random crashes of some D services running on a
Linux headless server.
I have compiled/linked my programs and Phobos with -g and without -O
(using DMD), and applied the gdb patches from
http://dsource.org/projects/gdb-patches/ .
I'm running a script based on
https://wiki.ubuntu.com/Backtrace#Summary%20in%20script%20form to
automatically record the backtrace and restart the program.
However, gdb still doesn't print very nice backtraces. Here's an example
[Thread debugging using libthread_db enabled]
[New Thread 0xf75506d0 (LWP 29591)]
Die: DW_TAG_<unknown> (abbrev = 9, offset = 449149)
has children: FALSE
DW_AT_byte_size (DW_FORM_data1) constant: 8
DW_AT_type (DW_FORM_ref4) constant ref: 449143 (adjusted)
Dwarf Error: Cannot find type of die [in module
/home/wnbot/wahostbot/HostingBuddy]
#0 0x080a72aa in _aaIn (aa={a = 0xf7461410}, keyti=0x8136969) at
internal/aaA.d:253
pkey = (void *) 0xff9439d8
len = 97
key_hash = 145249
i = 40
e = (struct aaA.aaA *) 0x180002
c = 50Die: DW_TAG_<unknown> (abbrev = 4, offset = 52531)
has children: FALSE
DW_AT_byte_size (DW_FORM_data1) constant: 8
DW_AT_type (DW_FORM_ref4) constant ref: 52523 (adjusted)
Dwarf Error: Cannot find type of die [in module
/home/wnbot/wahostbot/HostingBuddy]
So, what's the proper way to set up D debugging on Linux? Should I just
look into gdc?

It seems dmd is producing bad debug info here, you should report a bug
if you can narrow it down to a test case which still does that. I've
never had a problem using ldc with gdb and gdb-patches, so you could try
that if you're using D1/Tango. Otherwise I don't know what to suggest,
until the dmd bug is fixed... Other people have had similar problems, so
if you manage to get a test case that you can put in bugzilla it might
catch Walter's attention, getting gdb playing nicely with D seems to be
important to him :)

dmd producing buggy debugging information seems to be an old problem:
http://d.puremagic.com/issues/show_bug.cgi?id=1079

[Walter Bright]

Post by grauzone
http://d.puremagic.com/issues/show_bug.cgi?id=1079

The problem is that dmd produces dwarf debug info according to the dwarf spec.
When gdb fails, I have no clue why, and nobody has ever been able to figure out
just *what* in the dwarf info is causing gdb to fail.

Some problems were identified a few months back, and those were fixed (evidently
gdb cannot handle module records, and gdb also relied on some undocumented stuff).

If anyone wants to peruse the gdb source and figure out exactly *what* it is
choking on, that would be most appreciated.

[Robert Clipsham]

Post by Walter Bright

Post by grauzone
http://d.puremagic.com/issues/show_bug.cgi?id=1079

The problem is that dmd produces dwarf debug info according to the dwarf
spec. When gdb fails, I have no clue why, and nobody has ever been able
to figure out just *what* in the dwarf info is causing gdb to fail.
Some problems were identified a few months back, and those were fixed
(evidently gdb cannot handle module records, and gdb also relied on some
undocumented stuff).
If anyone wants to peruse the gdb source and figure out exactly *what*
it is choking on, that would be most appreciated.

I've spent about half an hour trying to track this down, and I can't
seem to reproduce the problem. The only time I can get the errors
mentioned here and in the listed bug is when using -g. When switching to
-gc the problems disappear. Does anyone have a test case that fails with
-gc too that I can play with?

The reason this happens when using -g is the D extensions to dwarf which
GDB doesn't support (even with the gdb-patches applied) are used here,
which causes the error, as gdb is unable to parse the debug info. When
using -gc, dmd doesn't use the D extensions, so the error doesn't occur.
To fix this, gdb would need patching to add support for the D
extensions, or alternatively we'll have to keep using -gc.

----
void main()
{
char[] foo;
foo ~= 'a';
}
----
When compiled using -gc, gdb works as expected. When compiled with -g,
gdb gives the following error:

(gdb) b test.d:3
Die: DW_TAG_<unknown> (abbrev = 4, offset = 77)
has children: FALSE
attributes:
DW_AT_byte_size (DW_FORM_data1) constant: 8
DW_AT_type (DW_FORM_ref4) constant ref: 69 (adjusted)
Dwarf Error: Cannot find type of die [in module /tmp/test]

Taking a look at the dwarf info for test:
% objdump --dwarf=abbrev test
-- SNIP --
4 Unknown TAG value: 41 [no children]
DW_AT_byte_size DW_FORM_data1
DW_AT_type DW_FORM_ref4
-- SNIP --

Unknown TAG value at 41... This isn't part of the dwarf spec, looking at
src/dmd/backend/dwarf2.h:
----
// D programming language extensions
DW_TAG_darray_type = 0x41,
DW_TAG_aarray_type = 0x42,
DW_TAG_delegate_type = 0x43,

DW_TAG_lo_user = 0x4080,
DW_TAG_hi_user = 0xFFFF,
----

Oh look, 0x41 is a D extension, no wonder GDB chokes :)

I'll look more into this if someone can provide me a test case which
doesn't require the D extensions to cause the error, currently this
looks like a non-bug to me as gdb doesn't support D extensions yet (yes,
this is even with a patched gdb).

Robert

[Bernard Helyer]

Post by Robert Clipsham
I'll look more into this if someone can provide me a test case which
doesn't require the D extensions to cause the error, currently this
looks like a non-bug to me as gdb doesn't support D extensions yet (yes,
this is even with a patched gdb).
Robert

I have definitely had programs choke up gdb even when using the `-gc`
flag. I'll have a poke around tonight and see what I can find.



-Bernard.

[grauzone]

Post by Bernard Helyer

Post by Robert Clipsham
I'll look more into this if someone can provide me a test case which
doesn't require the D extensions to cause the error, currently this
looks like a non-bug to me as gdb doesn't support D extensions yet (yes,
this is even with a patched gdb).
Robert

I have definitely had programs choke up gdb even when using the `-gc`
flag. I'll have a poke around tonight and see what I can find.

Me too, but always in larger amounts of code, where it's hopeless to try
to find the exact location. It's probably caused by a single obscure D
feature...

Post by Bernard Helyer
-Bernard.

[Robert Clipsham]

Post by grauzone

Post by Bernard Helyer
I have definitely had programs choke up gdb even when using the `-gc`
flag. I'll have a poke around tonight and see what I can find.

Me too, but always in larger amounts of code, where it's hopeless to try
to find the exact location. It's probably caused by a single obscure D
feature...

If either of you can manage to come up with a test case that fails with
-gc, even if it's huge, let me know and I'll see if I can figure out
what's happening :)

[Bernard Helyer]

Post by Robert Clipsham

Post by grauzone

Post by Bernard Helyer
I have definitely had programs choke up gdb even when using the `-gc`
flag. I'll have a poke around tonight and see what I can find.

Me too, but always in larger amounts of code, where it's hopeless to try
to find the exact location. It's probably caused by a single obscure D
feature...

If either of you can manage to come up with a test case that fails with
-gc, even if it's huge, let me know and I'll see if I can figure out
what's happening :)

Okay got it. It's 6000 lines and six megabytes large (and hacked to all
hell, as the std.string templates started breaking, for some reason),
but it doesn't debug with `-gc`! What's the best way to get it to you?

[Robert Clipsham]

Post by Bernard Helyer
Okay got it. It's 6000 lines and six megabytes large (and hacked to all
hell, as the std.string templates started breaking, for some reason),
but it doesn't debug with `-gc`! What's the best way to get it to you?

You can either upload it somewhere (eg http://omploader.org/ or some
other file upload site if you've not got hosting), or email it to me (my
address is the same as I use for the newsgroups). Either way if you
tar/zip it up you should be able to save a few megs on that :)

[Bernard Helyer]

Post by Robert Clipsham

Post by Bernard Helyer
Okay got it. It's 6000 lines and six megabytes large (and hacked to all
hell, as the std.string templates started breaking, for some reason),
but it doesn't debug with `-gc`! What's the best way to get it to you?

You can either upload it somewhere (eg http://omploader.org/ or some
other file upload site if you've not got hosting), or email it to me (my
address is the same as I use for the newsgroups). Either way if you
tar/zip it up you should be able to save a few megs on that :)

http://omploader.org/iM3ZudA

There you go. D2, run build.debug.sh to build. Modify the script to add
options and the like. Built with `-gc` list _Dmain or break _Dmain
doesn't work, if you force a segfault (remove the comments from the
first few lines of ator.main and compile with -noboundscheck to cause
one) bt doesn't work, etc, etc.

[Robert Clipsham]

Post by Bernard Helyer
http://omploader.org/iM3ZudA
There you go. D2, run build.debug.sh to build. Modify the script to add
options and the like. Built with `-gc` list _Dmain or break _Dmain
doesn't work, if you force a segfault (remove the comments from the
first few lines of ator.main and compile with -noboundscheck to cause
one) bt doesn't work, etc, etc.

Thanks! I've managed to work this down to a 2 line test case:
----
void function() myfunc;
void main(){}
----
Which errors with:
----
Die: DW_TAG_padding (abbrev = 0, offset = 0)
has children: FALSE
attributes:
Dwarf Error: Cannot find type of die [in module /tmp/test]
----

Compile with either dmd -gc test.d or dmd -g test.d. I'll spend some
more time with it now and see if I can figure out a cause so we can give
a more useful bug report, if not I'll open a bug as is. If anyone else
is getting different errors from gdb when using -gc, let me know and I
can narrow them down too.

[grauzone]

Post by Robert Clipsham

Post by Bernard Helyer
http://omploader.org/iM3ZudA
There you go. D2, run build.debug.sh to build. Modify the script to add
options and the like. Built with `-gc` list _Dmain or break _Dmain
doesn't work, if you force a segfault (remove the comments from the
first few lines of ator.main and compile with -noboundscheck to cause
one) bt doesn't work, etc, etc.

----
void function() myfunc;
void main(){}
----

Wow, good work...

Post by Robert Clipsham
----
Die: DW_TAG_padding (abbrev = 0, offset = 0)
has children: FALSE
Dwarf Error: Cannot find type of die [in module /tmp/test]
----
Compile with either dmd -gc test.d or dmd -g test.d. I'll spend some
more time with it now and see if I can figure out a cause so we can give
a more useful bug report, if not I'll open a bug as is. If anyone else
is getting different errors from gdb when using -gc, let me know and I
can narrow them down too.

If I compile it with dmd -gc test.d, start gdb (version "7.0.1-debian"),
and type "break _Dmain", I just get "Dwarf Error: Cannot find DIE at 0x0
referenced from DIE at 0x46 [in module /tmp/test]".

This is with dmd v1.055 without gdb patches (but some other custom
patches, though I think none should affect the debugging infos).

[Robert Clipsham]

Post by grauzone
Wow, good work...

Thanks :) Took a while, but I got there.

Post by grauzone
If I compile it with dmd -gc test.d, start gdb (version "7.0.1-debian"),
and type "break _Dmain", I just get "Dwarf Error: Cannot find DIE at 0x0
referenced from DIE at 0x46 [in module /tmp/test]".
This is with dmd v1.055 without gdb patches (but some other custom
patches, though I think none should affect the debugging infos).

I'm using dmd v2.040 with the gdb patches (gdb 6.8), which could explain
the difference in error messages. I might have to give it a shot, as
that error message looks far more useful than what I'm getting out of
6.8. I'll keep playing for a bit, and if I don't start getting somewhere
I'll try it :)

[Robert Clipsham]

Post by Bernard Helyer
http://omploader.org/iM3ZudA
There you go. D2, run build.debug.sh to build. Modify the script to add
options and the like. Built with `-gc` list _Dmain or break _Dmain
doesn't work, if you force a segfault (remove the comments from the
first few lines of ator.main and compile with -noboundscheck to cause
one) bt doesn't work, etc, etc.

http://d.puremagic.com/issues/show_bug.cgi?id=3987

Submitted to bugzilla with a test case, I couldn't narrow down the exact
reason for it happening though, so that will have to be left to someone
else. I'm pretty sure there's another bug in there somewhere too,
because I came up with some odd errors while narrowing down a test
case... I'll leave it until that is fixed though in case it's the same
bug, unless someone can come up with another test case for me to work down.

[Bernard Helyer]

Post by Robert Clipsham

Post by Bernard Helyer
Okay got it. It's 6000 lines and six megabytes large (and hacked to all
hell, as the std.string templates started breaking, for some reason),
but it doesn't debug with `-gc`! What's the best way to get it to you?

You can either upload it somewhere (eg http://omploader.org/ or some
other file upload site if you've not got hosting), or email it to me (my
address is the same as I use for the newsgroups). Either way if you
tar/zip it up you should be able to save a few megs on that :)

Found another, no function pointers in this one, I think. It's much
smaller, too! D2, dmd -ofnetmon.bin netmon/*.d -gc -debug. Linux only.

http://omploader.org/iM3h0cw

[Robert Clipsham]

Post by Bernard Helyer

Post by Robert Clipsham

Post by Bernard Helyer
Okay got it. It's 6000 lines and six megabytes large (and hacked to all
hell, as the std.string templates started breaking, for some reason),
but it doesn't debug with `-gc`! What's the best way to get it to you?

You can either upload it somewhere (eg http://omploader.org/ or some
other file upload site if you've not got hosting), or email it to me (my
address is the same as I use for the newsgroups). Either way if you
tar/zip it up you should be able to save a few megs on that :)

Found another, no function pointers in this one, I think. It's much
smaller, too! D2, dmd -ofnetmon.bin netmon/*.d -gc -debug. Linux only.
http://omploader.org/iM3h0cw

Excellent :) OmpLoader seems to be down right now, so I'll try again
later... As soon as I can get it I'll see what I can do. In the mean
time you could upload it somewhere else or email it to me directly (I'll
post here once I've got it :)).

[Bernard Helyer]

Post by Robert Clipsham

Post by Bernard Helyer

Post by Robert Clipsham

Post by Bernard Helyer
Okay got it. It's 6000 lines and six megabytes large (and hacked to all
hell, as the std.string templates started breaking, for some reason),
but it doesn't debug with `-gc`! What's the best way to get it to you?

You can either upload it somewhere (eg http://omploader.org/ or some
other file upload site if you've not got hosting), or email it to me (my
address is the same as I use for the newsgroups). Either way if you
tar/zip it up you should be able to save a few megs on that :)

Found another, no function pointers in this one, I think. It's much
smaller, too! D2, dmd -ofnetmon.bin netmon/*.d -gc -debug. Linux only.
http://omploader.org/iM3h0cw

Excellent :) OmpLoader seems to be down right now, so I'll try again
later... As soon as I can get it I'll see what I can do. In the mean
time you could upload it somewhere else or email it to me directly (I'll
post here once I've got it :)).

Okay, I sent it to you. You should receive it with any luck.

[Robert Clipsham]

Post by Bernard Helyer
Found another, no function pointers in this one, I think. It's much
smaller, too! D2, dmd -ofnetmon.bin netmon/*.d -gc -debug. Linux only.
http://omploader.org/iM3h0cw

I think it's the same bug. A test case:
----
import std.stdio;
void main()
{
writeln("");
}
----
I haven't narrowed it down further, because it requires most of phobos,
and does include function pointers. I could attempt to narrow it down
further, but I believe it has the same cause.

[Bernard Helyer]

Post by Robert Clipsham
----
I haven't narrowed it down further, because it requires most of phobos,
and does include function pointers. I could attempt to narrow it down
further, but I believe it has the same cause.

So the only way debugging will work is if we don't use Phobos? >_<;;

Thanks for your good work.

[Robert Clipsham]

Post by Bernard Helyer
So the only way debugging will work is if we don't use Phobos? >_<;;
Thanks for your good work.

Seems so, at least until
http://d.puremagic.com/issues/show_bug.cgi?id=3987 is fixed.
Unfortunately I don't know enough about DWARF to chase this any further,
at least without significant effort on my part, we we'll just have to
sit tight, and hope Walter take a look at it :)

[Robert Clipsham]

Post by Walter Bright

Post by grauzone
http://d.puremagic.com/issues/show_bug.cgi?id=1079

The problem is that dmd produces dwarf debug info according to the dwarf
spec. When gdb fails, I have no clue why, and nobody has ever been able
to figure out just *what* in the dwarf info is causing gdb to fail.
Some problems were identified a few months back, and those were fixed
(evidently gdb cannot handle module records, and gdb also relied on some
undocumented stuff).
If anyone wants to peruse the gdb source and figure out exactly *what*
it is choking on, that would be most appreciated.

Done. I've attached a patch to
http://d.puremagic.com/issues/show_bug.cgi?id=3987 which seems to fix
the issue. The problem was incorrect debug information being created for
function pointers, it seems there shouldn't be a type for both the
function and the pointer to it.

[Vladimir Panteleev]

On Fri, 12 Mar 2010 18:53:05 +0200, Robert Clipsham

Post by Robert Clipsham
It seems dmd is producing bad debug info here, you should report a bug
if you can narrow it down to a test case which still does that. I've
never had a problem using ldc with gdb and gdb-patches, so you could try
that if you're using D1/Tango. Otherwise I don't know what to suggest,
until the dmd bug is fixed... Other people have had similar problems, so
if you manage to get a test case that you can put in bugzilla it might
catch Walter's attention, getting gdb playing nicely with D seems to be
important to him :)

Thanks for the quick response! Unfortunately I'm still using D1/Phobos
(*cough* get off my lawn... hehe). I'm going to give gdc a shot, though.

I could e-mail Walter a binary, but it doesn't look like it's necessary,
according to grauzone's post.

[Robert Clipsham]

Post by Vladimir Panteleev
Hi all,
I'm trying to debug some random crashes of some D services running on a
Linux headless server.
I have compiled/linked my programs and Phobos with -g and without -O
(using DMD), and applied the gdb patches from
http://dsource.org/projects/gdb-patches/ .
I'm running a script based on
https://wiki.ubuntu.com/Backtrace#Summary%20in%20script%20form to
automatically record the backtrace and restart the program.
However, gdb still doesn't print very nice backtraces. Here's an example
[Thread debugging using libthread_db enabled]
[New Thread 0xf75506d0 (LWP 29591)]
Die: DW_TAG_<unknown> (abbrev = 9, offset = 449149)
has children: FALSE
DW_AT_byte_size (DW_FORM_data1) constant: 8
DW_AT_type (DW_FORM_ref4) constant ref: 449143 (adjusted)
Dwarf Error: Cannot find type of die [in module
/home/wnbot/wahostbot/HostingBuddy]
#0 0x080a72aa in _aaIn (aa={a = 0xf7461410}, keyti=0x8136969) at
internal/aaA.d:253
pkey = (void *) 0xff9439d8
len = 97
key_hash = 145249
i = 40
e = (struct aaA.aaA *) 0x180002
c = 50Die: DW_TAG_<unknown> (abbrev = 4, offset = 52531)
has children: FALSE
DW_AT_byte_size (DW_FORM_data1) constant: 8
DW_AT_type (DW_FORM_ref4) constant ref: 52523 (adjusted)
Dwarf Error: Cannot find type of die [in module
/home/wnbot/wahostbot/HostingBuddy]
So, what's the proper way to set up D debugging on Linux? Should I just
look into gdc?

I've spent a lot of time on this, and with the patch from bug #3987
applied debugging info should work, providing binaries are compiled with
-gc and not -g :)