When call stack is too deep and a DLL is being tested for leaks (not the executable), VLD hangs with a message "(abort)" and when I pause the debugger it takes me into some Windows DLL that I have no source for. It looks like VLD is trying to display some GUI with what is happening, and that fails.
Note about a special VLD build: I am debugging a custom XLL (DLL) under Excel. This means I cannot compile executable with VLD. I had to stop tracking Kernel32.dll in order to avoid endless recursion (stack overflow) in VLD. I am not sure if this contributed to me experiencing the issue described above, but I think it should not.
It looks like there are two things gong on:
- VLD fails (or at least thinks it failed) to truncate a long call stack and
- VLD tries to report it, which causes it to hang.
I have a partial fix to one of these issues, which I will post as an additional comment
Note about a special VLD build: I am debugging a custom XLL (DLL) under Excel. This means I cannot compile executable with VLD. I had to stop tracking Kernel32.dll in order to avoid endless recursion (stack overflow) in VLD. I am not sure if this contributed to me experiencing the issue described above, but I think it should not.
It looks like there are two things gong on:
- VLD fails (or at least thinks it failed) to truncate a long call stack and
- VLD tries to report it, which causes it to hang.
I have a partial fix to one of these issues, which I will post as an additional comment