When call stack is too deep and a DLL is being tested for leaks (not the executable), VLD hangs with a message "(abort)" and when I pause the debugger it takes me into some Windows DLL that I have no source for. It looks like VLD is trying to display some GUI with what is happening, and that fails.
Note about a special VLD build: I am debugging a custom XLL (DLL) under Excel. This means I cannot compile executable with VLD. I had to stop tracking Kernel32.dll in order to avoid endless recursion (stack overflow) in VLD. I am not sure if this contributed to me experiencing the issue described above, but I think it should not.
It looks like there are two things gong on:
- VLD fails (or at least thinks it failed) to truncate a long call stack and
- VLD tries to report it, which causes it to hang.
I have a partial fix to one of these issues, which I will post as an additional comment
Comments: increasing the MAXMESSAGELENGTH constant by 10 fold (and taking the assertion out, just for now) fixed the issue though. I didn't get a chance to look at what return value was (I would have to "unfix" and rebuild for that) because I am still chasing "important" memory leaks. The reason I went for this workaround, is because `messagea` was successfully getting a truncated call stack in the call that was going into the assert section. I made a few other minor changes to VLD, by the way, like sorting the output by block serial ID. (A small local set does it.) This is big help when diffing two different dumps to see what "leaks" went away (meaning they are not leaks) and what leaks survived (meaning they are likely leaks) from one call to another.
Note about a special VLD build: I am debugging a custom XLL (DLL) under Excel. This means I cannot compile executable with VLD. I had to stop tracking Kernel32.dll in order to avoid endless recursion (stack overflow) in VLD. I am not sure if this contributed to me experiencing the issue described above, but I think it should not.
It looks like there are two things gong on:
- VLD fails (or at least thinks it failed) to truncate a long call stack and
- VLD tries to report it, which causes it to hang.
I have a partial fix to one of these issues, which I will post as an additional comment
Comments: increasing the MAXMESSAGELENGTH constant by 10 fold (and taking the assertion out, just for now) fixed the issue though. I didn't get a chance to look at what return value was (I would have to "unfix" and rebuild for that) because I am still chasing "important" memory leaks. The reason I went for this workaround, is because `messagea` was successfully getting a truncated call stack in the call that was going into the assert section. I made a few other minor changes to VLD, by the way, like sorting the output by block serial ID. (A small local set does it.) This is big help when diffing two different dumps to see what "leaks" went away (meaning they are not leaks) and what leaks survived (meaning they are likely leaks) from one call to another.