When call stack is too deep and a DLL is being tested for leaks (not the executable), VLD hangs with a message "(abort)" and when I pause the debugger it takes me into some Windows DLL that I have no source for. It looks like VLD is trying to display some GUI with what is happening, and that fails.
Note about a special VLD build: I am debugging a custom XLL (DLL) under Excel. This means I cannot compile executable with VLD. I had to stop tracking Kernel32.dll in order to avoid endless recursion (stack overflow) in VLD. I am not sure if this contributed to me experiencing the issue described above, but I think it should not.
It looks like there are two things gong on:
- VLD fails (or at least thinks it failed) to truncate a long call stack and
- VLD tries to report it, which causes it to hang.
I have a partial fix to one of these issues, which I will post as an additional comment
Comments: It appears that the following fragment blows the session up: utility.cpp ====== if (wcstombs_s(&count, messagea, MAXMESSAGELENGTH + 1, messagew, _TRUNCATE) != 0) { assert(FALSE); // <<<<< this line return; } I am not a C++ wiz, but to me it looks like (1) wcstombs_s returns a warning code (my call stack does come back truncated), and (2) the blanket if() takes it as an error. To workaround, I commented out the assertion, and increased the buffer size by 10x. const size_t MAXMESSAGELENGTH = 50000; I did not get a chance to see what code wcstombs_s() actually returned.
Note about a special VLD build: I am debugging a custom XLL (DLL) under Excel. This means I cannot compile executable with VLD. I had to stop tracking Kernel32.dll in order to avoid endless recursion (stack overflow) in VLD. I am not sure if this contributed to me experiencing the issue described above, but I think it should not.
It looks like there are two things gong on:
- VLD fails (or at least thinks it failed) to truncate a long call stack and
- VLD tries to report it, which causes it to hang.
I have a partial fix to one of these issues, which I will post as an additional comment
Comments: It appears that the following fragment blows the session up: utility.cpp ====== if (wcstombs_s(&count, messagea, MAXMESSAGELENGTH + 1, messagew, _TRUNCATE) != 0) { assert(FALSE); // <<<<< this line return; } I am not a C++ wiz, but to me it looks like (1) wcstombs_s returns a warning code (my call stack does come back truncated), and (2) the blanket if() takes it as an error. To workaround, I commented out the assertion, and increased the buffer size by 10x. const size_t MAXMESSAGELENGTH = 50000; I did not get a chance to see what code wcstombs_s() actually returned.