When call stack is too deep and a DLL is being tested for leaks (not the executable), VLD hangs with a message "(abort)" and when I pause the debugger it takes me into some Windows DLL that I have no source for. It looks like VLD is trying to display some GUI with what is happening, and that fails.
Note about a special VLD build: I am debugging a custom XLL (DLL) under Excel. This means I cannot compile executable with VLD. I had to stop tracking Kernel32.dll in order to avoid endless recursion (stack overflow) in VLD. I am not sure if this contributed to me experiencing the issue described above, but I think it should not.
It looks like there are two things gong on:
- VLD fails (or at least thinks it failed) to truncate a long call stack and
- VLD tries to report it, which causes it to hang.
I have a partial fix to one of these issues, which I will post as an additional comment
Comments: OK, I finally got a chance to change the code, rebuild VLD, and rerun my tests to see what value was coming back to me. It was code 80 (int), which according to the page below is `STRUNCATE`, "String was truncated" https://msdn.microsoft.com/en-us/library/t3ayayh1.aspx The message code makes sense as this is exactly what is happening with my string: it is being truncated. Although this means that the `wcstombs_s` function is misdocumented, or has a bug because this page referenced by __thedailycommute__ https://msdn.microsoft.com/en-us/library/s7wzt4be.aspx states > __Return Value__ > Zero if successful, an error code on failure. and never mentions the `STRUNCATE` code. Since `wcstombs_s` is unlikely to change no matter how it is documented at the moment, I suggest to replace ``` CPP if (wcstombs_s(&count, messagea, MAXMESSAGELENGTH + 1, messagew, _TRUNCATE) != 0) { ``` with ``` CPP auto api_retval = wcstombs_s(&count, messagea, MAXMESSAGELENGTH + 1, messagew, _TRUNCATE); if (api_retval != 0 && api_retval != STRUNCATE) { ```
Note about a special VLD build: I am debugging a custom XLL (DLL) under Excel. This means I cannot compile executable with VLD. I had to stop tracking Kernel32.dll in order to avoid endless recursion (stack overflow) in VLD. I am not sure if this contributed to me experiencing the issue described above, but I think it should not.
It looks like there are two things gong on:
- VLD fails (or at least thinks it failed) to truncate a long call stack and
- VLD tries to report it, which causes it to hang.
I have a partial fix to one of these issues, which I will post as an additional comment
Comments: OK, I finally got a chance to change the code, rebuild VLD, and rerun my tests to see what value was coming back to me. It was code 80 (int), which according to the page below is `STRUNCATE`, "String was truncated" https://msdn.microsoft.com/en-us/library/t3ayayh1.aspx The message code makes sense as this is exactly what is happening with my string: it is being truncated. Although this means that the `wcstombs_s` function is misdocumented, or has a bug because this page referenced by __thedailycommute__ https://msdn.microsoft.com/en-us/library/s7wzt4be.aspx states > __Return Value__ > Zero if successful, an error code on failure. and never mentions the `STRUNCATE` code. Since `wcstombs_s` is unlikely to change no matter how it is documented at the moment, I suggest to replace ``` CPP if (wcstombs_s(&count, messagea, MAXMESSAGELENGTH + 1, messagew, _TRUNCATE) != 0) { ``` with ``` CPP auto api_retval = wcstombs_s(&count, messagea, MAXMESSAGELENGTH + 1, messagew, _TRUNCATE); if (api_retval != 0 && api_retval != STRUNCATE) { ```