When call stack is too deep and a DLL is being tested for leaks (not the executable), VLD hangs with a message "(abort)" and when I pause the debugger it takes me into some Windows DLL that I have no source for. It looks like VLD is trying to display some GUI with what is happening, and that fails.
Note about a special VLD build: I am debugging a custom XLL (DLL) under Excel. This means I cannot compile executable with VLD. I had to stop tracking Kernel32.dll in order to avoid endless recursion (stack overflow) in VLD. I am not sure if this contributed to me experiencing the issue described above, but I think it should not.
It looks like there are two things gong on:
- VLD fails (or at least thinks it failed) to truncate a long call stack and
- VLD tries to report it, which causes it to hang.
I have a partial fix to one of these issues, which I will post as an additional comment
Comments: I have no idea about what's happening with VLD, but it's possible your call to wcstombs_s is failing due to an _invalid wide character_. My first impression was an error due to the buffer being too small, but the use of the _TRUNCATE flag appears to preclude that. According to the MSDN documentation (https://msdn.microsoft.com/en-us/library/s7wzt4be.aspx): "The __wcstombs_s__ function converts a string of wide characters pointed to by _wcstr_ into multibyte characters stored in the buffer pointed to by _mbstr_. The conversion will continue for each character until one of these conditions is met: * A null wide character is encountered * A wide character that cannot be converted is encountered * The number of bytes stored in the mbstr buffer equals count. The destination string is always null-terminated (even in the case of an error). If _count_ is the special value `_TRUNCATE`, then __wcstombs_s__ converts as much of the string as will fit into the destination buffer, while still leaving room for a null terminator. If __wcstombs_s__ successfully converts the source string, it puts the size in bytes of the converted string, including the null terminator, into *_pReturnValue_ (provided _pReturnValue_ is not __NULL__). This occurs even if the mbstr argument is NULL and provides a way to determine the required buffer size. Note that if _mbstr_ is __NULL__, _count_ is ignored. If __wcstombs_s__ encounters a wide character it cannot convert to a multibyte character, it puts 0 in *_pReturnValue_, sets the destination buffer to an empty string, sets _errno_ to __EILSEQ__, and returns __EILSEQ__. If the sequences pointed to by _wcstr_ and _mbstr_ overlap, the behavior of __wcstombs_s__ is undefined.
Note about a special VLD build: I am debugging a custom XLL (DLL) under Excel. This means I cannot compile executable with VLD. I had to stop tracking Kernel32.dll in order to avoid endless recursion (stack overflow) in VLD. I am not sure if this contributed to me experiencing the issue described above, but I think it should not.
It looks like there are two things gong on:
- VLD fails (or at least thinks it failed) to truncate a long call stack and
- VLD tries to report it, which causes it to hang.
I have a partial fix to one of these issues, which I will post as an additional comment
Comments: I have no idea about what's happening with VLD, but it's possible your call to wcstombs_s is failing due to an _invalid wide character_. My first impression was an error due to the buffer being too small, but the use of the _TRUNCATE flag appears to preclude that. According to the MSDN documentation (https://msdn.microsoft.com/en-us/library/s7wzt4be.aspx): "The __wcstombs_s__ function converts a string of wide characters pointed to by _wcstr_ into multibyte characters stored in the buffer pointed to by _mbstr_. The conversion will continue for each character until one of these conditions is met: * A null wide character is encountered * A wide character that cannot be converted is encountered * The number of bytes stored in the mbstr buffer equals count. The destination string is always null-terminated (even in the case of an error). If _count_ is the special value `_TRUNCATE`, then __wcstombs_s__ converts as much of the string as will fit into the destination buffer, while still leaving room for a null terminator. If __wcstombs_s__ successfully converts the source string, it puts the size in bytes of the converted string, including the null terminator, into *_pReturnValue_ (provided _pReturnValue_ is not __NULL__). This occurs even if the mbstr argument is NULL and provides a way to determine the required buffer size. Note that if _mbstr_ is __NULL__, _count_ is ignored. If __wcstombs_s__ encounters a wide character it cannot convert to a multibyte character, it puts 0 in *_pReturnValue_, sets the destination buffer to an empty string, sets _errno_ to __EILSEQ__, and returns __EILSEQ__. If the sequences pointed to by _wcstr_ and _mbstr_ overlap, the behavior of __wcstombs_s__ is undefined.