Hi all,

we think we found a very fundamental issue with marshaling bool values from
..NET to COM. Our .NET server code looks like this (MC++, using VS.NET 2003
and .NET 1.1 SP1):

-----------------------------
namespace booltest
{
[Guid("466C7E09-E505-41ab-A331-39906FBBA20A"), ComVisible(true)]
public __gc __interface IBool {
bool Foo(void);
};

[ClassInterface(ClassInterfaceType::None), ComVisible(true)]
[Guid("B9E85D68-9284-4e62-A876-1B03EE8948B1")]
public __gc class Booltest : public IBool
{
public:
Booltest() {}
bool Foo(void) { return false; }
};
}
-----------------------------

So all we're trying to do is return a "bool" type to the caller. If we now
write a COM client which calls this code, we find that the automatically
generated marshaling code trashes the stack:

-----------------------------
#import "booltest.tlb"

int main(int argc, char * argv[])
{
::CoInitialize(0);
// Instantiate server code (COM wrapper on top of managed code)
Booltest::IBoolPtr pBool(__uuidof(Booltest::Booltest), 0, CLSCTX_ALL);
// call simple bool function; this trashes the stack
// (triggers alert when running in the debugger)
bool ret = pBool->Foo();

::CoUninitialize();
return 0;
}
-----------------------------

The generated ATL wrapper code assumes that the bool value returned from the
server is only one byte in size, but the CCW writes four bytes!

We are aware of the KB article 823072 which talks about incorrect bool
marshaling, but that problems occurs when calling unmanaged code from managed
code; in our case, we're marshaling in the other direction. Also, the KB
article only talks about cases where incorrect bool values are returned; in
our case, the stack is corrupted, and the code crashes.

Did anybody run into this before? If anybody's interested, I can provide a
trivial test project which shows the problem.

Thanks,

Claus

[quoted text, click to view]

No you don't have found an issue with marshaling, you have found what
happens under the covers when you are using the smart pointer wrapper
generated by the C++ compiler.
Take a look at the .tli and the .tlh files for details. Basically, each COM
interface method returns a HRESULT (a 32bit or 64bit integer), "your"
return value is passed as an argument.
The means that your bool Foo() in really looks like 'HRESULT Foo(unsigned
char* )', but the wrapper converts this into 'inline unsigned char Foo()',
the compiler happily casts the unsigned char ( a byte) into a bool.
In short what you have on the stack is the HRESULT code, which is the way of
COM to report errors. A 0 return means S_OK, negative values are errors.

Willy.

[quoted text, click to view]

I disagree. The marshaling code also overwrites the stack if we use COM
clients without the ATL wrappers.

bool ret;
HRESULT hr = pBool->Foo((unsigned char *)&ret);

When we run this client code, four bytes are written to 'ret', even though
the variable is also 1 byte in size - as a result, the stack is corrupted. I
still think this is a bug in the CCW which is generated for our .NET server
assembly.

Claus