Not sure if this is an Angelscript bug or somehow an MSVC bug. I'm compiling Angelscript 2.23.0 on 32 bit in release build using platform toolset v141 and SDK 8.1, 10.0.x.0 (I've tried 8.1, 10.0.16299.0, and 10.0.17134.0) Though it's quite strange, because this only started happening on 1 project after the most recent Visual Studio 2017 update. (It seems to be fine in another project, which is using the 8.1 SDK, but that project is also a custom build from GENie, this project is built from the msvc2015 folder that comes with Angelscript itself.)
The problem I'm seeing comes from the script engine message callback. I'm registering it like this, as a CDECL call:
int r = 0; m_engine->SetMessageCallback(asFUNCTION(ScriptMessageCallback), nullptr, asCALL_CDECL); assert(r >= 0);And the actual function is empty: (actually commented out just to test this problem)
static void ScriptMessageCallback(const asSMessageInfo *msg, void *param)
{
}Which results in a single "ret" instruction in memory. This makes sense, it's a cdecl call, so the caller does the cleanup.
So, this function is eventually called on some script error. The actual call to the function is in "asCScriptEngine::CallGlobalFunction". In the debug version of Angelscript, the assembly is fine:
03F22B9F 8B 45 0C mov eax,dword ptr [param2]
03F22BA2 50 push eax
03F22BA3 8B 4D 08 mov ecx,dword ptr [param1]
03F22BA6 51 push ecx
03F22BA7 FF 55 F0 call dword ptr [ebp-10h]
03F22BAA 83 C4 08 add esp,8 In the release build it works a little different, it sets up the call and calls the function, but it doesn't clean up the stack:
0400C461 56 push esi
0400C462 8B 30 mov esi,dword ptr [eax]
0400C464 83 FA 02 cmp edx,2
0400C467 74 5F je asCScriptEngine::CallGlobalFunction+88h (0400C4C8h)
...
0400C4C8 FF 75 0C push dword ptr [param2]
0400C4CB FF 75 08 push dword ptr [param1]
0400C4CE FF D6 call esi
0400C4D0 8B 4D F4 mov ecx,dword ptr [ebp-0Ch]
0400C4D3 64 89 0D 00 00 00 00 mov dword ptr fs:[0],ecx
0400C4DA 5E pop esi
0400C4DB 8B E5 mov esp,ebp
0400C4DD 5D pop ebp
0400C4DE C2 10 00 ret 10h Note that "esi" on the first instruction above is the thisptr, and is how the caller eventually picks up thisptr again. Since the stack is not cleaned up (before esi is restored at "pop esi"), esi actually ends up being param1.
Even if I explicitly write the function pointer as __cdecl, it doesn't clean up the stack in time: (results in the same assembly)
typedef void (__cdecl *func_t)(void *, void *);
func_t f = (func_t)(i->func);
f(param1, param2);For reference, here's the CallGlobalFunction C++ code directly from Angelscript:
void asCScriptEngine::CallGlobalFunction(void *param1, void *param2, asSSystemFunctionInterface *i, asCScriptFunction *s) const
{
if( i->callConv == ICC_CDECL )
{
void (*f)(void *, void *) = (void (*)(void *, void *))(i->func);
f(param1, param2);
}
else if( i->callConv == ICC_STDCALL )
{
typedef void (STDCALL *func_t)(void *, void *);
func_t f = (func_t)(i->func);
f(param1, param2);
}
else
{
// We must guarantee the order of the arguments which is why we copy them to this
// array. Otherwise the compiler may put them anywhere it likes, or even keep them
// in the registers which causes problem.
void *params[2] = {param1, param2};
asCGeneric gen(const_cast<asCScriptEngine*>(this), s, 0, (asDWORD*)¶ms);
void (*f)(asIScriptGeneric *) = (void (*)(asIScriptGeneric *))(i->func);
f(&gen);
}
}What's going on here? Am I missing something? Do I need to report this as an optimization bug to Microsoft? Do I need to make this a stdcall for some reason..?
