Intro

While testing a credential provider project that I inherited from a previous developer (written in native C++) I installed my "Windows Error Reporting" tool (or, WERSetup). I always use that small tool to trap any crashes in the application that I am developing. This is especially pertinent for a credential provider that runs (somewhat) early during the boot process, or before any user desktop is even initialized. That in itself significantly complicates testing and debugging. Thus, relying on the "Windows Error Reporting" mechanism was quite handy.

When set up through my WERSetup tool, it will instruct the OS to automatically generate crash dump files for crashing applications in the folder that I provided:

"Windows Error Reporting" Setup & Configuration Utility.

Soon enough, I started noticing sporadic crashes in the process called LogonUI.exe, that is a system process responsible for the user interface to display the credential provider. What ensued is my struggle to find the cause of the crash, and hopefully a lesson to anyone who reads this blog post.

Investigation

For the readers who are not familiar with credential providers in Windows, let me give you a very quick primer. Since about Windows Vista, to make a credential provider you would have to write a DLL and follow a strict set of callbacks that the host process will invoke, thus giving you some control of the user login process and the UI.

Not surprisingly, such DLL is loaded inside LogonUI.exe, or the host process that does the bulk of the UI work and the one that also dispatches the callbacks.

So after I set up my "Windows Error Reporting" I started noticing LogonUI.exe.*.dmp files that appeared in the CrashDump folder after my credential provider DLL was unloading. Note that crashes never happened during the operation of the credential provider, which made debugging such a bug quite troublesome.

Once I had a crash dump file, the first order of business was to move it to my development computer and open it with the same build of my credential provider that caused the crash. And after having downloaded .pdb symbols for the LogonUI.exe process from the Microsoft public symbol server, I was able to open it in the Visual Studio:

Visual Studio 2022, showing the location of the crash in LogonUI.exe.

To my surprise, the callstack that lead to the crash, had nothing to do with my code. The thread that crashed LogonUI.exe was a part of that process, that didn't directly deal with my credential provider. The following was the actual callstack:

KERNELBASE.dll!RaiseFailFastException() Unknown
combase.dll!RoFailFastWithErrorContextInternal2(HRESULT hrError, unsigned long cStowedExceptions, _STOWED_EXCEPTION_INFORMATION_V2 * * aStowedExceptionPointers) Line 1455 C++
Windows.UI.Xaml.dll!DirectUI::ErrorHelper::ProcessUnhandledError(DirectUI::ErrorInfo & errorInfo={...}, unsigned int fSkipFailFastIfNoErrorContext=0x00000001, unsigned int * pfHandled=0x00000072ed87f250) Line 616 C++
Windows.UI.Xaml.dll!DirectUI::FinalUnhandledErrorDetectedRegistration::OnFinalUnhandledErrorDetected(IInspectable * pSender=0x0000000000000000, Windows::ApplicationModel::Core::IUnhandledErrorDetectedEventArgs * pArgs) Line 1013 C++
[Inline Frame] Windows.UI.Xaml.dll!Microsoft::WRL::Callback::__l2::<lambda_fa46ac39691f4ca87fe78d9a3f2f4de1>::operator()(IInspectable * &&) Line 327 C++
Windows.UI.Xaml.dll!Microsoft::WRL::Details::DelegateArgTraits<long (__cdecl Windows::Foundation::IEventHandler_impl<Windows::Foundation::Internal::AggregateType<Windows::ApplicationModel::Core::UnhandledErrorDetectedEventArgs *,Windows::ApplicationModel::Core::IUnhandledErrorDetectedEventArgs *>>::*)(IInspectable *,Windows::ApplicationModel::Core::IUnhandledErrorDetectedEventArgs *)>::DelegateInvokeHelper<Microsoft::WRL::Implements<Microsoft::WRL::RuntimeClassFlags<2>,Windows::Foundation::IEventHandler<Windows::ApplicationModel::Core::UnhandledErrorDetectedEventArgs *>,Microsoft::WRL::FtmBase>,<lambda_fa46ac39691f4ca87fe78d9a3f2f4de1>,-1,IInspectable *,Windows::ApplicationModel::Core::IUnhandledErrorDetectedEventArgs *>::Invoke(IInspectable * <args_0>=0x0000019bc78718e0, Windows::ApplicationModel::Core::IUnhandledErrorDetectedEventArgs * <args_1>=0x00000072ed87f430) Line 245 C++
twinapi.appcore.dll!Windows::Internal::Details::GitInvokeHelper<struct Windows::Foundation::IEventHandler<class Windows::ApplicationModel::Core::UnhandledErrorDetectedEventArgs *>,class Windows::Internal::GitPtr,2>::Invoke(struct IInspectable *,struct Windows::ApplicationModel::Core::IUnhandledErrorDetectedEventArgs *) Unknown
twinapi.appcore.dll!Windows::ApplicationModel::Core::UnhandledErrorInvokeHelper::Invoke(struct IInspectable *,struct Windows::ApplicationModel::Core::IUnhandledErrorDetectedEventArgs *) Unknown
twinapi.appcore.dll!Microsoft::WRL::InvokeTraits<2>::InvokeDelegates<class <lambda_d3aa4dff9873c46d1acd593e89259f7b>,struct Windows::Foundation::IEventHandler<class Windows::ApplicationModel::Core::UnhandledErrorDetectedEventArgs *> >(class <lambda_d3aa4dff9873c46d1acd593e89259f7b>,class Microsoft::WRL::Details::EventTargetArray *,class Microsoft::WRL::EventSource<struct Windows::Foundation::IEventHandler<class Windows::ApplicationModel::Core::UnhandledErrorDetectedEventArgs *>,struct Microsoft::WRL::InvokeModeOptions<2> > *) Unknown
twinapi.appcore.dll!Microsoft::WRL::EventSource<struct Windows::Foundation::IEventHandler<class Windows::ApplicationModel::Core::UnhandledErrorDetectedEventArgs *>,struct Microsoft::WRL::InvokeModeOptions<2> >::DoInvoke<class <lambda_d3aa4dff9873c46d1acd593e89259f7b> >(class <lambda_d3aa4dff9873c46d1acd593e89259f7b>) Unknown
twinapi.appcore.dll!Microsoft::WRL::EventSource<struct Windows::Foundation::IEventHandler<class Windows::ApplicationModel::Core::UnhandledErrorDetectedEventArgs *>,struct Microsoft::WRL::InvokeModeOptions<2> >::InvokeAll<std::nullptr_t,class Windows::ApplicationModel::Core::UnhandledErrorDetectedEventArgs *>(std::nullptr_t,class Windows::ApplicationModel::Core::UnhandledErrorDetectedEventArgs *) Unknown
twinapi.appcore.dll!Windows::ApplicationModel::Core::CoreApplication::ForwardLocalError() Unknown
twinapi.appcore.dll!Windows::ApplicationModel::Core::CoreApplicationFactory::ForwardLocalError() Unknown
combase.dll!CallErrorForwarder(void * hProcess=0xffffffffffffffff, int fLocal=0x00000001, IRestrictedErrorInfo * pRestrictedErrorInfo=0x0000019bc78b6fc8) Line 1601 C++
combase.dll!RoReportFailedDelegate(IUnknown * punkDelegate=0x0000019bc75ebf50, IRestrictedErrorInfo * pRestrictedErrorInfo=0x0000019bc78b6fc8) Line 1691 C++
CredProvDataModel.dll!Microsoft::WRL::InvokeTraits<-2>::InvokeDelegates<class <lambda_4e9de4704cc9cdd88d60571ac4a784c2>,struct Windows::Foundation::Collections::VectorChangedEventHandler<class Windows::Internal::UI::Logon::CredProvData::Credential *> >(class <lambda_4e9de4704cc9cdd88d60571ac4a784c2>,class Microsoft::WRL::Details::EventTargetArray *,class Microsoft::WRL::EventSource<struct Windows::Foundation::Collections::VectorChangedEventHandler<class Windows::Internal::UI::Logon::CredProvData::Credential *>,struct Microsoft::WRL::InvokeModeOptions<-2> > *) Unknown
CredProvDataModel.dll!CCredentialGroupBase::RemoveAllCredentials(int) Unknown
CredProvDataModel.dll!CCredProvDataModel::_ForEachUser<<lambda_a79f157231530b58da103c954842cb8b>>() Unknown
CredProvDataModel.dll!CCredProvDataModel::_ForEachCredentialGroup<<lambda>(void)>() Unknown
CredProvDataModel.dll!CCredProvDataModel::_ClearState() Unknown
CredProvDataModel.dll!CCredProvDataModel::Shutdown() Unknown
Windows.UI.Logon.dll!LogonUX::RequestCredentialEntryViewModel::[LogonUX::__IRequestCredentialEntryViewModelPublicNonVirtuals]::Shutdown(void) Unknown
Windows.UI.Logon.dll!LogonUX::RootViewModel::_ShutdownRequestCredentialsViewModel() Unknown
Windows.UI.Logon.dll!LogonUX::RootViewModel::[Windows::Internal::UI::Logon::LogonUX::ILogonUXViewModel]::Shutdown(struct Windows::Internal::UI::Logon::Callbacks::IOperationComplete ^) Unknown
Windows.UI.Logon.dll!LogonUX::RootViewModel::[Windows::Internal::UI::Logon::LogonUX::ILogonUXViewModel]::__abi_Windows_Internal_UI_Logon_LogonUX_ILogonUXViewModel____abi_Shutdown(struct Windows::Internal::UI::Logon::Callbacks::IOperationComplete ^) Unknown
Windows.UI.Logon.dll!<lambda>(void)() Unknown
Windows.UI.XamlHost.dll!ASTAThreadHost::ASTAThreadHostThreadProc(void) Unknown
Windows.UI.XamlHost.dll!ASTAThreadHost::s_ASTAThreadHostThreadProc() Unknown
SHCore.dll!_WrapperThreadProc() Unknown
kernel32.dll!BaseThreadInitThunk() Unknown
ntdll.dll!RtlUserThreadStart() Unknown

With that, my original investigation of the crash came to a long search.

Resolution

I'll spare you a long tale of how and what I tried in effort to find the cause of the crash. What seemed to have worked for me was the method of exclusion, when by removing chunks of code from my credential provider I was able to isolate the line of code that was causing the crash. And unsurprisingly it was something totally unrelated to the area of the crash itself, i.e. the most difficult type of bug to hunt down.

The problem lied within the way a previous author (and many other people that I see in forums online) tend to initialize COM.

Most people know that in order to use Microsoft functions and interfaces that rely on the Component Object Model (or COM), they need to call CoInitialize, or CoInitializeEx from within the thread that will be invoking those functions. And, while at it, there's also CoUninitialize that has to be called at the end to release internal COM objects.

In practice though, only initialization function was called, while un-initialization CoUninitialize was usually ignored.

That actually, would probably have less dire consequences (albeit a memory leak) than calling those functions incorrectly. Microsoft doesn't stress this point enough, but an incorrect sequence of those calls was the cause of sporadic crashes that I experienced in LogonUI.

Let's review it in a code example. Most people call these functions like this:

//INCORRECT WAY - DO NOT USE IT!!!
CoInitialize(NULL);

//Do some work with COM here ...

CoUninitialize();

Or, even worse, what was done in my case: the author of the original credential provider needed to use some COM functions in his own function. But evidently he wasn't sure which thread that function would be called from. So instead of initializing COM during the initialization of the thread(s), he jammed it into the function itself. So it looked like this:

BOOL DoSomeWorkWithCOM()
{
	//INCORRECT WAY - DO NOT USE IT!!!
	CoInitialize(NULL);

	//Do some work with COM here ...

	CoUninitialize();

	return result;
}

Tell me honestly, how often do you see code like this in forums, and even in some online tutorials?

Explanation

The main problem with the code sample above is that the initialization function (CoInitialize or CoInitializeEx) and the un-initialization one (CoUninitialize) work differently.

CoInitialize* succeeds only if there was no previous COM initialization done on the same thread. And if there was, it doesn't load COM objects again, it simply returns an appropriate error.

CoUninitialize on the other hand, unconditionally unloads previously initialized COM objects.

To be honest, I partially blame it on a sloppy coding from the early Microsoft developers. They could've kept the state of invoking CoInitialize* within a required input parameter, that they could have passed back into the CoUninitialize call, that would check the state of calling CoInitialize* before unloading the COM objects.

I asked a question about this subject on a popular forum, and here's what Microsoft's Raymond Chen had to say:

To clarify: CoInitialize* functions have three categories of result:

1. S_OK means "COM was not initialized before - congratulations, you initialized it! The COM initialization count is now 1. Don't forget to uninitialize."
2. S_FALSE means "COM was already initialized, but it was initialized the same way you requested, so it's all good. The COM initialization count has been incremented. Don't forget to uninitialize!"
3. And RPC_E_CHANGED_MODE means "Oh no, COM was already initialized in an incompatible way. No initialization occurred, do not uninitialize!"

The expectation was that you would check the return value and call uninitialize only if your initialize succeeded. How you keep track of this is up to you. You could use an RAII type. You could use an "if" statement and skip the COM work if initialization failed. You could throw an exception. Not forcing you into a specific pattern. You do you.

How can this go wrong?

So picture a situation when DoSomeWorkWithCOM function from the code sample above was called from a thread that had COM libraries initialized already for a threading model other than the one that was requested in DoSomeWorkWithCOM. In that case the first call to CoInitialize in our function will fail to load COM, and return RPC_E_CHANGED_MODE, but the subsequent call to CoUninitialize will unload it. After that, it could lead to two possible outcomes in the host process:

1. Some COM function may continue using COM components without realizing that they are being unloaded, which could lead to a race-condition, that in turn would create a slew of unpredictable bugs and crashes.
2. The host process would notice that COM is unloaded and will have no way to recover from it.

In case of the LogonUI.exe, it seems like the shut-down logic responsible for unloading the credential provider DLL was checking the state of COM, and because of the messed up sequence of calls that I showed above, that check failed, resulting in LogonUI.exe terminating itself with a call to RaiseFailFastException.

The Fix

The fix in that sense was very simple. All I needed to do was to check the return value from the CoInitialize* call, and invoke CoUninitialize only upon success:

BOOL bComInitialized = SUCCEEDED(CoInitialize(NULL));

//Do some work with COM here ...

if(bComInitialized)
{
	CoUninitialize();
}

And that little check would have prevented the crash in LogonUI.exe.

Even Better Way

Or, having the power of C++ and the concept of RAII, we can put it into a class:

#include <Objbase.h>

enum COM_INITIALIZER_TYPE
{
	COM_SINGLE_THREADED = COINIT_APARTMENTTHREADED,
	COM_MULTI_THREADED = COINIT_MULTITHREADED,
};

struct COM_INITIALIZER
{
	COM_INITIALIZER(COM_INITIALIZER_TYPE type)
	{
		_hr = CoInitializeEx(NULL, type);
	}

	~COM_INITIALIZER()
	{
		if(SUCCEEDED(_hr))
		{
			CoUninitialize();
		}
	}

	BOOL is_COM_initialized()
	{
		//Checks if COM was initialized
		//RETURN:
		//      = TRUE if COM was initialized, or
		//      = FALSE if it was not (check GetLastError() for details)
		if(_hr == S_OK ||
			_hr == S_FALSE)
		{
			return TRUE;
		}

		//Error
		SetLastError(_hr);
		return FALSE;
	}

private:
	HRESULT _hr = E_UNEXPECTED;

	//Copy constructor and assignment operators are not allowed!
	COM_INITIALIZER(COM_INITIALIZER const&) = delete;
	COM_INITIALIZER(COM_INITIALIZER const&&) = delete;
	COM_INITIALIZER& operator=(COM_INITIALIZER const&) = delete;
	COM_INITIALIZER& operator=(COM_INITIALIZER const&&) = delete;
};

Conclusion

Sometimes even a slightest omission in checking the return value, or calling a releasing API at a wrong time, can have pretty serious consequences. Worse still, the types of bugs and security vulnerabilities that may ensue from these mistakes may be quite difficult to hunt down and fix.

By showing you this simple example, I truly hope that you will not make the same mistake in your code.