After determining that the DLL hang simply is not inside of the code that we maintain, my next step was to suspect that it had something to do with the various 3rd party controls we use or somewhere within Delphi 2006 itself.   My output debug strings in the finalization sections showed that the hang happens after the last finalization we use.  So I thought I'd drop some into the finalization sections of the 3rd party controls (DevExpress, ReportBuilder, SvCom, TeeChart, to name a few), then rebuild all of those packages (yes Virginia, there is a good reason to spend the extra money and buy the sourcecode versions of 3rd party software).  However after realizing I had 853 finalization sections that needed Outputdebugstring's (ODS) added, I nixed that idea.

The next step then was to get Delphi 2007 installed and do a rebuild.  For the first pass I just upgraded D2007 and used the existing versions of 3rd party recompiled in D2007.  Sent the newly recompiled test case off to our customer and they responded with "Yup, still happens.  But guess what we found out?  Remember we said that we tested it with hyperthreading turned off?  It seems that we must not have, because when we turn it off the lockup goes away.  We're looking into possible hardware / bios issues now."  Sweet ... 60+ hours into this issue and they now realize that one of the tests I asked for on day 1 they never did.

Still a number of unanswered questions though, got those queued up and will be researching & testing them once I get all 3rd party upgraded.

Looks like CodeGear has taken a step backwards with Delphi installs.  The RAD team at Borland went out of their weay to ensure that different versions of Delphi would live happily together on the same PC.   After installing Delphi 2007, my D2006 is now broken.

Something else that I found VERY irritating.  You use a download program to do the install.  I run the app, tells me that it's got 800mb gig of data to download and off it goes.  Downloads done, I run the setup, reboot the PC and start D2007.   It comes up with an option "There is a new version available, do you want to download Update 1?"  When I answered yes it informed me that it had to uninstall D2007, dowload another 800 mb of data, and then reinstall.  What the hell??  Why didn't they prompt me with that before the first download? 

Oh, and the best part of this is that you have to keep the local cache of the installation files if you want to be able to run any future updates - so your 800mb install is 1.6gig (give or take).

And now there's an update 2.. which is only a partial update, if you kept the Cache during the install of Update 1.  If you didn't keep it, time to uninstall D2007 - reinstall base image, reinstall update 1 (keeping the Cache for both of them), and then install update 2.

Chris Pattinson's blog about Update 2
The readme for Update 2

Been working on a problem for a customer of ours for the last couple weeks (off and on).  On one specific configuration of Dell computers, with a certain disk image on it, they're getting system lockups when the client application does an FreeLibrary.   Same image on different computers, no problem.  The new computer without this image on it, no problem.   Unfortunately this customer has purchased over 600 of these PC's, they *have* to use this image, and are in the process of rolling them out to their end users.  So this is an issue that must be resolved, we can't punt on it.  The real bitch of this issue is that we can't duplicate the issue here on any computer I've used, no other customer sees the problem, and due to security reasons they won't ship us one of their PC's so I can debug it, nor will they let come onto their site and do it there. 

So I'm debugging remotely, old school style.  Make up test cases, put in lots of outputdebug strings and ship it to their tech guy to try out, and he emails me the results back.  After 40+ hrs of research and test cases I'm no closer to the solution, but I am certain it happens somewhere after the FreeLibrary - something is hanging.

Research led me to a couple articles that I want to make note of for the future, I know I'll be needing this info again one day. 

Quick overview of how processes exit on Windows XP
LoadLibraryEx(DONT_RESOLVE_DLL_REFERENCES) is fundamentally flawed
Some reasons not to do anything scary in your DLLMiain
Why are DLLs unloaded in the "wrong" order?

We're using a bunch of packages (BPL's) in our client & dll, but compiling without packages the error still happens.  I've got a suspicion the problem is in the finalization code that happens, so I've built up another test case and added outputdebugstrings into all of the finalizations that occur.  There's no critical sections in these apps, so that shouldn't be an issue.   Ok, back to banging my head on the desk for a while.