Fix an Issue with Reboot Flag in NSIS Installer

Project Description

We have a NSIS installer that installs a product. It is a reasonably complicated legacy installer, that combines 32-bit and 64-bit files in itself to have one installer for both Windows platforms.

It does things that installer do - allows user to select a language, install directory, etc. Among other things it attempt to migrate an already existing installation of the product to new user settings.

The problem occurs when the new install directory is different from the original. The problem was first observed when trying to install a 64-bit product existing in C:\Program Files (x86)\ into C:\Program Files\. However it is not specific to PROGRAMFILES, PROGRAMFILES32, or PROGRAMFILES64. The problem is reliably reproduced when you first install it into a regular directory, such as C:\Temp1, and then user tries to install update into C:\Temp2 or any other directory different from the original.

Our investigation shows that when the directories are different, the installer attempts to exit the running program, uninstall a Windows service that it uses, then tries to recursively delete the original install directory. It looks like the reboot flag (IfRebootFlag) is set when removing this directory, and only when the installer is launched from the running program.

One peculiarity of the issue is that when you run the same installer interactively, it does not happen.

We need help with understanding the issue fully, and fixing it, if possible.

Completion Notes

Let's first see if we can reproduce the problem. The environment is:


Let's do a simple problem reproduction TEST1:


TEST2:


So, apparently, the problem only happens when a product launches the installer.

Reducing Complexity

Our setup is quite complex, we have a complicated legacy product consisting of many parts, and also a complicated legacy installer that does many things. We need to reduce the scope of complexity.

We will now try to catch the lyon in the desert by dividing the desert in half and determining where the lyon is.

Let us create a trivial test.exe that launches the installer in the same way the product does. This will be the only thing the exe does. So, our entire console application we'll be like:

int _tmain(int argc, _TCHAR* argv[])
{
  // Try to launch the installer.
  ShellExecute(0, _T("open"), _T("C:\\installer.exe"), NULL, 0, SW_SHOWNORMAL);
  return 0;
}

Now, it's time for TEST3:


Perhaps, the problem is with sub-directories? As the original installer suggests one directory for company name, and yet another for product name. Let's see if we can eliminate them and deal only with C:\Temp1 and C:\Temp2. No, installing without extra sub-directories does not fix the reboot issue.

Let us reduce the scope further by dealing with only one installer, not two. We'll use the build 4737, which is C:\installer.exe.

TEST4:


TEST5:

We will now try to run test.exe from outside of the original product installation directory. For example, from C:\test.exe. And this is the scenario where the reboot problem disappears, the product installs just fine into another directory without a reboot request.

Well then, our lion now runs in half of the desert, we know that it has nothing to do with the product or what the product might be doing.

The installer, however, is still a complex thingy, let us see if we can reduce it to something simple.

Reducing the Installer

Can we have a trivial installer that attempts to recursively delete a directory and then checks a reboot flag? Something like this NSIS script:

Outfile "remove_dir.exe"

Section

RmDir /r /REBOOTOK 'C:\Temp1'
IfRebootFlag 0 +2
MessageBox MB_OK "Reboot flag is set!"

SectionEnd

If we compile the above script, we'll have remove_dir.exe that attempts to remove C:\Temp1 and then displays a message box if a reboot flag is set. Let's see how much luck we'll have with it.

TEST6:


TEST6 shows, that a trivial installer, whose single task is to remove a directory, is exhibiting basically the same reboot problem.

Reducing the Directory

Notice that in all above tests, our product was fully installed, and it consists of many parts. We will now try to reduce this complexity by not installing the product at all, and instead by creating a directory manually.

TEST7:


TEST8:


We now reduced the scope of the original problem to a trivial installer and a trivial exe.

Solution

Could the problem be related to how we call ShellExecute()? Notice that the 5th parameter to ShellExecute is a working directory for the function, and it is NULL, meaning that the current working directory is used.

It looks like it really is the root of our problem, because the reboot issue goes away if, instead of NULL, we specify something else here. I tried "C:\Temp" and it worked.

If you need something done, feel free to post a project here. You can also leave a comment on this project.