Fix an Issue with Reboot Flag in NSIS Installer
We have a NSIS installer that installs a product. It is a reasonably complicated legacy installer, that combines 32-bit and 64-bit files in itself to have one installer for both Windows platforms.
It does things that installer do - allows user to select a language, install directory, etc. Among other things it attempt to migrate an already existing installation of the product to new user settings.
The problem occurs when the new install directory is different from the original. The problem was first observed when trying to install a 64-bit product existing in C:\Program Files (x86)\ into C:\Program Files\. However it is not specific to PROGRAMFILES, PROGRAMFILES32, or PROGRAMFILES64. The problem is reliably reproduced when you first install it into a regular directory, such as C:\Temp1, and then user tries to install update into C:\Temp2 or any other directory different from the original.
Our investigation shows that when the directories are different, the installer attempts to exit the running program, uninstall a Windows service that it uses, then tries to recursively delete the original install directory. It looks like the reboot flag (IfRebootFlag) is set when removing this directory, and only when the installer is launched from the running program.
One peculiarity of the issue is that when you run the same installer interactively, it does not happen.
We need help with understanding the issue fully, and fixing it, if possible.
Let's first see if we can reproduce the problem. The environment is:
- Windows 7, 32 bit (OS version 6.1 build 7601 Service Pack 1). The problem is also seen on 64 bit Windows 7, so a choice of OS probably does not matter.
- Customer product build 4720. This is what we install originally.
- Customer update build 4737. This is what is being downloaded and installed, and then the reboot problem occurs.
Let's do a simple problem reproduction TEST1:
- Install product 4720 into C:\Temp1
- Check for updates in the running product. It downloads the build 4737. Install it from the running product into C:\Temp2. Indeed, the problem is confirmed, as the installer for build 4737 asks for a reboot.
- Install product 4720 into C:\Temp1
- Run the installer for build 4737 manually (interactively by double-clicking). and install into C:\Temp2. The problem is NOT seen, the install completes successfully without a reboot.
So, apparently, the problem only happens when a product launches the installer.
Our setup is quite complex, we have a complicated legacy product consisting of many parts, and also a complicated legacy installer that does many things. We need to reduce the scope of complexity.
We will now try to catch the lyon in the desert by dividing the desert in half and determining where the lyon is.
Let us create a trivial test.exe that launches the installer in the same way the product does. This will be the only thing the exe does. So, our entire console application we'll be like:
int _tmain(int argc, _TCHAR* argv)
// Try to launch the installer.
ShellExecute(0, _T("open"), _T("C:\\installer.exe"), NULL, 0, SW_SHOWNORMAL);
Now, it's time for TEST3:
- Install the product into C:\Temp1
- Compile test.exe and put it into C:\Temp1
- Stop the product to keep things simple.
- Launch test.exe from Windows Explorer, and see the installer started, displaying the first screen. At this point test.exe exits. Close Windows Explorer to avoid any potential issues with open folders.
- Continue with install into C:\Temp2 to see the reboot issue again. We now confirmed that the problem exists when a trivial test.exe starts the installer.
Perhaps, the problem is with sub-directories? As the original installer suggests one directory for company name, and yet another for product name. Let's see if we can eliminate them and deal only with C:\Temp1 and C:\Temp2. No, installing without extra sub-directories does not fix the reboot issue.
Let us reduce the scope further by dealing with only one installer, not two. We'll use the build 4737, which is C:\installer.exe.
- Use C:\installer.exe to install the product in C:\Temp1.
- Put test.exe into C:\Temp1
- Run test.exe from Windows Explorer just to confirm that we still have the reboot problem.
We will now try to run test.exe from outside of the original product installation directory. For example, from C:\test.exe. And this is the scenario where the reboot problem disappears, the product installs just fine into another directory without a reboot request.
Well then, our lion now runs in half of the desert, we know that it has nothing to do with the product or what the product might be doing.
The installer, however, is still a complex thingy, let us see if we can reduce it to something simple.
Reducing the Installer
Can we have a trivial installer that attempts to recursively delete a directory and then checks a reboot flag? Something like this NSIS script:
RmDir /r /REBOOTOK 'C:\Temp1'
IfRebootFlag 0 +2
MessageBox MB_OK "Reboot flag is set!"
If we compile the above script, we'll have remove_dir.exe that attempts to remove C:\Temp1 and then displays a message box if a reboot flag is set. Let's see how much luck we'll have with it.
- Install the product in C:\Temp1.
- Run C:\remove_dir.exe interactively without stopping the product. See that reboot flag is set, because some files are in use, which is expected.
- Now repeat the test, but this time stop the product before running C:\remove_dir.exe. We can see that the directory is been removed okay, without reboot flag set.
- If we modify test.exe to launch C:\remove_dir.exe, and place it inside C:\Temp1 and run it from there, the reboot flag will be set.
TEST6 shows, that a trivial installer, whose single task is to remove a directory, is exhibiting basically the same reboot problem.
Reducing the Directory
Notice that in all above tests, our product was fully installed, and it consists of many parts. We will now try to reduce this complexity by not installing the product at all, and instead by creating a directory manually.
- Create a directory C:\Temp1 manually, and put one file into it for testing.
- Run C:\remove_dir.exe - see that the reboot flag is not set and the directory is removed.
- Create a directory C:\Temp1 manually, and put test.exe into it. Remember that it launches C:\remove_dir.exe to remove C:\Temp1.
- Launch C:\Temp1\test.exe and confirm that the reboot flag is set.
We now reduced the scope of the original problem to a trivial installer and a trivial exe.
Could the problem be related to how we call ShellExecute()? Notice that the 5th parameter to ShellExecute is a working directory for the function, and it is NULL, meaning that the current working directory is used.
It looks like it really is the root of our problem, because the reboot issue goes away if, instead of NULL, we specify something else here. I tried "C:\Temp" and it worked.
If you need something done, feel free to post a project
here. You can also leave a comment
on this project.