I
IIS
Guest
Throughout my time working on the IIS team I've seen a lot of cases where frequent unloading of the Application Domains caused several performance issues. While there are a lot of blog post covering the mechanism ASP.NET uses to detect changes in the files, each of them only covers one aspect of the problem. Hopefully this article can provide a more unified perspective and help you understand how you can troubleshoot or prevent these issues.
What is an Application Domain
The .NET Application Domain is the global data-structure where the data from your .NET application is loaded into. The .dlls (binaries) from your application get loaded inside this data-structure. All of the instances of all of the classes from your application, as well as the data these instances manipulate, all of these are found inside the .NET Application Domain for this application.
What Causes an Application Domain to shut down
Unlike native C++ applications, where a specific .dll can be unloaded from process memory, in .NET applications, when a .NET assembly (.NET dll) is loaded into the process address space, it cannot be unloaded. If we need to unload the assembly for any reason – eg: it has been updated on disk with a newer version, we will need to unload the entire Application Domain, so we unload all data regarding the application, and then we load the data back again: from your application’s point of view, it is equivalent to a worker process recycle. So, if for any reason you're still storing the session state InProc it will be lost. Also if the Application Domain unloads frequently, there will be a performance hit.
The full list of reasons for an Application Domain shutdown is available here: ApplicationShutdownReason Enum (System.Web)
How to tell if your Application Domain is being unloaded
Each .NET Application Domain has a security descriptor – you can think about this as a name for the .NET Application Domain. For example, the security descriptor for a .NET Application Domain can look something like this:
/LM/W3SVC/1/ROOT-10-131843727650365299
With ASP.NET Application Domains, the security descriptor is built in such a way as to reflect the location where the website (web-application) is hosted, as well as some runtime data about the Application Domain in question. In the example above the web-application is actually the root application of the Default Web Site in IIS (which is site W3SVC1). The number that comes after the ROOT part of the security descriptor is also of very special interest. It shows how many times we have loaded and unloaded this Application Domain from the memory of the worker process. From the -10- part of the security descriptor, we can know that this is the 10th time this Application Domain has been loaded into the process address space.
How to identify the reason for an Application Domain shut-down
ASP.NET has a built-in Health Monitoring system that logs events that occur during the application or request's lifetime. The event logged by the Health Monitoring system include application restarts along with the reason for the restart. To enable Health Monitoring you should add the following to the <healthMonitoring> section in the web.config:
<healthMonitoring>
<rules>
<add eventName="Application Lifetime Events" name="Application Lifetime Events Default" profile="Default" provider="EventLogProvider" />
</rules>
</healthMonitoring>
If you don't want to modify the web.config by hand, you can also enable Health Monitoring using appcmd.exe or PowerShell. The below example commands will enable Health Monitoring for the Default Web Site:
APPCMD
appcmd.exe set config "Default Web Site" -section:system.web/healthMonitoring /+"rules.[eventName='Application Lifetime Events',name='Application Lifetime Events Default',profile='Default',provider='EventLogProvider']"
POWERSHELL
Add-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST/Default Web Site' -filter "system.web/healthMonitoring/rules" -name "." -value @{eventName='Application Lifetime Events';name='Application Lifetime Events Default';profile='Default';provider='EventLogProvider'}
Once you enable Health Monitoring Events, you should see events like the ones below in the Application Event Logs:
Time: 12/18/2019 5:03:26 PM
ID: 1305
Level: Information
Source: ASP.NET 4.0.30319.0
Machine: ***
Message: Event code: 1002
Event message: Application is shutting down. Reason: Configuration changed.
Event time: 18/12/2019 17:03:26
Event time (UTC): 18/12/2019 16:03:26
Event ID: 24cd160d8330496a820b8a98c5600b11
Event sequence: 6684
Event occurrence: 1
Event detail code: 50004
Application information:
Application domain: /LM/W3SVC/1/ROOT-1035-132000756851951897
Trust level: Full
Application Virtual Path: ***
Application Path: ***
Machine name: ***
Process information:
Process ID: 7720
Process name: w3wp.exe
Account name: ***
Custom event details:
In the event above, the reason stated for the Application Domain shutdown is "Configuration changed".
The .NET Framework has a mechanism by which it asks the operating system to be notified when certain files change on disk, so that it can react to such changes. The mechanism in question is called: File Change Notifications, or FCNs in short. In order for ASP.NET to know when a page in your application has changed, and be able to reach to the change, it subscribes to file change notifications for the folders containing your application on disk.
Who Is Unloading my Application Domain ?
Now say that you checked the event logs and you found that the Application Domain for your .NET application is unloading quite frequently. You've also found that the reason is a configuration change. However, you're absolutely positive that you haven't changed the configuration in a while. So who did?
Well, a better word here instead of changed would be "touched" because really the FCNs get triggered whenever the application folder is updated in any way. That means even if the web.config is scanned by the antivirus, ASP.NET will consider that the file has changed. For that reason, we usually recommend implementing antivirus exclusions for the application folders. You can find a complete list of which files you should exclude from the antivirus scan in this article: https://support.microsoft.com/en-in...clude-from-antivirus-scanning-in-asp-net-apps. Some antivirus software will automatically exclude some folder paths if IIS is installed on the server. For example, Windows Defender will automatically exclude:
However, these default exclusions will only help if your application files are under the default %systemDrive%\inetpub\wwwroot folder. If you configured your application path to a different location, you will have to manually add that path to the exclusion list.
If you want to test whether the anti-virus might be the one triggering the FCNs, I would always recommend implementing the exclusions over disabling the antivirus. Even if you disable the anti-virus it does not mean it cannot be the cause of the FCN. The reason is that most antivirus software use kernel-mode filters. When you disable the software component, the filter driver is still loaded when you restart the computer. I don't recommend disabling the corresponding filter drivers since it may make your computer or your network more vulnerable to attack by malicious users or by malicious software such as viruses. My point here is that disabling the antivirus is not a conclusive test to prove whether the antivirus is the culprit.
If you're absolutely sure you implemented all the exclusions for the antivirus and you're still seeing the application shutting down due to configuration changes, the next step would be to collect a Process Monitor trace and see if there is any other software that is touching the application files.
What if I really don't want FCNs ?
It may happen that you have a valid reason for touching the files in the folders that ASP.NET monitors for File Change Notifications and you don't want the Application Domain to be unloaded every time a file is updated. Luckily, there is a registry key that you can use to control the FCN behavior.
As described in this article: https://support.microsoft.com/en-us...-applications-on-a-web-site-may-appear-to-sto, you can configure the registry key HKLM\Software\Microsoft\ASP.NET\FCNMode to one of the following values:
Value
Behavior
Does not exist
This is the default behavior. For each subdirectory, the application will create an object that will monitor the subdirectory.
0 or greater than 2
This is the default behavior. For each subdirectory, the application will create an object that will monitor the subdirectory.
1
The application will disable File Change Notifications (FCNs).
2
The application will create one object to monitor the main directory. The application will use this object to monitor each subdirectory.
Of course, if you don't want to make the change system-wide, you can also just modify it at the individual application level, by setting the <httpRuntime fcnMode="<setting>"/> property in the web.config of the respective application, where setting corresponds to:
Setting
Corresponding value
NotSet(0)
This is the same as setting 0 or greater than 2 in the registry.
Default(1)
This is the same as setting 0 or greater than 2 in the registry.
Disabled(2)
This is the same as setting 1 in the registry.
Single(3)
This is the same as setting 2 in the registry.
One last thing to note though would be that even if you disable FCN mode, changes to the web.config will still trigger an Application Domain reload.
Roxana Gheorghe
Continue reading...
What is an Application Domain
The .NET Application Domain is the global data-structure where the data from your .NET application is loaded into. The .dlls (binaries) from your application get loaded inside this data-structure. All of the instances of all of the classes from your application, as well as the data these instances manipulate, all of these are found inside the .NET Application Domain for this application.
What Causes an Application Domain to shut down
Unlike native C++ applications, where a specific .dll can be unloaded from process memory, in .NET applications, when a .NET assembly (.NET dll) is loaded into the process address space, it cannot be unloaded. If we need to unload the assembly for any reason – eg: it has been updated on disk with a newer version, we will need to unload the entire Application Domain, so we unload all data regarding the application, and then we load the data back again: from your application’s point of view, it is equivalent to a worker process recycle. So, if for any reason you're still storing the session state InProc it will be lost. Also if the Application Domain unloads frequently, there will be a performance hit.
The full list of reasons for an Application Domain shutdown is available here: ApplicationShutdownReason Enum (System.Web)
How to tell if your Application Domain is being unloaded
Each .NET Application Domain has a security descriptor – you can think about this as a name for the .NET Application Domain. For example, the security descriptor for a .NET Application Domain can look something like this:
/LM/W3SVC/1/ROOT-10-131843727650365299
With ASP.NET Application Domains, the security descriptor is built in such a way as to reflect the location where the website (web-application) is hosted, as well as some runtime data about the Application Domain in question. In the example above the web-application is actually the root application of the Default Web Site in IIS (which is site W3SVC1). The number that comes after the ROOT part of the security descriptor is also of very special interest. It shows how many times we have loaded and unloaded this Application Domain from the memory of the worker process. From the -10- part of the security descriptor, we can know that this is the 10th time this Application Domain has been loaded into the process address space.
How to identify the reason for an Application Domain shut-down
ASP.NET has a built-in Health Monitoring system that logs events that occur during the application or request's lifetime. The event logged by the Health Monitoring system include application restarts along with the reason for the restart. To enable Health Monitoring you should add the following to the <healthMonitoring> section in the web.config:
<healthMonitoring>
<rules>
<add eventName="Application Lifetime Events" name="Application Lifetime Events Default" profile="Default" provider="EventLogProvider" />
</rules>
</healthMonitoring>
If you don't want to modify the web.config by hand, you can also enable Health Monitoring using appcmd.exe or PowerShell. The below example commands will enable Health Monitoring for the Default Web Site:
APPCMD
appcmd.exe set config "Default Web Site" -section:system.web/healthMonitoring /+"rules.[eventName='Application Lifetime Events',name='Application Lifetime Events Default',profile='Default',provider='EventLogProvider']"
POWERSHELL
Add-WebConfigurationProperty -pspath 'MACHINE/WEBROOT/APPHOST/Default Web Site' -filter "system.web/healthMonitoring/rules" -name "." -value @{eventName='Application Lifetime Events';name='Application Lifetime Events Default';profile='Default';provider='EventLogProvider'}
Once you enable Health Monitoring Events, you should see events like the ones below in the Application Event Logs:
Time: 12/18/2019 5:03:26 PM
ID: 1305
Level: Information
Source: ASP.NET 4.0.30319.0
Machine: ***
Message: Event code: 1002
Event message: Application is shutting down. Reason: Configuration changed.
Event time: 18/12/2019 17:03:26
Event time (UTC): 18/12/2019 16:03:26
Event ID: 24cd160d8330496a820b8a98c5600b11
Event sequence: 6684
Event occurrence: 1
Event detail code: 50004
Application information:
Application domain: /LM/W3SVC/1/ROOT-1035-132000756851951897
Trust level: Full
Application Virtual Path: ***
Application Path: ***
Machine name: ***
Process information:
Process ID: 7720
Process name: w3wp.exe
Account name: ***
Custom event details:
In the event above, the reason stated for the Application Domain shutdown is "Configuration changed".
The .NET Framework has a mechanism by which it asks the operating system to be notified when certain files change on disk, so that it can react to such changes. The mechanism in question is called: File Change Notifications, or FCNs in short. In order for ASP.NET to know when a page in your application has changed, and be able to reach to the change, it subscribes to file change notifications for the folders containing your application on disk.
Who Is Unloading my Application Domain ?
Now say that you checked the event logs and you found that the Application Domain for your .NET application is unloading quite frequently. You've also found that the reason is a configuration change. However, you're absolutely positive that you haven't changed the configuration in a while. So who did?
Well, a better word here instead of changed would be "touched" because really the FCNs get triggered whenever the application folder is updated in any way. That means even if the web.config is scanned by the antivirus, ASP.NET will consider that the file has changed. For that reason, we usually recommend implementing antivirus exclusions for the application folders. You can find a complete list of which files you should exclude from the antivirus scan in this article: https://support.microsoft.com/en-in...clude-from-antivirus-scanning-in-asp-net-apps. Some antivirus software will automatically exclude some folder paths if IIS is installed on the server. For example, Windows Defender will automatically exclude:
- %SystemRoot%\IIS Temporary Compressed Files
- %SystemDrive%\inetpub\temp\IIS Temporary Compressed Files
- %SystemDrive%\inetpub\temp\ASP Compiled Templates
- %systemDrive%\inetpub\logs
- %systemDrive%\inetpub\wwwroot
However, these default exclusions will only help if your application files are under the default %systemDrive%\inetpub\wwwroot folder. If you configured your application path to a different location, you will have to manually add that path to the exclusion list.
If you want to test whether the anti-virus might be the one triggering the FCNs, I would always recommend implementing the exclusions over disabling the antivirus. Even if you disable the anti-virus it does not mean it cannot be the cause of the FCN. The reason is that most antivirus software use kernel-mode filters. When you disable the software component, the filter driver is still loaded when you restart the computer. I don't recommend disabling the corresponding filter drivers since it may make your computer or your network more vulnerable to attack by malicious users or by malicious software such as viruses. My point here is that disabling the antivirus is not a conclusive test to prove whether the antivirus is the culprit.
If you're absolutely sure you implemented all the exclusions for the antivirus and you're still seeing the application shutting down due to configuration changes, the next step would be to collect a Process Monitor trace and see if there is any other software that is touching the application files.
What if I really don't want FCNs ?
It may happen that you have a valid reason for touching the files in the folders that ASP.NET monitors for File Change Notifications and you don't want the Application Domain to be unloaded every time a file is updated. Luckily, there is a registry key that you can use to control the FCN behavior.
As described in this article: https://support.microsoft.com/en-us...-applications-on-a-web-site-may-appear-to-sto, you can configure the registry key HKLM\Software\Microsoft\ASP.NET\FCNMode to one of the following values:
Value
Behavior
Does not exist
This is the default behavior. For each subdirectory, the application will create an object that will monitor the subdirectory.
0 or greater than 2
This is the default behavior. For each subdirectory, the application will create an object that will monitor the subdirectory.
1
The application will disable File Change Notifications (FCNs).
2
The application will create one object to monitor the main directory. The application will use this object to monitor each subdirectory.
Of course, if you don't want to make the change system-wide, you can also just modify it at the individual application level, by setting the <httpRuntime fcnMode="<setting>"/> property in the web.config of the respective application, where setting corresponds to:
Setting
Corresponding value
NotSet(0)
This is the same as setting 0 or greater than 2 in the registry.
Default(1)
This is the same as setting 0 or greater than 2 in the registry.
Disabled(2)
This is the same as setting 1 in the registry.
Single(3)
This is the same as setting 2 in the registry.
One last thing to note though would be that even if you disable FCN mode, changes to the web.config will still trigger an Application Domain reload.
Roxana Gheorghe
Continue reading...