Sitecore Fetch Squad

I spent February 18 through 20 with a customer experiencing performance issues with a Sitecore 5.3.1 solution. We identified a configuration problem within the first few hours, but I stayed on the project for a little while to monitor the solution.

As most Sitecore developers are aware, Sitecore is thoroughly optimized for performance, relying heavily on caching. Caching consumes memory, and caching requirements for a solution with significant data or load can exceed available memory. Out of memory conditions can cause exceptions and can cause ASP.NET to crash or restart, resulting in errors, delays, and outages.

Sitecore provides a memory monitor hook to flush caches when memory exceeds a given threshold. You configure the memory monitor hook using the /configuration/sitecore/hooks section of web.config:

<hooks>
  <hook type="Sitecore.Diagnostics.HealthMonitorHook, Sitecore.Kernel" />
  <hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
    <param desc="Threshold">500MB</param>
    <param desc="Check interval">00:00:05</param>
    <param desc="Minimum time between log entries">00:01:00</param>
    <ClearCaches>true</ClearCaches>
    <GarbageCollect>true</GarbageCollect>
    <AdjustLoadFactor>true</AdjustLoadFactor>
  </hook>
</hooks>

The first parameter (the first <param> element) to the constructor for the Sitecore.Diagnostics.MemoryMonitorHook class specifies the threshold for the memory monitor hook, which is 500MB by default in Sitecore 5.3.1. Sitecore frequently checks the amount of memory consumed by ASP.NET and if it exceeds this limit, then Sitecore will flush caches. The following message repeated frequently in the logs should have made this pretty obvious:

WARN  Memory usage exceeded the MemoryMonitor threshold. All caches have been cleared and a forced GC has been induced.
WARN  Memory used before/after GC: 554,590,208 / 554,590,208

If you see a message like this, your cache may be too small. If you see it constantly, your cache may be way too small, especially if the before/after values are the same.

The problem for this customer was that the amount of memory consumed by ASP.NET was always greater than 500MB. Therefore, Sitecore was flushing caches constantly. It’s somewhat amazing that this didn’t completely overwhelm the content delivery and the database servers.

The solution is quite simple: either set the memory monitor threshold to an appropriate value, or set the ClearCaches property to false. In this case, we set the size of the cache to 1792MB and afterwards were unable to reproduce the issue.

Note that in Sitecore 6, the default value for the memory monitor threshold is higher (800MB), but it shouldn’t matter anyway because cache clearing is disabled by default:

<hooks>
  <hook type="Sitecore.Diagnostics.HealthMonitorHook, Sitecore.Kernel" />
  <hook type="Sitecore.Diagnostics.MemoryMonitorHook, Sitecore.Kernel">
    <param desc="Threshold">800MB</param>
    <param desc="Check interval">00:00:05</param>
    <param desc="Minimum time between log entries">00:01:00</param>
    <ClearCaches>false</ClearCaches>
    <GarbageCollect>false</GarbageCollect>
    <AdjustLoadFactor>false</AdjustLoadFactor>
  </hook>
</hooks>

I would like to thank Alexey Romaniuha of the Sitecore Support team for his identification of the issue.