Across my environment we are seeing sporadic issues appear on our database servers. The issue first appeared about 4 months ago and it is only resolved by rebooting the server. It has never appeared on the same server twice and it has now happened on a half dozen servers. The issue has occurred on a variety of server types, two different OS's and two different versions of SQL Server.
The issue is always the same, we start to notice high CPU utilization and poor performance in SQL server. The smallest of action, even when nothing else is hitting the database will cause the CPU to jump up. After checking all of the hardware, OS and SQL Logs and finding nothing we decided to check IOMeter. We now can conclude that this issue always has the same symptom that the IOPS drop by a huge amount (for example 15k IOPS on write drops to 2k). Note this happens when nothing else is going on in the server. When this happens we have tried resolving the issue by doing everything possible from stopping all services including SQL Server, resetting the cache policy on the disks, monitoring all file access in and processes in process explorer.
There's never a pattern, it always just "starts happening" at various points in the day. We have disabled the antivirus to eliminate it as a factor. The hardware and software vary. The only consistent thing is that this only happens on our database server and never any other server.
---
4 out of the 6 servers had this configuration:
Dell PE R720
RAID 1 - OS/Logs (2x 15k rpm drives)
RAID 6 - Data (14x 15k rpm drives)
Windows Server 2008 R2 SP1
SQL Server 2008 R2 SP2
96GB Memory
1x 6 core 2.5GHZ CPU
----
One scenario was on a HP Server with Windows Server 2003 running SQL Server 2000 against a SAN.
----
One scenario was Windows Server 2003 running SQL Server 2008 R2 against 4 spinning disks in RAID 6.
Has anyone seen this problem before? If so, were you able to find the cause of the problem and implement a permanent fix rather than just reboot to fix it. We are probably going to contact Microsoft to open a support case, but I figured I would see if any other Sys and/or SQL admins have seen this problem before. It's definitely a strange one.