We have a large 2 TB MS CRM SQL 2008 database running on 32 core, 32GB machines. We have 10 IIS server middle tier accessing the database via LINQ entities. The design looks like this:
MS Test Agents (20) -> Middle Tier WCF web services -> SQL Database1 -- SQL Database 2(MS CRM database)
Database1 has linked queries to Database 2 which is a Microsoft CRM database. The reason why we are not accessing the MS CRM directly is because it is a MS CRM SQL database that we don't want to modify directly.
While the SQL boxes are physical servers, the test agents and middle tier are VMs.
We were having performance problems under load test. The application starts out with fast response times averaging 400ms. However with time the perf degrades and after 20 minutes of constant load its 6-9secs response time. After around 20 minutes the response time plateaus to 6-9secs. With the help of SQL profiler we've narrowed the issue to SQL. Next I narrowed the problem domain by writing a simple stored procedure in database 1. This sproc does not access any table but rather is a simple static query select like the following:
Select 'x' as x, 'y' as y, 'z' as z
When we tried to apply a constant load of 100 concurrent users from 20 MS Test Agents to this sproc, that too exhibited the performance problem. No other apps are accessing the db.
Perfmon counters indicates no memory pressure, low I/O, and CPU hovers around 25-30% on SQL boxes. Also batch/secs goes down indirectly proportional to the response times. On IIS boxes the resource utilization is even lower. Like I mentioned profiler shows the sproc itself starts out with a fast transaction time of 400ms and gradually degrades to 6+ secs over 20 minutes. After that time its 6 secs constant response time.
I'm surprized at these results as I would've expected that by calling the same sproc the result would be cached and response time would improve. Please help out with this issue. After trying to resolve it over the course of 1 month I'm running out of ideas and in need to fresh ones to try.
MS Test Agents (20) -> Middle Tier WCF web services -> SQL Database1 -- SQL Database 2(MS CRM database)
Database1 has linked queries to Database 2 which is a Microsoft CRM database. The reason why we are not accessing the MS CRM directly is because it is a MS CRM SQL database that we don't want to modify directly.
While the SQL boxes are physical servers, the test agents and middle tier are VMs.
We were having performance problems under load test. The application starts out with fast response times averaging 400ms. However with time the perf degrades and after 20 minutes of constant load its 6-9secs response time. After around 20 minutes the response time plateaus to 6-9secs. With the help of SQL profiler we've narrowed the issue to SQL. Next I narrowed the problem domain by writing a simple stored procedure in database 1. This sproc does not access any table but rather is a simple static query select like the following:
Select 'x' as x, 'y' as y, 'z' as z
When we tried to apply a constant load of 100 concurrent users from 20 MS Test Agents to this sproc, that too exhibited the performance problem. No other apps are accessing the db.
Perfmon counters indicates no memory pressure, low I/O, and CPU hovers around 25-30% on SQL boxes. Also batch/secs goes down indirectly proportional to the response times. On IIS boxes the resource utilization is even lower. Like I mentioned profiler shows the sproc itself starts out with a fast transaction time of 400ms and gradually degrades to 6+ secs over 20 minutes. After that time its 6 secs constant response time.
I'm surprized at these results as I would've expected that by calling the same sproc the result would be cached and response time would improve. Please help out with this issue. After trying to resolve it over the course of 1 month I'm running out of ideas and in need to fresh ones to try.