First, some background.... Long-standing system, multiple applications and web apps communicating with SQL Server through a farm of load-balanced app servers. The app servers are running a common "business object" that we refer to as the data-source
factory (DSF, for short), it's purpose is to translate object calls into database queries (i.e., a client record from the database becomes a "Client" object with various attributes representing values from the database). This system has been in existence
for several years, and began life running against SQL Server 7, upgraded to 2000, 2005, and for the past year, 2008 R2. At any given point in time, there will be 400-500 connections to the database, any number of which can be doing work. Server
is not taxed, 48 cores, 512GB RAM, no performance issues of any kind.
From the beginning, we've seen a rare issue (2-3 times a year) where we'll suddenly have a handful of SPIDs start blocking, each claiming to be running "SELECT SCOPE_IDENTITY()". If ignored, more and more SPIDs will exhibit the same behaviour, to the
point where we've seen 200 or more, all blocking, all waiting on SCOPE_IDENTITY(). Often, attempting to kill the offending SPIDs is unsuccessful, as they are immediately respawned, presumably by our DSF. Multiple attempts to kill those SPIDs is
sometimes effective, but on occasion we are forced to restart the DSF's.
Trace log analysis of these incidents has not revealed a common table or query, the issue seems to affect all inserts into all tables. Of particular interest, "wait_type" and "wait_resource" for these offending SPIDs are both reported as NULL.
For the first time in over a year (since upgrading to 2008 R2), this "event" occurred again this morning. I'm tired of not being able to explain it, but I don't know where else to look, I've poked into every dark corner of SQL Server that I know to
poke, and I've found nothing. I'm not convinced that this is a SQL Server problem, I think it's something within our DSF, but I don't know how to prove (or disprove) that theory.
Suggestions? Web searches for "SCOPE_IDENTITY blocking" have come up empty, reaffirming my belief that this is not a SQL issue, but something unique to our app.
Below is a snippet from my blocking log for this morning's occurrence, showing the first two blockers. Within 10 minutes, there were over 50 blocking SPIDs, all waiting on SCOPE_IDENTITY():
********** Start time: 28 Dec 2011 05:11:00:427 ********** [SQLSTATE 01000]
session_id blocker status wait_type wait_resource wait_time command cpu_time logical_reads memory_usage login_time last_request_begin_time last_request_end_time client_net_address net_transport database_id
------------------------------- ------- -------------------- -------------------- -------------------------------------------------- ------------------------------ ---------------------------------------------------------------- ------------------------------------------------------------ ----------- -------------------------------- ----------- -------------------- ------------ ------------------------------ ------------------------------ ------------------------------ ---------------------------------------- ---------------------------------------- -----------
320* 0 a_web2 10.25.246.25 sleeping (null) (null) (null) (null) 1724659 40185412 2 27 Dec 2011 18:42:57:260 28 Dec 2011 05:10:19:420 28 Dec 2011 05:10:19:420 10.25.246.25 TCP (null)
564* 0 a_web2 10.25.246.25 sleeping (null) (null) (null) (null) 2010 22632 2 28 Dec 2011 05:10:31:830 28 Dec 2011 05:10:58:360 28 Dec 2011 05:10:58:360 10.25.246.25 TCP (null)
169 564 a_vts 10.25.246.24 running LCK_M_U KEY: 6:395455534989312 (34ca69ce595f) 664 UPDATE 873 55748 2 28 Dec 2011 05:08:49:543 28 Dec 2011 05:10:59:810 28 Dec 2011 05:10:57:527 10.25.246.24 TCP 6
514 320 a_vts 10.25.246.20 running LCK_M_U KEY: 6:72057594146652160 (86c97ce8fc81) 41210 UPDATE 16746 662206 2 28 Dec 2011 02:06:20:723 28 Dec 2011 05:10:19:230 28 Dec 2011 05:10:19:223 10.25.246.20 TCP 6
session_id blocker resource_type resource_subtype resource_database_id resource_associated_entity_id request_mode database_name object_name resource_description
---------- ------- -------------------- ------------------------------ -------------------- ----------------------------- ------------------------------------------------------------ -------------------- -------------------------------------------------- ----------------------------------------------
514 320 KEY 6 72057594146652160 U Vue (null) (86c97ce8fc81)
169 564 KEY 6 395455534989312 U Vue (null) (34ca69ce595f)
---------------------------------------
***** session_id=320 *****
SELECT SCOPE_IDENTITY()
***** session_id=564 *****
SELECT SCOPE_IDENTITY()
Blocking Time = 0:41 [SQLSTATE 01000]