Every couple weeks, the tempdb LOG fills and SQL Server is cycled to clear it out. It happens very quickly - everything is peachy, then BAM! Upgrading to SQL 2012 is the most significant change (I can think of) that happened before this started.
Service Broker is used as well as CLR procedures and temp table variables. Local temporary tables are used sparingly.
I've read several posts and articles, but so far none seem related to this. Service Broker endpoints are closed correctly (a few years back, we went through the problems associated with not doing this properly and fixed it). Selecting from tempdb..sysobjects shows a bunch of temporary tables that I don't think should be there. I assume their existence is not related to the log size.
We do not have long running transactions (more than 100 ms) and we have a performance system on which we can run tests to ensure this. DBCC OPENTRAN showed no open transactions when the tempdb LOG was in this state.
With the most recent near occurrence (the log filled during the weekend, then about half the space was freed), every related service (there are about 25 of them) was shutdown one by one in an effort to see if doing so would free log space. The 'Available Space In MB' value never significantly dropped.
Not sure how reliable the query used to determine available space:
Select name AS 'File Name', physical_name AS 'Physical Name', size/128 AS 'Total Size in MB'
, size/128.0 - Cast(FileProperty(name, 'SpaceUsed') AS int)/128.0 AS 'Available Space In MB'
, *
From sys.database_files;
The 'Available Space In MB' was constantly changing, but there was never a significant drop. The query was taken from other posts and seems the most popular (would prefer accurate) way to check space used in devices.
Eighty percent of the occurrences are after our system is upgraded (we develop and sell applications with the 25+ services, and we are upgrading about once a week right now). The upgrade sometimes requires a server reboot. The system on which this occurs is our in-house "dog food" server.
Another of our in-house systems, which stores the main database used for testing, is in a similar state. However, it has never experienced the tempdb LOG filling. It goes months without SQL Server being restarted. The query indicates only 425/1,300M of the tempdb log file is available. I would never create a tempdb more than 100M in the first place (optimist), so these values may give you an idea how wildly it has grown.
I'm stuck where to go next. I'd like to know how/why our test system shows over 800M of the tempdb transaction log is used. I doubt it is, but how would do you know? My only guess why this happens is SQL Server gets confused with log pointers / device fragments and thinks no more space is available when space actually is available.
The following would be helpful changes for SQL Server:
backup/restore tempdb (so state can be captured then analyzed).
ability to explicitly drop local temporary tables (and table variables).
Has been 15 years, and no developer has figured out how to ensure the system drops them.
Provide DBAs the opportunity to maintain their system.
Service Broker is used as well as CLR procedures and temp table variables. Local temporary tables are used sparingly.
I've read several posts and articles, but so far none seem related to this. Service Broker endpoints are closed correctly (a few years back, we went through the problems associated with not doing this properly and fixed it). Selecting from tempdb..sysobjects shows a bunch of temporary tables that I don't think should be there. I assume their existence is not related to the log size.
We do not have long running transactions (more than 100 ms) and we have a performance system on which we can run tests to ensure this. DBCC OPENTRAN showed no open transactions when the tempdb LOG was in this state.
With the most recent near occurrence (the log filled during the weekend, then about half the space was freed), every related service (there are about 25 of them) was shutdown one by one in an effort to see if doing so would free log space. The 'Available Space In MB' value never significantly dropped.
Not sure how reliable the query used to determine available space:
Select name AS 'File Name', physical_name AS 'Physical Name', size/128 AS 'Total Size in MB'
, size/128.0 - Cast(FileProperty(name, 'SpaceUsed') AS int)/128.0 AS 'Available Space In MB'
, *
From sys.database_files;
The 'Available Space In MB' was constantly changing, but there was never a significant drop. The query was taken from other posts and seems the most popular (would prefer accurate) way to check space used in devices.
Eighty percent of the occurrences are after our system is upgraded (we develop and sell applications with the 25+ services, and we are upgrading about once a week right now). The upgrade sometimes requires a server reboot. The system on which this occurs is our in-house "dog food" server.
Another of our in-house systems, which stores the main database used for testing, is in a similar state. However, it has never experienced the tempdb LOG filling. It goes months without SQL Server being restarted. The query indicates only 425/1,300M of the tempdb log file is available. I would never create a tempdb more than 100M in the first place (optimist), so these values may give you an idea how wildly it has grown.
I'm stuck where to go next. I'd like to know how/why our test system shows over 800M of the tempdb transaction log is used. I doubt it is, but how would do you know? My only guess why this happens is SQL Server gets confused with log pointers / device fragments and thinks no more space is available when space actually is available.
The following would be helpful changes for SQL Server:
backup/restore tempdb (so state can be captured then analyzed).
ability to explicitly drop local temporary tables (and table variables).
Has been 15 years, and no developer has figured out how to ensure the system drops them.
Provide DBAs the opportunity to maintain their system.