Access to one of our large tables suddenly got slow.
We have a table T1 which contains millions of rows. For each row in that table, call an sp which looks up rows in table T2 and
then updates T1. T2 has billions of rows. The lookup in T2 is done via clustered index. Each individual lookup in T2 accesses only
small number of rows. There is a clustered index on T2 and the lookup is on the clustered index fields.
We have a table T1 which contains millions of rows. For each row in that table, call an sp which looks up rows in table T2 and
then updates T1. T2 has billions of rows. The lookup in T2 is done via clustered index. Each individual lookup in T2 accesses only
small number of rows. There is a clustered index on T2 and the lookup is on the clustered index fields.
Suddenly this procedure went from taking 1 hour to 2.5 hours. We didn't change the SQL code. There is no conceivable way SQL can be choosing wrong plans because there is a clustered index exactly on the fields that the query uses.
We are puzzled about how can the access get 2.5x slower. Any suggestions on what we should look?