Monday, July 20, 2009
Database Mirroring Error on A cluster
Ran into a strange issue today on my cluster. We currently have a 3 node cluster in production which has been pieced together overtime with nodes going off of lease overtime and being replaced by new nodes. We have been moving more towards Database Mirroring over the past year and today when I came in the principal database in the cluster was in recovery and I couldn't do anything with it. After researching I could find anything that resembled my issue. I tried the standard fix which is breaking the mirror by running ALTER DATABASE
SET PARTNER OFF, no dice. After looking around I noticed something strange. I knew they had done maintenance on Sunday which involved rebooting the boxes and when they did they left the SQL Cluster groups running on different nodes. This led me to the root of the issue. When SQL was installed on the newest node for whatever reason this instance was not patched to Service Pack 2 as it should have been. Therefore, my database when running on the proper node has the patched binaries of Service Pack 2 and database mirroring works fine. However, on the node without a service pack applied to the binaries Mirroring doesn't run unless you set the trace flag to allow it to. Once I failed the instance back to the proper node where the Service Pack has been applied no issue, business as usual. I just wanted to post this in case someone runs across a similar issue in a clustered environment. I am going to be applying Service Pack 3 to the entire cluster next week so that should prevent this from happening in the future.