I built this up and expanded it over the past two years without problems but this week I added two new drives. Same as ones I have added before. The manager showed them as available. Then suddenly the manager said the rack was not found, the drive was still accessible, then the MD started to just keep rebooting. I can ping the controllers but I keep getting network errors if I try any cli commands.
Here is the putty capture:
-=<###>=-
Attaching interface lo0... done
Adding 9766 symbols for standalone.
Error
10/13/17-16:37:18 (GMT) (tRootTask): NOTE: I2C transaction returned 0x0423fe00
Current date: 10/13/17 time: 06:02:52
Send <BREAK> for Service Interface or baud rate change
10/13/17-16:37:34 (GMT) (tRAID): NOTE: SOD Sequence is Normal, 0
10/13/17-16:37:35 (GMT) (tRAID): NOTE: Turning on tray summary fault LED
10/13/17-16:37:36 (GMT) (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
esmc0: LinkUp event
10/13/17-16:37:38 (GMT) (tNetCfgInit): NOTE: Network Ready
10/13/17-16:37:39 (GMT) (tRAID): NOTE: Initiating Drive channel: ioc:0 bringup
10/13/17-16:37:42 (GMT) (tRAID): NOTE: IOC Firmware Version: 00-24-63-00
10/13/17-16:37:48 (GMT) (IOSched): NOTE: New Initiator: 1 - channel: 1,devHandle: x1d, SAS Address: 5001c234cb788100
10/13/17-16:37:48 (GMT) (IOSched): NOTE: New Initiator: 2 - channel: 0,devHandle: x13, SAS Address: 5001c234cb788101
10/13/17-16:37:53 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:0 prevNumActivePhys:2 numActivePhys:2
10/13/17-16:37:53 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:0 phy:1 prevNumActivePhys:2 numActivePhys:2
10/13/17-16:37:53 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:2 prevNumActivePhys:2 numActivePhys:2
10/13/17-16:37:53 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:1 phy:3 prevNumActivePhys:2 numActivePhys:2
10/13/17-16:37:53 (GMT) (tSasCfg017): NOTE: Alt Controller path up - chan:0 phy:18 itn:1
10/13/17-16:37:53 (GMT) (tSasCfg019): NOTE: Alt Controller path up - chan:1 phy:16 itn:2
10/13/17-16:37:54 (GMT) (tRAID): NOTE: IonMgr: Drive Interface Enabled
10/13/17-16:37:54 (GMT) (tRAID): NOTE: SOD: Instantiation Phase Complete
10/13/17-16:37:54 (GMT) (tRAID): NOTE: Inter-Controller Communication Channels Opened
10/13/17-16:37:54 (GMT) (tSasDiscCom): NOTE: SAS Discovery complete task spawned
10/13/17-16:37:54 (GMT) (tRAID): NOTE: LockMgr Role is Slave
10/13/17-16:37:54 (GMT) (sasCheckExpanderSet): NOTE: Expander Firmware Version: 0116-e05c
10/13/17-16:37:55 (GMT) (sasCheckExpanderSet): NOTE: Expander SAS address: Hi = x5001c234 Low = xcb79f110
10/13/17-16:37:55 (GMT) (tRAID): NOTE: spmEarlyData: No data available
10/13/17-16:38:05 (GMT) (tHckReset): NOTE: LockMgr Role is Master
10/13/17-16:38:05 (GMT) (tSasEvtWkr): NOTE: sasLinkDown: expDevHandle:x9 - WidePort: #1 - Alt Controller, chan:0 encl:0, subExp:255, OPTIMAL to DEGRADED
10/13/17-16:38:05 (GMT) (tSasEvtWkr): NOTE: sasLinkDown: expDevHandle:x9 - WidePort: #1 - Alt Controller, chan:0 encl:0, subExp:255, DEGRADED to FAILED
10/13/17-16:38:05 (GMT) (tSasEvtWkr): NOTE: sasLinkDown: expDevHandle:xa - WidePort: #0 - Alt Controller, chan:1 encl:0, subExp:255, OPTIMAL to DEGRADED
10/13/17-16:38:05 (GMT) (tSasEvtWkr): NOTE: sasLinkDown: expDevHandle:xa - WidePort: #0 - Alt Controller, chan:1 encl:0, subExp:255, DEGRADED to FAILED
10/13/17-16:38:05 (GMT) (IOSched): NOTE: Disconnect Initiator 2 - channel: 0, devHandle: x13, I/O: 0, SAS Address: 5001c234cb788101
10/13/17-16:38:05 (GMT) (IOSched): WARN: sasRemoveInitiator - No I/O's - send EventAck: event: 00000018, eventContext: 00020013
10/13/17-16:38:05 (GMT) (sasTgtSendEventAck): WARN: sasTgtSendEventAckTask - SendEventAck for host: 2
10/13/17-16:38:05 (GMT) (IOSched): NOTE: Disconnect Initiator 1 - channel: 1, devHandle: x1d, I/O: 0, SAS Address: 5001c234cb788100
10/13/17-16:38:05 (GMT) (IOSched): WARN: sasRemoveInitiator - No I/O's - send EventAck: event: 00000018, eventContext: 0002001d
10/13/17-16:38:05 (GMT) (sasTgtSendEventAck): WARN: sasTgtSendEventAckTask - SendEventAck for host: 1
10/13/17-16:38:07 (GMT) (tSasDiscCom): WARN: SAS: Initial Discovery Complete Time: 30 seconds
10/13/17-16:38:07 (GMT) (tRAID): NOTE: WWN baseName 0004001c-23cb79f1 (valid==>SoftRst)
10/13/17-16:38:07 (GMT) (tRAID): NOTE: sasEnableInterface: Enabling Host Interface for channel 2
10/13/17-16:38:07 (GMT) (tRAID): NOTE: IonMgr: Host Interface Enabled
10/13/17-16:38:07 (GMT) (tRAID): NOTE: SOD: Pre-Initialization Phase Complete
10/13/17-16:38:10 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:2 phy:6 prevNumActivePhys:0 numActivePhys:1
10/13/17-16:38:10 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: IOCPort: chan:2, FAILED to DEGRADED
10/13/17-16:38:10 (GMT) (tSasEvtWkr): WARN: sasIocPhyUp: Initializing Channel 2: Attached SAS Address: 514feb51fe9f6804
10/13/17-16:38:11 (GMT) (IOSched): NOTE: New Initiator: 3 - channel: 2,devHandle: x1f, SAS Address: 514feb51fe9f6804
10/13/17-16:38:12 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:2 phy:7 prevNumActivePhys:1 numActivePhys:2
10/13/17-16:38:22 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:2 phy:4 prevNumActivePhys:2 numActivePhys:3
10/13/17-16:38:22 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: chan:2 phy:5 prevNumActivePhys:3 numActivePhys:4
10/13/17-16:38:22 (GMT) (tSasEvtWkr): NOTE: sasIocPhyUp: IOCPort: chan:2, DEGRADED to OPTIMAL
10/13/17-16:38:25 (GMT) (tSAOGet): NOTE: Forcing getMeSAO due to GET_SAO_RESULTS_FAILED_GET chan:0 encl:0 slot:00 saoReq:0xa00000a Attempt:0
10/13/17-16:38:25 (GMT) (tSAOGet): NOTE: Forcing getMeSAO due to GET_SAO_RESULTS_FAILED_GET chan:0 encl:0 slot:01 saoReq:0xa00000c Attempt:0
10/13/17-16:38:25 (GMT) (tSAOGet): NOTE: Forcing getMeSAO due to GET_SAO_RESULTS_FAILED_GET chan:0 encl:0 slot:02 saoReq:0xa00000e Attempt:0
10/13/17-16:38:27 (GMT) (tSAOGet): NOTE: Forcing getMeSAO due to GET_SAO_RESULTS_FAILED_GET chan:0 encl:0 slot:03 saoReq:0xa000010 Attempt:0
10/13/17-16:38:27 (GMT) (tSAOGet): NOTE: Forcing getMeSAO due to GET_SAO_RESULTS_FAILED_GET chan:0 encl:0 slot:04 saoReq:0xa000012 Attempt:0
10/13/17-16:38:27 (GMT) (tSAOGet): NOTE: Forcing getMeSAO due to GET_SAO_RESULTS_FAILED_GET chan:0 encl:0 slot:05 saoReq:0xa000014 Attempt:0
10/13/17-16:38:29 (GMT) (tSAOGet): NOTE: Forcing getMeSAO due to GET_SAO_RESULTS_FAILED_GET chan:0 encl:0 slot:07 saoReq:0xa000016 Attempt:0
10/13/17-16:38:32 (GMT) (tRAID): PANIC: SubRecordStartOfDay::versionConvert exception caught DbmVersionException: recType: 19, Bad Version: 2