HI, we have dual controller system with md3200i and one enclosure…
Soon I find in MDSM that one of controllers is missed or removed, just empty space in manager, but controller is in storage system, and I can ping it.
We use VMware, and multipathing driver for failover switched ok, and nobody understand that controller is down...
After piece of bit searching I find how to connect to service console with PS2 serial cable… and I look that
Send <BREAK> for Service Interface or baud rate change
05/03/17-07:27:49 (tRAID): NOTE: Set Powerup State
05/03/17-07:27:49 (tRAID): NOTE: SOD Sequence is Normal, 0
05/03/17-07:27:50 (tRAID): NOTE: Turning on tray summary fault LED
05/03/17-07:27:50 (tRAID): NOTE: SODRebootLoop- Limit:5 Cnt:6
05/03/17-07:27:50 (tRAID): NOTE: Flashing tray summary fault LED
05/03/17-07:27:50 (tRAID): WARN: Controller entering SODReboot lockdown state...
I search again for solution, and find how to run command “lemClearLockdown”, and run it…
After that, controller can be viewed in failed state in MDSM, I’m happy now I can see it and switch it back in online mode, but my happy is short.
Now on service console, I see that:
-=<###>=-
Instantiating /ram as rawFs, device = 0x1
Formatting /ram for DOSFS
Instantiating /ram as rawFs, device = 0x1
Formatting...Retrieved old volume params with %38 confidence:
Volume Parameters: FAT type: FAT32, sectors per cluster 0
0 FAT copies, 0 clusters, 0 sectors per FAT
Sectors reserved 0, hidden 0, FAT sectors 0
Root dir entries 0, sysId (null) , serial number 10000
Label:" " ...
Disk with 1024 sectors of 512 bytes will be formatted with:
Volume Parameters: FAT type: FAT12, sectors per cluster 1
2 FAT copies, 1010 clusters, 3 sectors per FAT
Sectors reserved 1, hidden 0, FAT sectors 6
Root dir entries 112, sysId VXDOS12 , serial number 10000
Label:" " ...
RTC Error: Real-time clock device is not working
OK.
Adding 14630 symbols for standalone.
Reset, Power-Up Diagnostics - Loop 1 of 1
3600 Processor DRAM
01 Data lines Passed
02 Address lines Passed
3300 NVSRAM
01 Data lines Passed
4410 Ethernet 82574 1
01 Register read Passed
02 Register address lines Passed
6D40 Bobcat
02 Flash Test Passed
3700 PLB SRAM
01 Data lines Passed
02 Address lines Passed
7000 SE iSCSI BE2 1
01 Register Read Test Passed
02 Register Address Lines Test Passed
03 Register Data Lines Test Passed
3900 Real-Time Clock
01 RT Clock Tick Passed
Diagnostic Manager exited normally.
eth0: LinkUp event
05/03/17-08:20:14 (tNetCfgInit): NOTE: Network Ready
Current date: 05/03/17 time: 14:35:59
Send <BREAK> for Service Interface or baud rate change
05/03/17-08:20:22 (tRAID): NOTE: SOD Sequence is Normal, 0
05/03/17-08:20:22 (tRAID): NOTE: Turning on tray summary fault LED
05/03/17-08:20:22 (tRAID): NOTE: SODRebootLoop- Limit:5 Cnt:1
05/03/17-08:20:22 (tRAID): NOTE: Installed Protocols: <MTPs: INIC SAS USB > <ITPs: RDMA ISER IPOIB > <STPs: FCP RmtDMA iSCSI SAS >
05/03/17-08:20:22 (tRAID): NOTE: Required Protocols: <MTPs: INIC SAS > <ITPs: UNK > <STPs: iSCSI SAS >
05/03/17-08:20:22 (tRAID): NOTE: loading flash file: iSCSI
05/03/17-08:20:23 (tRAID): NOTE: soc is a stub
05/03/17-08:20:24 (tRAID): NOTE: SYMBOL: SYMbolAPI registered.
05/03/17-08:20:25 (tRAID): WARN: RCBBitmapManager total RPA size = 1828716544
05/03/17-08:20:25 (tRAID): NOTE: fcn is a stub
05/03/17-08:20:25 (tRAID): NOTE: In iscsiIOQLIscsiInitDq. iscsiIoFstrBase = 0x5
05/03/17-08:20:25 (tRAID): NOTE: qlg40xx is a stub
05/03/17-08:20:25 (tRAID): NOTE: loading flash file: MTLSEBET1G
MTLSE Driver version 2.300.310.144 built on Jun 13 2011 : 17:47:51
05/03/17-08:20:26 (tRAID): NOTE: mtlsebeDqInit
*** Disabled FULL TRACE ***
05/03/17-08:20:26 (tRAID): NOTE: mtlsebeDqInit return
Found 4 BE devices on this controller
05/03/17-08:20:26 (tRAID): NOTE: MTL sgl_pool pool 2080 objects created
05/03/17-08:20:26 (tRAID): NOTE: mtlsebe::Mtl::supportsDeviceId 0x21019a2 - TRUE
05/03/17-08:20:26 (tRAID): NOTE: mtlsebe::Mtl::supportsDeviceId 0x21019a2 - TRUE
05/03/17-08:20:26 (tRAID): NOTE: mtlsebe::Mtl::supportsDeviceId 0x21019a2 - TRUE
05/03/17-08:20:26 (tRAID): NOTE: mtlsebe::Mtl::supportsDeviceId 0x21019a2 - TRUE
05/03/17-08:20:26 (tRAID): NOTE: mtlsebe::Mtl::supportsDeviceId 0x21019a2 - TRUE
05/03/17-08:20:26 (tRAID): NOTE: mtlsebe::Mtl::supportsDeviceId 0x21019a2 - TRUE
05/03/17-08:20:26 (tRAID): NOTE: mtlsebe::Mtl::supportsDeviceId 0x21019a2 - TRUE
05/03/17-08:20:26 (tRAID): NOTE: mtlsebe::Mtl::supportsDeviceId 0x21019a2 - TRUE
05/03/17-08:20:26 (tRAID): WARN: init: ioc: 0, PLVersion: 06-000-05-00
05/03/17-08:20:26 (tRAID): NOTE: NumberOfCmndBuffs: 1024, NumberOfMsgFrames: 1950, NumberOfInitiators: 64, NumberOfTargets: 480
05/03/17-08:20:26 (tRAID): NOTE: NumberOfIocs: 1, NumberOfEnclosures: 40, NumberOfEnclosurePhys: 38, NumberOfRouteTable: 1024
05/03/17-08:20:26 (tRAID): NOTE: NumberOfExpanders: 40
05/03/17-08:20:26 (tRAID): WARN: Allocated for PL: DDR: 153842, CRAM: 89616
05/03/17-08:20:26 (tRAID): WARN: fast: 89616, DMAAble: 215506
05/03/17-08:20:26 (tRAID): WARN: CRAM allocated: HW Structs: 0, Total: 89616
05/03/17-08:20:26 (tRAID): NOTE: iopiConfig: set LED mode A
05/03/17-08:20:26 (tRAID): NOTE: iopiConfig: set LED mode A
05/03/17-08:20:26 (tRAID): NOTE: iopiConfig: set LED mode A
05/03/17-08:20:26 (tSasPlTmr): NOTE: sas::PLITimerMgr::process run, semaphore=47e8058
05/03/17-08:20:26 (tRAID): NOTE: eel::VirtualDriveEvent allocating 2048 objects
05/03/17-08:20:27 (tRAID): NOTE: SOD: Instantiation Phase Complete
05/03/17-08:20:27 (tSasPlTmr): NOTE: iopiConfig: set LED mode A
05/03/17-08:20:27 (IOSched): NOTE: SAS Expander Added: expDevHandle:x11 enclHandle:x2 numPhys:25 port:2 ioc:0 channel:0
05/03/17-08:20:27 (IOSched): NOTE: SAS Expander Added: expDevHandle:x12 enclHandle:x3 numPhys:25 port:3 ioc:0 channel:1
05/03/17-08:20:27 (IOSched): NOTE: SAS Expander Added: expDevHandle:x21 enclHandle:x4 numPhys:38 port:2 ioc:0 channel:0
05/03/17-08:20:27 (IOSched): NOTE: SAS Expander Added: expDevHandle:x22 enclHandle:x5 numPhys:38 port:3 ioc:0 channel:1
05/03/17-08:20:28 (tSasEvtWkr): NOTE: Alt controller path up on channel:0 devH:x20 expDevH:x11 phy:16 itn:2
05/03/17-08:20:28 (tSasExpChk): NOTE: Local Expander Firmware Version: 25.03.54.00
05/03/17-08:20:28 (tRAID): NOTE: Inter-Controller Communication Channels Opened
05/03/17-08:20:28 (tSasEvtWkr): NOTE: Alt controller path up on channel:1 devH:x1f expDevH:x12 phy:12 itn:16
05/03/17-08:20:28 (tSasExpChk): NOTE: Alternate Expander Firmware Version: 25.03.54.00
05/03/17-08:20:29 (IOSched): NOTE: discoveredEncl: trayId:1 slotCount:12 eli:500c04f21726fe00
05/03/17-08:20:29 (IOSched): NOTE: discoveredEncl: trayId:1 slotCount:12 eli:500c04f21726fe00
05/03/17-08:20:30 (IOSched): NOTE: New Initiator: channel: 1, devHandle: x1f, sasAddress: 5782bcb43dbcdf08
05/03/17-08:20:30 (IOSched): NOTE: New Initiator: channel: 0, devHandle: x20, sasAddress: 5782bcb43dbcdf0c
05/03/17-08:20:30 (tSasInitWkr): NOTE: Adding partner controller drive side initiator, chan:1 devH:x1f SAS Address:5782bcb43dbcdf08
05/03/17-08:20:30 (tSasInitWkr): NOTE: New Initiator: 1 - channel:1, devHandle:x1f, SAS Address:5782bcb43dbcdf08
05/03/17-08:20:30 (tSasInitWkr): NOTE: Adding partner controller drive side initiator, chan:0 devH:x20 SAS Address:5782bcb43dbcdf0c
05/03/17-08:20:30 (tSasInitWkr): NOTE: New Initiator: 2 - channel:0, devHandle:x20, SAS Address:5782bcb43dbcdf0c
05/03/17-08:20:31 (IOSched): NOTE: SAS: Opening ICON channel to alt controller on alt channel
05/03/17-08:20:32 (tRAID): NOTE: LockMgr Role is Slave
05/03/17-08:20:35 (tSasDiscCom): WARN: SAS: Initial Discovery Complete Time: 31 seconds since last power on/reset, 10 seconds since sas instantiated
05/03/17-08:20:40 (tRAID): NOTE: WWN baseName 0004d4ae-527cff77 (valid==>SoftRst)
05/03/17-08:20:40 (tRAID): NOTE: spmEarlyData: Using data from alternate
ServerEngines BladeEngine 2 Firmware Update Utility v2.0.310.144
Copyright 2009 ServerEngines CorpoFound BE card 0x0 at 2.0 bar1-base 0xf0200000 bar2-base 0xf0220000 dev 0x2df2898
flash_rbver = 2.0.38.0
flash_armfwver = 2.300.310.148
Bad File CRC after decompression
ERROR: VKI_READ failed while reading BE_FW from flash
ufi_rbver = 2.0.0.0
ERROR: VKI_READ failed while reading BE_FW from flash
ufi_armfwver = None
Flash on BladeEngine at 02 : 00 does not match UFI SEBE2T1G
05/03/17-08:20:42 (tRAID): WARN: MTLSE: Flashing BE Firmware at 2:0
05/03/17-08:20:42 (tRAID): NOTE: Delaying NMI watchdog during flash...
05/03/17-08:20:42 (tRAID): WARN: MTLSE: Updating BE flash at 2:0tThis may take few minutes...
Bad File CRC after decompression
ERROR: VKI_READ failed while reading BE_FW from flash
No signature found
05/03/17-08:20:43 (tRAID): ERROR: MTLSE: Flash of 2:0 FAILED
05/03/17-08:20:43 (tRAID): PANIC: Flash update failed at 2:0
Stack Trace for tRAID:
0x0025f8a4 vxTaskEntry +0x5c : sodMain (0, 0)
0x05cf6e60 sodMain +0x240: sodPreInitialization() ()
0x05cf591c sodPreInitialization()+0x1c : sodEnableHostInterface() ()
0x05cf5694 sodEnableHostInterface()+0x34 : sodLogStartup(void (*)()) ()
0x05cf4bb0 sodLogStartup(void (*)())+0xd0 : 0x064c72e4 ()
0x064c7314 ioni::enableHostInterfaces()+0x34 : ioni::IoniManager::enableHostInterfaceChannels() ()
0x064b3a44 ioni::IoniManager::enableHostInterfaceChannels()+0x1a4: ioni::ChannelData::enableInterface() ()
0x064c1b10 ioni::ChannelData::enableInterface()+0x50 : 0x0202bfe4 ()
0x0202c07c b_isn::IscsiNetworkManager::enableInterface(const ioni::ChannelData *)+0x9c : mtlsebe::Mtl::enableMtlInterface(unsigned long, unsigned int, unsigned int, unsigned long long, int) ()
0x02d7fda0 mtlsebe::Mtl::enableMtlInterface(unsigned long, unsigned int, unsigned int, unsigned long long, int)+0x160: update_be(unsigned long, bool) ()
0x02d762b4 update_be(unsigned long, bool)+0x1f4: _vkiCmnErr__link ()
0x0015a7c4 _vkiCmnErr +0x104: 0x0015aa00 (0x5a6e6c, 0x2de4888, 0x1dcd810)
0x0015adc8 vkiLogShow +0x588: psvJobAdd (0x15b840, 0x1dcddc8, 0, 0)
0x00141f44 psvJobAdd +0x64 : msgQSend ()
0x00404cfc msgQSend +0x61c: taskUnlock ()
05/03/17-08:20:44 (IOSched): WARN: channel: 0: Could not get connection
05/03/17-08:20:44 (IOSched): NOTE: IDR: devH:x20 expDevH:x11 IOCStatus:x8000 IOCLogInfo:x3112010c ioc:0 sasAddr:5782bcb43dbcdf0c
05/03/17-08:20:44 (IOSched): WARN: channel: 0: Could not get connection
-=<###>=-
And controller is in LOCK state again… I search for that error
“Bad File CRC after decompression
ERROR: VKI_READ failed while reading BE_FW from flash
ufi_rbver = 2.0.0.0
ERROR: VKI_READ failed while reading BE_FW from flash
ufi_armfwver = None”
but can’t find any solution…
Unfortunately support is ended long time ago, and I’m looking for help…
I miss some information, both controllers are with package version: 07.75.28.60
Regards and every help is appreciated.