The performance of my TL4000 tape library has recently decreased dramatically and I'm at a bit of a loss trying to determine why. I have been using this tape library with VEEAM 8 running on 2012 R2 for over a year now and sometime within the last month or so, performance has gone in the tank.
It was running pretty much as a set it and forget it system, but the other day I wandered by and heard the autoloader active in the middle of the day. Thinking that was odd, I decided to have a look at the backup logs and where I used to regularly see anywhere from 40-70MBps throughput to the unit (single LTO4 drive), I am now seeing a max of 14.5MBps.
Here are the things I've already done to try to troubleshoot. If anyone can offer additional insight, I'm all ears. None of the following had any meaningful impact.
Verified MS iSCSI initiator config, then removed and recreated it. Tried single path and MPIO. No affect
Tried intitator interfaces on the same subnet (normal config, L2 switched) as well as routed. No affect
Tested with IBM Tape Diagnostic Tool, same throughput
Set up iSCSI on a different machine, Tested with IBM Tape diagnostic, same result
Set up iSCSI initiator on a laptop, connected directly to iSCSI bridge, left port. Same result
Completed a full powerdown reboot of the entire backup infrastructure, blew away and reconfigured iSCSI-SAS bridge, same result.
Reinstalled latest recommended Dell drivers, no affect
Captured iSCSI traffic flow with wireshark, no dropped packets or pause frames. Only suspicious thing I notice are occasional TURs in the iSCSI command stream. I was under the impression that TUR issues were a thing gone since the days of 2008R2, could they have been recently reintroduced?
I'm just about at wits end with it. With backups going this slow, the next batch of actual full backups is going to take a lot longer than the allowed window to complete, not to mention the horriffic amount of shoe shining the drive head is being forced to endure. I can't imagine it'll last very long if this isn't resolved quickly.
Library is running D.10, Drive is LTO4 HH SAS v2 running F991, Bridge is running v1.02.04. Hoping there is something silly that I'm overlooking.
Thanks!
-Dave