LTO4 performance

Jeyendran

What is this storage?
Joined
Jul 3, 2009
Messages
7
Hi, WE are using HP LTO4 with ultra 320 SCSI interface in our facility, The LTO is difectly connected to a RAID machine which can deliver close to 600 MB/sec, When we try backing up or retriving content we are unable to get the desired throughput (close to theoretical performance). pls explaing where would I have gone wrong.

As per the spec the throughput should be 240MB/s but in reality we are getting very low throughput of 3 MB/s. we are on Linux cent os

http://h10010.www1.hp.com/wwpc/us/en/en/WF05a/12169-304612-3446236-3446236-3446236-3454484.html

thanks Jeyendran
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,269
Location
I am omnipresent
Is your setup cabled properly? Are you sure it's cabled properly? Are you REALLY, COMPLETELY SURE it's cabled properly?

What are you using for backup software? Is it supported by the drive or by HP?

Are there other things happening at the same time on your SCSI bus when you're running these backups? How are you getting 600MB/sec out of U320 SCSI?
 

Jeyendran

What is this storage?
Joined
Jul 3, 2009
Messages
7
Hi Thanks for you reply, The above mentioned machine is used for film playback, The machine has a RAID controller that has 16 SATA-II disk and the consolidated output will delver 600 MB/sec , We are using this machine for video play-back (it will stream 24 frames(files)/sec each of 13 MB). We tested this with a software call drastic preview (that can measure disk I/O).

We have installed a Ultra 320 SCSI card on the same machine. and we have connected the tape drive directly.

We are not using any backup software, we are just backing-up using tar -cvf /dev/nst0 command.

what do you exactly mean by connecting the cable properly. also let me know if there any method or software to test the connectivity and throughput of cable.
Thanks
Jeyendran
 

Fushigi

Storage Is My Life
Joined
Jan 23, 2002
Messages
2,890
Location
Illinois, USA
Absolutely the first thing to do is clean the tape drive. The drive should have come with a cleaning cartridge.

After that, the most common problem I've observed when trying to maximize tape throughput is that the machine being backed up cannot pump data fast enough. A 600MB/s disk subsystem is good, obviously, but if there is overhead involved then the throughput will drop. Off the top of my head, here are some things that induce overhead:
- Fragmentation of the files.
- Insufficient read caching on the RAID controllers.
- Host/server-based encryption.
- Compression.
- Lots of small files v. fewer but larger.
- Crappy backup software.
- Various combinations of the above.
Basically, a bottleneck occurs somewhere. That causes the tape to not only have to pause, but sometimes to have to rewind and reposition before writing can resume. This "see-saw" effect can take a few seconds and, over the course of an entire backup, can add significant time to the backup. LTO4 minimizes the see-saw effect by varying the write speed, but even then it is slowing down.

I would suggest defragmenting the file system if you can. Second, I would monitor memory & CPU consumption during the backup to see if the machine is hitting an internal bottleneck. On the CPU front, assuming a multi-core or multi-CPU environment, check to see if a single core/CPU is being hit hard while the others are idle; that would show a CPU bottleneck and non-multithreaded backup software. Also monitor the disk subsystem - how busy the disks are, how busy the RAID card controller is - to see what's going on there. Disk busy other 40% is bad & indicates thrashing.

Third, listen very closely to the tape drive during a backup so see if you can hear changes in the noise it makes. This will be tough as LTO4 is pretty quiet. If you hear variances, the tape drive is having to speed up/down/pause as it waits for data. That indicates a problem outside of the drive. Unless the media you have is of poor quality & you're getting read/write errors. Try a brand new tape. Also, make sure you're buying quality tapes. Some drives are finicky and like certain brands of tape better than others. Imation (3M), IBM, and Fuji seem to work well for us.

Oh, and verify you are using LTO4 media. LTO3 media will work but not at LTO4 speeds.

I would suggest disabling compression and encryption for your tests. If either of those solves your problem, and you're not using the LTO4's internal encryption capability, then you've found your bottleneck and it's most likely the server's CPU.

Another thought .. is the disk internal or is it actually a SAN or NAS?

And another .. is read-after-write verification on? That'll add significant time.

BTW, LTO4 should ideally be cabled via fibre channel & not SCSI. U320 in theory has adequate bandwidth for LTO4, but in practice there's not that much head room.
 

Fushigi

Storage Is My Life
Joined
Jan 23, 2002
Messages
2,890
Location
Illinois, USA
Thanks, paugie. I've done tape backups on midrange equipment since 1988 and have gone from the old "round reel" cabinet tape drives like this (1200-6250 bits per inch, 2400 foot per tape) through various cartridge tape systems like the 3480. In recent years that's included AIT and AIT2 and then LTO1/2/3/4. Right now I have one LTO3 and 2 LTO4 libraries with 2 drives each on the systems I manage. Basically we spend around $20K+ every 18-24 months upgrading some aspect of the tape environment. The LTO3 and one of the LTO4 libraries use SCSI; the newer LTO4 uses fibre.

Until recent times backup performance was a huge issue. The nightly & weekly windows were always under pressure to be minimized. Awhile back we changed to a save-while-active strategy that synchronizes all objects in the backup and snapshots their state, allowing for updates to occur while the backup is running.

So after all of these years I no longer have to care about backup performance. It also means we're up 24 hours a day without clustering. For part of the environment we've actually taken this to the next level. Since we have a High Availability setup (used for disaster recovery and to shift reporting demands off the main database), we're backing up the HA replica daily, the change journals on the main partition daily (but not the database itself) and only take the main system offline for backups once a month.

Restore performance is a different matter, of course. Restores can never be fast enough for the business. And that's where having fast drives and good tape management software comes into play.
 

Jeyendran

What is this storage?
Joined
Jul 3, 2009
Messages
7
Hi Fushigi

your solution and suggestions are very resourceful, It has given me lot of areas where I have to look for bottle neck, I will analyze the set up and inform you, thanks for your support and info. Jeyendran
 

Jeyendran

What is this storage?
Joined
Jul 3, 2009
Messages
7
Hi Fushigi, Thanks for the support. The problem was with the PCI-e slotht to which the SCSI card was connected. after changing the sloght. and matching the clock speed, the data were retrived faster. thanks once again. -Jeyendan
 
Top