Optimising for very small files

[ddrueding]

I have an unusual situation that I'm hoping for some tuning advice on.

I currently have a drive with 25,000,000 files averaging 16kb, and they need to be read a half-million at a time. 750GB of Intel SSDs isn't really an option, so I bought 3 300GB Velociraptors that I intend to run in RAID-0 (I will have a backup).

Having never really studied stripe or cluster sizes, I suspect this is an application that could use some special tuning. Any advice would be most appreciated.

 

[LunarMist]

Why use the consumerish WD stuff? Don't they make 15K SAS drives anymore?

 

[ddrueding]

I don't have that kind of money for this project. It will be going into my workstation, and I'll be using the on-board SATA controller.

 

[LunarMist]

Dare we ask what personal apps read a half-million files at a time?

 

[Mercutio]

That sounds like an awful lot of image thumbnails to me.

 

[LunarMist]

pr0n files?

 

[ddrueding]

I've gotten kinda hooked on soaring in MS' now cancelled Flight Simulator X. Unfortunately, the terrain sucks in that program.

A program called TileProxy takes a local cache of MS Virtual Earth files and replaces the default terrain in real time. The files cached from MS are 256x256x32 JPGs, and I have about 30 million of them now. It loads an area around your plane and tries to keep up as you fly around. In really high settings (30cm/pixel) and high flight speeds (300kts), it takes a whole lot to keep up. The initial load time can be 5 minutes on the 1TB drive it is on now.

 

[Mercutio]

I would think the main thing you have any operational control over would be NTFS cluster size. The files are 16kB apiece, there's a 16kB cluster size option. That would be the first thing I'd try. That certainly will be the most efficient use of disk space, but I'm not sure that it will lead to any increased performance; that's going to depend much more on the drive's internal operation.

 

[ddrueding]

Thanks Merc. Considering I will be going to RAID-0 (3x Velociraptors), I was also thinking about the stripe size. Any thoughts?

 

[P5-133XL]

I think this is a very good application for the use of an SSD. They are so small and so many, it is highly unlikely that they are read sequentially which means you are seek limited: raid-0 is unlikely to produce a large benefit and it also means the smaller the stripe size the better.

I agree with merc: 16KB clusters is likely to be optimium. I think that will produce some performance improvement, but not much. performance depends on how often the files are actually read sequencially and how often they have to be seeked to before being read.

 

[mubs]

Also, whatever the cluster size on the disk is, the OS reads/writes certain KB at a time (this is fixed), and if the two are different (disk and OS) you end up with overheads. I have forgotten what WinNT OSes use; getting old, forgetting things and not actually caring anymore.

 

[ddrueding]

Thanks all. It is very unlikely that these files will be read sequentially. Based on the design of the file structure, it is virtually guaranteed to be 100% random across the drive.

 

[P5-133XL]

Thanks all. It is very unlikely that these files will be read sequentially. Based on the design of the file structure, it is virtually guaranteed to be 100% random across the drive.

Thats why SSD: Super-fast random reads ...

 

[ddrueding]

If I were to get some SSDs for this, I take it the access pattern here would highlight the performance issues of the cheaper/larger MLC SSDs? So I would need the Intel ones, or something equally fast?

 

[P5-133XL]

If I were to get some SSDs for this, I take it the access pattern here would highlight the performance issues of the cheaper/larger MLC SSDs? So I would need the Intel ones, or something equally fast?

It depends on how often the SSD gets written to. If the application is still getting more of these files, then yes you will need an Intel drive to eliminate pauses. If it has become basicly read only then you can get away with a lower-end SSD. Here is a very good article on SSD performance characteristics

 

[timwhit]

If I were to get some SSDs for this, I take it the access pattern here would highlight the performance issues of the cheaper/larger MLC SSDs? So I would need the Intel ones, or something equally fast?

I think you could get away with something cheaper than the Intel drive. Something like the G.SKILL TITAN Series FM-25S2S-256GBT1 would probably work well. As long as you aren't trying to write to it random reads should be very fast.

 

[ddrueding]

Here is a very good article on SSD performance characteristics

Excellent (so far, I'm on page 8 ), recommended for anyone interested in SSDs.

 

[ddrueding]

I think you could get away with something cheaper than the Intel drive. Something like the G.SKILL TITAN Series FM-25S2S-256GBT1 would probably work well. As long as you aren't trying to write to it random reads should be very fast.

There are writes, but they are few and far between at this point. Not sure the impact it would have.

 

[timwhit]

There are writes, but they are few and far between at this point. Not sure the impact it would have.

I think your best bet would be to try. If it doesn't live up to your expectations, then I'm sure you could find another home for the drive.

 

[ddrueding]

Based on that article, the only drive currently on the market, other than Intel, that doesn't suck, is the OCZ Vertex. Unfortunately, I would need at least 3 of the 250GB models to do what I want, and that is prohibitively expensive. I'll try implementing the Velociraptors that I have sitting here (following the above recommendations) and see what performance is like.

Thanks!

 

[blakerwry]

I think you'll gain more performance from file system optimization here... disable last access, disable 8dot3 names, set cluster size to 16k.

I would think smaller stripe sizes would work better for your workload (random access of small files), this allows multiple spindles to be used to lower latency rather than increase STR. But you'll need to try it out to get a good idea.

For lowest latency, match your stripe size to your cluster size and align your file system to your RAID array.


*Since these files are exactly 16kb, I would also pay attention to how much space they take up on disk. There is a possibility that the files will need 16kb + another cluster due to FS overhead. If so, I would revert back to the default 4kb clusters to reduce overhead.

*If you are downloading textures of the whole earth, is there a possibility to download one region at a time, so as to localize the data stored on disk and reduce seek distance and improve read-ahead optimization?

 

[ddrueding]

Thanks for the tips blakerwry. File overhead is significant, but I don't really care (other than potential performance impacts). I'm not downloading the whole earth, just the bits I like to fly over (my location, and the Italian Alps). When it is downloading, it does so on-demand while flying, so it is location-specific.

 

[MaxBurn]

What about a mirror with a controller that can read data back from both drives at the same time like a 3Ware twin store feature? With so many small files I don't see where the throughput of RAID 0 helps here? Kind of an interesting problem with the presented bazillion tiny random files situation.

 

[ddrueding]

If I got one more Velociraptor I could do a RAID-10 with a 4-port 7000-series 3Ware I have sitting around. IIRC, it does twin store. 300GB wouldn't be enough.

 

[LunarMist]

What kind of crap game software requires several thousand dollars of HD systems to work properly? Crazy.

 

[timwhit]

What kind of crap game software requires several thousand dollars of HD systems to work properly? Crazy.

It works fine until he added an add-on to the game which uses better textures.

 

[ddrueding]

It'll work just fine on an $800 rig, but I want it to work better. Still cheaper than a PPL and a 172.

 

[MaxBurn]

Well he has some ground there, I was thinking myself that this could use some software optimization as others have already mentioned but you clearly don't have control over that.

The 7000 series controller is it native SATA, thought that was PATA?? I had a couple 6xxx and 7xxx a long time ago and I wasn't impressed with its speed, but then again I wasn't using 0+1, I was using RAID 5 and they have done a LOT of work on 5 and 6 speed. I am sort of thinking a modern on board ICH9R or ICH10R is going to be faster in 0+1 than a 7xxx series controller. Assuming you even had a 64bit slot for the thing on a modern motherboard.

 

[ddrueding]

Yeah, I'll probably stick to the onboard controller. I don't have the slots available anyway.

 

[Bozo]

The 7000 series controller is it native SATA, thought that was PATA?? I had a couple 6xxx and 7xxx a long time ago and I wasn't impressed with its speed, but then again I wasn't using 0+1, I was using RAID 5 and they have done a LOT of work on 5 and 6 speed. I am sort of thinking a modern on board ICH9R or ICH10R is going to be faster in 0+1 than a 7xxx series controller. Assuming you even had a 64bit slot for the thing on a modern motherboard.

The Escalde 7000 I have sitting here is PATA.

 

[ddrueding]

Ah, right you are. This is actually a 8500-4, which uses individual SATA-PATA bridge chips, but is otherwise a 7500. 64-bit PCI is pretty old, though. Perhaps Greg wants it?