Slow network with large numbers of files.

mangyDOG

Learning Storage Performance
Joined
Feb 15, 2003
Messages
161
Location
Ballarat, Vic, Aust.
I have a client running a small network with five computers, all running Windows XP SP2. One of the computers is acting as a file server. The client PCs each have three network shares (X, Y, Z).

1) Accessing X, Y or Z from any computer works fine, with files and folders appearing close to instantly.

2) in the Y drive there is a folder called Jobs, which has over 6500 small files (total combined size is only about 430Mb). Accessing this folder results in the swinging flashlight and a significant delay of anywhere from 15 seconds to almost a minute. Reducing the number of files (not a permanent option) fixes the problem.

The problem always occurs no matter how the folder is accessed, eg using the mapped drive, typing the UNC address or browsing through My Network Places.

The local access to the files (on the server pc) is instantaneous.

CPU and network activity on the client PC during the delay are both 1% or less.

Once the files are displayed there is no delay in using any of the files, but closing or leaving the folder then going back to it results in another long delay.

No software firewalls are on and all antivirus programs have exclusions set for these folders (on the clients and the server). I have tested the problem with all antivirus software turned off completely.

Does anyone know how to fix this issue?
 

timwhit

Hairy Aussie
Joined
Jan 23, 2002
Messages
5,278
Location
Chicago, IL
Could you try creating some folders and spreading the files across the folders. Windows doesn't deal with a large number of files very well.
 

mangyDOG

Learning Storage Performance
Joined
Feb 15, 2003
Messages
161
Location
Ballarat, Vic, Aust.
Could you try creating some folders and spreading the files across the folders. Windows doesn't deal with a large number of files very well.

That is what I would normally do, but in this case the folder is used by a specific piece of software which throws a tantrum if any of the files it expects to be in the folder are missing :-(

We tried creating a range of subfolders but the error messages from the software are worse than the wait to open the folder.

cheers,
mangyDOG
 

mangyDOG

Learning Storage Performance
Joined
Feb 15, 2003
Messages
161
Location
Ballarat, Vic, Aust.
Disable Indexing Service??

Tried that, on both, on the server and on the client. made no difference at all.

Seems to me that there is a threshold number of files in a folder, and once it is exceeded XP just cannot handle it. My feeling is there is a cache setting somewhere that can deal with a few hundred files to be listed but cannot cope with more. I may be wrong there and it is something else entirely.

Thanks
mangyDOG
 

timwhit

Hairy Aussie
Joined
Jan 23, 2002
Messages
5,278
Location
Chicago, IL
Do you think this is a client-side or a server-side problem?

If it's server side you could try a samba share.

I have a feeling it's client side though.
 

mangyDOG

Learning Storage Performance
Joined
Feb 15, 2003
Messages
161
Location
Ballarat, Vic, Aust.
I think it is client side. I found a few similar problems when googling for a fix, and some mentioned that the problem did not occur with Win2k or Win98 clients. If I get time I will put together a linux NAS and test it with that, but I don't want to tell my client they need to spend more money. :sad:
And no, it is not an option to go back to win98 :smile:
 

timwhit

Hairy Aussie
Joined
Jan 23, 2002
Messages
5,278
Location
Chicago, IL
Is this a problem if all the files are stored locally or is it only when you access it from a mapped network drive?
 

mangyDOG

Learning Storage Performance
Joined
Feb 15, 2003
Messages
161
Location
Ballarat, Vic, Aust.
The files on the server machine appear in the directory immediately without any delay. If the files are copied to the client machines they also work properly. But they need to be shared someware so all the computers are working with the same set of files.

cheers,
mangyDOG
 

Bozo

Storage? I am Storage!
Joined
Feb 12, 2002
Messages
4,396
Location
Twilight Zone

LiamC

Storage Is My Life
Joined
Feb 7, 2002
Messages
2,016
Location
Canberra
Could be, though according to this:

http://www.theeldergeek.com/thumbnail_cache.htm

it might make things worse rather than better. All I've been able to find so far indicates that Windows will always try to cache, or rebuild it on selected folders. Can't find what the "selected" criteria is though. It should be fairly quick to try both approaches though...
 

Bozo

Storage? I am Storage!
Joined
Feb 12, 2002
Messages
4,396
Location
Twilight Zone
You don't need to root around in the registry to change the setting. Open Windows Explorer. At the top click on Tools>Folder Options>View>check 'Do Not Cache Thumbnail'

Bozo :joker:
 

Bozo

Storage? I am Storage!
Joined
Feb 12, 2002
Messages
4,396
Location
Twilight Zone
Think this might help?

"How to turn off Windows auto-expand (and search) for compressed folders
Friday, September 10, 2004 9:36:46 PM UTC ( Windows Server 2003 )
I have a lot of .zip files floating around my computer and Windows XP/2003 includes them with regular folders when browsing in Explorer. Also, when you search for files, Windows searches inside .zip files which can take a long time because they need to be uncompressed every time to be searched.
I personally don't want this feature, and after a bit of looking around, i came across a simple solution. Just unregister the component that does the searching within compressed folders. Note: this doesn't stop you opening compressed folders, it just means they don't get included in the Windows Explorer tree-view, and they don't get included in searching with the Indexing Service.
Run the following command (Start > Run) or you can put it in the Run section of the Windows Registry in case Windows automatically adds back the zip dll.

regsvr32 /u C:\windows\system32\zipfldr.dll
If you go need to put it back again, then run:
regsvr32 C:\windows\system32\zipfldr.dll Many thanks to 'SJB' for his/her post on http://www.annoyances.org/exec/forum/winxp/1054284272"

Bozo :joker:
 

mangyDOG

Learning Storage Performance
Joined
Feb 15, 2003
Messages
161
Location
Ballarat, Vic, Aust.
Hi Bozo,

That isn't the problem, I have the auto-unzipping turned off on all the PCs I build and there arn't any zip files in the problem folder.
I really did think it might be the Icon cache causing the problem but a couple of hours of experimentation resulted in no improvment.

A couple of new pieces of information:

There are three PCs accessing the slow network folder (not all simutaneously), the oldest and slowest does not have the problem, it can open the folder without any lag. It has the same version of Windows (XP Home SP2) as one of the other computers, it has the same antivirus program and as far as I can tell exactly the same network settings.

I tried replacing the existing 100Mbit switch with a Gigabit one, which made no difference.

I put it a FreeNAS box, copied all the files to it - all three computers can access the NAS without LAG!!! So it must be something to do with the way the two newer computers are interacting with the Server PC but I just cannot figure out what.

cheers,
mangyDOG.
 

BigRon

What is this storage?
Joined
Apr 12, 2012
Messages
1
Hi mangyDOG,

I stumbled on this old thread and having the same issue with Win7 clients and Win XP server. Did you finally find the cause of the problem?
Or never looked into it anymore because the NAS solved it?

Thanks,

BigRon
 

Bozo

Storage? I am Storage!
Joined
Feb 12, 2002
Messages
4,396
Location
Twilight Zone
Apparently MS limits file copying in Win 7 to 10% of the network speed. I have been searching for a fix, but no luck yet.
 

Bozo

Storage? I am Storage!
Joined
Feb 12, 2002
Messages
4,396
Location
Twilight Zone
After much head scratching, I found that a 100Mb switch had been installed. Gb comming soon. :cursin:
 

mangyDOG

Learning Storage Performance
Joined
Feb 15, 2003
Messages
161
Location
Ballarat, Vic, Aust.
Hi mangyDOG,

I stumbled on this old thread and having the same issue with Win7 clients and Win XP server. Did you finally find the cause of the problem?
Or never looked into it anymore because the NAS solved it?

Thanks,

BigRon

Although the NAS fixed it this problem, it created a few more. In the end the problem was resolved when the client needed a new application using SQL databases so I sold him a Server 2008R2 machine and this also didn't have a problem with lag to the client machines. The particular folder in question now has over 20k of files and access is lag free from all computers on the network. :)

Cheers,
mangyDOG
 
Top