Slow network with large numbers of files.

[mangyDOG]

I have a client running a small network with five computers, all running Windows XP SP2. One of the computers is acting as a file server. The client PCs each have three network shares (X, Y, Z).

1) Accessing X, Y or Z from any computer works fine, with files and folders appearing close to instantly.

2) in the Y drive there is a folder called Jobs, which has over 6500 small files (total combined size is only about 430Mb). Accessing this folder results in the swinging flashlight and a significant delay of anywhere from 15 seconds to almost a minute. Reducing the number of files (not a permanent option) fixes the problem.

The problem always occurs no matter how the folder is accessed, eg using the mapped drive, typing the UNC address or browsing through My Network Places.

The local access to the files (on the server pc) is instantaneous.

CPU and network activity on the client PC during the delay are both 1% or less.

Once the files are displayed there is no delay in using any of the files, but closing or leaving the folder then going back to it results in another long delay.

No software firewalls are on and all antivirus programs have exclusions set for these folders (on the clients and the server). I have tested the problem with all antivirus software turned off completely.

Does anyone know how to fix this issue?

 

[timwhit]

Could you try creating some folders and spreading the files across the folders. Windows doesn't deal with a large number of files very well.

 

[ddrueding]

Maybe disabling the "last accessed" timestamp for the files?

 

[Bozo]

Disable Indexing Service??

Bozo :joker:

 

[mangyDOG]

Could you try creating some folders and spreading the files across the folders. Windows doesn't deal with a large number of files very well.

That is what I would normally do, but in this case the folder is used by a specific piece of software which throws a tantrum if any of the files it expects to be in the folder are missing :-(

We tried creating a range of subfolders but the error messages from the software are worse than the wait to open the folder.

cheers,
mangyDOG

 

[mangyDOG]

Maybe disabling the "last accessed" timestamp for the files?

Where do I find that setting?

Cheers,
mangyDOG

 

[mangyDOG]

Disable Indexing Service??

Tried that, on both, on the server and on the client. made no difference at all.

Seems to me that there is a threshold number of files in a folder, and once it is exceeded XP just cannot handle it. My feeling is there is a cache setting somewhere that can deal with a few hundred files to be listed but cannot cope with more. I may be wrong there and it is something else entirely.

Thanks
mangyDOG

 

[timwhit]

Do you think this is a client-side or a server-side problem?

If it's server side you could try a samba share.

I have a feeling it's client side though.

 

[mangyDOG]

I think it is client side. I found a few similar problems when googling for a fix, and some mentioned that the problem did not occur with Win2k or Win98 clients. If I get time I will put together a linux NAS and test it with that, but I don't want to tell my client they need to spend more money. :sad:
And no, it is not an option to go back to win98 :smile:

 

[timwhit]

Is this a problem if all the files are stored locally or is it only when you access it from a mapped network drive?

 

[ddrueding]

Where do I find that setting?

Cheers,
mangyDOG

Looks like it only affects directories and not files, but it is worth a try.

http://www.pctools.com/guides/registry/detail/50/

 

[mangyDOG]

The files on the server machine appear in the directory immediately without any delay. If the files are copied to the client machines they also work properly. But they need to be shared someware so all the computers are working with the same set of files.

cheers,
mangyDOG

 

[mangyDOG]

Maybe disabling the "last accessed" timestamp for the files?

Tried it and no joy :sad:

cheers,
mangyDOG

 

[LiamC]

Offhand, I'd say that the client is trying to build the icon cache, and it sounds like the default cache is too small. Not sure how to increase the default though...

Ohh, tabbed browsing and Google works wonders (Aside: I like Firefox 3 RC1)

http://articles.techrepublic.com.com/5100-10878_11-5164407.html

 

[Bozo]

Offhand, I'd say that the client is trying to build the icon cache, and it sounds like the default cache is too small. Not sure how to increase the default though...

Ohh, tabbed browsing and Google works wonders (Aside: I like Firefox 3 RC1)

http://articles.techrepublic.com.com/5100-10878_11-5164407.html

You can turn off 'cache thumbnails'...is that the same thing?

Bozo :joker:

 

[LiamC]

Could be, though according to this:

http://www.theeldergeek.com/thumbnail_cache.htm

it might make things worse rather than better. All I've been able to find so far indicates that Windows will always try to cache, or rebuild it on selected folders. Can't find what the "selected" criteria is though. It should be fairly quick to try both approaches though...

 

[Bozo]

You don't need to root around in the registry to change the setting. Open Windows Explorer. At the top click on Tools>Folder Options>View>check 'Do Not Cache Thumbnail'

Bozo :joker:

 

[Bozo]

Think this might help?

"How to turn off Windows auto-expand (and search) for compressed folders
Friday, September 10, 2004 9:36:46 PM UTC ( Windows Server 2003 )
I have a lot of .zip files floating around my computer and Windows XP/2003 includes them with regular folders when browsing in Explorer. Also, when you search for files, Windows searches inside .zip files which can take a long time because they need to be uncompressed every time to be searched.
I personally don't want this feature, and after a bit of looking around, i came across a simple solution. Just unregister the component that does the searching within compressed folders. Note: this doesn't stop you opening compressed folders, it just means they don't get included in the Windows Explorer tree-view, and they don't get included in searching with the Indexing Service.
Run the following command (Start > Run) or you can put it in the Run section of the Windows Registry in case Windows automatically adds back the zip dll.

regsvr32 /u C:\windows\system32\zipfldr.dll
If you go need to put it back again, then run:
regsvr32 C:\windows\system32\zipfldr.dll Many thanks to 'SJB' for his/her post on http://www.annoyances.org/exec/forum/winxp/1054284272"

Bozo :joker:

 

[mangyDOG]

Hi Bozo,

That isn't the problem, I have the auto-unzipping turned off on all the PCs I build and there arn't any zip files in the problem folder.
I really did think it might be the Icon cache causing the problem but a couple of hours of experimentation resulted in no improvment.

A couple of new pieces of information:

There are three PCs accessing the slow network folder (not all simutaneously), the oldest and slowest does not have the problem, it can open the folder without any lag. It has the same version of Windows (XP Home SP2) as one of the other computers, it has the same antivirus program and as far as I can tell exactly the same network settings.

I tried replacing the existing 100Mbit switch with a Gigabit one, which made no difference.

I put it a FreeNAS box, copied all the files to it - all three computers can access the NAS without LAG!!! So it must be something to do with the way the two newer computers are interacting with the Server PC but I just cannot figure out what.

cheers,
mangyDOG.

 

[BigRon]

Hi mangyDOG,

I stumbled on this old thread and having the same issue with Win7 clients and Win XP server. Did you finally find the cause of the problem?
Or never looked into it anymore because the NAS solved it?

Thanks,

BigRon

 

[Bozo]

Apparently MS limits file copying in Win 7 to 10% of the network speed. I have been searching for a fix, but no luck yet.

 

[Stereodude]

Apparently MS limits file copying in Win 7 to 10% of the network speed. I have been searching for a fix, but no luck yet.

I haven't seen that. My Windows 7 box with gigabit ethernet will do 125MB/sec.

 

[ddrueding]

Same here

 

[sdbardwick]

Apparently MS limits file copying in Win 7 to 10% of the network speed. I have been searching for a fix, but no luck yet.

I had a dodgy NIC report gigabit speeds to Windows when the physical link was 100...

 

[Bozo]

After much head scratching, I found that a 100Mb switch had been installed. Gb comming soon. :cursin:

 

[mangyDOG]

Hi mangyDOG,

I stumbled on this old thread and having the same issue with Win7 clients and Win XP server. Did you finally find the cause of the problem?
Or never looked into it anymore because the NAS solved it?

Thanks,

BigRon

Although the NAS fixed it this problem, it created a few more. In the end the problem was resolved when the client needed a new application using SQL databases so I sold him a Server 2008R2 machine and this also didn't have a problem with lag to the client machines. The particular folder in question now has over 20k of files and access is lag free from all computers on the network.

Cheers,
mangyDOG