Corrupt files after network transfer

[Handruin]

I'm finding sporadic files that are corrupt after transferring them over the network from my other computer. They've all been photos (CR2 canon raw). I'm transferring them in large quantities from my winXP machine over to my new PC (Win 7) via GigE switch. There is no reported error in the transfer process, but after I begin processing to jpg windows occasionaly finds and complains that one or two are corrupt in a give set. So far about a total of 8-10 files have had this issue in 1000+ I've done. When I recopy the corrupted file from my old machine, all is fine and I'm able to process it.

I've run memtest-x86 for a few hours on the new system and done a complete hard drive sector scan and there have been no apparent problems with the system. Stability has been fine so far besides this issue.

Could I have a problem with my network card driver? Windows 7 seem to have all the 'right' drivers from initial install. I didn't update any of them with the ones on the motherboard CD ROM. Could this be an issue? I was actually quite surprised that Windows 7 found all my new hardware and had no items in device manager as undefined or without a proper driver.

 

[Stereodude]

Do you have a 3rd computer you can use to see if it's the old or new system?

Also, I'd check to see if you've got jumbo frames turned on only one of the two system. (I'd suggest you keep them off)

 

[Handruin]

I do have a 3rd system (my laptop) and I've never had this corruption issue when processing on my laptop. The files that I processed were done the same way in that I've copied them first from the XP machine.

I just noticed a new corruption in the following situation:
I converted a CR2 raw file into a jpg (as part of a series of 100 photos) and the content of the picture within the jpg is all distorted, yet I'm able to open it. This happened to 2 photos in this series that I processed. When I go look at the RAW file from which it was generated from, there is no corruption and I can open it fine. I wonder if I'm having a memory issue with this new system and it's not a network related problem?

I've uploaded an example to show you what is happening. I'm using Canon DPP to process the RAW files. I'm using the same version (3.61), but this time I'm on Windows 7, 64-bit rather than XP. I've used DPP 3.6.1 on my laptop which is Vista 64-bit and never had this issue. I've transferred many gigabytes worth of data from my XP machine to my laptop in the past.

 

[sechs]

Does it matter whether you pull or push the files to the new computer?

 

[Stereodude]

If you go back to the same file on your XP machine it is still intact?

 

[Mercutio]

rsync doesn't look so silly now!

 

[Handruin]

Does it matter whether you pull or push the files to the new computer?

I haven't tried a push of the files, I've done 100% pull from my XP machine. I'll see if that makes any difference.

 

[Handruin]

If you go back to the same file on your XP machine it is still intact?

Yes, the files are still OK on the XP machine. If I copy it over a second time it's fine.

How long should I let something like memtest86 run for? I only left it for a couple hours. Does it need longer than that to find problems?

 

[MaxBurn]

Yes, the files are still OK on the XP machine. If I copy it over a second time it's fine.

How long should I let something like memtest86 run for? I only left it for a couple hours. Does it need longer than that to find problems?

I think in general you want to see a couple of full cycles, however long that takes on whichever computer.

 

[Handruin]

I'll give that a try. It did about one and a half full cycles with no errors. Is prime95 a reasonable way of testing the system also?

 

[MaxBurn]

I'll give that a try. It did about one and a half full cycles with no errors. Is prime95 a reasonable way of testing the system also?

Yes, there is even a test suite to automate it for stability testing but I can't remember the name of it. Basically it just automated the running of prime95. Maybe it was orthos.

Found these:
http://www.ocforums.com/showthread.php?t=335813

http://forums.anandtech.com/messageview.aspx?catid=28&threadid=1901991

All seems to be focused on CPU/memory though. I will also mention burn in tester only because I see it has a network loopback test, but I know that doesn't test the hardware...

Also maybe use quickpar to build file level parity, move the files and verify them at the destination? Surely that would be easier than looking at pictures and not catching all the errors you can't see.

 

[Handruin]

Thanks for the links MaxBurn. I'll give those a try when I get home tonight. I made one small change last night before I went to bed related to memory speed. The motherboard I have allows for changing memory speed based on profiles. I had it set to 1600MHz (which is what the memory is rated for) where I believe the default is 1300MHz for this CPU. I set it back down to 1300 MHz and I'll try a bunch more file copies and then check some parity. I wasn't trying to test this issue with my images, it just happened to be the symptoms I was seeing.

 

[Stereodude]

I like running Furmark & multiple copies of CPU burn-in (one for each core, but leaving one free to babysit Furmark) for at least 8 hours simultaneously.

 

[Handruin]

I can also give that a try. If something turns up from this, what typically happens to indicate a problem?

As an aside, I've been doing some searching and I've found this thread from other users seeing similar issues. There doesn't seem to be a common cause to the problem from what I've read. A couple of the example images posted match what I'm seeing.

If this continues, I'll have to switch back to Vista or XP and do further testing to see if it's reproducible. I need to at least isolate if this is only affecting images so far. I haven't transferred any other data in bulk from my other machine like I've done for RAW files.

I did notice when I burnt a DVD-R last night (which contained jpgs and RAW files) that during the verify stage it reported differences in the blocks written when compared to the source. Not sure if this has any relation to my problem...

 

[Stereodude]

I can also give that a try. If something turns up from this, what typically happens to indicate a problem?

Well CPU Burn-in checks for errors. I've never had it fail, but it's great to check your cooling and that your power supply is up to the task. It probably won't help you find the cause of your problem.

If this continues, I'll have to switch back to Vista or XP and do further testing to see if it's reproducible. I need to at least isolate if this is only affecting images so far. I haven't transferred any other data in bulk from my other machine like I've done for RAW files.

I did notice when I burnt a DVD-R last night (which contained jpgs and RAW files) that during the verify stage it reported differences in the blocks written when compared to the source. Not sure if this has any relation to my problem...

That would be a serious problem in my opinion. I assume you burnt the data on your new PC? The question I have is your data getting corrupted on the write, or on the read. When you burn a DVD it reads from the HD, then on the verify it reads it again. You should get the same result every time to read the file on the HD. That makes it seem like you've at least got a HD read issue.

Can you compare the files on the DVD in your old XP machine? Maybe copy them to the HD and then FC them against the originals.

Is this all using your new WD VR that you bought?

IMHO, I'd basically stop using the new machine for anything other than testing until you track down the source of the problem. Data corruption is nothing to mess with it, and it seems like it isn't just network related.

I'd try using another HD in the new machine in place of the VR. I'd try a different SATA cable. I'd try other SATA ports. My best guess is that you're getting data corruption somewhere between the platters and the output of the SATA controller.

 

[Handruin]

It is still a test machine. All my real data still lives on my other system and I've been temporarily copying it via the network to process pictures. If the processed pictures are ruined, it's only time lost, not my actual RAW files because a copy still lives on my older machine (and two other locations).

For cooling, I'm using the stock Intel HSF until I can find a decent socket 1156. The only change I made was to replaced Intels thermal grease for artic silver. I'm not overclocking right now and the temperatures seem to be in check...but I'll monitor closer when processing. The PSU should be up to the task unless it's defective. It's a PC Power & cooling 750W which is the same as in my old machine (I bought a second one for this machine). The new machine is on a new APC, so hopefully that's keeping the power clean.

I have two new drives on the way from newegg. I ordered two 1TB Samsung F3 7200RPM drives to put into the new machine. I'll play around with those once they get here tomorrow. Yes, so far all this odd corruption has been on the WD VR (I know, I know). I've already run the full disk sector scanning but no errors were reported.

Right now I only have the WD VR and a Sony DVD-R drive in this system. I do have an external eSATA drive connected, but not powered on. I'll try swapping the cables and SATA ports. I also want to try updating all the drivers from the default windows 7 drivers to the ones for my motherboard.

I've done other intensive work with this system without issue. I've played games on it for multiple hours at a time with no crashing or corrupted saved game files. I've ripped a DVD to the WD VR and then encoded the iso into a video file suitable for an iPhone and there was no corruption. I've ripped a CD using EAC and encoded it into an MP3 without errors. I know that doesn't clear the system from having these issue in the future, but I'm curious why it's only happening with graphical images so far. I would have expected this to happen with system files, installed application files, or some of the large files associated with the games I've installed.

For the weird DVD issue, I'll try comparing the files with the originals on my older machine to see if they match. I did burn this on the new system.

 

[Stereodude]

I've done other intensive work with this system without issue. I've played games on it for multiple hours at a time with no crashing or corrupted saved game files. I've ripped a DVD to the WD VR and then encoded the iso into a video file suitable for an iPhone and there was no corruption. I've ripped a CD using EAC and encoded it into an MP3 without errors. I know that doesn't clear the system from having these issue in the future, but I'm curious why it's only happening with graphical images so far. I would have expected this to happen with system files, installed application files, or some of the large files associated with the games I've installed.

I'm not sure any of those thing clear your storage subsystem. There's no way to know there wasn't corruption on those tasks. Take a mp3 open it in a hex editor change a few random bits and play it back. I'd bet 99 times out of 100 it plays back fine. That doesn't mean it didn't have corruption. The same is true of the compressed video too. It's also lossy compression and in both cases there's no checksum on the decode. Random errors in the data are not likely to make themselves known.

I'd try raring some larger files, copy the rar file around the HD a few times to and from different locations, then try to unrar 'em and see what happens.

That or make a PAR2 file set from a larger file, or set of files, then copy them around (don't move) and then check 'em against the PAR2 you created.

 

[Fushigi]

IMO you're looking at the wrong area. I'd be focusing on network transfer errors more than the local PC. A straight network file copy does not significantly stress CPU or memory. It doesn't stress the storage subsystem all that much either unless you're running at full Gb speeds and getting 80+MB/s throughput.

Try using netcap or some other network monitoring utility and running a fresh transfer.

Some years ago I had similar issues but with USB. If I transferred lots of files from our digicam to the PC in one shot, about 1 in 15 would be corrupted. They always were fine if I re-copied them individually or in smaller groups.

 

[Stereodude]

IMO you're looking at the wrong area.

I disagree. The failure of the verification of the burned DVD would seem to point to data corruption in the storage subsystem, not the network (unless he got a super rare bad DVD blank and also has a network data integrity issue).

It doesn't stress the storage subsystem all that much either unless you're running at full Gb speeds and getting 80+MB/s throughput.

FWIW, Windows file sharing tops out at about 47MB/sec over gigbit Ethernet. FTP is good for mid 80's over gigabit.

 

[Handruin]

I did a file compare on a binary level between the DVD I burned last night of those images with the same files on my new computer and they claim to match 100%.

I copied (not moved) around the 5.6 GB DVD iso and compared the source and destination (both on the same WD raptor) and they match from a hex comparison check (I'm using a tool called beyond compare). Now I'm compressing the DVD iso using 7-zip into a .7z file that I'll also copy around and compare.

Prior to these tests I updated the drivers for the Intel chip, the storage controller, the audio, and network using the latest for my motherboard on gigabytes website. I'm also going to do some more batch conversions of CR2 files into jpgs and see if I can reproduce this. All these tests I'm doing right now are with the changed memory speed that I made last night before I went to bed (changed from 1600 to 1300).

I forgot to mention another symptom I just remembered. One of the first few times I noticed a corruption problem was within Canon DPP. I clicked on an image (CR2) that I had just batched processed and DPP said the file was corrupt. I rebooted the computer and went back to the same image. I could then open it without problem. That's why I was thinking this could be memory issues in my first post.

 

[Stereodude]

Now you'll never know if you solved the problem or not because you don't have a confirmed procedure that manifests the error.

 

[Handruin]

The most identifiable method was CR2 file processing. I probably should have made one change at a time, but I still couldn't get this to happen regularly.

 

[Stereodude]

I probably should have made one change at a time...

I usually don't make the changes one at a time either. I usually have a bunch of ideas and make several of them at once too. :cheers:

Then I never quite know what fixed the problem. :rofl:

 

[Handruin]

I was too impatient to try one by one. I want this solved so I can move on and get some more work done.

I've processed about 400 images and no corruption (yet). I'm working on another set of 400+ doing just about all the same tasks I was doing before. Web browser open with 5-6 tabs, listening to pandora, flickr uploader uploading files, DPP, and photoshop.

I'm noticing the network transfer is rather slow for GigE even with the change in network driver. I can only get about 10-11MB/sec transfer when I pull from my XP machine. I checked and I'm not using Jumbo frames on either side. I'll have to try some transfers with my laptop running Vista to see if it's any better. I've seen much better transfer using my laptop to my XP machine. It works for now, but still surprisingly slower than expected for large monolithic file transfer.

 

[LunarMist]

Are you converting files on the other server or is it just that they were previously copied? Sometimes the OS or an app is creating an evil preview of the embedded thumbnail that slows down copies - even if you can't see it. However, that is usually with TIF (/RAW) or JPG files.

 

[Handruin]

I'm converting them locally. I've been copying them first from my old machine to the new one. Once that's done I've been processing locally.

As for the copy, I also transferred an iso file and it was also around 10-11MB/sec so I don't think it was the thumbnail causing a problem.

 

[CougTek]

Right now I only have the WD VR and a Sony DVD-R drive in this system.

Merc might commit suicide after reading this.

 

[Stereodude]

I'm noticing the network transfer is rather slow for GigE even with the change in network driver. I can only get about 10-11MB/sec transfer when I pull from my XP machine.

Are you pushing or pulling across the network?

FWIW, I found GigE network transfers sped up immensely when the gigbit NiC's were no longer on the 32-bit PCI 33MHz bus. PCIe x1 gigabit NiCs are great (most NiC's on motherboards are now PCIe). Interesting enough one of the gigabit NiCs in my P4 backup server connects directly to MCH (Memory Control Hub) on the chipset bypassing the PCI bus.

 

[LunarMist]

I copied 7.45GB of CR2 files from one computer to the other in 252 sec. I don't know if that is normal speed between computers.

 

[Stereodude]

That's 30MB/sec which is respectable.

 

[Handruin]

Three times faster than what I'm seeing so far.

 

[Handruin]

Well, I processed about 800 photos, shrunk a DVD and burned it successfully, uploaded tons of images, and copied files all over the place...no corruption tonight. I'll have to give it a few more days to see if I'm out of the water yet.

 

[MaxBurn]

I'm curious, this problem went away because of the memory adjustment?

Have you run file level parity tests with PAR2?

 

[Handruin]

The problem has not yet returned since I reduced the speed of the memory and updated drivers with the ones from Gigabyte (vs using Windows 7 default). This motherboard supports memory profiles and it found one profile for the RAM I purchased. I used the faster profile which increased the speed of the memory, but it seemed to have caused system instability with this corruption problem.

I have not tried parity testing with PAR2 but I did do some binary comparisons using a tool called Beyond Compare. I compared copies of files to see if they had been corrupted and after numerous GB of copying files, I didn't find any to be corrupt or different.