Copying from one location to another

[Jimshady]

In a W2K machine, if I am copying (not moving) a file from, and to, the same physical and logical drive, then what is the process that the data goes through. Does data need to travel on the bus? Or does a drive usually perform this function internally? I guess I could just see what perfmon has to say about this, but I'm a little slack.

Basically, is it performed internally? Or is it read to memory, and written back?

 

[Howell]

In a W2K machine, if I am copying (not moving) a file from, and to, the same physical and logical drive, then what is the process that the data goes through. Does data need to travel on the bus? Or does a drive usually perform this function internally? I guess I could just see what perfmon has to say about this, but I'm a little slack.

Basically, is it performed internally? Or is it read to memory, and written back?

The drive itself does not know anything about the file structure that resides on it. It doesn't even know what file system is on it. This operation must be handled by the OS and you can guess what that means.

 

[Jimshady]

Wow, I really didn't think of it like that

 

[jtr1962]

As far as the drive is concerned, all it deals with are commands to write or read to a sector or group of sectors starting at whatever LBA. As Howell mentioned, it is up to the O/S to make sense of the data it stores on the hard drive.

Also, whenever anything is done with a hard drive, the data goes from the drive to the bus, even if it is simply copied from partition to partition. However, it might make sense to have drives which are "smart" enough to bypass a few steps when copying the same data from one part of the drive to another. Perhaps everything could be done internally, without putting the data on the bus, and then it could let the O/S know that the operation has been successfully completely. I'm not really sure if this would speed anything up, however, for most file copies since the bus is usually faster than the STR of the drive.

 

[sechs]

I might be satisfied if my controller could be given enough information to not have to drag data in and out of memory for a disk-to-disk transfer.

 

[LunarMist]

Has anybody else experienced slow interadrive copy speeds on SCSI drives? For some reason my 15K.3 and the X15-36LP before it were both very slow in this operation and heavy seeking was observed. Perhaps it is a difference in cache utilization, but my S-ATA and ATA drives are much faster. For example the 15K.3 is only good for ~6 MB/sec intradrive copying, whereas 7K250, DM+9, and WD740GD drives copy at 21-24 MB/sec. It has always been a mystery since the 15K drives have otherwise been stellar performers.

 

[Howell]

Same partition of different partitions?

 

[Howell]

I might be satisfied if my controller could be given enough information to not have to drag data in and out of memory for a disk-to-disk transfer.

That would be tough because the controller doesn't even know what "data" is. The biggest problem I see is not for the drive to know what you want ot move but where to put it. The drive wouldn't know if the area was used or available.

The drive would have to have knowledge/understanding of file systems so that the MFT could be passed to it. Then we would need either a drive model for each file system or a switch. I'm not sure even 8M drives can hold the MFT much less a (or several as tends to be the case) *nix file system too.

And this is for a copy with a root destination. It's even more complicated with any kind of folder structure.

 

[Pradeep]

I posted some test times on SR.com many moons ago, took around 5 times longer going intradrive copy on an x15-36LP as opposed to copy interdrive. I also had IDE times. I'll have to hunt them down.

 

[mubs]

On a related note, I copied entire partitions back and forth between drives as in drive D --> drive E --> replace drive D --> copy from drive E to new drive D. POS W2k created all folders and sub-folders in the new drive with the current date instead of the original create date. The files are ok, though. This has messed me up big time, because I regularly look for some critical info based on folder date, now they are all the same. I'll even go through the bother of changing each one, one at a time, but don't have a tool handy. The tool I have will let me change date stamps on files, but not folders. Why oh why does W2k do this?? :arge: :arge: :arge:

 

[jtr1962]

mubs,

IIRC the utilities that come with new retail Maxtor drives include a way to copy one partition to another, and I'm pretty sure that the folder dates are copied correctly. I'll have to double check that for you one of these days, but I think it works that way.


Yes, it's very annoying when I switch to a new drive to have all my folders with the same date. The only plus here is that I know I installed the drive on that particular date. Other than that, there is no upside. BTW, Win9x does this also, and I wouldn't be surprised if WinXP also does it. Besides Maxtor's utilities, I believe there may be command line utilities similar to XCOPY which copy with the folder date intact. I know there is something called XXCOPY, which is an enhanced version of XCOPY, but I'm not sure if it has this option.

 

[mubs]

jtr,

I have Partition Magic and Drive Image, both of which will not touch datestamps. I used WinExplorer in this particular instance because I neded to rearrange a lot of stuff, and since Ii was going to be replacing drives, decided to solve two issues at one stroke. And I moved the folders, didn't copy them.

 

[sechs]

I might be satisfied if my controller could be given enough information to not have to drag data in and out of memory for a disk-to-disk transfer.

That would be tough because the controller doesn't even know what "data" is. The biggest problem I see is not for the drive to know what you want ot move but where to put it. The drive wouldn't know if the area was used or available.

Not really. A set of commands to the effect of copy these sectors on drive a to these sectors over here on drive b is all that the controller needs; it would probably be in the form of a large series of commands, read sector x on drive a and write to sector y on drive b. A lot of bus traffic, but not nearly the amount of data as moving the information in and out of memory.

 

[ddrueding]

I might be satisfied if my controller could be given enough information to not have to drag data in and out of memory for a disk-to-disk transfer.

That would be tough because the controller doesn't even know what "data" is. The biggest problem I see is not for the drive to know what you want ot move but where to put it. The drive wouldn't know if the area was used or available.

Not really. A set of commands to the effect of copy these sectors on drive a to these sectors over here on drive b is all that the controller needs; it would probably be in the form of a large series of commands, read sector x on drive a and write to sector y on drive b. A lot of bus traffic, but not nearly the amount of data as moving the information in and out of memory.

That wouldn't really work, as the drive wouldn't know if sector y oon drive b was available. If the OS provided that information as part of the command I can see it working; but on more fragmented drives the commands might take longer than the data itself.

Further, isn't the copying process faster if it is done in bulk to RAM and back? Near (Sustained Read+seek+Sustained Write) speeds vs (Block-level read+seek+block-level write)?

 

[sechs]

That wouldn't really work, as the drive wouldn't know if sector y oon drive b was available. If the OS provided that information as part of the command I can see it working; but on more fragmented drives the commands might take longer than the data itself.

That's not a controller's job anyway. It's up to the OS to keep the file system coherent. It shouldn't be sending commands to put data in sectors that aren't where the file is going.

Further, isn't the copying process faster if it is done in bulk to RAM and back? Near (Sustained Read+seek+Sustained Write) speeds vs (Block-level read+seek+block-level write)?

This is kind of like saying it's quicker to pour water from one bucket to another by putting all of the water in a third bucket first (even if it is a very nice bucket).

 

[ddrueding]

Further, isn't the copying process faster if it is done in bulk to RAM and back? Near (Sustained Read+seek+Sustained Write) speeds vs (Block-level read+seek+block-level write)?

This is kind of like saying it's quicker to pour water from one bucket to another by putting all of the water in a third bucket first (even if it is a very nice bucket).

Which it is under certain circumstances. The ability to "pour" the entire transfer at once to a 3rd location, then "pour" the data back @ STR speeds versus only being able to pour as much as the hard drive cache allows before having to stop, seek, read, seek again, and continue.

I mught even suspect that this is why the SCSI drives are slower for intra-drive copies. They probably take the integrity of their data more seriously and don't allow large chunks of it to be sitting in RAM only.

 

[sechs]

Assuming that even half of file copy operations are sequential at one end or the other is a lot to ask.

I honestly don't see how sustained reads and writes are better, when you're still limited by the media transfer rate for anything larger than the write buffer of the receiving drive (at which point it may not matter).

 

[Fushigi]

And how will the drive(s) handle parity calculations & writes in a RAID 5 array?

 

[sechs]

The same way that they always have?

 

[Fushigi]

The same way that they always have?

Nope. Can't do that becuase the drive is handling the transaction; not the OS or controller.

My point is that, for all situations other than a standalone drive, the complexity required to maintain integrity goes way up. You can't easily take the intelligence away from the controller and OS even if it would improve performance.

Frankly, if Windows had slightly more intelligence when it came to copying this would likely be a non-issue. If the OS would read source file(s) from the drive until physical RAM ran out (w/out resoring to page/swap file), then stream the target file(s) back to the drive, then repeat until nothing's left to copy, you would maximize performance by getting the best STR possible. If going between multiple physical drives, each file copy could be a thread such that after each file is fully read into RAM (or RAM is exhausted), a thread is dispatched to a single-threaded queue to write that file to the target drive & free up the RAM for more reading. Doing that should maximize throughput. Reducing the amount of RAM used for caching in order to make more available to the copy function would help even more as long as reducing that cache didn't involve paging cache to disk.

 

[sechs]

I think you've missed it. It's the controller that's handling the transaction, not the OS. Since the controller runs the array, it would *still* be responssible for these tasks. Since RAID 5 controllers tend to be "smarter," I'd say it'd be easier for them to do something like this.

Your suggestion concerning RAM optimises storage at the expense of applications -- which is a really bad idea.

 

[Fushigi]

I think you've missed it. It's the controller that's handling the transaction, not the OS. Since the controller runs the array, it would *still* be responssible for these tasks. Since RAID 5 controllers tend to be "smarter," I'd say it'd be easier for them to do something like this.

Your suggestion concerning RAM optimises storage at the expense of applications -- which is a really bad idea.

No, the discussion involved the drive having sufficient logic to do the file copy on it's own. That implies minimal commands from the controller/OS on the order of 'there's a file starting here; copy it to this other place'. I'm saying the difficulty involved in the implementation outweighs any possible performance benefits. Can it be done, yes, but it would involve a lot of work in the firmware of the drives and controllers. And would be made more complicated by JBOD and striping arrays that span controllers.

As to RAM optimization, please explain how applications are impacted when unused memory is temporarily allocated for the copy operation. If there is a misunderstanding, let me explain what I'm talking about: I have 1GB of physical RAM, about 350MB of which is used by various applications. That would leave 650MB free to use as a buffer for copying files without forcing any applications to be swapped to disk. Files could be copied in (up to) 650MB streams much more efficiently than whatever puny buffer MS does now. Since this is a temporary allocation of otherwise unused RAM, how can there be a negative impact on performance?

 

[sechs]

No, the discussion involved data transfer going from one disk to another on the controller, without having to touch main memory or an external bus.

If you read what you previously wrote, you clearly said, "If the OS would read source file(s) from the drive until physical RAM ran out...." Nothing there about memory first being "unused"; sounds to me like all physical memory.

Nonetheless, if we now do what you currently suggest, you'd simply be flushing the system cache in favor of a file copy operation. Once again, not a good idea.

Furthurmore, as I intimated, getting a sequential read or write is nigh impossible. There is no apparent advantage to a sustained read or write where the media transfer comes into greater play.