Rsync for Windows (or similar) app suggestions?

S

Stereodude

Not really a
Joined
Jan 22, 2002
Messages
10,865
Location
Michigan
I'm after the ability to automate the back up of a large amounts of data where very little of it has changed or is new on a day to day basis, but the sheer size of the data set vs the network throughput (even with gigabit Ethernet) makes it extremely impractical to copy the entire thing over daily.

I figure there's bound to be some good knowledge here so I don't have to try a bunch of programs to find the one that works. I came across DeltaCopy (an rsync port for Windows that doesn't require cygwin) that looks promising and another similar program called Unison File Synchronizer that also looks like a potential option. There are lots of other options too So, does anyone have any suggestions for good rsync like program, or running rsync itself on Windows?
 
Microsofts free 'SyncToy'.
Xcopy has a switch that will tell it to only copy files that have changed. It is included in all Windows systems.
XXCopy- a free program that is basically xcopy on steroids.
 
Bozo said:
Microsofts free 'SyncToy'.
Click to expand...
This looks like a potential option
Xcopy has a switch that will tell it to only copy files that have changed. It is included in all Windows systems.
XXCopy- a free program that is basically xcopy on steroids.
Click to expand...
I'm not sure I want to rely on the archive bit (which both of those programs do) in order to ensure that my backup is complete. :bomb:
 
I guess I should also mention that if possible I would like the backup server to pull the data from the main server as opposed to having the main server push the data to the backup server (I think this is not how rsync works).

Here's the current process I'm envisioning:
- Main server sends WOL to backup server
- Backup server powers on
- Backup server automatically logs into Windows XP (using an account with read only access to main server)
- Backup server runs synchronization / backup program
- Backup server powers itself off

This should minimize the power on time for the backup server and take out any variables out of play like backup server boot time, or how long the copy / sync takes.
 
I use this one to keep my TBs of files backed up daily. It probably doesn't do what you need, but I don't understand all the evil things IT people do on computers. ;)
 
At least for my use, deltacopy isn't that important. Each file isn't that big, and isn't likely to be edited that much. If it has to copy an entire file each time it is changed, that's OK. So long as it doesn't have to copy unchanged files every day.
 
ddrueding said:
At least for my use, deltacopy isn't that important. Each file isn't that big, and isn't likely to be edited that much. If it has to copy an entire file each time it is changed, that's OK. So long as it doesn't have to copy unchanged files every day.
Click to expand...
FWIW, I read that Rsync isn't very fast on a high speed connection (ie: LANs). Supposedly it is unbeatable for backing up over the internet, but apparently it can't take advantage of the capability of a LAN connection.

Most the data I'm messing with doesn't change. There are plenty of big files, but they typically don't change. Once a file is put on the server it typically is left alone. So the whole Deltacopy thing is not an issue for me.
 
Ghetto method: xcopy /h/i/c/k/e/r/y/d

Of course, that's at the file level rather than the block level, but for most purposes that's probably OK.

Or if you have proper Windows Servers you can manage replication with dfs.
 
Where did people end up with this? I've been using FileBack PC, but it is starting to choke on this size of data set (2TB, ~5,000,000 files). I've also come to the conclusion that I need the target to simply be an exact copy of the original (including permissions), so Merc's suggestion would work if it were fast enough (though decent error logging would be nice).
 
ddrueding said:
This only went about 25GB in before it threw an "insufficient memory" error. This machine has 8GB of RAM, and the target has 8TB free.

Thoughts?
Click to expand...

I've never seen that before, and I've used xcopy in that fashion to move hundreds of gigabytes of .jpg and .flac files.

As has been pointed out, there are xcopy replacements. Teracopy, xxcopy etc.
Teracopy is handy, but it hooks into the OS and brings up a user interface every time you copy a file, which is not IMO desirable behavior. I've always stuck with that simple xcopy command because it IS always available and reliable.

The only thing that comes to my mind as far as troubleshooting is that you might want to do a chkdsk on that volume.
 
Stereodude said:
I'm using Syncback.
Click to expand...

I second this. I'm using an older copy of the Freeware version of SyncBack as a means to keep one of our Windows servers in sync with a backup for years. This involves a regularly scheduled backup of a large MSSQL db dump, several GB worth of small files (many of which do not change), and a few larger files.

There are several types of backups that can be performed (sync, one way, restore, etc), a ton of file comparison options to determine which files need backed up, and many destination types (archive, drive/folder, FTP, CIFS, etc). It also includes a scheduler and task grouping - so my 3 step backup/restore process works in the correct order, every time I run the group, and not as 3 separate tasks.


Regarding rsync - One thing that rsync does, that is pretty unique is that it can choose whether to copy the whole file or only the parts of the file that have changed. This is why it is considered great over a WAN. However, I use it on a local gbit network just fine - as long as the HDDs can push the data, I can get 200-300Mbps when working with small files (large files do better, of course). If you're on a high speed LAN, copying whole files really isn't going to slow you down much though (assuming that you're not dealing with a very small number of very large files as your backup case).
 
I use rdiff-backup via Cygwin on Windows for this sort of stuff.

It works the same way as rsync. It just adds deltas and a few other backup-specific (rather than sync-specific) features. I'm going to checkout SyncBack though. This is a problem I regularly have to solve.



I've also been meaning to checkout Novell's iFolder: iFolder. That's more a synchronization tool than a backup tool however.

Synkron was another I had on my list. It's written in Qt so it has native GUI integration on Windows, Mac OS X, & Linux. I haven't tried this out yet though.
 
You must log in or register to reply here.
Back
Top