Rsync for Windows (or similar) app suggestions?

[Stereodude]

I'm after the ability to automate the back up of a large amounts of data where very little of it has changed or is new on a day to day basis, but the sheer size of the data set vs the network throughput (even with gigabit Ethernet) makes it extremely impractical to copy the entire thing over daily.

I figure there's bound to be some good knowledge here so I don't have to try a bunch of programs to find the one that works. I came across DeltaCopy (an rsync port for Windows that doesn't require cygwin) that looks promising and another similar program called Unison File Synchronizer that also looks like a potential option. There are lots of other options too So, does anyone have any suggestions for good rsync like program, or running rsync itself on Windows?

 

[Fushigi]

Maybe look for something that backs up like Windows Home Server. It uses VSS to back up only the changed blocks.

 

[Bozo]

Microsofts free 'SyncToy'.
Xcopy has a switch that will tell it to only copy files that have changed. It is included in all Windows systems.
XXCopy- a free program that is basically xcopy on steroids.

 

[Stereodude]

Maybe look for something that backs up like Windows Home Server. It uses VSS to back up only the changed blocks.

Isn't WHS the OS you can't buy? That rules it out.

 

[Stereodude]

Microsofts free 'SyncToy'.

This looks like a potential option

Xcopy has a switch that will tell it to only copy files that have changed. It is included in all Windows systems.
XXCopy- a free program that is basically xcopy on steroids.

I'm not sure I want to rely on the archive bit (which both of those programs do) in order to ensure that my backup is complete. :bomb:

 

[Stereodude]

I guess I should also mention that if possible I would like the backup server to pull the data from the main server as opposed to having the main server push the data to the backup server (I think this is not how rsync works).

Here's the current process I'm envisioning:
- Main server sends WOL to backup server
- Backup server powers on
- Backup server automatically logs into Windows XP (using an account with read only access to main server)
- Backup server runs synchronization / backup program
- Backup server powers itself off

This should minimize the power on time for the backup server and take out any variables out of play like backup server boot time, or how long the copy / sync takes.

 

[ddrueding]

I'm looking for something like this as well, only I need it to support shadow copy, as some files are always in use.

 

[Stereodude]

I found this Wikipedia article with some program comparisons.

 

[Fushigi]

Isn't WHS the OS you can't buy? That rules it out.

Nope.

 

[LunarMist]

I use this one to keep my TBs of files backed up daily. It probably doesn't do what you need, but I don't understand all the evil things IT people do on computers.

 

[Stereodude]

I'm looking for something like this as well, only I need it to support shadow copy, as some files are always in use.

SyncBackPro might fit the bill. According to their comparison chart it can "Copy open/locked (in-use) files (XP/Vista/2003/2008)".

They have a trial you can download to try out.

Note it does not have deltacopy like functionality like Rsync.

 

[ddrueding]

At least for my use, deltacopy isn't that important. Each file isn't that big, and isn't likely to be edited that much. If it has to copy an entire file each time it is changed, that's OK. So long as it doesn't have to copy unchanged files every day.

 

[Stereodude]

At least for my use, deltacopy isn't that important. Each file isn't that big, and isn't likely to be edited that much. If it has to copy an entire file each time it is changed, that's OK. So long as it doesn't have to copy unchanged files every day.

FWIW, I read that Rsync isn't very fast on a high speed connection (ie: LANs). Supposedly it is unbeatable for backing up over the internet, but apparently it can't take advantage of the capability of a LAN connection.

Most the data I'm messing with doesn't change. There are plenty of big files, but they typically don't change. Once a file is put on the server it typically is left alone. So the whole Deltacopy thing is not an issue for me.

 

[Mercutio]

Ghetto method: xcopy /h/i/c/k/e/r/y/d

Of course, that's at the file level rather than the block level, but for most purposes that's probably OK.

Or if you have proper Windows Servers you can manage replication with dfs.

 

[ddrueding]

Where did people end up with this? I've been using FileBack PC, but it is starting to choke on this size of data set (2TB, ~5,000,000 files). I've also come to the conclusion that I need the target to simply be an exact copy of the original (including permissions), so Merc's suggestion would work if it were fast enough (though decent error logging would be nice).

 

[Stereodude]

I'm using Syncback.

 

[ddrueding]

Ghetto method: xcopy /h/i/c/k/e/r/y/d

This only went about 25GB in before it threw an "insufficient memory" error. This machine has 8GB of RAM, and the target has 8TB free.

Thoughts?

 

[Chewy509]

This only went about 25GB in before it threw an "insufficient memory" error. This machine has 8GB of RAM, and the target has 8TB free.

Thoughts?

What is the source and target OS?

Win7/Win2K8R2 adds the /J switch to xcopy which turns off buffering.

 

[ddrueding]

This is Server 2003, so no luck there. Source and Target are different volumes on the same machine.

 

[time]

Guy with same problem

He used XXCopy, which is a formidable but compatible piece of kit that costs about $25/server (or something like that).

For a freebie, use Robocopy, or better yet, RichCopy.

 

[Mercutio]

This only went about 25GB in before it threw an "insufficient memory" error. This machine has 8GB of RAM, and the target has 8TB free.

Thoughts?

I've never seen that before, and I've used xcopy in that fashion to move hundreds of gigabytes of .jpg and .flac files.

As has been pointed out, there are xcopy replacements. Teracopy, xxcopy etc.
Teracopy is handy, but it hooks into the OS and brings up a user interface every time you copy a file, which is not IMO desirable behavior. I've always stuck with that simple xcopy command because it IS always available and reliable.

The only thing that comes to my mind as far as troubleshooting is that you might want to do a chkdsk on that volume.

 

[blakerwry]

I'm using Syncback.

I second this. I'm using an older copy of the Freeware version of SyncBack as a means to keep one of our Windows servers in sync with a backup for years. This involves a regularly scheduled backup of a large MSSQL db dump, several GB worth of small files (many of which do not change), and a few larger files.

There are several types of backups that can be performed (sync, one way, restore, etc), a ton of file comparison options to determine which files need backed up, and many destination types (archive, drive/folder, FTP, CIFS, etc). It also includes a scheduler and task grouping - so my 3 step backup/restore process works in the correct order, every time I run the group, and not as 3 separate tasks.


Regarding rsync - One thing that rsync does, that is pretty unique is that it can choose whether to copy the whole file or only the parts of the file that have changed. This is why it is considered great over a WAN. However, I use it on a local gbit network just fine - as long as the HDDs can push the data, I can get 200-300Mbps when working with small files (large files do better, of course). If you're on a high speed LAN, copying whole files really isn't going to slow you down much though (assuming that you're not dealing with a very small number of very large files as your backup case).

 

[mubs]

I third SyncBack.

 

[Gilbo]

I use rdiff-backup via Cygwin on Windows for this sort of stuff.

It works the same way as rsync. It just adds deltas and a few other backup-specific (rather than sync-specific) features. I'm going to checkout SyncBack though. This is a problem I regularly have to solve.



I've also been meaning to checkout Novell's iFolder: iFolder. That's more a synchronization tool than a backup tool however.

Synkron was another I had on my list. It's written in Qt so it has native GUI integration on Windows, Mac OS X, & Linux. I haven't tried this out yet though.