File synchronization options for large amounts of files

[Gilbo]

Hey guys, I've spent the last 3 months trying to get two computers to sync a large directory of photos (about 1.8 TB).

I just can't make it work. There have been BSOD errors and mysterious reboots, and it seems like Live Mesh just can't handle comparing that amount of data. I can never get out of the initial synchronization stage.



Are there any other options for synchronization that will let me keep a large directory like this in sync?

I could just create FTP servers on each and manage it manually, but I find things get out of sync when I get busy and it's a huge PITA to fix it afterwards. I was really hoping Live Mesh would fix this, but it just doesn't seem stable enough .

There must be tools that do this. I can't believe in this day and age that Live Mesh is the only thing that'll work for me. (Dropbox, SugarSync, etc. all require cloud storage. I don't want to pay out the nose for 1.8TB of cloud storage!)

 

[timwhit]

Rsync.

 

[Gilbo]

Rsync isn't really what I need. It's a 1-way solution, that has to be scheduled.

I need a 2-way (or n-way) solution like Windows Live Mesh, that works in real time, so when I go home or go to the office or open my laptop all my files are on each computer, and are up to date.

In this day and age this is still almost impossible so far as I can tell if you have more than a couple GB of data and can't use Dropbox or SugarSync ...

 

[Mercutio]

Windows Server has dfs, if your files are hosted there.

 

[timwhit]

How about this? http://www.cis.upenn.edu/~bcpierce/unison/

 

[Adcadet]

Microsofts' SynToy? I've used it with good success. Nice and simple. The downside is that you need to manually run it.

http://www.microsoft.com/download/en/details.aspx?id=15155

 

[ddrueding]

Just to clarify, these two machines aren't on the same LAN are they? If so, SyncToy can easily do what you ask. I'm not sure how efficient it is with network resources, or I would suggest running it over a VPN.

 

[ddrueding]

Microsofts' SynToy? I've used it with good success. Nice and simple. The downside is that you need to manually run it.

http://www.microsoft.com/download/en/details.aspx?id=15155

Just edged me out

There are command line options to automate SyncToy.

 

[Mercutio]

Synctoy is hugely inefficient for large numbers of files. It also generates a tremendous number of very small (1 byte? 1kB? Don't remember) temp files that can create a large overhead in the filesystem and also annoy the piss out of you when a sync fails.

 

[Bozo]

RoboCopy will work.

 

[Gilbo]

How about this? http://www.cis.upenn.edu/~bcpierce/unison/

Unison doesn't work with Windows 7 . (Also it's PITA to setup...)

Just to clarify, these two machines aren't on the same LAN are they? If so, SyncToy can easily do what you ask. I'm not sure how efficient it is with network resources, or I would suggest running it over a VPN.

Ya different networks. I want to sync over the internet.

Even though it's a large amount of data, the data doesn't change that much. At most about 30 GB / month. It's just that Live Mesh can't seem to do the initial sync, even if all the data is already copied over and all it has to do is verify everything matches.



My googling brought me to this page, which recommends Allway Sync, and FreeFileSync. Sigh. Both of which don't necessarily appear polished. I'm wary of trusting all my photos and other data to random programs.

I'm really surprised Microsoft's Live Mesh doesn't work. They specifically mentioned software updates 6-12 months ago that they claimed would allow it to work with a near-unlimited number of files. I'm going to file a bug report. Maybe they can help me...

 

[Gilbo]

Oh, and this is for 2 Windows 7 Professional computers, so I can't use the Windows Server option you mentioned Merc...

Will Photoshop and things like that run well on Windows Server 2008 R2? I could get a Technet Sub if I had to...

 

[Bozo]

RoboCopy has a 'mirror' switch (mir) that sounds like what you are looking for.
I've used it for copying 300GB to 600GB of files of all sizes without problems.

 

[LunarMist]

I use SmartSync to sync 10s of TB of local data between primary/backup drives and between two computers over the network. You can run multiple syncs simultaneously, which is nice.

 

[Stereodude]

I use SyncBack Pro. It can do what you're after (assuming the systems have mapped drives or UNC path access to each other).

 

[Howell]

I believe Beyond Compare can be configure with scheduled syncs. At least it is easy to do manual syncs though not free.

 

[Gilbo]

I think DFS on Windows Server may be the way to go.

Will I be able run Photoshop, Lightroom, etc. on Windows Server? Or will I have to build separate boxes?

 

[Mercutio]

I've been using Server editions of Windows on desktop machines for a long time. Yes, Adobe crap runs on Windows Server.

 

[LunarMist]

PS and LR are crap?

 

[Handruin]

Can you ever recall him saying anything positive about Adobe?

 

[LunarMist]

Of course not, but they should not all be lumped together. I abhor the awful Flash and I don't particularly like the Acrobat for example, but PS is not crap.

 

[Mercutio]

Of course not, but they should not all be lumped together. I abhor the awful Flash and I don't particularly like the Acrobat for example, but PS is not crap.

Even if I didn't think that the UI was total shit, I'd still thing Photoshop is worthless shit by association. And the UI is total shit that has unfortunately contaminated every other program with similar functionality. Fuck Adobe and everyone who works there or supports its products.

 

[ddrueding]

PS is the best, most powerful, most feature complete product in that space. I don't like many things about it (it's product activation scheme for students is awful), but there is a reason I use it every day.

 

[Mercutio]

I really can't even begin to care what you think you need it for. If you're paying for it, and I sincerely hope you aren't, you're supporting something that does not deserve to exist, let alone your money.

 

[LunarMist]

Even if I didn't think that the UI was total shit, I'd still thing Photoshop is worthless shit by association. And the UI is total shit that has unfortunately contaminated every other program with similar functionality. Fuck Adobe and everyone who works there or supports its products.

The newer versions have become increasingly difficult, but it's not like there are practical alternatives for serious work.

 

[Handruin]

I really can't even begin to care what you think you need it for. If you're paying for it, and I sincerely hope you aren't, you're supporting something that does not deserve to exist, let alone your money.

What alternative photo editing and graphic design tool is of equal or better alternative? If you even write gimp I'll sign you up for magazine subscriptions of male porn for a year.

 

[Handruin]

Dammit, I thought we were in the something random thread. How the hell did this file synch thread turn into an adobe sucks thread...

 

[ddrueding]

Dammit, I thought we were in the something random thread. How the hell did this file synch thread turn into an adobe sucks thread...

I've been using Server editions of Windows on desktop machines for a long time. Yes, Adobe crap runs on Windows Server.

 

[Mercutio]

What alternative photo editing and graphic design tool is of equal or better alternative? If you even write gimp I'll sign you up for magazine subscriptions of male porn for a year.

Hey, if you want a bunch of that stuff on YOUR credit card statement I'm not going to stop you. I'm sure I could find a nerd girl who would be happy to have it.

But anyway, there's a lot of power and potential in some other photo manipulation applications, like Paint.Net. The biggest problems those other guys have is that they are chasing the tampaxtastic Photoshop UI (I really, really hate how cluttered and busy Photoshop is), and the lack of dev mindshare that can lead to plugin development and improvement.

Like it or not, Gimp was an effort to think of photo manipulation outside the tiny little brain-box that photoshop puts its users into, and that is a reason to praise rather than curse it.

 

[time]

Gilbo, with regard to your question as the OP ...

PowerFolder might be worth a try. Although there's a cloud storage component, it's optional; you can use the product to sync directly between any two PCs without touching the cloud.

I've only given it a quick test drive so far. It's written in Java but importantly it's not from Microsoft, so I figure your chances of it coping with something serious will be greatly enhanced.

Howell's suggestion of Beyond Compare may work - it has FTP support for nested directory comparisons - but I've never used it across the internet for such a large number of files, so you would have to try it and see. Really good product though, no Microsoft tools involved.

 

[sechs]

Fuck Adobe and everyone who works there or supports its products.

Fuck you too, Merc.

Are we done yet?

 

[Mercutio]

Fuck you too, Merc.

Are we done yet?

Fuck you right back. And LM and dd and everyone else who support Adobe.

 

[ddrueding]

Making sure our year starts on the right foot?

 

[Mercutio]

I'm just making sure that I blame the right people for Adobe's continued existence.

 

[Handruin]

I'm just making sure that I blame the right people for Adobe's continued existence.

Now that you've blamed them and released vulgar words, has it made the difference you hoped to achieve?

 

[Mercutio]

It feels better every single time I type it.

 

[Handruin]

It feels better every single time I type it.

Let's summarize...You feel better by being insulting to people in a thread about file synchronization which has little to nothing to do with Adobe and accomplish absolutely nothing except to waste their time with childish insults just because they don't agree with you. Where's the constructiveness or usefulness in this except to appease yourself?

 

[Mercutio]

I already made my useful and on-topic contribution to the thread. Since Gilbo seems to have gotten his answer, it's as good a place as any to conduct a little Adobe-bashing.

 

[Handruin]

I already made my useful and on-topic contribution to the thread. Since Gilbo seems to have gotten his answer, it's as good a place as any to conduct a little Adobe-bashing.

Your hatred for an inanimate object extends to a personal level. It's no longer just a little Adobe-bashing.

 

[Mercutio]

All hatred is personal. Once you accept that, there's only the question of degree.