File synchronization options for large amounts of files

Gilbo

Storage is cool
Joined
Aug 19, 2004
Messages
742
Location
Ottawa, ON
Hey guys, I've spent the last 3 months trying to get two computers to sync a large directory of photos (about 1.8 TB).

I just can't make it work. There have been BSOD errors and mysterious reboots, and it seems like Live Mesh just can't handle comparing that amount of data. I can never get out of the initial synchronization stage.



Are there any other options for synchronization that will let me keep a large directory like this in sync?

I could just create FTP servers on each and manage it manually, but I find things get out of sync when I get busy and it's a huge PITA to fix it afterwards. I was really hoping Live Mesh would fix this, but it just doesn't seem stable enough :( .

There must be tools that do this. I can't believe in this day and age that Live Mesh is the only thing that'll work for me. (Dropbox, SugarSync, etc. all require cloud storage. I don't want to pay out the nose for 1.8TB of cloud storage!)
 

Gilbo

Storage is cool
Joined
Aug 19, 2004
Messages
742
Location
Ottawa, ON
Rsync isn't really what I need. It's a 1-way solution, that has to be scheduled.

I need a 2-way (or n-way) solution like Windows Live Mesh, that works in real time, so when I go home or go to the office or open my laptop all my files are on each computer, and are up to date.

In this day and age this is still almost impossible so far as I can tell if you have more than a couple GB of data and can't use Dropbox or SugarSync :( ...
 

ddrueding

Fixture
Joined
Feb 4, 2002
Messages
19,511
Location
Horsens, Denmark
Just to clarify, these two machines aren't on the same LAN are they? If so, SyncToy can easily do what you ask. I'm not sure how efficient it is with network resources, or I would suggest running it over a VPN.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
21,564
Location
I am omnipresent
Synctoy is hugely inefficient for large numbers of files. It also generates a tremendous number of very small (1 byte? 1kB? Don't remember) temp files that can create a large overhead in the filesystem and also annoy the piss out of you when a sync fails.
 

Gilbo

Storage is cool
Joined
Aug 19, 2004
Messages
742
Location
Ottawa, ON

Unison doesn't work with Windows 7 :( . (Also it's PITA to setup...)

Just to clarify, these two machines aren't on the same LAN are they? If so, SyncToy can easily do what you ask. I'm not sure how efficient it is with network resources, or I would suggest running it over a VPN.


Ya different networks. I want to sync over the internet.

Even though it's a large amount of data, the data doesn't change that much. At most about 30 GB / month. It's just that Live Mesh can't seem to do the initial sync, even if all the data is already copied over and all it has to do is verify everything matches.



My googling brought me to this page, which recommends Allway Sync, and FreeFileSync. Sigh. Both of which don't necessarily appear polished. I'm wary of trusting all my photos and other data to random programs.

I'm really surprised Microsoft's Live Mesh doesn't work. They specifically mentioned software updates 6-12 months ago that they claimed would allow it to work with a near-unlimited number of files. I'm going to file a bug report. Maybe they can help me...
 

Gilbo

Storage is cool
Joined
Aug 19, 2004
Messages
742
Location
Ottawa, ON
Oh, and this is for 2 Windows 7 Professional computers, so I can't use the Windows Server option you mentioned Merc...

Will Photoshop and things like that run well on Windows Server 2008 R2? I could get a Technet Sub if I had to...
 

Bozo

Storage? I am Storage!
Joined
Feb 12, 2002
Messages
4,396
Location
Twilight Zone
RoboCopy has a 'mirror' switch (mir) that sounds like what you are looking for.
I've used it for copying 300GB to 600GB of files of all sizes without problems.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
16,625
Location
USA
I use SmartSync to sync 10s of TB of local data between primary/backup drives and between two computers over the network. You can run multiple syncs simultaneously, which is nice.
 

Stereodude

Not really a
Joined
Jan 22, 2002
Messages
10,865
Location
Michigan
I use SyncBack Pro. It can do what you're after (assuming the systems have mapped drives or UNC path access to each other).
 

Howell

Storage? I am Storage!
Joined
Feb 24, 2003
Messages
4,740
Location
Chattanooga, TN
I believe Beyond Compare can be configure with scheduled syncs. At least it is easy to do manual syncs though not free.
 

Gilbo

Storage is cool
Joined
Aug 19, 2004
Messages
742
Location
Ottawa, ON
I think DFS on Windows Server may be the way to go.

Will I be able run Photoshop, Lightroom, etc. on Windows Server? Or will I have to build separate boxes?
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
21,564
Location
I am omnipresent
I've been using Server editions of Windows on desktop machines for a long time. Yes, Adobe crap runs on Windows Server.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
16,625
Location
USA
Of course not, but they should not all be lumped together. I abhor the awful Flash and I don't particularly like the Acrobat for example, but PS is not crap.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
21,564
Location
I am omnipresent
Of course not, but they should not all be lumped together. I abhor the awful Flash and I don't particularly like the Acrobat for example, but PS is not crap.

Even if I didn't think that the UI was total shit, I'd still thing Photoshop is worthless shit by association. And the UI is total shit that has unfortunately contaminated every other program with similar functionality. Fuck Adobe and everyone who works there or supports its products.
 

ddrueding

Fixture
Joined
Feb 4, 2002
Messages
19,511
Location
Horsens, Denmark
PS is the best, most powerful, most feature complete product in that space. I don't like many things about it (it's product activation scheme for students is awful), but there is a reason I use it every day.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
21,564
Location
I am omnipresent
I really can't even begin to care what you think you need it for. If you're paying for it, and I sincerely hope you aren't, you're supporting something that does not deserve to exist, let alone your money.
 

LunarMist

I can't believe I'm a Fixture
Joined
Feb 1, 2003
Messages
16,625
Location
USA
Even if I didn't think that the UI was total shit, I'd still thing Photoshop is worthless shit by association. And the UI is total shit that has unfortunately contaminated every other program with similar functionality. Fuck Adobe and everyone who works there or supports its products.

The newer versions have become increasingly difficult, but it's not like there are practical alternatives for serious work.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
I really can't even begin to care what you think you need it for. If you're paying for it, and I sincerely hope you aren't, you're supporting something that does not deserve to exist, let alone your money.

What alternative photo editing and graphic design tool is of equal or better alternative? If you even write gimp I'll sign you up for magazine subscriptions of male porn for a year.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
Dammit, I thought we were in the something random thread. How the hell did this file synch thread turn into an adobe sucks thread...:eek:
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
21,564
Location
I am omnipresent
What alternative photo editing and graphic design tool is of equal or better alternative? If you even write gimp I'll sign you up for magazine subscriptions of male porn for a year.

Hey, if you want a bunch of that stuff on YOUR credit card statement I'm not going to stop you. I'm sure I could find a nerd girl who would be happy to have it. :p

But anyway, there's a lot of power and potential in some other photo manipulation applications, like Paint.Net. The biggest problems those other guys have is that they are chasing the tampaxtastic Photoshop UI (I really, really hate how cluttered and busy Photoshop is), and the lack of dev mindshare that can lead to plugin development and improvement.

Like it or not, Gimp was an effort to think of photo manipulation outside the tiny little brain-box that photoshop puts its users into, and that is a reason to praise rather than curse it.
 

time

Storage? I am Storage!
Joined
Jan 18, 2002
Messages
4,932
Location
Brisbane, Oz
Gilbo, with regard to your question as the OP ...

PowerFolder might be worth a try. Although there's a cloud storage component, it's optional; you can use the product to sync directly between any two PCs without touching the cloud.

I've only given it a quick test drive so far. It's written in Java but importantly it's not from Microsoft, so I figure your chances of it coping with something serious will be greatly enhanced.

Howell's suggestion of Beyond Compare may work - it has FTP support for nested directory comparisons - but I've never used it across the internet for such a large number of files, so you would have to try it and see. Really good product though, no Microsoft tools involved.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
It feels better every single time I type it.

Let's summarize...You feel better by being insulting to people in a thread about file synchronization which has little to nothing to do with Adobe and accomplish absolutely nothing except to waste their time with childish insults just because they don't agree with you. Where's the constructiveness or usefulness in this except to appease yourself?
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
21,564
Location
I am omnipresent
I already made my useful and on-topic contribution to the thread. Since Gilbo seems to have gotten his answer, it's as good a place as any to conduct a little Adobe-bashing.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,737
Location
USA
I already made my useful and on-topic contribution to the thread. Since Gilbo seems to have gotten his answer, it's as good a place as any to conduct a little Adobe-bashing.

Your hatred for an inanimate object extends to a personal level. It's no longer just a little Adobe-bashing.
 
Top