Cloud Backup

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
What are folks using for cloud back-up?

I've used Crashplan for some years, but they've been going into the pooper at an accelerating rate. They're hamstrung by their Java-based client, and the servers have been getting slower and slower. I need to start looking for a replacement well in advance of my current subscription ending.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
Crashplan still suites me fine. Them having a java client works out good for me since I use it on Linux as well as Windows. My biggest complaint would be that it needs a decent amount of RAM when dealing with a lot of files as well as a few other configuration changes. I disable their dedup as much as possible to reduce RAM consumption. Their servers have been slow but have you tried reaching out to them to see if maybe they can migrate you? With Crashplan, I set my local server as one of the crashplan servers to get a quick backup and then their cloud service for my remote data. Works pretty well.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,297
Location
I am omnipresent
You could potentially roll your own thing with btsync or owncloud, but I've found that btsync doesn't like huge (multi-terabyte) backup jobs. I did try Acronis, since it was "Free" with my annual license, but I found it to be even slower for uploading data than Crashplan. I've been tempted to try Backblaze, but it doesn't retain old file versions indefinitely, which is a bigger selling point for me than I might've suspected a couple years ago, as I've recently been asked to dig up weekly copies of someone's accounting data over a two-year period. Mozy's fairly small cap on backup size takes it entirely out of the running in my opinion.
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
Crashplan still suites me fine. Them having a java client works out good for me since I use it on Linux as well as Windows. My biggest complaint would be that it needs a decent amount of RAM when dealing with a lot of files as well as a few other configuration changes. I disable their dedup as much as possible to reduce RAM consumption. Their servers have been slow but have you tried reaching out to them to see if maybe they can migrate you? With Crashplan, I set my local server as one of the crashplan servers to get a quick backup and then their cloud service for my remote data. Works pretty well.
I have plenty of memory and processing power, but their dedup doesn't seem to work right. About a month ago, there was an upgrade, and the system suddenly decided that it couldn't match any of my files with their existing back-ups and started over. It'll take about six months for it to catch up on the 7TB+.

I've been migrated twice. I'm now sending to their Seattle facility, which I think was new when I was moved there. It was better at the beginning, but, recently, it's been randomly dropping connections during resync, which makes them take days instead of hours. Also, the client has been randomly corrupting its own cache, which requires me to manually fix and triggers a resync.

I recently needed to restore some files. They have a 500MB limit via the web, which forced me to use the client. It failed on the download three times.

Their support has become garbage and just tells me to jack around with my settings. This is never a long-term fix the system seems incapable of handling the ever-growing amount of data. They handed out "unlimited" subscriptions in the early days, but unlike the cell phone companies, they haven't figured out how to throttle it.
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
I've been tempted to try Backblaze, but it doesn't retain old file versions indefinitely, which is a bigger selling point for me than I might've suspected a couple years ago, as I've recently been asked to dig up weekly copies of someone's accounting data over a two-year period.
I was looking at Backblaze, and it does seem like they have their stuff together.

That they don't keep versions indefinitely is a negative but, in my experience, I rarely need more than a few months back. I have tiers of back-up that covers everything but serious user error and catastrophic loss. So, I'm less concerned about that than never having a complete back-up.
 

Howell

Storage? I am Storage!
Joined
Feb 24, 2003
Messages
4,740
Location
Chattanooga, TN
About a month ago, there was an upgrade, and the system suddenly decided that it couldn't match any of my files with their existing back-ups and started over. It'll take about six months for it to catch up on the 7TB+.

That has happened to me at my own destination when the service comes up and can't see the files. Say, if the drive mount failed. Not saying this is what happened to you as I don't know what their back end storage looks like. I'd say it might be worth a call but maybe not of their service is as bad as you say.
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
That has happened to me at my own destination when the service comes up and can't see the files. Say, if the drive mount failed. Not saying this is what happened to you as I don't know what their back end storage looks like. I'd say it might be worth a call but maybe not of their service is as bad as you say.
I've had a ticket open with them for about a month, and all they tell me to do is change settings. It's clear that I've simply gotten to the point that the client and server can't keep track of all of the files and versions that I backed-up over the years.

I was recently told to remove some files from my back-up list, which means I permanently lose any backed-up versions. I don't even know why they'd suggest it, especially since they didn't mention that result.
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
FYI, I made every settings change that Crashplan support suggested, and it still didn't fix the problem. So, they had me deauthorize my client so that they could off-line "maintain" my archive on their servers. I figured that it'd take a couple days. One week in, they told me that it'll take three weeks to "maintain" my 10TB archive.

Anybody have any experience with BackBlaze? Considering taking in their two-week trial while I wait....
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
FYI, I made every settings change that Crashplan support suggested, and it still didn't fix the problem. So, they had me deauthorize my client so that they could off-line "maintain" my archive on their servers. I figured that it'd take a couple days. One week in, they told me that it'll take three weeks to "maintain" my 10TB archive.

Anybody have any experience with BackBlaze? Considering taking in their two-week trial while I wait....

Wow, 3 weeks is a long time to do an offline maintenance. My Crashplan client estimates another 8.8 months until the remainder of my data is synched with their servers. I'm 10.8TB out of 16.3TB uploaded. My local crashplan server is 100% but I see what you mean about the slow progress.

Last I remember trying BackBlaze a couple years ago they didn't have all the features I wanted and I believe their client was a bit hands-off in terms of what it selected to backup. That may have changed since I last used their service. I'll be curious to know what your results are if you decide to try them for their two-week trial.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
I found a suggestion for cloud backups for those of us with fairly large data sets. Several recommendations I've read was to use Arq backup with Amazon Cloud Drive. The Arq backup software would be one time fee of $49.99 per user (with an option for lifetime updates for an additional $29.99) and the Amazon Cloud Drive would cost you $59.99 a year for unlimited space. Both Arq and Amazon offer a limit-time 30-day trial if you want to try it out and see what kind of upload speeds you might get. I'm considering giving it a try just to understand how it works and to see if it's worth switching to. I've read small reports of people claiming they can max out their 100Mb internet connection backing up to ACD but take that with a grain of internet salt. The one down side is that Arq only seems to have a client for Windows and Mac right now. It would be nice if they had one for Linux.

This Arq backup software also gives you the option to pick different back-end cloud providers if you want to pay for something like Amazon S3, Google nearline, etc. They seem to support versioning and there is no limitation of time if you disconnect a system from the backup, you just need to continue paying Amazon for the cloud space. I've not tried this yet so I don't know if it'll work but if it does it might be worth the small amount of extra money for the Arq backup software.
 

Howell

Storage? I am Storage!
Joined
Feb 24, 2003
Messages
4,740
Location
Chattanooga, TN
Based on our corporate research of Amazon as backup destination, pay attention to which storage solution you are sending to, and the cost of getting it back from them.

IIRC, Glacier is the slowest and cheapest and you could migrate to it from their other product after the bulk of your data was up.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
Based on our corporate research of Amazon as backup destination, pay attention to which storage solution you are sending to, and the cost of getting it back from them.

IIRC, Glacier is the slowest and cheapest and you could migrate to it from their other product after the bulk of your data was up.

I agree, I only meant to point out this offering allows for more flexibility in storage options. For most of us the unlimited ACD might suffice in most cases. I'd be careful with Glacier due to the cost of retrieving the data in the event it's needed but more to your point that understanding the data structure is key before choosing the solution. Arq has a rough chart of storage costs here but it's not the complete picture.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
I'd be interested in seeing what the memory footprint on Arq is like. Crashplan is a pig in that respect.

Me too; I agree Crashplan is a pig with memory. I'll see if I can get it setup tonight after work and give it a try.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
I've downloaded the demo of Arq and I've connected it to my Amazon Cloud Drive. I get 5GB of space with my Prime account so I'm testing the backups with that. I'm syncing up about 2.5 GB of videos that are located on an SSD and I'm seeing about 35-40Mb/ps upload speeds so far (compared to Crashplan which is around 0.9Mb/ps). Their software is hovering between 255MB and 400MB during the backup. I may not have a large enough data set to check how hungry it might be.

So far the Arq utility is pretty basic with a no-frills UI. What I mean by this is that there is no visible progress bar to watch the status of your backups. I have to watch upload rates based on my PC and router because there's nothing built in to show this in the software. The UI doesn't seem to have the ability to minimize into the system tray which is annoying. I'm unclear how the scheduling would even work if I close it. I don't believe there is any kind of background service.

The data backups are scheduled by clicking the option to add a folder to backup. I am able to select any file or folder and there is the ability to exclude file types based on extension which is helpful. Their utility creates a directory on my ACD with the name Arq Backup Data and inside there are lots of folders and small files which must be the encrypted structure of their backup sets.

During my testing the UI proved to be a little buggy. I made a change to my only defined backup to set a budget of 5GB and after saving the change it aborted my upload and reported two errors even though I was almost done. Now it has restarted but it doesn't seem to be sending much data but rather it looks to be syncing the content which is a good sign.

After the sync completed, I used their restore abilities and I downloaded one of the three videos which weighed in around 305MB in size. The restore speeds seem to be a bit slower than the upload which is interesting. I'm seeing between 10-20Mb/ps on the download with ocassional pauses in bandwidth which may be the result of piecing the file back together? The CPU usage is around 1% and the memory usage around 50-70MB which is small. The file restored fine and I was able to play the video without any issues.

Overall this may make for an interesting utility for backing up data into an encrypted format. Given the version of the software is labeled as v5.1.23 I'm a bit surprised at how much it feels like "my first windows utility" given the price.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,297
Location
I am omnipresent
Hrm. I misread the cost for Glacier storage. I thought there was another decimal place. Even ignoring my vast collection of media files, I'd be looking at about $1000/year to keep things my personal data there. Suddenly running buying an extra Crashplan Pro subscription doesn't seem so bad.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
Hrm. I misread the cost for Glacier storage. I thought there was another decimal place. Even ignoring my vast collection of media files, I'd be looking at about $1000/year to keep things my personal data there. Suddenly running buying an extra Crashplan Pro subscription doesn't seem so bad.

Why use Glacier when you can use the unlimited Amazon Cloud Drive for $60/year?
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
For whatever it's worth there does appear to be a Arq agent/service installed and running in the background independent from the GUI.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,297
Location
I am omnipresent
Why use Glacier when you can use the unlimited Amazon Cloud Drive for $60/year?

That's a particularly interesting question since I do appear to have Cloud Drive Unlimited (not just Unlimited Photos). I don't remember paying for it but it looks like I'm cool until at least next February.
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
Any update on how Arq is working? Have you tested the versioning?

I just read your report, but I also currently have unlimited Amazon Cloud Drive on both my personal and business accounts. Certainly worth looking into.
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
In case anybody cares, received a message from Crashplan that the "first round" of maintenance has now completed after three weeks. A "second round" should take less time. So, I'll probably be down a full month before this is complete.

Also downloaded the Arq demo. It seems to upload pretty quick. Other than a clunky interface, my only complaint is that it doesn't appear to track changes as they happen. It just rescans all of folders I've chosen for back-up and uploads the changed files. Once I've added all of my files, it'll probably never be in a state of not scanning, and it will spend a lot of time looking over unchanged files and not uploading the changed ones.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,297
Location
I am omnipresent
What I'm not seeing when I look at Syncovery is any indication that it stores files in a backup format. It looks more like a front-end to rsync. I need to sit down and play with this stuff more than I have.
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
I think that rsync is just one of the protocols that it can use.

I haven't looked at it yet, but it does seem more geared towards syncing than backing-up.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
I managed to get this Dokan (FUSE for Windows) project working well enough to upload at about 10Mbit from what appears on my PC as a mounted drive instead of Amazon's lame-ass software. It's not perfect but Amazon's software is also terrible and I don't want to buy $300 worth of ExpandDrive or Arq licences, either.

Nice find with the ACD Dokan on github. I'd love to find a way to fuse the incremental utilities from Crashplan to backend storage such as ACD. I wonder if there is some code that can be leveraged in this project to make this work...hmm.
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
If I understand correctly, this just makes ACD appear as drive?

How reliable is it?
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,297
Location
I am omnipresent
If I understand correctly, this just makes ACD appear as drive?

I haven't seen it disconnect yet, if that's what you're asking. It takes a minute for Explorer windows to populate, but having the ability to address the storage outside a web client makes it vastly more useful to me. I'm in the process of moving over 4TB of FLAC files and mirroring ~400GB of Google Drive stuff on it. It's going to take a while no matter what I do, although I'm contemplating running the upload from two or three locations just to speed it up a bit.
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
Merc, I'm sure that you'll stress it far more than I would.

Any suggestions on systems for syncing files?
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
Preferably perfect synchronization with version back-up.

However, something far more reasonable will do.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,297
Location
I am omnipresent
I mean, you could rsync or robocopy every two hours and probably get pretty close for any normal use case.

edit: Also, one of Windows, the FS driver or Amazon Cloud has a problem with long file paths. God. Dammit.
 

sechs

Storage? I am Storage!
Joined
Feb 1, 2003
Messages
4,709
Location
Left Coast
I'm really more concerned about versioning. 90%+ of my files don't change.

I haven't had a chance to install. How long is long?
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,297
Location
I am omnipresent
255 characters, apparently. Some Windows applications don't have a problem with it and some do. It's an inconsistency I really wish somebody would fix.
 

Chewy509

Wotty wot wot.
Joined
Nov 8, 2006
Messages
3,358
Location
Gold Coast Hinterland, Australia
255 characters, apparently. Some Windows applications don't have a problem with it and some do. It's an inconsistency I really wish somebody would fix.
Easy...

Enable long paths in Windows. ( http://betanews.com/2016/05/29/long-paths-windows-10/ )
Update/reprogram all your applications so they solely uses the Unicode version of the Windows32 File API, and uses the UNC filename convention.
And mark the application's manifest as being long filename aware. ( https://msdn.microsoft.com/en-au/library/windows/desktop/aa365247(v=vs.85).aspx )

Actually, that doesn't appear so easy... sorry...

PS. The last link gives you all the details, but basically:
MAX_PATH (maximum filename/path length) is limited to 260 characters. (defined way back in NT 3.x days).
Because backwards compatibility is paramount, MS never updated the limit due to fear of breaking heaps of applications.
Windows 10 allows a reg-key to up the limit from 260 to 32767, but it may/will break some applications.
 

Mercutio

Fatwah on Western Digital
Joined
Jan 17, 2002
Messages
22,297
Location
I am omnipresent
I've read up on it. In a lot of cases the easiest way for me to deal with it is to handle my file manipulation with shell scripts, but here's a case where I'm having to work with something external to my control. I can't imagine that Amazon Cloud has arbitrary path length limits, but both the Amazon Cloud Uploader and the Dokan-based tool I've been using seem to puke on the same files, so clearly something does.
 

Handruin

Administrator
Joined
Jan 13, 2002
Messages
13,931
Location
USA
I've been following a few other communities seeking out a solution to the same problem with Crashplan that you've brought up. I'm growing more concerned at the slow performance of Crashplan the more I see and read of others having issues.

This utility called Syncovery was recommended as a possible solution. I've not tried it yet but I'm going to give the demo a try now. It's also another paid solution for the utility and then you have to provide your own storage solution in the cloud so the cost obviously increases. It looks to have incremental with encryption support. They don't explicitly say ACD is supported (that I could find) but other Amazon cloud is supported. Edit: I should also note they have clients for Windows, Linux, and Mac which is a bonus.
Edit 2: This does appear to support ACD.

Another option recommended to me by people at work was Duplicity. I've also not tried this one yet but I may give it a try to see how well it works. I'll try to report back on what I find once I give these enough time to understand their capabilities.
 
Top