CSC - The largest unplanned outage in years and how we survived it - Blog Post

[Howell]

"A month ago [2/2016] CSC's high-performance computing services suffered the largest unplanned outage in years. In total approximately 1.7 petabytes and 850 million files were recovered."

https://csc.fi/web/blog/post/-/blogs/the-largest-unplanned-outage-in-years-and-how-we-survived-it

 

[CougTek]

This was a neat article about losing a lot of data and problems with restoring it: The largest unplanned outage in years and how we survived it

Handruin posted it in the "Something Random" thread at 12:26pm Eastern Time, although it might be more appropriate in this section of the forum.

I guess reading that users placed critical data in a storage space designated for temporary work really put a smile on anyone here administering a computer park.

 

[Handruin]

I debated if posting the story warranted it's own thread but I'm glad Howell started a new thread for it. I found it to be an interesting story with creative problem solving when it comes to very large restores.

 

[Mercutio]

There's some IT Operations superhero stuff going on there and I see that this is really a matter where their intended temp space got to be stable enough to be seen as permanent storage, but this issue was entirely created by users doing things they weren't supposed to be doing. It's temp space. If important data is in there, put it someplace you know is going to get backed up.

I'm a bit peeved that the end of the article mostly turned in to someone shilling for Slack. The solution to not having a single communications tool is not to add yet another communications platform.

 

[timwhit]

Not all communication tools are created equal. Slack is the best tool I've used.

 

[Handruin]

I've been interested to try Slack given the amount of press that surrounds it and people who praise it. I've been working during my free time to demo one of their open source competitors named MatterMost. I haven't gotten a setup working yet but I'd like to try it just for the sake of learning to understand the pros and cons of this kind of environment. If it works and has real benefit I may consider suggesting it to my company's engineering group.

 

[timwhit]

You can setup a free Slack team account. It has limitations on the number of messages that it will store, but it's a good way to test it out. If your team decides they like it, then you can convert your free account to a paid account and get access to all your messages.

Hipchat is alright too, it just lacks a lot of features that Slack has.

 

[Mercutio]

Not all communication tools are created equal. Slack is the best tool I've used.

I have a few problems with it as a concept. It's web based, which is fine and everything since at least it doesn't require another goddamned client, but it also makes every conversation available by virtue of being searchable. Which just means that some conversations will wind up "out of band" because somebody doesn't want to communicate on the record (I know there are direct messages, but they're analogous to email at that point). Which just pushes everything off onto another goddamned platform. AGAIN. And it's one more thing to back up. And it's one more thing to administer. And why the hell not just use Email, which already does the things that Slack does?

 

[timwhit]

I have a few problems with it as a concept. It's web based, which is fine and everything since at least it doesn't require another goddamned client, but it also makes every conversation available by virtue of being searchable. Which just means that some conversations will wind up "out of band" because somebody doesn't want to communicate on the record (I know there are direct messages, but they're analogous to email at that point). Which just pushes everything off onto another goddamned platform. AGAIN. And it's one more thing to back up. And it's one more thing to administer. And why the hell not just use Email, which already does the things that Slack does?

You can use Slack with the webclient, but there's also clients for OSX, Windows, Linux, Android, iOS, and even Windows Phone.

Slack is a hosted solution, so you don't have to worry about backup, administration, etc.

Email is a terrible solution for group chat or even direct communication. With Slack and tools like it I can see everything ever sent in a channel without having to search Outlook or Gmail. It's much easier to share files, code, etc. When you're working on a big distributed team, email just doesn't work.

 

[timwhit]

I have a few problems with it as a concept. It's web based, which is fine and everything since at least it doesn't require another goddamned client, but it also makes every conversation available by virtue of being searchable. Which just means that some conversations will wind up "out of band" because somebody doesn't want to communicate on the record (I know there are direct messages, but they're analogous to email at that point). Which just pushes everything off onto another goddamned platform. AGAIN. And it's one more thing to back up. And it's one more thing to administer. And why the hell not just use Email, which already does the things that Slack does?

You can use Slack with the webclient, but there's also clients for OSX, Windows, Linux, Android, iOS, and even Windows Phone.

Slack is a hosted solution, so you don't have to worry about backup, administration, etc.

Email is a terrible solution for group chat or even direct communication. With Slack and tools like it I can see everything ever sent in a channel without having to search Outlook or Gmail. It's much easier to share files, code, etc. When you're working on a big distributed team, email just doesn't work.

 

[Mercutio]

It looks like yet another IM client to me. If Email sucks or the tools that you're using for Email suck maybe the answer is to fix your habits and/or your client.