A Case Study

Once upon a time, a small group of people joined a club. This club was using a discussion board platform none of them had heard of but they signed up to its hosts, EzBoard, by setting up user accounts.

Then one day they fell out with the board's Administrator/ezOp and left to form their own club. Not knowing any better, they too set their board up on EzBoard. Having looked at EzBoard's different packages, they decided to go for the premium Gold Community package from almost the outset. This promised:

  • "powerful features"

  • "blistering ad-free performance"

  • "unparalleled support"

They delved further and found that the features of the Gold Community included:

  • "Weekly backups and free restores protect your board"

How good did that sound? They signed up.

The price for Gold Community status was calculated by ezboard based upon the amount of bandwidth the board consumed although how those statistics were derived was never revealed to the Administrators, just the overall daily totals. Deducted from those usage figures were the page visits of ezSupporters.

Of course, from time to time there were some serious glitches, usually when ezboard - as they re-branded themselves - decided to "upgrade" their service. There were often times when members couldn't access the board, getting instead a "Server maintenance alert" page. Other times, members would find themselves logged out by ezboard and would have to log in again, often many times a day.

But the members were loyal and stuck with the club and the club stuck with ezboard. The members would donate money by way of ezboard's Community Chest feature and the Administrators would spend hundreds of dollars every year for the Gold Status - in 2004, the renewal cost was $254. In order to keep the cost of renewal as low as possible, the Administrators and the Moderators paid an additional $7 to $12 to buy ezSupporter status.

Then on 31 May 2005, disaster struck!

ezboard initially claimed it was a hacker:

Content removed due to ezboard, Inc. threats
You may be able to view what was being quoted here:
http://p080.ezboard.com/fezboardfrm29.showMessage?topicID=1633.topic

On our board, we simply saw no posting or replying capabilities from about 10.00am BST to 4.30pm BST. We were able to read, post and reply that afternoon and evening and our board data was intact ... until 1 June 2005, when we realised that overnight and post the claimed attack, we had lost all our data from 31 May 2005 back to the end of April 2004, coincidentally around the same time that Rob Labatt took over as CEO and managed to get ezboard profitable... On the basis of the board's claimed stats., that amounted to some 70,000 posts by our members.

Note that at that point how Robert Labatt referred to his customers as the "ezboard Family". Thank goodness for those backup systems, eh? Still, 48 hours and our Gold board should be up and running at least with thanks to the backups. But wait! The attacker "deleted current board and back-up data." How could that be? Surely the backup should be secure? Most sources on the Internet agree that the backup should be kept at different location(s) from the original data.

1 June 2005

An update from ezboard:

Content removed due to ezboard, Inc. threats
You may be able to view what was being quoted here:
http://p080.ezboard.com/fezboardfrm29.showMessage?topicID=1633.topic

What is wrong with that picture?

"The attacker erased all historical post data on all boards and a significant amount of back-up data". Hmm. Surely if the data was backed up properly, any external attack could not have compromised them? Either that or the attacker was somehow able to use the Internet to log into ezboard's servers, delete the data, then leave their servers and mysteriously get into the data safes to delete the secured backup media. Wow! How l33t was the h4x0r!

"...we have back up data from the May 9th incremental back-up (we perform back-ups every day, but the attacker deleted many of them)". See above. Also, what of the promise by ezboard that: "Weekly backups and free restores protect your board". We weren't feeling too 'protected' at this stage but Labatt was stating categorically that they had "restored a significant number of boards and we are continuing to restore boards", that they had "restored data to many boards at this point ... and will be able to restore over half of all Gold Boards". Our hackles did rise at the way he was using an important announcement to spin the data loss by saying he was diverting resources to working on the next release of the ezboard software for a "vastly improved experience". He did, however, promise a daily noon update.

Another update the same day said:

Content removed due to ezboard, Inc. threats
You may be able to view what was being quoted here:
http://p080.ezboard.com/fezboardfrm29.showMessage?topicID=1633.topic

So we carried on using the board as directed.

2 June 2005

So, 48 hours into the data loss - remember this? - and ezboard post this update:

Content removed due to ezboard, Inc. threats
You may be able to view what was being quoted here:
http://p080.ezboard.com/fezboardfrm29.showMessage?topicID=1633.topic

So, after 48 hours, the new ETA is a further 24 to 48 hours - twice the original estimate. Whither the backups now?

3 June 2005

ezboard starts to realise that they have messed up:

Content removed due to ezboard, Inc. threats
You may be able to view what was being quoted here:
http://p080.ezboard.com/fezboardfrm29.showMessage?topicID=1632.topic

What is interesting here is what ezboard will and won't admit to. Let's break it down:

"it will be impossible to restore full data on all boards"
Well, to be fair to ezboard, it's only the Gold boards that were promised backups.

"Data restoration began the day of the attack"
That was good to hear.

"The process is automated and each server takes approximately 6 hours to restore. We can restore between 8 and 10 servers in parallel. To some of you this may seem like a long time to restore data on a single server. And it is. However, we are using an innovative process to maximize the data recovery and testing it thoroughly to make sure that (a) we don't erase the posts that have been made since the attack and (b) the data is being properly restored."
Bearing mind there were servers starting with p080 up to p209 and possibly many others, that's a lot of servers.

"With quick thinking and fast action on the part of your technical team we were able to do an emergency backup of active data."
w00t! Now all we need is the data from the backups...

"The attacker was successful in erasing historical board posts and the backup systems, which were password protected and on a separate network."
Excuse me? Let's put to one side the notion that backups should be made on removable media that is then stored securely to allow for disaster recovery. Instead, let's focus on the data being on "a separate network". How then did the attacker manage to log in remotely to one network and then magically hop onto another separate one without raising any alarms? He's a jolly clever chap... Either that or it's an inside job.

Of course, the cynic in me simply thinks it's ezboard messing about with their systems and making an almighty cock-up.

"What more can you tell me about the back-ups?
The issue was not the back-up systems themselves. Our back-up systems have been working well and are separate from our production environment. They are not accessible without inside technical information and passwords. The issue was that either through luck or (more likely) specific information, the attacker was able to delete both the historical files and the back-ups."

OK, so that narrows your enquiries down, but I'm sorry, the issue most certainly is the backup system itself.

"Since the attack we have put out five global messages and have had two conference calls with ezOps and admins."
Funny, but neither our ezOp or any of us Admins. was told about any conference calls.

"I would anticipate further news on credits to be available in the next week."
Well at least that might be some small comfort, but never mind: we've still got the restoral process to fall back on.

"...we do have a back up of that data from a few days before the attack. We need to recalculate the actual Community Chest balances by taking the balance as of the date of the backup, adding Community Chest contributions and subtracting payments from Community Chest. We hope you agree that restoring Community Chest data is a second priority to getting the board posts restored"
This is good news and means we'll be able to keep tabs on the $566 that our members had contributed and was in our Community Chest, even if ezboard were now reporting it as $800+.

4 June 2005

Well, actually, the 5th: the noon announcement was made at just before 1.00am the following morning:

Content removed due to ezboard, Inc. threats
You may be able to view what was being quoted here:
http://p080.ezboard.com/fezboardfrm29.showMessage?topicID=1648.topic

Good to see that ezboard decided not to work 9-5 when the whole of their network is in meltdown. Such dedication...

"We were able to increase the number of server restorations to more than ten at a time."
Excellent news - that should mean the restoration process is completed more quickly.

"Talking time to generate timelines and debate the nuance of a changing and complex restoral process will only delay the completion of the restores. Given this thinking it is my hope that you agree with our decision to provide you with regular updates and an announcement when the restoration is complete."
Or
"We haven't got a clue frankly about how long this restoration will take so we've decided just to keep crashing on regardless."

"If a thread was not replied to in the time between the attack and when the server your board is on was rebooted, then the information in that thread is most likely permanently lost. We released a fix for this on Thursday last week. It should no longer be an issue, but you may just be seeing the effects of this now."
But surely if only data was lost, then there shouldn't still be an issue with continuing to lose data? The hacker's long gone now ... isn't he? A fix? For what?

"I apologize for not getting the noon update out to you today. We made security changes to the site that made it impossible for the Global Announcements to go out for a few hours."
Or
"We managed to lock ourselves out of the system and it took us 12 hours to work out how."

 

To be continued.... {last updated 04 Oct 2006}

16 February 2006

ezboard, Inc. have threatened to take action because they claim breach of copyright in their statements previously reproduced above. Whilst we claim their use is allowable as comment or review, we have removed the 'offending' elements as requested. If the comments now appear out of context then so be it: it was their choice to demand the original elements' removal.

14 March 2006

Oh dear...

ezboard’s CEO has stated that - contrary to what its customers were being told by its Customer Services [sic]/Help [sic] Forum staff - all ezboards will have to move to Yuku when it’s “ready for prime time” (a curious expression that, especially for a supposedly market leading message board company).

The message from the ezboard CEO - whose wife is or was CFO at the venture capitalist that supported it a few years ago - is here: http://www.ezboard.com/component/option,com_mamblog/Itemid,55/task,show/action,user/id,69/ and/or aliased here: http://www.ezboard.com/labatt.blog

Except that that very important announcement wasn’t actually announced! ezboard has a system whereby important news used to be propagated as a “Global Announcement”, so that the ezOps who operate ezboard boards would see them when they visited their boards and be able to choose whether or not to show all the message board visitors. It wasn’t even announced in the help forums, but slipped in by one of their staff in one of their minor forums. See for yourself - take a look here and see if you can find anything: http://p080.ezboard.com/fezboardimportantannouncements

That contains such earth-shattering things as suggesting that ezboard users upgrade their browsers. Woohoo!

And yet, when it comes to its customers being forced to change their message boards or communities from one platform to another, it chooses to keep this quiet from the vast majority of its users, most of whom will never even look at the help [sic] forums or the main ezboard web site. Why?

Maybe it’s to ensure that ezboard, Inc. continues to gain an income stream from its ezSupporter feature? ezboard operates its free boards on the basis of advertising revenue. Knowing that people don’t like advertisements all over the place, they presently offer two ways to for users and board ‘owners’ to operate ad-free:

  • Gold Communities, where the board operators pay to have the adverts removed and are charged on the basis of (unverifable by the ezOps) page views;
  • and ezSupporter where the user pays for a subscription and doesn’t see adverts on any ezboard.

In the latter case, visits by ezSupporters to Gold Communities are claimed to be deducted from the overall page views when ezboard calculates how much the annual or six-monthly cost will be so it’s always been the case that Gold Community ezOps are urged either to activate their Community Chest (members can make payments to ezboard that can then be used to fund the Gold status for that board and nothing else, so they cannot be paid out to the ezOps if the boards move away from ezboard) or to solicit their members to subscribe as individuals to ezSupporter.

But if you read Labatt’s blog entry, he says that in order to remain ad-free, existing ezboard Gold Communities “will need to have a minimum of 6 months left on their subscription when they move”. So tough luck if you renew your Gold subscription and then have the move over to Yuku foist upon you the following week, for instance, as you’ll lose that benefit (and one that isn’t exactly cheap for a busy board).

Likewise, if you renew for a year’s ezSupporter and the boards you visit transfer or are transferred over to Yuku, you’ll then see adverts on them because the ezSupporter programme isn’t being carried over to Yuku.

Cynical? Maybe, but then ezboard promised Gold Communities weekly backups as part of our paid subscription and yet when push came to shove and 70,000 or so messages on our ezboard alone were lost (according to ezboard’s average stats. for it) the backups weren’t available. A year’s messages and threads gone. ezboard’s answer to criticism (apart from banning those critical of them) was that the service was provided “as-is”, so tough: they were only transitory messages anyway.

Oh and one of their support/customer service [sic] staff has posted a message saying that she thought it “hilarious” that people who’d been saying that ezboard wasn’t working properly - it isn’t, by ezboard’s own admission - were now complaining that “if it ain’t broke, don’t fix it”. I’m sure it’s “hilarious” that people don’t like change - one reason why people stay with ezboard - or that they’re not impressed by a new system that may well cost them more, will serve ads. at them and presently at least causes regular browser crashes and is slow as swimming in treacle.

 

In the meantime, you could always try some of these services: we have, to make sure our web sites and discussion forums work, unlike some other players in the market...

DreamHost - loads of features and cheap too


Here are some other ads. to help us make even more money. We've probably not used these advertisers' goods or services ourselves before, so we don't vouch for them.