As many of you noticed, AnandTech has spent several hours offline today. We are still in recovery mode at the moment (as I write this, the site has been restored to a copy from November 25th), but now that our major restoration efforts are completed, I wanted to offer you guys a brief update on the status of AnandTech.
At around 13:00 UTC (5am PT) today, the on-site cloud storage for AnandTech’s hosting provider became corrupted. As a result, AnandTech (and some other sites) were brought offline. Due to the nature of the corruption and the need to begin restoration efforts ASAP, we opted to restore the site from an off-site cold storage backup, rather than trusting the questionable on-site storage.
This is the first time we’ve ever had to execute our off-site data recovery plan before. And while it meant AT took a bit longer to restore than would be ideal, ultimately everything worked out and proved the necessity for off-site backups.
We’re still working to restore content from the last few days. Articles will be back, but we’ve likely lost any comments and user account registrations/updates made since midday Friday. Sorry about that! And thank you for bearing with us during today's outage.
Facebook
Twitter
RSS
31 Comments
View All Comments
ballsystemlord - Thursday, December 1, 2022 - link
I re-entered my latest comments. The others should restore fine with the rest of the site.Thanks for telling us.
GreenReaper - Thursday, December 1, 2022 - link
Someone must've grabbed a Cyber Monday deal a year or two ago and forgot to renew...Sorry you've had such a stressful start to December, hope you have a good rest of the month and end up with a nice present under the tree to make up for it.
linuxgeex - Thursday, December 1, 2022 - link
@Ryan Smith: Time to add a witness log to your off-site recovery plan, so that you can replay the incremental changes between those backups.at_clucks - Friday, December 2, 2022 - link
Also a good time to plan for testing the procedures periodically (even yearly). No point in taking backups if you don't know if the restore works, how long it takes, how to do it in a safe, effective manner, etc. Once per year do a restore on a separate site and you get to see what goes wrong and put it in the procedure.Threska - Thursday, December 1, 2022 - link
" Articles will be back, but we’ve likely lost any comments and user account registrations/updates made since midday Friday."Time to hack our caches and get them back.
Silver5urfer - Thursday, December 1, 2022 - link
Good to hear, thanks for explanation. I was worried on what could have happened.GeoffreyA - Friday, December 2, 2022 - link
Thanks. Glad the recovery plan worked. Maybe this could be an opportunity for an article on backups.Threska - Friday, December 2, 2022 - link
Connect storage to the torrent networks and give everything saucy names. Soon you'll be backed up.GeoffreyA - Friday, December 2, 2022 - link
Indeed, little chance of failure there!James5mith - Friday, December 2, 2022 - link
Onsite cloud storage?Can you elaborate on what that means?