{"id":1060,"date":"2022-01-25T02:52:05","date_gmt":"2022-01-25T01:52:05","guid":{"rendered":"https:\/\/irg-net.com\/?p=1060"},"modified":"2024-02-07T10:50:36","modified_gmt":"2024-02-07T09:50:36","slug":"big-disaster-irg-net-datacenter","status":"publish","type":"post","link":"https:\/\/irg-net.com\/index.php\/big-disaster-irg-net-datacenter\/","title":{"rendered":"BIG DISASTER @ irgNET Datacenter"},"content":{"rendered":"\n

Well I think the title is telling a lot. Unfortunately, in my main server the RAID 5 array has drifted apart due to 2 defective HDDs.<\/p>\n\n\n\n

First I got the information from iLO Advanced, the OnBoard Management solution of HPE, that one hard disk failed. Then I went to Datacenter 01 and exchanged the defective disk for a new one. During the rebuild, another disk failed and my RAID 5 array shot apart. This resulted a data loss and looked like this in the ESXi Host: <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

As every company should have a functional backup, I could also rely myself on my backup. As a backup solution, I use Veeam, a fairly simple solution that runs as a VM on ESX 04. (Fortunately I thought)<\/p>\n\n\n\n

So I connected to my backup server via RDP and selected the restore function of the Veeam appliance. When I tried to select and restore the respective VMs from the backup, I got an error message that the backup repository was corrupt, which is a 2TB WD Black USB 3.0 disk. (see Home Lab Stage V, 5.0 U2<\/strong><\/a><\/em>)<\/p>\n\n\n\n

A total disaster, I not only lost my datastore but also my backup. <\/p>\n\n\n\n

What do we learn from this?<\/strong> Everyone should keep an eye on their backup environment and run restore tests every now and then, so that there are no surprises in moments like this.<\/p>\n\n\n\n

I am currently in the process of restoring my entire environment by hand. Fortunately, my blog and some other services were running here in Datacenter 02 and I lost the infrastructure services but not other applications such as; <\/p>\n\n\n\n