The Road to Recovery - How Databarracks Assisted WTB with Massive Data Loss
13th January 2011
Back to Press Releases
WTB Group Limited is the UK's leading supplier of civil
engineering and building materials to the infrastructure,
environmental and industrial markets. With 75 years of experience
and having grown to produce an annual turnover in excess of £320
million, WTB is definitely a force in their industry. WTB employs
cutting edge technology in their warehouses and utilizes integrated
stock management control systems to coordinate the logistics
involved in offering a product range of over 18,000 lines. Depots
where customers can choose products in a more traditional shopping
atmosphere are available in over 50 locations around the UK and
Ireland, but a very large contributor to WTB's reputation as the
pinnacle in customer service can be attributed to the online
enquiry management system they employ. The Enquiry List Service
allows customers to select products they are interested in and
provides communication via email with their local WTB which
supplies price and availability information on the items in
question.
Email communication is of the utmost importance to WTB as a
means of maintaining the high standard of Customer Service and
flexibility they offer. Much communication involving critical
details is done between employees and their customers as well as
with suppliers. In March of 2007, WTB decided to use Databarracks
to provide fully managed online backup services in order to
safeguard nearly 3 Terabytes of their valuable emails, databases,
and files on 21 servers. Databarracks employs Asigra software to
safeguard WTB's data. The functions of the software that are
currently utilised for the account are the local storage,
archiving, and backup lifecycle management capabilities.
"Blue Screens of Death"
On Wednesday, 23 January 2008, Disaster struck at WTB. A SAN
with one Terabyte of data and eight virtual servers experienced
massive problems. The virtual management logs revealed that at
approximately 6:00am, a drive failed; then when it attempted to
recover itself, the SAN suffered corruption across the volume
group. Initially, IBM was contacted to come in and attempt to
repair the damage, however, it was determined that there was no
solution at hand that would not result in severe and permanent data
loss. Tim Brice, IT Systems Manager for WTB recalls, "The situation
continued to worsen throughout the morning, with blue screens of
death appearing on several virtual servers hosted on the SAN and
key databases losing data and becoming corrupted. In the end we
made the decision to rebuild the SAN volume group, rebuild the
virtual machines and restore data and configurations from
backup."
At approximately 12 noon, restoration efforts and the rebuilding
of machines began. That afternoon, Databarracks was contacted for
help. Within four hours, two of Databarracks most senior
engineers were on site with data necessary for recovery which they
had collected from the Data Centre en-route. The engineers stayed
through the night, doing everything in their power to assist with
the recovery.
"I was amazed at the service we received, as the engineers were
wonderful! They acted professionally and were in good spirits all
night, which helped to raise our morale. You don't expect people to
come out immediately and stay working until morning, but they did!
It felt like Databarracks were part of our IT team. They were with
us every step of the way as a safety net and really helped us get
through this difficult time," reveals Laura Smith, the Helpdesk
manager who was involved in the recovery efforts. Databarracks'
devoted staff achieved recovery of sufficient systems to enable WTB
to be up and running the following day.
Deep Impact
Overall, the disaster hindered the normal flow of business quite
significantly for more than half of WTB's 1,000 internal users. One
SAN failure resulted in impact across servers designated for their
Exchange and SQL in addition to other data being lost. As WTB
performs vital business transactions via electronic messages,
sending and receiving an average of 25,000 emails every day in
addition to the millions of archived historical communications they
maintain, the loss of Exchange functionality was no small problem.
The corruption impacted the largest of three Exchange Servers which
hosts 400 users, containing 50% of the estate of around 600GB.
Included in the loss were mailboxes used for electronic ordering as
well as those belonging to senior level management. In addition to
Exchange, half of WTB's user profiles were wiped out, resulting in
hundreds of employees being unable to log on to the company
system.
The SQL server affected was home to five major databases, each
of which provided crucial operational value to WTB. Overnight, key
financial reports are generated by business intelligence software,
and automatically emailed to senior management, business partners,
and financial institutions. The database required to assist with
the production of the reports was compromised, resulting in
unavailability of essential business analysis and performance
information. The Blackberry Database required for facilitating
email and intranet access via mobile devices was lost, resulting in
no access to information for remote users. Databases necessary to
run systems monitoring and fleet management applications became
corrupt and unusable, impacting most notably the fleet management
system which is responsible for orchestrating the logistics and
maintenance for WTB's 250 trucks. Remote VPN access to company
computers was lost as was access to the databases required for the
IT Service Desk software, which is used by the helpdesk to support
users and also used by systems administrators to manage the IT
systems infrastructure.
"Because half of office based users and all remote users were
without Email, we obviously missed out on important correspondence
with business partners including potential customers. In addition,
business operations were severely impacted by having fifty percent
of users unable to access the systems and several key applications
unavailable. Luckily, we were able to run with much higher
availability the following day. The resulting delays and costs
would have been much worse if Databarracks had not acted so
quickly" summarised Tim.
In the days following the disaster at WTB, efforts had to be
made to recover additional historical data and further reinstate
working order. Databarracks had the same two engineers, working in
shifts 24 hours a day, retrieving data from the data centre and
restoring it to WTB's systems through the weekend.
A New Peace of Mind
Since January, WTB's confidence in its choice of online backup
service provider has been reaffirmed once more. In early May 2008,
a couple of workers cabling computer sockets in the WTB
computer room shorted a mains circuit resulting in the main
electrical feed into the computer room being tripped and the
Uninterruptible Power Supply automatically shutting down to protect
itself. All of the server, storage, and network systems crashed and
some data had to be restored through remote connection to ensure
systems were operational once the electrical fault had been
fixed.
Tim says he chose Databarracks based on the fact that it offered
"the right product at the right price," and continues "I used to
have nightmares when backing up to tape due to reliability, now I
can just forget about backups in the knowledge that backup data is
safe and sound in a managed data centre environment. The company
has benefitted from the solution, and I have too! I have time to
focus on other areas of the IT systems and projects since the daily
chore of backup managing has been removed. I feel comfortable since
the disaster. I know I can rely on Databarracks solution 100%. It
has been proven and it works."