Canadian Blood Services: a life saving data center migration
The ability to gather, interpret and protect its data is increasingly becoming the metric by which a company survives or perishes. Data has become, in short, the lifeblood of the modern organization. However, in the past five years, the staggering speed at which IT advancements have swept across every industry has placed pressure on enterprises looking to house IT systems and data in onsite legacy infrastructure. “The sheer volume of IT services and the explosion of data means that it all has to be stored somewhere, and if it's not in your own data center, it needs to be in somebody else's. It's led to enormous growth in capacity across the commercial space, and those new data centers have been built with the latest technologies which can quite often put your own in-house data center to shame,” explains David Grant, Associate Director, Enterprise Services at Canadian Blood Services. “When you look at the needs of the modern digital workplace, then attempt to retrofit your own data center real estate to give it the same capabilities as some of those commercial ones, it becomes enormously expensive and, in some cases, just isn’t feasible.”
With a career spanning more than 20 years in the data center space, Grant came to Canadian Blood Services in 2016. We sat down with him to discuss his role in the organization’s recently completed migration from two legacy data centers in Ottawa to commercially operated colocation facilities, a move calculated to ensure the safety and resilience of Canadian Blood Services’ vital IT operations.
Founded in 1998, Canadian Blood Services is the country’s lifeline, providing a link between the generosity of over 410,000 annual blood, plasma and organ donors from Vancouver to Newfoundland, and the patients that need it. “We drive the donor experience so that it's as pleasant as possible in order to encourage people to continue in their generosity. At the same time, we continue to innovate and improve the products we offer, so that we can improve patient outcomes and ultimately save lives. It's quite a mission to be involved in,” enthuses Grant. “I've worked in IT for a long time, but Canadian Blood Services is the first place where I've had the opportunity to make a difference to somebody's personal life, as opposed to just improving the bottom line or increasing shareholder value. It's a nice feeling.”
Canadian Blood Services has already taken steps over the last three years to improve its front end customer experience, digitalising booking processes and streamlining workflow. “Donors generously give not only their blood, but also their time,” notes Grant. “The last thing that we want to do is waste that time chasing bits of paper around our clinics. By doing this digital transformation at the front end, we have improved that donor experience.” However, as in most aspects of industry, Grant admits that the attention paid to upgrading systems at the front end had outstripped that paid to the internal workings of the organization. “That front end user experience stuff tends to move pretty fast, but not so much the back end plumbing. But, without that plumbing, the front end isn't really much use,” he explains.
In 2016, Canadian Blood Services engaged Gartner to review its data center strategy and the results revealed a particularly worrying vulnerability. “We realized that our data centers were both located in Ottawa – one in our HQ and one in a building that houses most of the IT team. The distance between the two buildings was less than four miles,” Grant says. “In terms of disaster recovery (DR), that’s basically next door to one another.”
In September 2018, while Canadian Blood Services were midway through their migration, an unprecedented total of six tornadoes swept across Ottawa and Quebec, causing damage in excess of $300mn and leaving hundreds of thousands of residents without power. “It was a timely reminder that the ongoing change in weather patterns we’ve seen over the past five to 10 years is going to continue, and having two data centers that were literally next door to one another was probably not a good idea,” says Grant. The review of Canadian Blood Services’ systems not only exposed areas in the organization’s data centers that had single points of failure – areas which would have been both costly and disruptive to upgrade to Uptime Institute Tier Three certification – but also the threat posed by the state of its DR plan. “Our contingency at the time was a cold site about four hours to the south in an IBM data center. In the event of the loss of data centers in Ottawa, the DR strategy was to drive down there, collect the necessary hardware and then restore our systems from backup tapes. The estimated recovery time for an event like that was between one and six days,” Grant explains. “When you’re a digital business, being down or out for 24 hours is a major issue. Being out for five or six days – well, you’re lucky if you survive.”
The need for revision of Canadian Blood Services’ data center strategy was, to Grant, clear. Change, however, needed to be effected as safely as possible. “We are a risk-averse organization, as suits people who are in the business of saving lives,” says Grant. “So, we adopted the colocation approach and went looking for a partner who could provide facilities in locations that were geographically diverse.” Canadian Blood Services eventually settled on Rogers Communications, which now provides data center services in Calgary and near to Toronto (both locations are near one of the organization’s major manufacturing sites).
A veteran of more than a few migration strategies, Grant is well aware that each has its own unique challenges. “We’re regulated by Health Canada, so there are a lot of protocols around testing and validation, which all had to be embedded into our strategy,” he explains. “We broke it down into a series of overlapping waves based around business capabilities: our blood management system, our donor management system, etc. Then, we further subdivided those into our non-production and production-based systems.” The process saw a huge number of careful, methodical tests, warranty periods and planned outages, that allowed the migration to be carried out effectively and with minimal risk. “We did stretch our team quite a bit just to stay on top of the overlapping waves, but they were real troopers and they came through for us,” Grant recalls.
Another key metric in the success of the migration is the increased resilience of Canadian Blood Services’ DR plan. “It was calculated that the business could only tolerate four hours of downtime,” says Grant. To shrink the process of getting the organization’s IT infrastructure back on line from six days to four hours, Canadian Blood Services bought enough capacity in its new colocation sites that the entire business can be run from either one. “We replicate our VMware workloads from one data center to the other continuously. Should we lose a data center site, we use Zerto to reactivate those VMs in the other data center.” Zerto is a Tel-Aviv-based virtual replication software company that specializes in providing operational continuity in the direst of circumstances. The company’s duplication software was also, according to Grant, a key tool in the migration itself. “We used Zerto replication to move the data from our Ottawa data centers to our new Rogers partners data center site,” he explains.
10 years ago, if a company embarked on a digital project, or bought a new piece of software, the expectation was that there was a finite price tag and completion date. In 2019, if executed correctly, a digital transformation project will cost an unlimited amount of money and take an unlimited amount of time. With the completion of the migration, Grant at the team are preparing to immediately begin reevaluating Canadian Blood Services’ data center and DR strategies. “When we were first doing this, a lot of the options and ideas getting thrown around were things like, ‘Why don't we move DR to the cloud? Why don't we move this to the cloud? Why don't we move that to the cloud?’ Even three years ago, a lot of those services were in their infancy and weren't really to be trusted, and certainly not by blood operator - somebody who is committed to patients and donors and saving lives wouldn’t want to risk that,” Grant explains. “In the intervening three years, we've seen a lot of development in that space. I think our next data center strategy will look at what we’re still running in-house and in our colocation facility, and determine what’s the next step for these systems.”
