With over 22 million customers worldwide, bet365 is the world's largest online sports betting company. Employing over 4,000 people, operating in 22 languages and managing thousands of services globally, it has accumulated a server fleet that numbers in the 10,000s, to support its business-critical systems.

THE CHALLENGE
Launched in 2001, the betting platform was built on a classic Windows architecture. The operating system was tightly coupled to the products it served, and this meant that the Infrastructure team had to work within a specific set of parameters that dictated the way each server was administrated. It was an approach that became challenging as the server estate grew.

"bet365's Linux revolution was forged in the fires of necessity," says James Nightingale, Principal Infrastructure Architect at Hillside Technology, bet365's own technology business. "Despite serving the company admirably, it had become clear that the existing platform couldn't sustain our planned growth. Looking ahead, we could see the point where Windows would be a barrier to our expansion and our growth objectives."

With a problem of server scale, Nightingale explains that, "When you've got tens of servers, you have the luxury of loving each one. When they number in the tens of thousands, this level of attention is unworkable, with hundreds of people dedicated to the task. With that many people making tweaks to individual servers, finding the source of any issue is a problem."

To maintain the integrity of the existing architecture it was essential that each cluster and sometimes each machine received the love it demanded. However, as the server footprint grew, this was neither sustainable nor scalable.

Complexity of any kind makes scalability and maintenance more difficult. Keeping things simple was vital but proving difficult to deliver.

THE SOLUTION
"If Infrastructure is like a motorway, we require a flat, straight road for our products to motor on. Tight coupling of applications to the OS introduced too many corners. As our product architecture transitioned with technologies like Riak, Erlang and Golang, moving to Linux made sense. Maintenance is reduced, uptime is improved and resource management is more efficient: the Linux tooling and ethos enables a more scalable approach," explains Nightingale.

Rather than dealing with individual machines, Linux provides the necessary tooling and ways of working to support a herd mentality. Embracing Ansible and Python, which operate smoothly on Linux is now the Infrastructure team's core model. Nightingale explains that, "We like the Pets v Cattle analogy made famous by CERN and others; avoid expending too much energy, lovingly caring for each server instance (Pets), make servers the same each time, and try and treat them collectively (like Cattle)."

bet365 has achieved further efficiencies from building applications in-house. Key Infrastructure platforms, including network, storage and management have transitioned to Linux. The Infrastructure teams continue to leverage vendors and work with communities, but the flavour is always determined by bet365.

THE RESULT
Nightingale enthuses that, "The move to open source has influenced both our methods and the way we recruit. We look for a more inquisitive mindset. People who are interested in getting under the hood, who want to get on the command line, drive usage and act more creatively in their problem solving."

Whilst Enterprise Linux 7 is now the default platform, bet365 has several different distributions supporting its workloads and use cases, including specialist and lightweight flavours such as CoreOS. This enables them to utilise the same core skills and tools to manage different application product requirements at scale.

Nightingale concludes, "At bet365 we've moved beyond core systems with dedicated Linux clusters. As we continue to innovate with new products, our starting point is Linux." NC