CELAR Supports Cancer Research Centre to Utilise Public Cloud for Improved Processing Efficiency

The processing demands of The University of Manchester’s Paterson Institute for Cancer Research (PICR) are subject to high levels of variation, meaning that at times of high demand the limitations of their private infrastructure can result in delays. Greatly extending this infrastructure to better meet peak demand would be wasteful, as frequently the available resources would be under-utilised.

Therefore PICR require the ability to deploy a complex cancer research application with the flexibility to utilize a public cloud service to cover excess demand for processing at busy periods. This will allow the centre to add additional public resources only when required, resulting in optimal performance with minimal waste. In order to assist the delivery of this, CELAR helps to automate the complex workflow application which links the available resources and the data generated.

This workflow production application is named SCAN, and can either run fully on the Cloud or part of it on the cloud with other parts on the existing PICR private infrastructure, thus providing the flexibility required. The cloud on which SCAN will be deployed is GRNET’s ‘Okeanos’, the largest academic cloud infrastructure in Europe with around 7,000 VMs running with different configurations in terms of OS, number of cores and memory capacity. Okeanos provides a user-friendly web application where users can deploy and manage VMs as well as an OpenStack compatible API for programmatically performing the same tasks. However, such tools cannot perform workflow management and orchestration of deployments, let alone manage VMs that could reside in different IaaS infrastructures.

Therefore, in order to create a single point of interaction that provides mechanisms to deploy different VMs (with various combinations of OS, cores, memory, storage) at different cloud infrastructures, SixSq’s SlipStream was required. Slipstream provides a Web UI, command line and API modes of accessing the system, meaning the aim could be achieved via following steps:

1. First, a SlipStream server was deployed in a VM running in Okeanos. CELAR made this simple, as using the CELAR repositories a “yum install” was possible, which was followed by configuring to the target cloud infrastructures. Whilst SlipStream does provide connectors to some cloud providers, a specific connector for Synnefo (the software that powers Okeanos) was required and so written. This has now been released as an open source component available to the community.
2. Following this, GRNET used the SlipStream web UI to create the SCAN project, declare the different VM classes, and create the deployment configuration.
3. Next, GRNET wrote some bash shell scripts to install and configure software within the deployed VMs, using the SlipStream “in” and “out” variables as well as the “wait for” directives, in order to orchestrate software deployment among the various deployed VMs.
4. Starting a SCAN deployment was then simply a matter of a single click action in the SlipStream web UI. After that the rest of the CELAR components took over to control elasticity and apply user defined elasticity rules.

The CELAR platform greatly simplified the process by automating elements of the deployment, through the use of the CELAR repositories and by controlling the elasticity following deployment. Further, the c-Eclipse IDE can be used to define SlipStream configurations, offering the ability to fine tune the elasticity parameters of the application and drive the decision making of CELAR components.

Utilising CELAR allowed for a more simple successful integration of Okeanos and SlipStream in order to run the SCAN application on the public cloud infrastructure. It is capable of enabling a multi cloud application deployment that supports VM and application level workflows involving file servers, database servers, scientific software in different OSs and programming languages, and moreover provides a flexible API that enable higher level tools such as c-Eclipse and CELAR elasticity module to interact with the platform. Most importantly, it has provided SCAN with the capability to control compute and data elasticity and exploit Okeanos cloud resources through the advanced automation middleware provided by CELAR. Researchers at PICR can now simply create desirable configurations and run complex workflows on the provided IaaS infrastructure, enabling a much more efficient provisioning of resources.