Workflow Cloudification

 
 

The key challenge of the execution infrastructure for the IDEaliSM framework is the on-demand and flexible delivery of the necessary resources in the infrastructure (such as virtual machines, virtual networks, storage). To achieve this goal, a cloud-based architecture has been incepted by the KU Leuven that scales the execution environment over multiple servers and multiple data centres.
The execution infrastructure supports the deployment and execution on hybrid cloud environments, including both private as well as public cloud infrastructures.

The key aspect is in the flexible scaling and deployment of the simulation workflows, e.g. created with Optimus from Noesis Solutions N.V.. These simulation workflows require a more flexible, agile and on-demand approach to acquiring resources than the business processes, with infrequent intervals and peaks that are hard to predict in size and time.

The current state-of-practice approaches to execution infrastructure for simulation workflows are two-fold:

  1. Desktop-based execution with desktop tools. Engineers use their desktop environment with limited capacity and performance to execute the simulation, and deploy the required software on their desktop manually. This often results in license management problems.
  2. Batch execution on HPC clusters. Engineers have to reserve a predefined time-slot to execute a larger-scale simulation on a HPC cluster that still has only limited capacity.

 
The IDEaliSM cloud-based architecture supports a more flexible, on-demand approach that deploys the necessary infrastructure and tools when they are needed – right before execution. When the limits of the private cloud are constraining the timely execution of the simulation, a spill-over to the public cloud is supported.

To avoid a large amount of repetitive, manual work, the architecture supports the following automation features when a simulation workflow is executed:

An example of how this infrastructure was utilized in the context of UC3 is described in the following three videos. An overview of the performances obtained while executing a DOE with 10 parallel experiments is also provided.

 

Initialization, analysis and deployment

The video describes:

  1. the definition of the simulation workflow developed in context of UC3
  2. how this simulation workflow is submitted to the cloud infrastructure
  3. the automatic instantiation of the virtual resources required execute the workflow “in the cloud”

 

 
 

Execution

The video describes:

  1. how the simulation workflow is executed on the cloud architecture
  2. how the execution of simulation tools is coordinated by the Optimus Parallel System

 

 
 

Cleanup

The video describes how the virtual resources are destroyed.
 

 
 

Performance of cloud execution infrastructure