HemeLB within CRESTA: computational haemodynamics en route to patient-specific treatment planning using the exascale
Author: Timm Krueger, UCL.
It is estimated that about 1 - 5% of the entire population have aneurysms in their blood vessels; these are pathological balloon-like bulges of blood vessels which can form following the weakening of the vessel wall. With a cranial aneurysm, there is an increased risk of artery rupture and internal haemorrhage, which is a life-threatening situation.
Aneurysms are often found during routine investigations as they are usually asymptomatic. Most patients with a cranial aneurysm never suffer consequences, but in some cases, aneurysms rupture with severe, if not fatal, outcomes (e.g., stroke). Once a rupture has occurred, physicians must plan and begin treatment rapidly. The choice of treatment strongly depends on patient-specific factors (e.g., location of aneurysm, vessel geometry, blood pressure). Therefore, the ultimate goal is the development of a real-time predictive tool accounting for the short time-scale of clinical decision-making.
But even in the case of a non-ruptured aneurysm it is not directly obvious whether it might turn into a life-threatening situation in the future and whether intervention is necessary at all. Although there are means to stabilise or remove aneurysms (e.g., inserting stents into the artery to modify the flow field or clipping the aneurysm from the outside), these interventions are not without risks and should be applied only when required. Unfortunately, the factors affecting the potential danger of a given aneurysm are not yet known. It is thus desired to provide methods predicting the risk of a future vascular malfunction in individual patients.
There is strong evidence that the blood flow pattern plays a role in the formation process of aneurysms. This brings us to the question as to whether it may be possible to find significant correlations between specific features of the flow near the aneurysm and how it eventually develops.
Computational fluid dynamics (CFD) comprises a wide class of numerical algorithms designed to tackle problems related to the flow of fluids. In recent years, larger and larger volumes have been simulated with higher and higher resolution. This development calls for more efficient large-scale simulations. Therefore, nowadays, many CFD applications belong in the realm of high performance computing (HPC). Current supercomputers are capable of achieving peak performance of a few petaflops per second (1 Pflops/s = 1015 floating point operations per second), whereas modern laptop computers typically reach of the order of 1010 flops/s. The aim of CRESTA is to reach the exascale (1018 flops/s) by 2018. This endeavour strongly relies on designing software solutions which have to run with hundreds of thousands, if not millions, of parallel processes. As blood flow is essentially a fluid dynamics problem, it seems natural to take advantage of recent developments, in both the CFD and HPC communities, in order to better understand the effects of blood flow on vessel topologies. Eventually, it will be possible to provide a simulation tool assessing the personal risk of aneurysm formation and rupture since each patient’s circulatory system is different.
HemeLB is designed as a specialised predictive tool for fluid flows in complex geometries. Its main focus is simulating blood flow in parts of the cerebral arterial network. HemeLB employs a highly efficient implementation of the lattice Boltzmann (LB) algorithm which found its way into the CFD community around two decades ago. Due to its locality, the LB algorithm is intrinsically easy to parallelise and therefore a method of choice for HPC. A sophisticated communication approach based on the Message Passing Interface (MPI) allows HemeLB to be run on large supercomputers. Therefore, the HemeLB project fits perfectly into CRESTA, as it stimulates the development of technologies leading to the next generation of supercomputers on the exascale.
The typical workflow of a HemeLB simulation has three main steps: (1) obtain patient-specific data from a medical scan, (2) pre-process the scan to extract geometry data suitable for a simulation, and (3) simulation, visualisation, and data assessment by running HemeLB. At the beginning, an X-ray rotational angiography scan is performed to obtain patient-specific data. It provides the highest resolution of the available imaging methods. The dataset provided by the scan typically contains between $10^5$ and $10^8$ voxels whose brightness carries data about their locations – inside or outside the blood vessels. In the next step, the data has to be processed in order to construct a geometrical representation of the blood vessel network. At this point, the intervention of the user is required: the voxel data is converted into a surface representation which defines the blood vessel boundaries. There is ongoing work at the University of Jyväskylä in Finland to automate this currently labour intensive procedure. Once the surface data is available, the region of interest (ROI, containing the aneurysm and connected blood vessels) has to be selected employing the graphical user interface of the HemeLB Setup Tool. The user has to specify the simulation resolution and the so-called inlet and outlet planes through which the blood enters and leaves the network.
Only now does the main HemeLB application come into play. The prepared geometry file is loaded together with an additional input file. The latter contains information about the pressure values applied at the inlets and outlets. The geometry is mapped onto a regular lattice where each lattice site can either be fluid (within the blood vessel) or tissue (outside). Due to the sparseness of the arterial network, usually only 5 - 10% of the volume consists of fluid sites. This underlines how important it is to use software optimised for sparse geometries, like HemeLB. Since the simulation is running on a highly parallelised computer, the total fluid volume has to be subdivided into domains of approximately equal size to ensure computational load-balancing. This is performed at runtime by the parallel partitioner ParMETIS which eventually provides the final partition of fluid sites between processes.
After reading the geometry and boundary conditions, the flow simulation is executed. For large-scale simulations, it is impractical to write complete snapshots to the hard disks: a single snapshot can easily require several gigabytes. Due to the ever-growing size and complexity of computer simulations, new data-handling approaches are necessary. Indeed, the German Aerospace Centre (DLR) institutes in Cologne and Braunschweig explore innovative data-processing ideas within CRESTA. The fundamental idea is to enable the interactive inspection of a coarse data subset at runtime, which drastically decreases the amount of data to be transferred. In the future, the user will be able to select regions of particular interest and request highly resolved snapshots of these. The HemeLB Steering Client for remote connections receives real-time visualisations of the simulation and allows the user to define which data to visualise (e.g., streamlines, wall stresses, or pressure distributions). An advantage of this approach is that both the computation and visualisation are executed on the same machine and share the same data and resources, producing data quickly and in real-time.
There are several ideas for enhancing the efficiency and physical relevance of HemeLB in the future. In order to increase the computational performance and to eventually approach exascale capability, EPCC is working on a hybrid parallelisation using GPUs and OpenMP. It is also possible to apply steering not only to the visualisation pipeline, but also to the simulation itself. This way, simulation parameters could be changed at runtime to investigate different scenarios without performing a restart. There are additional model components planned, including magnetic microparticles (targeted drug delivery for cancer treatment), vascular remodelling (predicting effects of flow on aneurysm formation), and a coupling to coarse-grained circulation models (improving estimates of the inlet/outlet pressure conditions). The latter multiscale project overlaps with the scope of the MAPPER (Multiscale APPlications on European e-infRastructures) project. The ultimate goal of the HemeLB project is to reach a state where the entire workflow between patient scan and assessment of the simulation results is automatised and takes no longer than one hour which is the typical time-scale on which physicians have to make decisions about to procedures with the patient on the operating table. CRESTA provides the optimal environment to approach this ambitious aim.
