UCL breaks through scaling roadblock with development tools at scale and achieves first simulations of complex regions of the human brain
 

Researchers at University College London (UCL) modelling intracranial blood flow have been able to reach a major milestone by using development tools from Allinea Software.  

A massive simulation using 50,000 processor cores on the UK’s largest supercomputer, ARCHER, became possible after solving an application crash that only occurred at this high scale.
 
UCL’s HemeLB software applies computational fluid dynamics to model blood flow around cerebral vessels and simulate pressure at points of weakness such as aneurysms.  Using data from MRI scans of a patient’s blood vessels, the group anticipates that simulations will one day help to decide the best clinical option for individual patients.
 
The complexity of the simulations requires the UCL team to use ARCHER, the UK’s flagship Cray XC30 system, which is managed by EPCC on behalf of EPSRC and other UK research councils.           
 


The UCL team used Allinea Software’s performance profiling tool, Allinea MAP, to increase the performance of HemeLB on the system. Having already improved performance on some test cases by over 25% they wanted to run a larger simulation.


 
“We’d never been able to look at this many cores – and get a clear view of how the time was being used – we were keen to see it in Allinea MAP,” UCL Post-doctoral Researcher Derek Groen said. 
However, the application crashed when using 50,000 processor cores, which stopped them in their tracks.
 
“The crash was totally unexpected. I didn’t know how I would diagnose or fix it at that scale – it was beyond anything I had tried to do before.” says Groen. “Allinea Software helped us straight away – they knew that if we could run the simulation with their debugger, we would find the problem.” 
 


The UCL team and Allinea Software were brought together by the EU CRESTA project – which is preparing applications for future extreme-scale computing.  Running applications at extreme scales poses challenges, which Allinea Software’s tools help to solve. 
 
Allinea DDT is the only parallel debugger that handles that scale of the problem – which was resolved quickly by debugging all 50,000 application processes simultaneously.
 
“Getting HemeLB to scale to 50,000 ARCHER cores is a real achievement”, says Professor Peter Coveney, Director, Centre for Computational Science, UCL.

“We are thankful for the productive collaborations we enjoy with Allinea Software that have allowed us to reach these intoxicating heights,  which are enabling us to study hemodynamics within the Circle of Willis for the first time.”
 
“Working alongside the HemeLB team has been a great experience for us,” says David Lecomber, CEO of Allinea Software. “It helps us to understand the issues they face better. It is inspiring to see the impact HPC could make to lives in the future – and we’ve seen at first-hand the impact on progress that the challenges make. Tools designed to handle scale are changing the way developers think about those hard challenges.”
 


“Seeing HemeLB used at this scale on ARCHER is very impressive, and highlights the success of the co-design approach used by UCL and Allinea Software on the CRESTA project”, says Dr Lorna Smith, CRESTA project manager and Deputy Director of the ARCHER CSE team. “This achievement with Allinea Software is a great example of the importance of providing a rich tooling environment for our users on ARCHER”.
 

CRESTA’s remit involves preparing applications and software for future exascale platforms, utilising a co-design approach. Beyond this, there is an extensive programme of research activities looking to develop new techniques and algorithms to exploit future exascale systems.

Our White Paper series is designed to showcase the social and economic importance of these activities and to highlight the benefits of the co-design approach, disseminating our work to the wider HPC community.

CRESTA is therefore pleased to announce the release of three new case studies and one new White Paper. The case studies showcase the work of the HemeLB, NEK5000 and Ludwig codes while the new White Paper looks at the latest developments in exascale architectures and discusses the implications for the CRESTA co-design applications.  

The new case studies and White Paper are available from the Publications section of our website.

Copies of the slides and Presentations from the Joint Exascale Projects Workshop in Edinburgh

Day 1

Professor Mark Parsons - Welcome

Panagiotis Tsarchopoulos - HPC in H2020 - Slides

Overview of the projects:

Wilfried Verachtert - EXA2CT - Slides

Stefano Markidis - EPIGRAM

Riccardo Rossi - NUMEXAS- Slides

Alex Ramirez - Mont-Blanc and Mont-Blanc 2- Slides

Estela Suarez - DEEP and DEEPER - Slides

Lorna Smith - CRESTA- Slides

Bernd Mohr - EESI-2 - Slides

Tools for Exascale - including new user requirements:

Bernd Mohr - DEEP and Mont-Blanc - Scalasca - Slides

Judith Giménez - DEEP and Mont-Blanc - Extrae/Paraver- Slides

Tobias Hilbrich - CRESTA - Vampir and CRESTA Tools - Slides

Use cases:

• Stefano Markidis - EPiGRAM - Use Case of Tools in EPiGRAM - Slides

• Riccardo Rossi - NUMEXAS - Slides

• Eric Petit - EXA2CT

Day 2

Programming models for Exascale:

Daniel Holmes - EPiGRAM - Message passing and PGAS - Slides

Jesus Labarta - Mont-Blanc (and DEEP) - OmpSs

Alistair Hart - CRESTA - OpenACC - Slides

Norbert Eicker - DEEP - ParaStation Global MPI

Eric Petit - EXA2CT

Innovative Algorithms for Exascale:

Wim Vanroose - EXA2CT - Slides

Riccardo Rossi - NUMEXAS

Dmitry Khabi - CRESTA - Slides

Pictures from the Collaboration Workshop that was held in Edinburgh on the 18 & 19th March 2014, thank you to everyone that attended.

Please email Andy (a.mcdonald at epic.ed.ac.uk) if you would like a hi-res version of any of the photos. Website file size limitations don't allow us to upload them here.

The European Projects Meeting being held in Edinburgh over the course of today and tomorrow has started.  With talks from representives from CRESTA, DEEP, MONT-BLANC, EPIGRAM, NUMEXAS and EXA2CT, the next two days promise to be very interesting for all of those attending. 

We're pleased to report that our colleagues from Europe have also brought the sun with them to Edinburgh today!

Copies of the talks and a group photograph will be available on the CRESTA website following the meeting.