2018

High-Performance Computing Center

Stuttgart

ANNUAL REPORT

ENERGY

HEALTH

CLIMATE

MOBILITY

PHILOSOPHY

2018

HLRS ANNUAL REPORT The High-Performance Computing Center Stuttgart (HLRS) was established in 1996 as the first German national high-performance computing (HPC) center. As a research institution affiliated with the University of Stuttgart and a founding member of the Gauss Centre for Supercomputing, HLRS provides HPC services to academic users and industry. We operate leading-edge HPC systems, provide training in HPC programming and simulation, and conduct research to address key problems facing the future of supercomputing. Among our areas of expertise are parallel programming, numerical methods for HPC, visualization, grid and cloud computing concepts, data analytics, and artificial intelligence. Our system users conduct research across a wide range of scientific disciplines, with an emphasis on computational engineering and applied science.

Director‘s Welcome Grußwort

2 /3

HLRS ANNUAL REPORT 2018

Welcome to the HLRS 2018 Annual Report. We are

Wir freuen uns, mit Ihnen auf die zahlreichen Erfolge der

pleased to present this look back at our activities over

letzten zwölf Monate am HLRS zurückzublicken.

the past 12 months, which were marked by many suc-

An oberster Stelle der wichtigsten Ereignisse steht

cesses.

sicherlich die Beschaffung unseres neuen Höchstleis-

At the top of our list of achievements was certainly our

tungsrechners. Nach einer intensiven, einjährigen Aus-

procurement of a new high-performance computer.

schreibungsphase, an der führende Hardware-Anbieter

After an intense, yearlong competition among leading

teilnahmen, unterschrieb das HLRS im November 2018

hardware vendors, HLRS signed a contract with Hewlett

einen Vertrag mit Hewlett Packard Enterprise (HPE), um

Packard Enterprise (HPE) in November to build Hawk,

„Hawk“ — unseren Höchstleistungsrechner der nächs-

our forthcoming next-generation supercomputer. As

ten Generation — in Betrieb zu nehmen. Als Teil dieser

part of this collaboration, HLRS and HPE will also work

Zusammenarbeit werden das HLRS und HPE gemein-

together on improving services, optimizing codes, and

sam Services ausbauen, Codes optimieren und neue

developing new types of high-performance computing

Anwendungen für Maschinelles Lernen und Künstliche

(HPC) applications for machine learning and artificial

Intelligenz vorantreiben. Wir freuen uns darauf, unse-

intelligence. In the coming years we look forward to

ren Nutzern in den kommenden Jahren deutlich leis-

being able to offer our users a much more powerful

tungsfähigere Ressourcen für das Hoch- und Höchst-

supercomputing resource and will be well positioned to

leistungsrechnen (HPC) anbieten zu können und damit

lead as HPC technologies continue to advance.

unsere führende Position für die Weiterentwicklung

HLRS also successfully participated in the launch of

zukünftiger HPC-Technologien auszubauen.

several new EU-funded projects, including new Euro-

Das HLRS konnte seine Arbeit in zahlreichen neuen,

pean Centres of Excellence (CoE’s) in high-performance

EU-geförderten Projekten beginnen — darunter vier

computing that will form the backbone of the European

neue europäische Exzellenzzentren (CoEs) für HPC-An-

HPC strategy. As a member of multiple new CoE’s cre-

wendungen, die das Rückgrat der europäischen

ated in 2018 — EXCELLERAT, HIDALGO, CHEESE, and

HPC-Strategie bilden sollen. Als Mitglied von EXCEL-

POP2 — HLRS is involved in four of the nine European

LERAT, HIDALGO, CHEESE und POP2 ist das HLRS in

centers. As project leader of the new European Centre

vier der neun in 2018 gestarteten europäischen Exzel-

of Excellence for Engineering Applications (EXCEL-

lenzzentren vertreten. Als koordinierender Partner des

LERAT), HLRS will strengthen its position as Europe’s

neuen European Center of Excellence for Engineering

leading center for simulation in engineering, a critical

Applications (EXCELLERAT), wird das HLRS seine Posi-

tool for high-tech innovation. Our support for the com-

tion als führendes europäisches Zentrum für Simula-

putational engineering and applied science communi-

tion in den Ingenieurwissenschaften stärken. Unsere

ties is the focus of this annual report’s spotlight article.

Unterstützung für die computergestützte Ingenieur-

Also important in 2018 was the positive evaluation

und angewandten Wissenschaften ist ein Schwerpunkt

of several key research projects that HLRS leads or

dieses Jahresberichts.

participates in as a major partner. In September the

occasion to talk with us for this annual report about the

Ein weiteres Ereignis in 2018 war die positive Bewertung

und Forschung arbeiten wird, um technischen Heraus-

Deutsche Forschungsgemeinschaft (DFG, or German

economic and scientific benefits of networking the Ger-

einiger wichtiger Forschungsprojekte, die das HLRS lei-

forderungen aus den Bereichen Medien und Kunst zu

Research Foundation) once again recognized the Uni-

man automotive industry.

tet oder daran als bedeutender Partner teilnimmt. Die

begegnen. Das MSC ist der neueste Bestandteil unse-

versity of Stuttgart’s Cluster of Excellence in Simula-

In terms of key performance indicators HLRS can

Deutsche Forschungsgemeinschaft hat zum Beispiel

rer industriellen Strategie, die mit dem Automotive

tion Technology (SimTech) as a German national center

proudly look back on a very successful year. Our third-

im September das Exzellenzcluster SimTech der Uni-

Simulation Center Stuttgart (asc(s) begann. In diesem

of excellence for its proposal to expand our focus on

party funding saw an increase over previous years, our

versität Stuttgart wieder als deutsches Exzellenzzen-

Jahr feierte das asc(s sein zehnjähriges Bestehen und

data-integrated simulation science. As the prominent

income from industrial HPC users was again at a high

trum anerkannt, das seinen Fokus auf daten-integrierte

demonstrierte somit, wie nachhaltig diese Strategie

facility for high-performance computing at the Univer-

level, and the number of people benefiting from our

Simulationswissenschaft legen wird. Als herausra-

sein kann. Wir danken asc(s Geschäftsführer Alexander

sity, HLRS looks forward to the coming seven years, in

training and education activities remained strong.

gende Einrichtung für das Höchstleistungsrechnen an

Walser dafür, dass er für diesen Jahresbericht mit uns

which we will work closely with our colleagues here to

Users of our HPC systems also had a very productive

der Universität Stuttgart, freut sich das HLRS auf die

über die Vorteile des Networking innerhalb der Auto-

push the limits of using big data in simulation.

year. This annual report showcases some examples of

nächsten sieben Jahre, in denen wir in enger Zusam-

mobilindustrie sprach.

HLRS recognizes that the future of high-performance

our users’ research, including a new sound prediction

menarbeit mit unseren Kollegen die Grenzen der Simu-

In Bezug auf unsere Leistungskennzahlen kann das

computing lies not just in new technologies, but also

model that could help reduce machinery noise, appli-

lation unter Benutzung von Big Data erweitern werden.

HLRS stolz auf ein erfolgreiches Jahr zurückblicken.

in addressing humanity’s greatest challenges. Con-

cations of HPC and machine learning to make power

Für uns liegt die Zukunft des Höchstleistungsrechnens

Unsere Drittmitteleinnahmen stiegen über die letz-

sidering important relationships among supercom-

plants more efficient, and a computational approach

nicht nur in der Entwicklung von neuen Technologien,

ten Jahre hinweg, unser Einkommen aus industrieller

puting, society, and politics we created a new Socio-

that is helping to point the way toward new semicon-

sondern auch darin, die größten Herausforderungen

HPC-Nutzung war wieder auf einem hohen Niveau und

political Advisory Board, which held its first meeting

ductor technologies. Such applications show how sim-

der Menschheit zu adressieren. Unter Berücksichtigung

die Anzahl der Personen, die von unserem Weiterbil-

this year. The gathering identified many topics that

ulation is making important contributions in areas that

wichtiger Zusammenhänge zwischen dem Höchst-

dungsprogramm und unseren Vorlesungen profitieren,

will feed back into our research and enable HLRS to

are critical for our wellbeing.

leistungsrechnen, der Politik und der Gesellschaft

blieb hoch.

remain focused on society’s needs and requirements.

With this annual report we express our heartfelt thanks

haben wir einen neuen Gesellschaftspolitischen Beirat

Auch die Benutzer unseres HPC-Systems hatten ein

Throughout the year we were also glad to welcome

to our supporters and funders, who have made our suc-

geschaffen, der 2018 zum ersten Mal zusammengetrof-

sehr produktives Jahr. Dieser Jahresbericht verdeutlicht

prominent politicians from Germany and Europe to

cesses in 2018 possible. At the same time, we look for-

fen ist. Die Sitzung hat mehrere Themen identifiziert,

anhand einiger Beispiele aus der Forschung unserer

HLRS to discuss the future of high-performance com-

ward to tackling new challenges in 2019.

die unsere Forschung voranbringen können und es

Nutzer, wie wichtig Simulation für unser Wohlergehen

dem HLRS ermöglichen werden, sich stärker als bis-

ist. Dazu gehören ein Geräusch-Vorhersage-Modell,

her auf die Bedürfnisse und Anforderungen der Gesell-

das helfen könnte, den Lärm von Maschinengeräu-

HLRS’s focus on HPC training grew in strength this year

schaft zu konzentrieren. Dieses Jahr haben wir uns

schen zu reduzieren; eine Machine Learning-Anwen-

with the completion of the first course offered by the

auch darüber gefreut, viele prominente Politikerinnen

dung, die Kraftwerke effizienter machen könnte; sowie

Supercomputing-Akademie. Its “blended learning” cur-

und Politiker aus Deutschland und Europa am HLRS

einen Berechnungsansatz, der den Weg zu einer neuen

riculum is designed to address the needs of HPC users

willkommen zu heißen, um die Zukunft des Höchstleis-

Halbleitertechnologie ebnet.

puting and how its potential could be maximized to With best regards,

benefit society.

in industry, and complements HLRS’s existing state-

Prof. Dr.-Ing. Dr. h.c. Dr. h.c. Prof. E.h. Michael M. Resch

tungsrechnens und dessen Nutzen für die Gesellschaft

Mit diesem Jahresbericht bringen wir auch unseren

of-the-art training program. Trainers and trainees were

Director, HLRS

zu diskutieren.

Dank an unsere Unterstützer und Förderer zum Aus-

overwhelmingly positive about the Supercomputing-

Der Schwerpunkt des HLRS auf die HPC-Weiterbildung

druck, die uns diese Erfolge in 2018 ermöglicht haben.

Akademie’s first test-run, and we look forward to

verstärkte sich in diesem Jahr durch den Abschluss

Gleichzeitig freuen wir uns darauf, die neuen Herausfor-

watching this innovative program grow in the coming

des ersten Moduls der Supercomputing-Akademie. Ihr

derungen 2019 in Angriff zu nehmen.

years.

Kursplan im „Blended Learning“-Format entspricht den

Another highlight of 2018 was the inauguration of the

Bedürfnissen der HPC-Nutzer aus der Industrie und

Media Solution Center Baden-Württemberg (MSC),

ergänzt das wegweisende Schulungsprogramm des

through which HLRS will work closely with partners

HLRS. Nach den vielen positiven Reaktionen der Aus-

from industry and academia to find solutions to chal-

bildenden und Teilnehmenden auf den ersten Testlauf

lenges facing the fields of media and the arts. The MSC

der Supercomputing-Akademie freuen wir uns darauf,

is the latest product of our industrial strategy, which

dieses innovative Programm in den nächsten Jahren

began with the Automotive Simulation Center Stuttgart

wachsen zu sehen.

(asc(s). This year the asc(s celebrated its 10th anniver-

Ein weiteres Highlight von 2018 war die Gründung des

sary, showing just how sustainable our strategy can be.

Media Solution Centers Baden-Württemberg (MSC), in

We thank asc(s director Alexander Walser for using the

dem das HLRS eng mit unseren Partnern aus Industrie

4 /5

HLRS ANNUAL REPORT 2018

8

Spotlight

9

Engineering the Future

14

News & Highlights

15

News in Brief

22

New European Centers of Excellence Funded

23

HLRS Visualization Supports Stuttgart 21 Construction

24

Supercomputing-Akademie Completes First Course

25

Sociopolitical Advisory Board to Help Guide HLRS Vision

26

SimTech Named National Center of Excellence

27

Media Solution Center to Promote Innovation in Film and Digital Art

28

EXCELLERAT Will Bring HPC Applications to Engineering Industry

29

Building a Sustainable HPC Infrastructure

30 Interview: Networking the German Auto Industry 33

Student Dives into Data to Predict Train Delays

36

Enhanced User Support Improves Performance of HPC Systems

38

EuxDat: Taking Agriculture into the Cloud

39

PhD Graduates 2018

IMPRINT

40 Looking Inside Simulation‘s ”Black Box“

6 /7

HLRS ANNUAL REPORT 2018

34

HLRS by the Numbers

42

User Research

43

Supercomputer Enables Sound Prediction Model for Noise Control

46

Using Computational Chemistry to Investigate New Semiconductor Technologies

48

Simulation and Machine Learning Could Make Power Plants More Efficient

50 Selected User Publications

56

About HLRS

57

Inside Our Computing Room

59

User Profile

60 Third-Party Funded Research Projects 66

HPC Training Courses in 2018

68

Workshops and Conferences 2018

69 Structure 70

Divisions and Departments

Engineering the Future

HLRS has decades of experience supporting computational engineering for academic and industrial research. With recent expansions in its infrastructure and services, it is now laying a foundation that will enable engineers and applied

SPOTLIGHT

© Mathis Bode, Institute for Combustion Technology, RWTH Aachen

scientists to take full advantage of the next generation of faster supercomputers.

8 /9

HLRS ANNUAL REPORT 2018

Located in the capital of the state of Baden-Württem-

industrial research and development — for example, in

berg, the High-Performance Computing Center Stutt-

the automotive, energy, and aerospace industries. In

gart (HLRS) sits at the epicenter of one of Germany’s

fact, approximately 10% of HLRS computing hours are

most dynamic regions for high-tech engineering. The

dedicated to industrial users, a figure that stands out as

region is not only home to prominent international con-

particularly noteworthy in Germany’s academic super-

cerns such as Daimler, Bosch, and Porsche, but also

computing landscape.

to a large and thriving ecosystem of small and medi-

Bridging the academic and industrial worlds in this way

um-sized companies with a long tradition of manufac-

has enabled HLRS to develop expertise in applying

turing high-quality mechanical, electronic, and chemi-

high-performance computing (HPC) to many kinds of

cal products. Together, these communities have been a

real-world computational engineering problems. This

major engine of Germany’s economic prosperity.

knowledge has, in turn, guided its strategy for growth.

This regional strength in engineering has also played an

HLRS’s goal of providing resources that optimally serve

important role in shaping HLRS’s evolution. Based on

the needs of computational engineers has been a key

the campus of the University of Stuttgart — itself a cen-

driver of investments in its new computing infrastruc-

ter for research and higher education with an empha-

ture, professional training offerings, and in-house com-

sis on engineering and the applied sciences — HLRS

puter science research. As HLRS lays the groundwork

has for decades maintained a state-of-the-art infra-

for ever-larger supercomputers in the future, the center

structure for computational simulation, visualization,

is also focused on providing the tools and services the

and data analytics. Such tools have become essential

engineering community will need to take full advantage

for creating models of natural phenomena that are so

of these resources as they become available.

complex or that occur at such small scales that they are difficult to study in any other way. In engineering, the

HPC improves automotive manufacturing

design of many precision products for which Germany

For Stefan Hildenbrand at chemicals manufacturer

is renowned has become unimaginable without such

Pfinder KG, HLRS has become an important partner,

computational models.

enabling the company to develop new services for cli-

HLRS has long provided essential supercomputing

ents in the automotive industry.

resources for scientists in disciplines such as com-

Pfinder supplies wax-based anticorrosive coatings that

putational fluid dynamics, physics, chemistry, and

are typically applied by assembly line robots during car

materials science, among other fields. This includes

manufacture. As team leader for digital engineering,

not only academic researchers, but also scientists in

Hildenbrand performs computational simulations to

Sample simulation of the application of wax to a sill. The colors indicate the thickness of coverage.

advance applications in energy, climate, mobility, and

Writing software for large-scale parallel computing sys-

health. The 5,000-node Hawk system will have a theo-

tems like Hazel Hen or Hawk is much more complex

retical peak performance of 24 petaFLOPs.

than running the same algorithm on a desktop com-

An important feature of Hawk is that its technical spec-

puter or a small computing cluster. Indeed, developing

ifications were chosen with the specific needs of com-

and implementing effective and efficient codes for HPC

help optimize this process. Spe-

putational engineers in mind. Hawk’s architecture is

systems requires specialized knowledge, and will be

cifically, he uses OpenFOAM, a

based on chip manufacturer AMD’s next-generation

especially important as supercomputers approach the

widely used software package for

EPYC processor, code-named Rome. HLRS picked the

exascale, the next major plateau in computing power

computational fluid dynamics, to predict

Rome processors because they utilize a memory sub-

over that of current petascale systems. Optimizing

how the coatings will behave when injected by a nozzle,

Keeping up with changes in HPC technologies would

system that makes them particularly well suited for

codes for such extremely parallelized systems will be

the paths the droplets will follow through the air, and

be impossible for a small company.

simulation applications in fields such as computational

necessary to accelerate individual computational sci-

how they will form films on surfaces. These models are

Having access to this kind of computing power became

fluid dynamics, molecular dynamics, and other research

entists’ research and to make the operation of super-

used to optimize the amount of fluid that needs to be

particularly important for Hildenbrand in 2018, when

areas in which many of HLRS’s users are engaged. In

computers more efficient — this will, in turn, help save

sprayed for a specific application, to identify the best

for the first time his team simulated the application of

addition, the AMD EPYC processors complement the

energy and make HPC resources available to the largest

nozzle layout, and to ensure they are fully protective

Pfinder coatings not just for one component, but for an

use of competing processors at the other two centers

number of potential users.

and durable.

entire automobile simultaneously. This jump in scale

in the Gauss Centre for Supercomputing — the alliance

For many years, HLRS has recognized that the future

Pfinder has a small parallel computing system in-house,

required a leap in computing power that was far beyond

of Germany’s three national supercomputing centers.

of high-performance computing lies in developing and

but when complex questions arise or when results need

the reach of its own system. He anticipates continuing

The choice supports GCS’s goal of offering its users a

increasing access to state-of-the-art codes capable of

to be delivered quickly, Hildenbrand turns to HLRS.

to utilize the HLRS supercomputer in the future, as this

diverse set of computing architectures.

scaling to ever-larger systems. For these reasons, the

“When we are deep in a project for a client that requires

ability to model entire systems could offer a new mar-

“We are excited that Hawk constitutes a sizable

center has undertaken several projects aimed at prepar-

a large-scale simulation,” Hildenbrand explained, “we

ket opportunity for his firm to provide complete coating

increase in the performance of our flagship supercom-

ing computational engineers for the future.

can’t wait four weeks to run the computation on our own

application solutions.

puting system,” said Prof. Dr. Michael M. Resch, Director of HLRS. “But the real winners will be our user commu-

Bringing industrial engineering up to scale

nity of computational engineers in academic research

In 2018, for example, HLRS became the lead coordi-

we have here. Instead of waiting for up to 4 weeks, this

Hawk to offer new capabilities for industrial production

and industry who will benefit from the ability to run

nating center of a new Horizon 2020-funded research

can deliver a result in as little as 3 to 4 days.”

Users of HLRS’s HPC systems will soon have even

much more complex simulations.”

program called EXCELLERAT (European Centre of

Being able to call on HLRS also has other advantages

more power at their fingertips. In November 2018, HLRS

system. HLRS makes it possible for us to quickly run a job on many hundreds of cores as opposed to on the 16

Excellence for Engineering Applications). A collabora-

over installing its own larger HPC system. Because

and Hewlett Packard Enterprise announced a joint col-

Getting to the next level

Pfinder’s computational workload fluctuates widely, it

laboration to build a next-generation supercomputer

There is no doubt that larger supercomputers enable

EXCELLERAT will provide the HPC and engineering

would not make economic sense to install a large sys-

that will be 3.5 times faster than Hazel Hen, HLRS’s

larger simulations and increase the number of core

communities access to scalable applications and

tem of its own, only to have it sit idle for multiple months

current flagship system. The upcoming supercomputer,

hours available for the research community. At the

facilitate their availability through technology transfer.

of the year. Moreover, Hildenbrand can be confident

called Hawk, will be the fastest in the world for indus-

same time, however, it has become clear that build-

(See page 28 to learn more about EXCELLERAT.)

that HLRS uses reliable, state-of-the-art hardware that

trial production, powering computational engineering

ing hardware alone is not enough to meet the growing

Recognizing the need for such resources and expertise,

continually grows in power as HPC systems evolve.

and research across science and industrial fields to

needs of computational engineering.

companies including Porsche, Ansys, and Festo have

10 / 11

HLRS ANNUAL REPORT 2018

tive network including 13 partners from across Europe,

Representatives of the University of Stuttgart and Hewlett Packard Enterprise celebrated the contract signing for the new Hawk supercomputer.

expressed support for EXCELLERAT. Aircraft manufac-

In July, the Supercomputing-Akademie — which is sup-

turer Airbus has also been an early participant in the

ported by the Baden-Württemberg Ministry for Social

project, as EXCELLERAT will enable academic knowl-

Affairs and Integration through the European Social

edge to be connected with industrial needs.

Fund, and by the Baden-Württemberg Ministry of Sci-

In addition to EXCELLERAT, HLRS participates in proj-

ence, Research, and Art — completed its first 15-week

ects focused on identifying and addressing opportuni-

course on parallel programming. In 2019, it will begin

ties for supercomputing in specific industries. For more

offering a second course, focusing on simulation. In

than 10 years it has been a key partner in the Automo-

the coming years, the curriculum will grow to include

tive Solution Center Stuttgart (see page 30), an alliance

additional modules on visualization; performance

of companies that use simulation in automotive devel-

optimization; cluster, cloud, and high-performance

opment and design. In 2018, it also joined the Hoch-

computing; ecology and economics of supercomput-

schule der Medien und the Center for Art and Media in

ing; and data management — topics that are of great

Karlsruhe (ZKM) in founding the Media Solution Cen-

interest to system administrators, programmers, and

ter, an alliance of organizations interested in explor-

HPC software users in industry.

ing opportunities for high-performance computing in

The Supercomputing-Akademie is just one compo-

the media arts. (see page 27). In coming years, HLRS

nent of an ongoing dialogue between HLRS and com-

intends to implement this solution center model to

putational engineers in industry. Working with the

focus on increasing access to high-performance com-

nonprofit organization SICOS-BW, the center has for

puting for health research and other areas.

the last two years also organized the Industrial HPC User Roundtable (iHURT), a full-day event designed to

Supercomputing-Akademie trains HPC experts in industry

promote information exchange in industrial R&D. This

Providing HPC training has long been a key activity for

important challenges that HPC users in industry face,

HLRS. In 2018, however, the center opened a new con-

and will continue to guide its development of new infra-

tinuing education program designed with the needs of

structure and services in the coming years.

dialogue also enables HLRS to understand the most

Aron Precht (VP Sales, DACH & Russia, Hewlett Packard Enterprise)

Heiko Meyer (General Manager und Vice President, Enterprise Group, Hewlett-Packard GmbH) Jan Gerken (Chancellor, University of Stuttgart) Michael M. Resch (Director, HLRS)

computational engineers in industry in mind. To make these training activities as accessible as pos-

Putting all the pieces together

Installing the next-generation Hawk supercomputer

sible, the new program, called the Supercomputing-

In the coming decade, simulation, visualization, data

in 2019 will be one important step toward this goal.

Akademie, uses a “blended learning” format. This

analytics, and machine learning will remain critical for

Additionally, HLRS’s ambitious training program and

means that only a small part of the training actually

research and development, and will play important

its outreach and training programs focused on indus-

takes place in a classroom. Approximately 80% of a

roles in addressing some of our greatest challenges.

try are important parts of this effort. Through such

participant’s time spent learning can be done online,

By identifying and focusing on the needs of engineers

activities, HLRS is positioned to be a key partner for

making it easy for someone with a full-time job to join in

and applied scientists, HLRS aims to provide the wid-

innovation in Baden-Württemberg, in Germany, in

the course alongside his or her normal responsibilities.

est possible access to such tools.

Europe, and beyond.

12 / 13

HLRS ANNUAL REPORT 2018

(CW)

Günther Oettinger

HLRS Director Michael Resch

NEWS HIGHLIGHTS

Commissioner Oettinger Visits HLRS

14 / 15

HLRS ANNUAL REPORT 2018

As a longtime supporter of HLRS in his previous role as minister president of the state of Baden-Württemberg, European Commissioner for Budget and Human Resources Günther Oettinger is acquainted with the needs and challenges facing high-performance computing. On May 17 Oettinger and HLRS Director Michael Resch discussed the importance of a European strateg­y for high-performance computing capable of providing European scientists access to state-of-the-art technology. One topic of discussion was how European HPC will be supported through the EuroHPC Joint Undertaking (EuroHPC), which plans to fund two pre-exayears. Both Oettinger and Resch expressed a need to

HLRS Supports Crime Investigation with 3D Visualization

make high-performance computing available and pro-

An October 8, 2018, episode of the ARD television pro-

ductive for both scientific applications and industrial

gram Kriminalreport focused on how 3D modelling and

research.

simulation can help solve complex crimes. The episode

flop systems and two exaflop systems in the coming



(LB)

detailed methods the State Office of Criminal Investigations, Baden-Württemberg (LKA) uses in analyzing 3D images of crime scenes, as well as HLRS’s recent advances in developing such approaches. HLRS Visualization Department leader Uwe Wössner explained that investigators can use visualization environments to accurately study bullet trajectories or conduct autopsies, among other aspects of crime scene investigation. Such methods can provide critical forensic evidence that would be hard to gather using other methods. In recent years, HLRS has been collaborating with the LKA to develop new methods for 3D-digital reconstruction of crime scenes to study them even more effectively.

(EG)

Golden Spike Awards Recognize Outstanding Research Each year, the Results and Review Workshop offers an opportunity for HLRS supercomputer users to discuss their research and best practices in HPC. On October 4-5, the 21st annual workshop featured 41 user projects in 22 scientific talks and 19 posters. The event showcased research from computational fluid dynamics, climate research, computer science, chemistry, physics, and biology. At the workshop HLRS also presented Golden Spike Awards to representatives of three outstanding projects: Mathis Bode of RWTH Aachen (“Towards clean propulsion with synthetic fuels”), Travis Jones of the Fritz Haber Institute of the Max Planck Society (“Sulfur in ethylene epoxidation on sil-

Girls’ Day 2018

ver”), and Christoph Wenzel of the University of Stutt-

Girls’ Day is an annual event that takes place across

gart (“DNS of compressible turbulent boundary layers

Germany to encourage girls to pursue careers in sci-

with adverse pressure gradients”).

ence, engineering, and information technology. As

HPC User Forum Considers State of European Supercomputing

the meeting featured presentations by Leonardo Flores (European Commission) on the EuroHPC Joint Under-

part of the 2018 festival, ten students between the

Organized by HPC industry consultants Hyperion

taking; by Hyperion’s Earl Joseph and Steve Conway

ages of ten and twelve visited HLRS to learn about

Research, the 2018 HPC User Forum took place at HLRS

on trends in the HPC industry; by leading HPC hard-

the world of high-performance computing. Visitors

on October 1-2. It offered an intimate venue for promoting

ware manufacturers about technical challenges they

received a hands-on introduction to computing hard-

interactions among HPC users and technology suppli-

face; by industrial HPC users focusing on autonomous

ware, a tour of the HLRS computing room, and training

ers, and provided insights into how supercomputers and

driving; and by SICOS-BW, the Fortissimo Project, and

in using the programming language Scratch. On their

their uses are evolving. In addition to updates from HLRS

the EXCELLERAT project looking at unique challenges

last stop, they hopped into a flight simulator in HLRS’s

and its partners in the Gauss Centre for Supercomputing,

faced by HPC users in industry.

3D visualization room to paraglide virtually above the

(CW)

(CW)

Black Forest. After learning to “fly,” each girl obtained an HLRS pilot’s license. Girls’ Day offered the students

Stuttgart and Wuhan Sign Collaboration Agreement

enjoyable experiences that would raise their interest in

On April 3, HLRS took its next step in strengthening

would take home good memories.

technology; at the end of the day all agreed that they (AL)

its scientific cooperation in Asia. In a short ceremony in Wuhan, China, Supercomputing Center of Wuhan University Director Dengyi Zhang and HLRS Director

Illustris Project Featured on Postage Stamp

project released five simultaneous papers document-

Michael Resch signed an agreement pledging collabo-

In December the Deutsche Post announced the release

ing results of the first two IllustrisTNG simulations. The

ration for the next three years. HLRS and the Supercom-

of a new postage stamp design featuring the Illustris

project is a two-time winner of the Golden Spike Award,

puting Center at Wuhan University plan to exchange

project. The international scientific collaboration, led

an annual prize recognizing outstanding research using

scientists and to focus on key research topics in

by Volker Springel at the Heidelberg Institute for The-

HLRS computing resources.

high-performance computing. Both sides will also share

oretical Studies in partnership with researchers at the

experience in installing large-scale computing systems.

Massachusetts Institute of Technology, Harvard Uni-

In addition to its cooperation with Wuhan University,

versity, and the Max Planck Institute for Astrophysics,

HLRS maintains agreements with many other HPC cen-

has created the world’s largest cosmological simulation

ters in Asia, Europe, Russia, and the United States with

of galaxy formation. Currently, the Illustris team is working with HLRS to run its next-generation simulations

the goal of promoting the exchange of HPC knowledge and expertise.

(CW)

16 / 17

Dengyi Zhang

HLRS ANNUAL REPORT 2018

Michael M. Resch

on Hazel Hen, a project called IllustrisTNG. In 2018 the

(CW)

Artist Visualizes Air Quality Problems in Stuttgart

optimal platform for users whose computation relies

For the Drehmoment Produktionskunst-Festival, orga-

on millions of small data files. Because the cloud-

nized by the KulturRegion Stuttgart, artist Michael Saup

based platform is connected to Baden-Württemberg’s

collaborated with HLRS to make the city’s notorious air

BelWü high-speed digital network, it also offers the

pollution visible. His virtual reality project, called Staub

possibility for users to access their data directly from

(Dust), interpreted and visualized sensor measure-

their home institutions, speeding their research. In

ments of particulate emissions, integrating them with

addition, the scalability and affordability of the new

open-source street imagery and real-time data about

platform will permit users to save data on the system

traffic conditions. Visitors to the installation could wear

© FSE/Hikisch Gergely

for the duration of their project allocations, rather than 60 days as required on Lustre. Moving suitable proj-

ment. “It is inspirational to work with people who use

New Cloud-Based File System Offers Users Greater Flexibility

similar tools but have a completely different sense of

To enhance its user service, HLRS implemented a new

solution but also reduces bottlenecks on the Lustre

what to do with them,” Saup said. “Exchanging ideas

Quobyte cloud-based file system. The system comple-

system so that it can be dedicated to projects for which

with people working in computing ... can lead to new

ments HLRS’s existing Lustre file system, offering an

it is best suited.

a VR headset to explore this otherwise hidden environ-

ects to the Quobyte system not only offers a cheaper

(CW)

narratives about the future and what it could look like.” The installation was shown in St. Maria’s Church in

Stuttgart Wins Formula SAE Michigan for Third Consecutive Year

Stuttgart for two weeks in October.



(CW)

Minister of Hungary Visits to Discuss HPC in Europe, Industry On October 19, Hungarian Minister for Innovation and

of Stuttgart won the 2018 Formula SAE Michigan, the

Technology Prof. Dr. Laszlo Palkovics and Hungarian

unofficial world cup of the international Formula Stu-

Foreign Trade Attaché Dr. David Bencsik visited HLRS.

dent network. It was the third consecutive year that a

Following an introduction to HLRS research and indus-

team from the University of Stuttgart won the contest.

trial activities — focusing on collaboration between

In addition to constructing a racecar and testing it on

HLRS and the automotive engineering industry, par-

the track, team members presented it to experts from

ticularly with respect to research on autonomous driv-

the motorsport, automotive, and aerospace industries.

ing vehicles — Minister Palkovics and HLRS Director

Teams are judged on the car’s performance, design,

Michael Resch discussed the current high-performance

construction, cost, and their sales presentations. HLRS

computing landscape in Europe, including the EuroHPC

has been an official supporter of the University of Stutt-

joint undertaking and the Partnership for Advanced

Joseph Schuchart Wins Best Paper Award at IWOMP 2018

gart team since 2016, sponsoring computing hours for

Computing in Europe. HLRS has existing collaborations

At the International Workshop on OpenMP in Barce-

with the University of Pécs and Hungarian Academy of

lona, HLRS scientist Joseph Schuchart was recognized

Science, and both sides agreed to continue searching

for his investigation of taskyield, an important piece

for new avenues for cooperation in autonomous driving

of software for scheduling parallel computing tasks.

and high-performance computing.

Schuchart examined several potential implementations

computational fluid dynamics simulations to improve the car’s aerodynamics. “The support from HLRS is one of the cornerstones for developing our race cars,” said team member Johannes Burgbacher.

© Michael Saup

The Formula Student Racing Team of the University

(AL)

(EG)

Joseph Schuchart

of taskyield, comparing their correctness and performance. Because it can be difficult to know how task-

Third Annual German-Russian Workshop

of Applied Mathematics, Russian Academy of Sci-

yield prioritizes tasks, he and his colleagues presented

On April 23-26, investigators from academic insti-

ences) and Prof. Dr.Sci. Andrei Klemeshev (Rector,

a test that can detect which variant of taskyield is used

tutes across Russia and Germany met in Kalinin-

Immanuel Kant Baltic Federal University), the packed

in different OpenMP compilers and runtimes. “Our

grad to forge closer ties between the two research

three-day conference explored a wide range of topics

motivation is to help developers and users by informing

communities and identify opportunities for potential

related to HPC and its applications. Talks focused on

them about the weak spots of the taskyield feature and

shared research projects. The meeting was the third

scientific challenges in fields such as computational

providing a tool that offers insights into the inner work-

in a series of annual workshops organized to promote

fluid dynamics, cosmology, and structural mechanics,

ings of OpenMP implementations,” says Schuchart. In

international dialogue. Following welcoming remarks

and on cutting-edge methods in high-performance

by Prof. Dr.Sci. Boris Chetverushkin (Keldysh Institute

computing.

18 / 19

HLRS ANNUAL REPORT 2018

(CW)

the future he intends to explore other types of applicaMichael Resch

Laszlo Palkovics

tions and add to the research on OpenMP tasks.

(AL)

Run for Digitalization On July 27, digitalization experts ran around the Uni-

New Book: High Performance Computing in Science and Engineering ‘17

versity of Stuttgart’s Vaihingen campus — literally.

Including findings from researchers using HLRS

Held with special guests Thomas Strobl, Baden-Würt-

high-performance computing systems in 2017, this

temberg’s Minister of the Interior, Digitalization and

book presents the state-of-the-art in supercomputer

Migration, and ultramarathon runner Jürgen Mennel,

simulation. The reports cover all fields of computational

the event, titled “Digitalisierung: Läuft,” raised aware-

science and engineering, ranging from computational

ness of Baden-Württemberg as a location of digital

fluid dynamics to computational physics, and from

expertise. During a press conference, HLRS Director

chemistry to computer science, with a special emphasis

HLRS Opens its Doors on Tag der Wissenschaft

Michael Resch emphasized that the state has the ideal

on industrially relevant applications. Presenting find-

On June 30 HLRS took part in the Tag der Wissenschaft

foundation for a successful digitalization strategy,

ings made possible by HLRS’s Hazel Hen supercom-

(Science Day) at the University of Stuttgart. Welcoming

pointing out that HLRS’s successful collaborations

puter, this volume covers the main methods in high-per-

more than 300 visitors, the event offered the general

with municipalities and German industry could be fur-

formance computing

public an opportunity to experience high-performance

ther developed through collaboration among politics,

and

computing and to learn about its use in research and

industry, and the scientific community. Strobl praised

that deliver a high

The spring of 2018 arrived at HLRS with more color

technology development. Visitors toured the HLRS

Baden-Württemberg’s progress in digitalization and

level

sustained

than in the past. Led by the HLRS Sustainability in HPC

computer room and visited the CAVE, a 3D virtual real-

encouraged ongoing persistence in the effort.

applications of

HLRS Blooms

performance. Scien-

Centers project, employees volunteered in late 2017 to

ity facility. There, they enjoyed the thrills of piloting a

tists and engineers

plant beds of flower bulbs on the HLRS grounds. Their

driving simulator through city streets or flying a para-

will find accomplish-

efforts paid off in April, when fields of crocuses and

gliding simulator high above a virtual landscape. Staff

ments in achieving

narzissuses burst through the earth. The activity was

also demonstrated visualization applications for city

the best performance

organized with the twin goals of strengthening biodi-

planning, and HLRS Director Michael Resch presented

for production codes

versity on the HLRS grounds and promoting feelings

of particular interest.

of wellbeing among the staff. In this way, it supported

The book includes a

the general mission of the HLRS sustainability project,

wealth of color illus-

which aims through a wide array of activities to simul-

trations and tables of

taneously address environmental, economic, and social

results.

dimensions of sustainability.

derwerke der Technik und des Geistes.” The talk provided a lively introduction to supercomputing as well as its cultural and philosophical context.

© courtesy of KISTI

a lecture titled “Simulation auf Supercomputern: Wun-

(LB)

(CW)

(CW)

Sustainability Days

HLRS and KISTI Identify Collaboration Opportunities

This year‘s Nachhaltigkeitstage (Sustainability Days)

Meeting in Daejeon, South Korea, HLRS Director

2nd Industrial HPC User Roundtable

at the University of Stuttgart took place in June as

Michael Resch and Korean Institute of Science and

Organized by HLRS and SICOS-BW, the second annual

including from Pfinder (supplier for the automobile

part of N!-Tage, a program focusing on sustainability

Technology Information (KISTI) President Dr. Choi

Industrial HPC User Roundtable (iHURT) focused on

industry), Putzmeister (cement pumping technology),

efforts across the state of Baden-Württemberg. Bring-

Heeyoon identified several collaboration opportunities.

the diverse challenges that industrial users of HPC

Bosch, Porsche, Stihl, and BASF. Lectures by HPC users

ing together sustainability initiatives at the University

One specific topic involves developing new tools for

face, including complex programming codes, new

provided insights into a variety of industrial HPC user

— including HLRS’s sustainability in supercomputing

digital product design and fabrication. By combining

technologies, licensing models, user support, and the

scenarios, covering topics including methods for sim-

program — the event promoted networking and idea

HEMOS-Fluid (a software platform developed at KISTI

organization of workflows, among others. Among the

ulating fire protection systems, the use of HPC in the

exchange. Over the course of two days, it featured talks

for fluid analysis) with COVISE (a software environment

more than 30 participants were representatives from

development of new cement pumps, and the impor-

about sustainability challenges and progress the univer-

developed at HLRS for simulation post-processing and

companies spanning a wide assortment of industries,

tance of HPC for the chemical industry.

sity is making, with a focus on the challenges of energy

visualization), the centers see opportunities to develop

consumption and cooling in high-performance comput-

a powerful new approach for visualizing simulation

ing. “With the Nachhaltigkeitstage we want to demon-

data. The centers anticipate that pooling their institutes’

strate that sustainable action has to be an overarching

expertise and strengthening collaboration on this topic

goal that is based on a common conviction and will ben-

could benefit the competitiveness of small- and medi-

efit all parties involved in the long run,” says Sabine Eger,

um-scale enterprises (SMEs) that use simulation in

a member of the HLRS sustainability team.

product design and development.

(AL)

(CW)

(CW)

Steffen Poppitz (Bosch) Stefan Hildenbrand (Pfinder KG) Matthias Sitte (BASF)

20 / 21

HLRS ANNUAL REPORT 2018

Susanne Kilian (hhpberlin)

Christian Simander (Putzmeister Engineering)

Markus Klietmann (SICOS-BW GmbH)

New European Centres of Excellence Funded

HLRS Visualization Supports Stuttgart 21 Construction

Four new COEs will address important social and industrial challenges.

3D virtual reality models created at HLRS helped engineers to successfully build a complex new kind of reinforced concrete column for the new Stuttgart train station.

In 2018 the European Commission approved all propos-

HiDALGO will combine data analysis, machine learning,

als for “Centres of Excellence on HPC” (CoE’s) contain-

and HPC to address global challenges in finance, health

In 2018 construction began on the most architectonically

staff created an interactive digital model of one of the

ing an HLRS contribution, with HLRS participating in

care, migration, and energy. In this unique consortium,

distinctive elements of the new Stuttgart central train

columns for display in the CAVE, a walk-in virtual reality

four out of nine funded EU CoE’s in this call. HLRS will

the computational and social sciences will join forces,

station. The so-called “Kelchstütze” are a set of mas-

environment.

receive three-year funding totalling €7 million. The proj-

with HLRS providing HPC expertise and its infrastruc-

sive, chalice-shaped columns that will cradle windows

Meeting at HLRS, representatives of the Deutsche Bahn

ects started operation in late 2018.

ture for high-performance data analysis.

to bring natural light into the subterranean station. At a

and construction managers at Ed. Züblin AG could then

HLRS will play the largest role in EXCELLERAT (Euro-

HLRS is also part of FocusCoE, a collaboration support

height of over 12 meters and a diameter of approximately

explore and interact with the visualization to develop a

pean Center of Excellence for Engineering Applica-

action that will bring all nine CoEs together to facilitate

32 meters, the elegantly curving forms of reinforced con-

strategy for erecting the column most efficiently and at

tions), acting as project coordinator thanks to its experi-

knowledge exchange. HLRS will contribute to training

crete promise to become a landmark in the city.

the highest possible structural integrity. Once the first

ence with industrial engineering partners. EXCELLERAT

activities aimed at identifying and solving shared prob-

For construction engineers working on the project —

column was poured onsite, the HLRS team also created

will support industrial HPC users by improving simula-

lems among the CoEs.

called Stuttgart 21 — the columns’ complex geometry

3D scans of the physical structure to compare the vir-

tion technologies for engineering applications, mak-

created unique challenges. Although CAD drawings of

tual model to the actual column and determine whether

ing the development and testing of high-tech products

the structures existed, the curving volumes and need to

it had been erected to specifications.

safer and more efficient. The goal of EXCELLERAT is to

precisely arrange an enormous number of steel bars to

According to Uwe Wössner, who heads the HLRS

support industry in scaling such applications to (pre)-

reinforce the concrete proved more challenging than is

Visualization Department, “The effort was extremely

exa­scale HPC systems.

the case with more conventional pillars.

productive, as the participants had an easier time visu-

To better plan the structures’ fabrication, engineers at

alizing the complex geometry, identified problems in

the Deutsche Bahn called upon HLRS to develop a 3D

the original design, and were able to develop effective

visualization. Using CAD and point cloud data, HLRS

solutions.”

HLRS is contributing to four new European Commission COE’s.

(LB)

POP 2 will support scientific users and software developers by helping them more efficiently parallelize and, in turn, improve the performance of their codes. Researchers who optimize their codes with HPC experts ultimately make better use of European supercomputers and improve their individual productivity. The CheeSE project will use HPC to predict natural disasters quicker and more reliably in order to reduce

A 3D visualization of one of the Stuttgart 21 “Kelchstütze,” in the HLRS CAVE.

response time.

22 / 23

HLRS ANNUAL REPORT 2018

(CW)

Supercomputing-Akademie Completes First Course

Sociopolitical Advisory Board to Help Guide HLRS Vision

Using a “blended learning” approach, the new training program will provide engineers and computer scientists with

A new multidisciplinary committee will provide counsel on how computer simulation can benefit society and help

essential HPC skills.

address its key challenges. in partnership with SICOS-BW, which promotes access

three-month training module offered by the Supercom-

to high-performance computing for small and medi-

puting-Akademie. Participants included engineers and

um-sized enterprises (SMEs). The Supercomputing-

computer scientists interested in developing their skills

Akademie aims to provide continuting education pro-

for working in high-performance computing environ-

gram that is attractive for both large and small compa-

ments.

nies.

The course, which focused on code development

The Supercomputing-Akademie courses are conceived

for parallel computing systems, included interactive

using a “blended learning” format, meaning that only

online content, animations, programming exercises,

a small part of the training takes place in a classroom.

and explanatory videos. Participants also met with one

Approximately 80% of a participant’s course time is

another and with instructors in virtual online meetings.

spent online, reducing the need to travel to participate.

The course covered the architecture of parallel systems

Courses are held in the German language, focusing on

and programming models such as MPI and OpenMP,

the needs of the country’s precision technology com-

and taught participants how to use programming librar-

munity.

From predicting weather and discovering new medi-

of society, representing socially relevant disciplines

ies and key parallelization concepts efficiently.

The Supercomputing-Akademie’s training program

cines, to enabling the more efficient design of cars and

including nursing, architecture, design, art, education,

With funding from the European Union and the State

continues to develop with future modules to cover

traffic flow, simulation indirectly benefits society in

and journalism.

of Baden-Württemberg, HLRS has been developing

topics such as simulation; visualization; performance

many ways. However, it is also often driven by scientific

“Simulation can make social phenomena more acces-

the Supercomputing-Akademie in collaboration with

optimization; cluster, cloud and HPC; ecology and eco-

or economic interests, while knowledge about it can be

sible to decision-makers and affected people,” said

the University of Freiburg and University of Ulm, and

nomics of supercomputing; and data management.

limited to experts in a particular field. This reality can

advisory board chairman Prof. Ortwin Renn, Scientific

lead to limitations in vision about how simulation could

Director at the Institute for Advanced Sustainability

best benefit society.

Studies in Potsdam. “Against the background of this

“There is no question that computer simulations pro-

progress, however, it must be ensured that ethical val-

vide added value, but they must not be a privilege of

ues are not violated and that social preferences are

elites,” says HLRS Director Michael Resch. For this

taken into account.”

reason, he and Andreas Kaminski, head of the HLRS

Discussions at the inaugural meeting focused on topics

Department of Philosophy of Computer Simulation,

in which simulation can have an impact, such as urban

convened the inaugural meeting of a new sociopoliti-

development, social inequality, and populism, as well as

cal advisory board on April 9. “We want to ensure that

questions regarding the methodology of simulations. In

a wider cross-section of society benefits from the use

the future the board will gather and refine such thematic

of simulation,” Resch explained. The sociopolitical advi-

and methodologic ideas, with the goal of developing a

sory board includes 13 members from different areas

pilot proposal for a research project.

24 / 25

HLRS ANNUAL REPORT 2018

(CW)

© Julian Rettig

Attendees at the first HLRS Sociopolitical Advisory Board meeting.

On July 17, fourteen participants completed the first

(LB)

SimTech Named National Center of Excellence

Media Solution Center to Promote Innovation in Film and Digital Art

A University of Stuttgart research consortium focused on data-integrated simulation technologies was awarded a

A new association will facilitate the integration of applications of high-performance computing and simulation into

prestigious seven-year grant.

media production.

Since 2007 the Excellenzcluster SimTech (SimTech

to see it once again recognized as a national center of

In October HLRS, together with the Center for Art

organize scientific continuing education activities that

Center of Excellence) has nurtured a multidisciplinary

excellence,” said Prof. Resch. “Our collaborations with

and Media Karlsruhe (ZKM) and the Hochschule der

train people working in media to use high-performance

community at the University of Stuttgart focused on the

scientific investigators will not only support exciting

Medien Stuttgart (HdM) founded a new association

computing more effectively. This includes focusing on

field of simulation. The cluster includes developers of

new discoveries, but also enable HLRS to continue to

called the Media Solution Center Baden-Württemberg

the use and development of software for high-perfor-

models, methods, and other simulation technologies as

improve its capabilities.”

lenges in media production where high-performance

The MSC is open to membership of other individuals

research in molecular and particle physics, mechanics,

computing and simulation could help, and to accel-

and companies working in the media industry — includ-

dynamic systems, and numerical mathematics.

erate the development of technologies that address

ing animation studios, visual effects specialists, and

Recognizing the University of Stuttgart’s role as

these needs.

arts organizations — as well as leaders of scientific

a national leader in simulation technology and

The MSC will undertake collaborative research and

and technical research groups in Baden-Württemberg

applications, the Deutsche Forschungsgemeinschaft

development projects aimed at solving problems that

whose work has relevance for media production. Ulti-

media producers face. It will also facilitate sharing of

mately, the MSC aims to facilitate a precompetitive

the knowledge and technologies that result from these

exchange of knowledge and information among all

collaborations and organize events focusing on themes

organizations that choose to participate.

that are important to the media industry. One key set

Prof. Dr. Bernd Eberhardt of the Hochschule der

of application areas in which the MSC will be engaged

Medien was named chairman of the association. HLRS

includes animation, simulation, and visual effects

Director Prof. Dr. Michael Resch will serve as deputy

(VFX). In coordination with the HdM, HLRS will also

chairman.

This highly selective program supports key research centers that are advancing Germany’s national strategy for scientific excellence. The new grant will enable SimTech to grow in new directions. For the next seven years, it will focus on increasing capacities for data-driven simulation. Such applications are important in fields in which large amounts of data are produced using sensors, experiments, simulations, and other methods. High-performance computing could enable simulation based on such datasets and so improving supercomputing infrastructure to enable efficient and effective high-performance data analytics is one of the key problems facing the field. For this reason, HLRS is an important partner in the SimTech cluster, with HLRS Director Michael Resch participating as a principal investigator for high-performance computing systems. “HLRS is well networked within SimTech and so we are delighted

26 / 27

HLRS ANNUAL REPORT 2018

(CW)

HLRS was recognized in 2017 with an HPC Innovation Award for its work with M.A.R.K. 13 and Studio 100 Media, who used high-performance computing to accelerate production of the animation film Die Biene Maja (Maya the Bee).

a national center of excellence in its latest funding cycle.

Perfusion MRI is a promising method for supporting therapy for multiple sclerosis. In one project, SimTech researchers conduct detailed small-scale simulations of how MR contrast agents spread in the brain. The method could potentially improve interpretation of images of MS-lesions.

announced on September 27 its selection of SimTech as

(CW)

© Studio 100 Media. Studio B Animation

mance computing environments.

© University of Stuttgart, Max Kovalenko

(MSC). The MSC’s mission is to identify technical chal-

well as computational scientists who use simulation for

Building a Sustainable HPC Infrastructure

A new European Union Centre of Excellence coordinated by HLRS will support industrial HPC users in adopting more

A two-day meeting at HLRS focused on the challenges of designing environmentally responsible facilities for super-

powerful hardware and software technologies.

computing.

Participants at the kick-off meeting of the EXCELLERAT Project.

EXCELLERAT Will Bring HPC Applications to Engineering Industry

HLRS launched EXCELLERAT (excellerat.eu), a new

industry to take advantage of the opportunities that the

The march toward faster, more powerful supercom-

planning can make HPC more sustainable, while at the

European Union Centre of Excellence (CoE) for engineer-

latest HPC technologies offer.

puters shows few signs of slowing down. Although

same time highlighting some important challenges and

ing applications. With roughly €8 million in EU funding,

EXCELLERAT will work with consortium partners who

such growth is good for science and technology devel-

problems to be addressed.

the center includes 13 partners in 7 European countries.

have developed important codes for academic appli-

opment, it also leads to increased physical needs for

The goal of any design for a supercomputing center is

Its mission is to accelerate technology transfer of lead-

cations in engineering fields such as aerospace, auto-

supercomputing centers — building spaces able to

to have the newest technology installed when the cen-

ing-edge HPC developments to the engineering sector.

motive, combustion, and fluid dynamics. To facilitate

accommodate larger machines, cooling systems that

ter opens. But considering the fact that erecting a brick

EXCELLERAT will facilitate the further development of

the codes’ integration into real-life industrial applica-

efficiently manage greater heat production, and energy

and mortar structure can take 5-7 years, the best tech-

important codes for high-tech engineering, maximizing

tions, these partners will work closely with end users

to keep everything running. How can the need for more

nologies often only become available after a project is

their scalability to increasingly large computing archi-

outside the consortium. This will ultimately lead to fast

resources be managed in a sustainable way?

underway. As the meeting revealed, this fact requires

tectures and supporting the technology transfer that

feedback-cycles in all areas of the HPC engineering

On October 23-24, IT specialists and research infra-

close communication among stakeholders and a plan-

will enable their uptake within the industrial environ-

application lifecycle, from consultation on method-

structure experts from the German-speaking HPC

ning approach that accounts for future expansion or

ment. These activities will support engineers through

ology and code implementation to data transfer and

community met at HLRS to discuss holistic strategies

other potential requirements that could arise when new

the entire HPC engineering application lifecycle, includ-

code optimization. End users will benefit by gaining

for addressing such challenges. The event — the sec-

technologies materialize. In addition, close cooperation

ing data pre-processing, code optimization, application

first-hand access to the project results. This concept of

ond annual meeting organized by HLRS on the topic

with funding agencies and implementing better metrics

execution, and post-processing. In addition, EXCEL-

a one-stop-shop for all services is unique in the area of

of sustainability in supercomputing centers — revealed

for tracking progress toward sustainability are import-

LERAT will provide training that prepares engineers in

industrial HPC.

how sensors, new cooling technologies, and creative

ant parts of the puzzle.

28 / 29

HLRS ANNUAL REPORT 2018

(LB)

(CW)

Networking the German Auto Industry: An Interview with Alexander Walser

The projects we work on are very diverse in terms of the number of computing cores used or of the parallelization necessary to achieve research goals. By including HLRS in project planning, we also learn about potential

asc(s Director Alexander F. Walser

problems and how to address them. Another important question for OEMs and small businesses is how to design a workflow that connects the workstation under your desk with the HLRS cluster system. In the past we have built workflows in which data and results are automatically transferred between classical workstations and HLRS using only the necCompetition in the automobile industry is intense, but the Automotive Simulation Center Stuttgart has showed that

essary software resources. In practice this means that

cooperation can offer important advantages.

engineers don’t just push everything onto the HLRS supercomputer, but rather they can integrate it into their

Car manufacturer Porsche has long used computer sim-

of interested parties necessary to get projects off the

ulation resources at HLRS to develop better vehicles. In

ground, in terms of expertise or financing.

the early 2000s, however, it became clear that exist-

It is also important to our mandate to integrate small

ing commercial software packages were not keeping

and medium-sized enterprises (SMEs), as well as scien-

pace with the constantly doubling performance of new

tific institutions. SMEs sometimes develop very innova-

supercomputing hardware. To accelerate the develop-

tive software packages, but if the interfaces aren’t built

Ten years ago no one was talking about self-driving

ment of more efficient and effective simulation soft-

correctly or if the workflow doesn’t fit, industry might

vehicles. Electric cars were a theme that was in the air,

ware, Porsche joined forces in 2008 with HLRS, Daim-

not be able to use them. Getting OEMs and SMEs to

but not something that was as prominent as it is today.

ler, Opel, and other companies — with support of the

work together reduces this risk. Building these relation-

Now everything must develop much, much faster.

Baden-Württemberg Ministry for Science, Research,

ships also incorporates industry’s needs into software

Virtual crash tests — which are necessary for safe and

and the Arts — to found the Automotive Simulation Cen-

development early and makes it possible to design

efficient development — are also becoming increas-

ter Stuttgart, also called asc(s.

solutions that meet the needs of multiple OEMs at the

ingly important. Countless test scenarios need to be

Since then the asc(s has built a network of large and

same time.

conducted, raising the question of how HPC and driving

small companies from across the automobile indus-

Ten years ago the foundational idea of the asc(s was

simulation can be better combined. In the medium-term,

try, with HLRS playing a central role. Recently, we

very innovative. Today, it fits contemporary needs per-

such tests will have to be conducted on a supercomputer.

spoke with asc(s Director Alexander F. Walser about

fectly and in the future it will be indispensable for driv-

Technological trends related to artificial intelligence

the association’s activities, HLRS’s contributions to

ing innovation.

are also gaining in importance. This will result not only

What are the most important topics in simulation for the future of automobile design and development?

in the use of the newest AI-technologies in driving sys-

them, and why simulation is growing in importance What role does HLRS play in this network?

for the field.

workflows.

tems, but also in asking how we can use AI to lessen

High-performance computing is a key focus at the

process. How can we apply AI in designing simulation

asc(s, a fact that distinguishes our organization from

models? How can we extract more knowledge from the

The asc(s is a nonprofit association that brings car

other interest groups in the automotive industry. It is

simulation results that now exist?

manufacturers, software developers, hardware manu-

only through our work with HLRS that we can maintain

Because of these new trends, the horizontal networking of

facturers, engineering service providers, start-ups, and

this strong focus on HPC.

automobile manufacturers that the asc)s promotes is very

scientific organizations together to steer the future of

Cooperation with HLRS enables us to have access to

important, particularly as new key players enter the mar-

virtual automobile development. Just as an OEM (orig-

the newest hardware architecture. HLRS is always a step

ket that have never been integrated into the supply chain.

inal equipment manufacturer) can buy a screwdriver

ahead of what’s available in industry, where the update

to build a car, we try to ensure that manufacturers

cycles are longer and the cluster sizes are smaller. Using

What plans does the asc(s have to make the auto-

have access to the simulation methods they need. To

HLRS resources, we can show relatively early that new

motive industry better and to prepare for the future of

make this possible, the asc(s gathers the critical mass

methods will work on future cluster systems.

simulation?

30 / 31

HLRS ANNUAL REPORT 2018

© Benjamin Stollenberg

the burden on engineers during the virtual development How do you describe the asc(s?

Student Dives into Data to Predict Train Delays

As a participant in the HLRS program Simulated Worlds, high school student Niklas Knöll used machine learning to investigate a common problem.

Simulation is becoming ever more important for the

Smartphone apps from the Deutsche Bahn and Stutt-

for solving big data problems on supercomputers. He

gart’s local transit networks can show delays of trains

developed models and trained them using machine

and buses in real-time. But wouldn’t it be nice to know

learning algorithms to predict train departures to the

earlier whether there’s time to stop at the bakery for a

minute. In 8 out of 10 cases his approach could clearly

pretzel?

predict whether a city train would leave at least three

As a high school student living in Stuttgart, Niklas

minutes late.

Knöll is very familiar with this problem. In 2018 he was

From a strictly statistical point of view, this result was

one of eight scholars who participated in Simulated

not fully satisfying because the database Niklas used

Worlds, an initiative of the Baden-Württemberg Minis-

was too small for a big data problem. But Simulated

try of Science, Research, and the Arts. Over the school

Worlds’ aspiration is not to produce scientific break-

motivate students to study virtual automobile develop-

year, students receive €1,000 to conduct a research

throughs. Most importantly, it provides the participat-

auto industry. This presents an important question:

ment and to select the best program of study.

project at HLRS, receiving supervision from HLRS

ing scholars with advanced technical competencies

what qualifications will tomorrow’s simulation engineer

The need for experts in simulation is constantly grow-

employees. The aim is to prepare young people for the

and problem-solving skills. These skills were on dis-

or even an engineer in another field need?

ing. By promoting young investigators and helping

digital realm.

play at an awards ceremony at HLRS on July 4, when

At the asc(s we have been asking ourselves how we

them to quickly become integrated into the develop-

At HLRS students were encouraged to analyze big

Niklas and the other participating students presented

could support our member organizations by helping to

ment process, we can not only offer advantages for

datasets. Niklas accessed public data from the Deut-

their projects and results. According to Simulated

ensure that university students gain the right knowl-

industry but also strengthen Baden-Württemberg as a

sche Bahn and learned to analyze it using the pro-

Worlds project coordinator Doris Lindner, “Everyone

edge during their studies. This includes engaging in

center for research. We expect that this kind of cooper-

gramming language Python and the programming

was impressed by how thoroughly the students had

dialogue with universities, as well as finding ways to

ation will grow stronger in the future.

framework Apache Spark, tools that are often used

studied their subject.”

32 / 33

HLRS ANNUAL REPORT 2018

(CW)

(LB)

System Usage

HLRS by the Numbers

Education and Training

1.28

Visiting Researchers

Staff

155

BILLION

107

36

41

63

25 Teilnehmer 997 Participants

129 Course-Days

CORE HOURS PRODUCED USER PROJECTS INDUSTRIAL CUSTOMERS USER PUBLICATIONS

23

CONTINUING EDUCATION COURSES

12

SCIENTIFIC WORKSHOPS 451 Participants 24 Days

22

UNIVERSITY LECTURES 2,733 Participants 36 SWS

78

SCIENTISTS

49

NONSCIENTISTS

Visitors at HLRS

RESEARCH ASSISTANTS

2,142

5

Third-Party Funds Talks

8,136,13 8

44

€

52 %

Staff Publications

20 2 34 / 35

PAPERS IN JOURNALS, BOOKS, AND CONFERENCE PROCEEDINGS

33 %

11 % 3 %

BOOKS

HLRS ANNUAL REPORT 2018

1 %

FEDERAL GOV. EU STATE GOV. DFG OTHER

Providing personalized guidance to HPC users results in faster codes and more efficient use of computing resources. Since 2007 the Gauss Centre for Supercomputing

from the help of HPC experts in selecting the best tools

(GCS) — the network of Germany’s national supercom-

and optimizing their performance. This can make a big

puting centers that includes HLRS, the Jülich Super-

difference, making it possible to ask new, more com-

computing Centre, and the Leibniz Supercomputing

putationally demanding kinds of scientific questions.

Centre — has coordinated the installation and oper-

Since 2016 HLRS has held a biannual optimization

ation of world-class systems for high-performance

workshop in which Hazel Hen users meet with sup-

computing. In 2018 it expanded in an important new

port staff to identify opportunities for improving their

direction, implementing enhanced user support ser-

codes’ performance. These meetings have been very

vices that will empower scientists to use these sys-

productive, resulting in improvements in runtime and

tems more efficiently.

data transfer speeds; in some cases, codes can end up

“Every time you help individual users to utilize the

running twice as fast, or faster.

machine more efficiently you enhance the capabilities

With grants provided by the Baden-Württemberg Min-

of the computer as a whole,” says Thomas Bönisch,

istry of Science, Research, and Art, HLRS added new

leader of the HLRS user management team. “By ensur-

staff positions in 2018, each of which is dedicated to

ing that applications are optimized for the computing

supporting specific user groups all year round. In this

hardware they are run on, we can get a lot more science

way, deeper collaborative relationships can develop

out of the same computing resources.”

in which experts in scientific domains and experts

Enhanced user support involves more than helping

in high-performance computing regularly exchange

users port codes or transfer data. Instead, scientists

ideas, working together to continually improve com-

receive hands-on, personalized advice for improving

putational tools.

their codes’ performance on HPC resources. This can

This enhanced user support initiative has also made

occur at several levels: selecting and optimizing the

it possible to improve coordination of user support

best algorithm for a particular problem and supercom-

activities between HLRS and its GCS partner institu-

puter, improving its ability to scale up to parallel com-

tions. Representatives of the three centers now meet

puting networks, structuring it to take advantage of

regularly to share knowledge about their accomplish-

unique characteristics of the specific processors that

ments in improving computational efficiency. This

make up the system, and eliminating bottlenecks in

dialogue also enables the three centers to address

the movement of large datasets.

the needs of users who compute at multiple GCS sites

While some computational scientists develop their

in a more coordinated way. In this way, the initiative

own computational tools for analyzing a scientific

promises to raise the performance of users’ codes

problem, others rely on community-built codes or

across Germany’s entire national supercomputing

open-source software. In either case, they can benefit

infrastructure.

36 / 37



HLRS ANNUAL REPORT 2018

(CW)

Attendees at an HLRS optimization workshop receive guidance on improving their codes.

Enhanced User Support Improves Performance of HPC Systems

EUXDAT: Taking Agriculture into the Cloud

PhD Graduates 2018

High-performance data analytics holds enormous potential to improve efficiency in food production.

Three students completed their doctoral studies at HLRS in 2018. Read more about their dissertation topics below.

In the past, farmers consulted books like The Farmer’s

datasets is needed, users could access high-perfor-

Björn Bliese

Almanac to know which crops to plant, when to plant

mance computing resources at HLRS through the web.

An Augmented Reality System Suitable for Geometric Analysis in Product

them, and when to harvest them. As in other industries

Having access to such a product could, for example,

Development

being transformed by the Internet of Things, however,

enable farmers to identify the optimal time to fertilize

Augmented reality offers the potential to perform geometric analyses between

digital tools have initiated new thinking about the future

their crops to produce the best harvest.

physical and virtual models in product development and production planning,

of agriculture. High-resolution weather and climate

Atos, a European provider of business technology,

although today it is rarely used in practice. Bliese’s thesis defines basic require-

models and sensors for measuring soil moisture and

invited HLRS to join the project because of its exper-

ments for a usable AR system for geometric analyses and improves upon the

composition, for example, offer rich sources of data that

tise in high-performance computing, cloud comput-

chosen system components. It provides a process-driven user interface to guide

could make farming more productive. The challenge

ing, and big data analytics. According to Dr. Michael

a user through an AR-assisted investigation. The developed AR system is finally

now is to design systems to integrate, interpret, and

Gienger, who heads the cloud computing research

evaluated within a usability study.

apply this information.

group at HLRS, “For us, this is the best of all possible

As part of a multidiscplinary European Union-funded

projects, because it combines many key areas of exper-

Marius Feilhauer

innovation project called EUXDAT (European e-Infra-

tise at HLRS, while offering an opportunity to test new

Simulation-Supported Safeguarding of Driver Assistance Systems

structure for Extreme Data Analytics in Sustainable

approaches for integrating cloud and HPC technologies

In theory, ensuring that an autonomous vehicle drives safely would require pro-

Development), researchers at HLRS are developing a

in a field that could really benefit from it.”

gramming it to recognize all possible situations, including various traffic or light-

(CW)

software platform to address this need. The goal is to

ing conditions. This is impossible, however. In his thesis, Feilhauer proposes an

create an off-the-shelf product hosted in the cloud that

expandable catalog of traffic scenarios that would grow as it incorporates lessons

could make big data analytics available to anyone in

Learn more about how data analytics and cloud com-

learned based on real or virtual driving test data. Driving scenarios could be inves-

agriculture. In cases where analysis of extremely large

puting can support agriculture at www.euxdat.eu.

tigated in virtual test drives on computers, with the results being fed back into the catalog. Such a resource would improve iteratively, benefiting the training of machine learning applications for autonomous driving.

Matthias Nachtmann Model-Centric Task Debugging at Scale With the increasing complexity of HPC hardware and software, there is a need for tools to support computer scientists and engineers in efficiently porting and optimizing their computational codes regardless of what system they are using. In addition, researchers want to be able to debug their applications within their respective programming models. In his thesis, Nachtmann focuses on further developing this “model-centric” debugging approach, spearheading the redevelopment of the backend of the HLRS-developed TEMANEJO graphical debugging tool.

© Shutterstock/ESB Professional

38 / 39

HLRS ANNUAL REPORT 2018

Looking Inside Simulation‘s “Black Box”

Nicole J. Saam

Johannes Lenhard

Visiting scholars in the HLRS Department of Philosophy of Computer Simulation work toward a clearer understanding

The opacity of simulation

from individual to individual? As defined now, opacity

of computational research and its limitations

Perhaps the most interesting philosophical problem in

is a relational concept. We would love to have a way to

computer simulation is the result of what philosophers

measure this.”

Simulation and other applications of high-perfor-

that a conceptual framework was needed for discuss-

call its “opacity.” Often, a computer simulation behaves

Saam also anticipates a practical use for such a tool.

mance computing have become indispensable tools for

ing the deeper implications of this trend, “For philoso-

in unexpected ways. In many cases it is impossible

“If one could know in advance which kinds of simula-

research in the basic and applied sciences. Employing

phers of science,” Lenhard says, “one important prob-

to determine why, because it is impossible to mecha-

tion models typically suffer from specific problems due

HPC has had many positive impacts, helping to strive

lem is what it means for a scientist to know something.

nistically observe how an algorithm functions. “This

to opacity,” she suggests, “one might be able to iden-

for a deeper understanding of our world and enabling

In the case of computer simulation, is the knowledge it

turns the idea of mathematical modeling on its head,”

tify and avoid typical problems early in the process of

the development of new technologies. As our reliance

produces different from that produced by older meth-

Lenhard says, “because we used to think that mathe-

implementing a model.” What might at first seem to be

on computer simulation grows, however, many wonder

ods? If so, how? As a historian, one can also ask how

matical models should be able to make the causes of a

a theoretical issue could thus enable tangible improve-

how it is changing science, technology, politics, and

the conditions in which computer simulation is used

behavior crystal clear.”

ments in how simulation is done.

society, and how we should react.

have changed over the years. This gives us a better

During her residency at HLRS, Nicole J. Saam has been

As a supercomputing center, HLRS is unusual in that

understanding of how simulation is used today.” In his

developing a systematic method for defining, catego-

The value of interdisciplinary dialogue

for many years it has prioritized discussion about such

research Lenhard has written about features that distin-

rizing, and measuring such opacity. Although her proj-

Bridging the conceptual divides between scientists and

issues. Through its Department of Philosophy of Sim-

guish today’s computer simulation from earlier mathe-

ect is still evolving, such a model might address factors

researchers in the humanities and social sciences is not

ulation, HLRS promotes interdisciplinary dialogue

matical modelling.

such as instability in the numerical model itself, meth-

easy, but Saam’s project suggests why it is important to

among simulation scientists, philosophers, historians

One example is the change in experiments using sim-

ods to simplify a model to reduce computational effort,

make the effort. “Sometimes it can take years or even

of science, sociologists, and other scholars whose

ulation. Supercomputers enable scientists to easily

and the number of different research teams involved in

decades to get scientists and researchers from diverse

insights enable reflection on the nature of simulation.

adjust a mathematical model’s parameters so that it

developing a model, each of whom only understands a

disciplines such as natural sciences, engineering sci-

In 2018, the Department of Philosophy of Simulation

approaches observed data. Deep learning algorithms

small piece of the project. Such criteria address the fact

ences, humanities, or the social sciences to have a pro-

hosted two visiting scholars — PD Dr. Johannes Lenhard

using neural networks are perhaps the best exam-

that opacity in computer simulation can have technical,

ductive conversation,” Saam remarks. “The opportunity

(Philosophy Department, University of Bielefeld) and

ples of this approach; by adjusting parameters, scien-

mathematical, and social origins.

for me to work directly with simulation scientists here at

Prof. Dr. Nicole J. Saam (Institute for Sociology, Fried-

tists gain the ability to model almost any behavior. But

After defining the dimensions of opacity, Saam has

HLRS is very unusual.”

rich-Alexander-Universität Erlangen-Nürnberg). Their

although such a capability can be useful, Lenhard asks,

developed a scientific framework and questionnaire.

Lenhard agrees. “I find it wonderful that HLRS has a

research highlights some key questions facing the phi-

is it still science?

Her team has started conducting interviews with

group whose job it is to talk with interested scientists

losophy of simulation today, and their experiences at

“The idea that general laws can reveal order in chaos is

groups of scientists who run their simulations at HLRS

about their work, and to promote greater self-reflection

HLRS show the benefits of promoting multidisciplinary

historically connected to the idea that these laws can be

in order to gain a more precise understanding of opacity

about what they are doing,” he observes. “It is some-

discourse.

formulated mathematically,” Lenhard says. “But newer

in all of its different forms.

thing that the scientific community and society in general need to do more often.”

kinds of computer modelling have nothing to do with

“Opacity can look quite different from the perspec-

Simulation yesterday and today

finding such general laws. Today, we hope to describe

tive of a principal investigator in comparison with that

Johannes Lenhard initially trained as a mathematician,

and manage things in a predictable way, even if we don’t

of a graduate student,” Saam points out. “Because of

but in 2001 became captivated by philosophical and

have a law for it.” In this sense, computer simulation is

each’s degree of experience, their perceptions of opac-

historical questions related to the growing use of com-

different from some earlier scientific approaches, but

ity can be different. Is opacity an objective condition of

puting across many scientific disciplines. He realized

offers completely new options.

simulation, or is it a subjective experience that varies

40 / 41

HLRS ANNUAL REPORT 2018

(CW)

USER RESEARCH 42 / 43

HLRS ANNUAL REPORT 2018

Supercomputer Enables Sound Prediction Model for Noise Control

Combining principles from computational fluid dynamics and acoustics, researchers at the TU Berlin have developed a model that could simplify the process of designing noise cancelling structures for airplanes, ships, and ventilation systems. Noise-cancelling headphones have become a popular

In a paper published in the journal Acta Mechanica,

accessory for frequent flyers. By analyzing the back-

Lewin Stein and Jörn Sesterhenn of the TU Berlin

ground frequencies produced by an airplane in flight

describe a new analytical model for sound predic-

and generating an “anti-noise” sound wave that is per-

tion that could make the design of Helmholtz cavities

fectly out of phase, such headphones eliminate dis-

cheaper and more efficient. The development of the

turbing background sounds. Although the headphones

model was facilitated by a dataset produced using

can’t do anything about the cramped seating, they can

direct numerical simulation at the High-Performance

make watching a film or listening to music in flight

Computing Center Stuttgart (HLRS).

nearly as enjoyable as at home.

The analytical model can predict, in a way that is more

To minimize the disturbing noise caused by loud

generally applicable than before, a potential Helm-

machines like cars, ships, and airplanes, acoustic engi-

holtz cavity’s sound spectrum as turbulent air flows

neers use many strategies. One technology, called a

over it. The authors suggest that such a tool could

Helmholtz cavity, is based on a similar concept to that

potentially be used to tune Helmholtz cavities to can-

used in noise-cancelling headphones. Here, engineers

cel out or to avoid any frequency of interest.

build a resonating box that opens to a slit on one side. As air passes over the slit, the box vibrates like a church

Simulation approaches all the scales of nature

organ pipe, producing a tone. By adjusting the size and

When moving air passes over the slit of a Helmholtz

shape of the cavity and its slit, acoustic engineers can

cavity, its flow becomes disrupted and turbulence is

tune it to produce a specific tone that — like the head-

enhanced. Vortices typically arise, detaching from the

phones — cancels a dominant, irritating sound pro-

slit’s upstream edge. Together they form a sheet of

duced by machinery.

vortices that covers the slit and can interact with the

Historically, the process of tuning a Helmholtz cavity was

acoustic vibrations being generated inside the cavity.

a brute force undertaking involving costly and time-con-

The result is a frequency-dependent damping or exci-

suming trial and error. Engineers had no other choice but

tation of the acoustic wave as air passes through this

to physically build and test many different geometries

vortex sheet.

experimentally to find an optimal shape for a specific

In the past it was difficult to study such interactions

application, especially in an environment of turbulent flow.

and their effects numerically without making crude

Today, however, high-performance computing offers

approximations. For the first time, Stein’s simulation

the potential to undertake such tests virtually, making

realistically integrates turbulent and acoustic phe-

the design process faster and easier.

nomena of a Helmholtz cavity excited by a turbulent

30 milliseconds of physical time. Each run of the

predict acoustic proper-

numerical model on HLRS’s Hazel Hen supercom-

ties of other Helmholtz

puter required approximately four 24-hour days, using

cavity geometries and

some 40,000 computing cores. Whereas a physical

airflow

experiment is spatially limited and can only track a few

comparing the extrapo-

physically relevant parameters, each individual DNS

lated model results with

run provides a 20-terabyte dataset that documents all

experimental data pro-

flow variables at all time steps and spaces within the

vided by Joachim Golliard

mesh, delivering a rich resource that can be explored

at the Centre de Transfert

conditions.

By

de Technologie du Mans in France, Stein found that the model did so with great accuracy. The model reported in the paper is optimized for low speed airflows and for low frequencies, such as those found in ventilation systems. It is also designed to be modular so that a cavity that includes complex materials like foam instead of a hard wall can be investigated as well. Stein anticipates that gaining more computing time and access to faster supercomputers would enable him to numerically predict a wider range of © Lewin Stein, TU Berlin

potential resonator shapes and flow conditions. Having recently completed his PhD and now working as a postdoc at the Institute of Fluid Dynamics and Technical Acoustics in the group of Prof. Sesterhenn (TU Berlin), Stein foresees some attractive opportunities to cooperate with industrial partners and possibly to

flow passing over its slit. At an unprecedented reso-

equations — to get as close as possible to the actual

in detail. Stein says that running the simulation over

apply his model in real-life situations. “Although I stud-

lution, it makes it possible to track the flow–acoustic

phenomenon in nature while using as little approxima-

this time period provided a good compromise between

ied theoretical physics,” he explains, “it is fulfilling to

interaction and its implications for the cavity’s reso-

tion as necessary,” Stein says. “Our DNS enabled us to

being able to set up a reliable database and getting

work on problems that reach beyond pure academic

nance.

gain new insights that weren’t there before.”

results in a practical amount of time.

research and can be applied in industry, where people

This achievement is possible using a method called

Stein’s direct numerical simulation divides the sys-

direct numerical simulation (DNS), which describes a gas

tem into a mesh of approximately 1 billion grid points

Moving toward a general sound prediction model

This latest paper is an opportunity to prove the utility

or liquid at a fundamental level. “I’m using the most com-

and simulates more than 100 thousand time steps,

Once the details of the acoustic model were devel-

and applicability of our work. It’s a great moment after

plex form of fluid equations — called the Navier–Stokes

in order to fully resolve the system dynamics for just

oped, the next challenge was to confirm that it could

years of working on a PhD.”

44 / 45

HLRS ANNUAL REPORT 2018

can potentially profit from what you’ve accomplished.

(CW)

Using Computational Chemistry to Investigate New Semiconductor Technologies

molecules develops over time, at the level of atoms and

“To do this,” Tonner explains, “we really need to under-

electrons and at time scales of picoseconds (one pico-

stand how the interfaces between silicon and these

second is one trillionth of a second).

organic compounds look and behave. The reaction

“Increasing computing power has made it possible for

between these two material classes needs to proceed

computational chemistry and quantum chemistry to

in a very controlled manner so that the interface is as

describe real molecular systems. Just 15-20 years ago,

perfect as possible. With computational chemistry we

people could only look at small molecules and had to

can look at the elemental details of these interactions

make rather strong approximations,” Tonner explains.

and processes.”

“The computational chemistry and solid state theory

For example, to cover a slab of silicon, liquid precursor

communities have now solved the problem of paralleliz-

molecules for the constituent atoms of gallium arsenide

ing their codes to operate efficiently on HPC systems.

are placed in a bubbler, where they are then brought into

University of Marburg researchers are exploring how functionalizing silicon with other compounds could make light-

As supercomputers get bigger, we anticipate being able

the gas phase; following a chemical process gallium and

based signaling in semiconductors feasible.

to develop increasingly realistic models for experimen-

arsenide atoms attach to the silicon, forming a GaAs film.

tal systems in materials science.”

How atoms are arranged when they adsorb to a surface

As new methods have become available for under-

Bringing computation to chemistry

standing and manipulating matter at its most funda-

Atoms bond together to form molecules and com-

Toward light-based semiconductors

how this takes place is an open question. Previously, it

mental levels, researchers in the interdisciplinary field

pounds when they approach one another and then

One area in which Tonner is currently using computa-

had been suggested that repulsive relationships among

of materials science have successfully synthesized

trade or share electrons orbiting around their nuclei.

tional chemistry is to study ways to improve silicon for

atoms is the most important factor in “steering” atoms

new kinds of materials. Often the goal is to design

The specific atoms involved, the physical shapes that

use in new kinds of semiconductors. This problem has

into place when they adsorb on a surface. By using DFT

materials that incorporate properties that can be use-

the molecules take, their energetic properties, and how

gained urgency in recent years, as it has become clear

and looking at intriguing features of how electrons are

ful for performing specific functions. Such materials

they interact with other nearby molecules are all prop-

that the microelectronics industry is reaching the limits

distributed, Tonner determined that the ability of atoms

can, for example, be more chemically stable or resis-

erties that give a compound its unique properties. Such

of its ability to improve semiconductors using silicon

to steer other atoms into place on the surface can also

tant to physical breakage, have advantageous electro-

characteristics can determine whether compounds are

alone.

result from attractive dispersive interactions.

magnetic characteristics, or react in predictable ways

likely to remain stable, or whether stresses such as

Tonner and experimental colleagues have been investi-

Gaining a better understanding of these fundamental

to specific environmental conditions.

changes in temperature or pressure could affect their

gating how functionalizing silicon with compounds such

interactions should help designers of optically active

Dr. Ralf Tonner and his research group at the Uni-

reactivity.

as gallium arsenide (GaAs) could enable the design of

semiconductors to improve adsorption of the precur-

versity of Marburg are addressing the challenge of

Tonner uses a computational approach called density

new kinds of semiconductors. This research posits that

sor molecules onto silicon. This, in turn, would make it

designing functional materials in an unusual way — by

functional theory (DFT) to explore such characteris-

such new materials would make it possible to use light

possible to combine light signal conduction with sili-

applying approaches from computational chemistry.

tics at the quantum scale; that is, at the scale where

instead of electrons for signal transport, supporting the

con based microelectonics, bringing together the best

Using computing resources at HLRS, Tonner models

Newtonian mechanics becomes replaced by the much

development of improved electronic devices.

of both worlds in optical and electronic conduction. (CW)

phenomena that happen at the atomic and subatomic

stranger world of quantum mechanics. DFT uses infor-

scale to understand how factors such as molecular

mation about variations in the density of electrons

structure, electronic properties, chemical bonding,

within a molecule — a quantity that can also be exper-

and interactions among atoms affect a material’s

imentally measured using x-ray diffraction — to derive

behavior.

the energy of the system. This, in turn, enables the

Tonner and his group have highlighted the ability of

researchers to infer interactions among nuclei as well

computational chemistry and high-performance com-

as between electrons and nuclei, factors that are critical

puting to reveal interesting phenomena that occur

to understanding chemical bonds and reactions.

between organic molecules and surfaces, and have

DFT can provide useful, though static, information

demonstrated how these interactions can be under-

about the energy profiles of the compounds they study.

stood with respect to the molecular and solid state

To gain a better understanding of how systems of mol-

world. The knowledge they have gained could help

ecules actually behave when interacting with a surface,

design patterned surfaces, a goal of scientists working

Tonner’s group also uses high-performance comput-

on the next generation of more powerful, more effi-

ing at HLRS to perform molecular dynamics simula-

cient semiconductors.

tions. Here, the scientists look at how the system of

rL

ab

46 / 47

HLRS ANNUAL REPORT 2018

Artist’s rendering of organic molecules adsorbing on a silicon surface.

ne

© Aaron Beller

©

n To

is determined by chemical bonding, though precisely

whether carbon dioxide could replace water in power generation. In conventional steam power plants, residual water is

the high-resolution fluid dynamics simulations they

separated from power-generating steam. This process

required.

limits efficiency, and in early generation power plants

© IKE, University of Stuttgart

Using high-performance computing and data-driven machine learning, University of Stuttgart researchers investigate

The simulation shows the structure and the (red) high and (blue) low speed streaks of the fluid during a cooling process. The researchers observed a major difference in turbulence between downward flowing (left) and upward flowing (right) supercritical carbon dioxide.

Simulation and Machine Learning Could Make Power Plants More Efficient

could be volatile, leading to explosions. In the 1920s,

The heat of the moment

Mark Benson realized that the risk could be reduced

Water is often used in power generation and heat trans-

do so, the team turned to a method called direct numeri-

Critical next steps

and power plants could be more efficient by bringing

fer because it is easily accessible, well-understood on

cal simulation, which is very computationally expensive

To date, the team has been using OpenFOAM, a com-

water to a supercritical state — when a fluid exists as

a chemical level, and predictable under a wide range

and requires HPC resources.

munity code, for its DNS simulations. While Open-

both a liquid and gas at the same time. Though it was

of temperature and pressure conditions. That said, in

FOAM is a well-established code for a variety of fluid

too expensive to use in practice during his day, the

order to become supercriticial. water must be heated

Neural networks for commercial computers

Benson Boiler offered the world its first glimpse of

to 374°C and placed under extremely high pressure.

Using the stress and heat transfer data coming from its

wanted to use a higher-fidelity code. The researchers

supercritical power generation.

Further, when a material enters its critical state, even

high-fidelity DNS simulations, the team worked with

are working with a team from University of Stuttgart’s

Almost a century later, researchers at the University

slight changes to temperature or pressure can have a

SIT’s Dr. Wanli Chang to train a deep neural network

Institute of Aerodynamics and Gas Dynamics (IAG) to

of Stuttgart’s Institute of Nuclear Technology and

large impact. For instance, supercritical water does not

(DNN), a machine learning algorithm modeled roughly

use its FLEXI code, which offers higher accuracy and

Energy Systems (IKE) and Institute of Aerospace Ther-

transfer heat as efficiently as it does in a purely liquid

after networks of neurons in the brain.

can accommodate a wider range of conditions.

modynamics (ITLR) are revisiting Benson’s concepts

state, and the extreme heat needed to reach supercriti-

Using Hazel Hen, the team ran 35 DNS simulations,

Pandey also mentioned he is using a method called

to explore how it can improve safety and efficiency in

cal levels can lead to degradation of piping and, in turn,

each focused on one specific operational condition,

implicit LES in addition to the DNS simulations. While

modern power plants. Using high-performance comput-

catastrophic accidents.

and then used the generated dataset to train the DNN.

implicit LES simulations do not have quite the same

ing, the researchers are developing tools that can make

Although using carbon dioxide (CO2) to make power

The team uses inlet temperature and pressure, heat

high resolution as the team’s DNS simulations, it does

supercritical heat transfer more viable.

plants cleaner may sound like an oxymoron, Pan-

flux, pipe diameter, and heat energy of the fluid as

allow them to run simulations with higher Reynolds

Sandeep Pandey, a PhD candidate at IKE, and Dr.-Ing.

dey and his colleagues are investigating its use as an

inputs, and generates the pipe’s wall temperature and

numbers, meaning it can account for a wider range of

Xu Chu of ITLR are leading the computational aspects

alternative. The common molecule offers a number of

wall shear stress as output. Eighty percent of the data

turbulence conditions.

of this research. In cooperation with computer science

advantages, chief among them being that it reaches

generated in the DNS simulations is randomly selected

The team wants to continue to enhance its database

researchers at the Singapore Institute of Technology

supercriticality at just over 31°C, making it far more effi-

to train the DNN, while researchers use the other 20

in order to further improve its DNN tool. Further, it is

(SIT), they are employing machine learning techniques

cient than water. In addition, supercritical CO2 (sCO2)

percent of data for simultaneous, but separate, vali-

collaborating with IKE experimentalists to conduct pre-

informed by high-fidelity simulations. They are also

needs far less space and can be compressed with far

dation.

liminary experiments and to build a model supercritical

developing a tool that can be easily used on commer-

less effort than subcritical water. This, in turn, means

After the team felt confident with the agreement, they

power plant in order to test the agreement between

cial computers.

that it requires a smaller power plant.

used the data to start creating a tool for commercial

experiment and theory. The ultimate prize will be if the

To achieve these goals, the team needed to run com-

In order to replace water with carbon dioxide, though,

use. Using the outputs from the team’s recent work as

team is able to provide an accurate, easy-to-use, and

putationally intensive direct numerical simulations

engineers need to understand its properties on a fun-

a guide, they were able to use their DNN to simulate the

computationally efficient tool that helps engineers and

(DNS), which is only possible using high-performance

damental level, including how the fluid’s turbulence

heat energy of specific operational conditions in 5.4

power plant administrators generate power safer and

computing. HLRS’s Hazel Hen supercomputer enabled

transfers heat and in turn, interacts with machinery. To

milliseconds on a standard laptop computer.

more efficiently.

48 / 49

HLRS ANNUAL REPORT 2018

dynamics simulations, Pandey indicated that the team

(EG)

Selected User Publications

Ahmad N, Farooqi MN, Unat D. 2018. Load balancing for parallel multiphase flow simulation. Sci Programming-Neth.

Coppola E, Sobolowski S, Pichelli E, et al. 2018. A first-of-its-kind multi-model convection permitting ensemble for

2018:6387049.

investigating convective phenomena over Europe and the Mediterranean. Climate Dynamics. DOI:10.1007/s00382018-4521-8.

Albers M, Meysonnat PS, Schröder W. 2018. Closed-loop control of turbulent boundary layers using spanwise traveling surface waves. Proc Appl Math Mech. 18:201800336.

Cormier M, Caboni M, Lutz T, et al. 2018. Numerical analysis of unsteady aerodynamics of floating offshore wind turbines. J Phys Conf Ser. 1037:072048.

Alon OE, Cederbaum LS. 2018. Attractive Bose-Einstein condensates in anharmonic traps: Accurate numerical treatment and the intriguing physics of the variance. Chem Phys. 515:287-298.

Donnert J, Jang H, Mendygral P, et al. 2018. Towards exascale simulations of the ICM dynamo with WENO-Wombat. Galaxies. 6(4):104.

Babb J, McLaughlin BM. 2018. Radiative charge transfer between the helium ion and argon. Astrophys J. 860(2). Dumbser M, Guercilena F, Köppel S, et al. 2018. Conformal and covariant Z4 formulation of the Einstein equaBangga G, Lutz T, Krämer E. 2018. Energy assessment of two vertical axis wind turbines in side-by-side arrangement.

tions: Strongly hyperbolic first-order reduction and solution with discontinuous Galerkin schemes. Phys Rev D.

J Renew Sustain Energy. 10(3):10.1063/1.5028199.

97(8):084053.

Biastoch A, Sein D, Durgadoo JV, et al. 2018. Simulating the Agulhas system in global ocean models — nesting vs.

Forrey R, Babb J, Stancil P, McLaughlin B. 2018. Rate constants for the formation of CS by radiative association. Mon

multi-resolution unstructured meshes. Ocean Model. 121:117-131.

Not R Astron Soc. 479(4):4727–4734.

Bordia P, Alet F, Hosur P. 2018. Out-of-time-ordered measurements as a probe of quantum dynamics. Phys Rev A.

Fröhlich K, Schneiders L, Meinke M, Schröder W. 2018. Validation of Lagrangian two-way coupled point-particle mod-

97:030103(R).

els in large-eddy simulations. Flow Turbul Combust. 101(2):1-25.

Börner PC, Kinyanjui MK, Björkman T, et al. 2018. Observation of charge density waves in free-standing 1T-TaSe2

Galindo-Lopez S, Salehi F, Cleary MJ, et al. 2018. A stochastic multiple mapping conditioning computational model in

monolayers by transmission electron microscopy. Appl Phys Lett. 113:173103.

OpenFOAM for turbulent combustion. Comput Fluids. 172:410-425.

Borsanyi S, Fodor Z, Guenther JN, et al. 2018. Higher order fluctuations and correlations of conserved charges from

Goela S, Cross G, Stukowski A, et al. 2018. Designing nanoindentation simulation studies by appropriate indenter

lattice QCD. J High Energy Phys. 2018:205.

choices: Case study on single crystal tungsten. Comput Mater Sci. 152:196-210.

Bose S, Chakravarti K, Rezzolla L, et al. 2018. Neutron-star radius from a population of binary neutron star mergers.

Harutyunyan A, Nathanail A, Rezzolla L, Sedrakian A. 2018. Electrical resistivity and Hall effect in binary neutron-star

Phys Rev Lett. 120:031102.

mergers. Euro Phys J A. 54:191.

Bradley C, Emamy N, Ertl T, et al. 2018. Enabling detailed, biophysics-based skeletal muscle models on HPC systems.

Hötzer J, Reiter A, Hierl H, et al. 2018. The parallel multi-physics phase-field framework PACE3D. J Comp Sci. 26:1-12.

Front Physiol. 9:816. Jarniven R, Vainio R, Palmroth M, et al. 2018. Ion acceleration by flux transfer events in the terrestrial magnetosheath. Chu X, Chang W, Pandey S, et al. 2018. A computationally light data-driven approach for heat transfer and hydraulic characteristics modeling of supercritical fluids: From DNS to DNN. Int J Heat Mass Tran. 123:629-636.

50 / 51

HLRS ANNUAL REPORT 2018

Geophys Res Lett. 45(4):1723-1731.

Juusola L, Pfau-Kempf Y, Ganse U, et al. 2018. A source mechanism for magnetotail current sheet flapping. Ann

Liu X, Huang G, Hu K, et al. 2018. Sharing of Na+ by three −COO– groups at deprotonated carboxyl-terminated self-

Geophys. 36:1027–1035.

assembled monolayer-charged aqueous interface. J Phys Chem-US. 122(16):9111–9116.

Kaltenbach C, Laurien E. 2018. CFD simulation of spray cooling in the model containment THAI. Nucl Eng Design.

Lode AUJ, Diorico FS, Wu E, et al. 2018. Many-body physics in two-component Bose-Einstein condensates in a cavity:

328:359-371.

fragmented superradiance and polarization. New J Phys. 20:055006.

Kaltenbach C, Laurien E. 2018. CFD simulation of aerosol particle removal by water spray in the model containment

Luitz DJ, Lazarides A, Bar Lev Y. 2018. Periodic and quasiperiodic revivals in periodically driven interacting quantum

THAI. J Aerosol Sci. 120:62-81.

systems. Phys Rev B. 97:020303(R).

Klyushin A, Jones T, Lunkenbein T, et al. 2018. Strong metal support interaction as a key factor of Au activation in CO

Marquard P, Smirnov AV, Smirnov VA, Steinhauser M. 2018. Four-loop wave function renormalization in QCD and

oxidation. ChemCatChem. 10(18):3985-3989.

QED. Phys Rev D. 97:054032.

Knodel MM, Nägel A, Reiter S, et al. 2018. Quantitative analysis of hepatitis C NS5A viral protein dynamics on the ER

Nicholson CW, Lücke A, Schmidt WG, et al. 2018. Beyond the molecular movie: dynamics of bands and bonds during

surface. Viruses. 10(1):28.

a photoinduced phase transition. Science. 362(6416):821-825.

Kohns M, Horsch M, Hasse H. 2018. Partial molar volume of NaCl and CsCl in mixtures of water and methanol by

Pandey S, Chu X, Laurien E, Weigand B. 2018 Buoyancy induced turbulence modulation in pipe flow at supercritical

experiment and molecular simulation. Fluid Phase Equilibria. 458:30-39.

pressure under cooling conditions. Phys Fluids. 30(6):065105.

Kokh DB, Amaral M, Bomke J, et al. 2018. Estimation of drug-target residence times by τ random acceleration molec-

Pásztor A. 2018. Recent results on small μB QCD from the lattice. J Phys Conf Ser. 1024:012026.

ular dynamics simulations. J Chem Theory Comput. 14(7):3859–3869. Pecher L, Schmidt S, Tonner R. 2018. Dispersion-mediated steering of organic adsorbates on a precovered silicon Köster A, Thol M, Vrabec J. 2018. Molecular models for the hydrogen age: hydrogen, nitrogen, oxygen, argon, and

surface. Beilstein J Org Chem. 14:2715-2721.

water. J Chem Eng. 63(2):305-320. Pecher L, Tonner R. 2018. Computational analysis of the competitive bonding and reactivity pattern of a bufunctional Kühne M, Börnert F, Fecher S, et al. 2018. Reversible superdense ordering of lithium between two graphene sheets.

cyclooctyne on Si(001). Theor Chem Accounts. 137:48.

Nature. 564:234-239. Pfeiffer M. 2018. Particle-based fluid dynamics: comparison of different Bhatnagar-Gross-Krook models and the Lesnicki D, Sulpizi M. 2018. A microscopic interpretation of pump−probe vibrational spectroscopy using ab initio

direct simulation Monte Carlo method for hypersonic flows. Phys Fluids. 30:106106.

molecular dynamics. J Phys Chem B. 122(25):6604–6609. Pietracaprina F, Macé N, Luitz D, Alet F. 2018. Shift-invert diagonalization of large many-body localizing spin chains. Letzgus P, Lutz T, Krämer E. 2018. Detached eddy simulations of the local atmospheric flow field within a forested

SciPost Phys. 5:045.

wind energy test site located in complex terrain. J Phys Conf Ser. 1037:072043. Pindzola MS, Loch SF. 2019. Proton impact excitation of the Na atom. J Phys B-Atom Mol Opt. 52:025202.

52 / 53

HLRS ANNUAL REPORT 2018

Pogorelov A, Schneiders L, Meinke M, Schröder W. 2018. An adaptive Cartesian mesh based method to simulate tur-

Tsoutsanis P, Antoniadis AF, Jenkins KW. 2018. Improvement of the computational performance of a parallel unstruc-

bulent flows of multiple rotating surfaces. Flow Turbulence Combust. 100(1):19-38.

tured WENO finite volume CFD code for implicit large eddy simulation. Comput Fluids. 173:157-170.

Pogorelov A, Meinke M, Schröder W. 2018. Large-eddy simulation of the unsteady full 3D rim seal flow in a one-stage

Wang B, Kronenburg A, Tufano GL, Stein OT. 2018. Fully resolved DNS of droplet array combustion in turbulent con-

axial-flow turbine. Flow Turbulence Combust. DOI:10.1007/s10494-018-9956-9.

vective flows and modelling for mixing fields in inter-droplet space. Combust Flame. 189:347-366.

Rodriguez-Gomey V, Snyder GF, Lotz JM, et al. 2018. The optical morphologies of galaxies in the IllustrisTNG simula-

Weeber R, Kreissl P, Holm C. 2018. Studying the field-controlled change of shape and elasticity of magnetic gels using

tion: a comparison to Pan-STARRS observations. Mon Not R Astron Soc. 483(3):4140-4159.

particle-based simulations. Arch Appl Mech. 1.

Roy S, Bar Lev Y, Luitz DJ. 2018 Anomalous thermalization and transport in disordered interacting Floquet systems.

Weir DJ. 2018. Gravitational waves from a first order electroweak phase transition: a brief review. Philos Trans A Math

Phys Rev B. 98:060201(R).

Phys Eng Sci. 376(2114).

Sanna S, Lichtenstein T, Mamiyev Z, et al. 2018. How one-dimensional are atomic gold chains on a substrate? J Phys

Wyrwich R, Jones TE, Günther S, et al. 2018. LEED-I⁄(V) structure analysis of the (7x√3)rect S04 phase on Ag(111):

Chem C. 122:25580-25588.

precursor to the active species of the Ag-catalyzed ehylene epoxidation. J Phys Chem C 122:26990-27004.

Saveleva VA, Wang L, Teschner D, et al. 2018. Operando evidence for a universal oxygen evolution mechanism on

Zhu Q, Xu D, Gaspari M, et al. 2018. Formation of a Malin 1 analogue in IllustrisTNG by stimulated accretion. Mon Not

thermal and electrochemical iridium oxides. J Phys Chem Lett. 9(11):3154–3160.

R Astron Soc. 480(1):L18-L22.

Schäferlein U, Keßler M, Krämer E. 2018. Aeroelastic simulation of the tail shake phenomenon. J Am Helicopter Soc. 63(3):1-17. Shterenlikht A, Margetts L, Cebamanos L. 2018. Modelling fracture in heterogeneous materials on HPC systems using a hybrid MPI/Fortran coarray multi-scale CAFE framework. Adv Eng Softw. 125:155-166. Steinmetz P, Hötzer J, Kellner M, et al. 2018. Study of pattern selection in 3D phase-field simulations during the directional solidification of ternary eutectic Al-Ag-Cu. Comput Mater Sci. 148:131-140. Tavelli M, Dumbster M. 2018. Arbitrary high order accurate space-time discontinuous Galerkin finite element schemes on staggered unstructured meshes for linear elasticity. J Comput Phys. 366:386-414. Tsoutsanis P. 2018. Extended bounds limiter for high-order finite-volume schemes on unstructured meshes. J Comp Phys. 263:69-94.

54 / 55

HLRS ANNUAL REPORT 2018

ABOUT HLRS

Inside Our Computing Room

56 / 57

HLRS ANNUAL REPORT 2018

Cray XC40 Hazel Hen Hazel Hen is at the heart of HLRS’s HPC system infrastructure. With a peak performance of 7.42 Petaflops, it is one of the most powerful HPC systems in the world (position 30 in the TOP500, November 2018). Hazel Hen entered operation in October 2015, is based on the Intel Haswell Processor and the Cray Aries network technologies, and is designed for sustained application performance and high scalability.

CPU

Intel® Xeon CPU E5-2680 v3 12 core @ 2.5 GHz

Number of nodes / cores

7,712 / 185,088

Peak performance

7.42 PFLOPS

Memory

128 GB/node

Disk storage

15 PB

Cray Urika-GX Increasingly, projects running on the Cray XC40 Hazel Hen generate large amounts of data. To provide a powerful tool for analyzing such results, HLRS installed a specialized Data Analytics Platform in December 2016. This research project gives users the ability to adapt data analytics methods for engineering applications.

Optimized software for

Spark Hadoop CGE (CrayGraph Engine)

Number of nodes

48 + 16

Cooperation with academic and industrial partners

Daimler, Porsche, Sicos BW, among others

User Profile In 2018 the Gauss Centre for Supercomputing approved 10 new large-scale projects for Hazel Hen (each project requiring more than 35 million core hours within one year), for a total of 766 million core-hours. The Partnership for Advanced Computing in Europe (PRACE) also approved 4 international simulation projects for HLRS, for a total of 113 million core-hours. In total, 129 projects were active on Hazel Hen in 2018 with 1.28 billion core-hours used.

System Usage by Scientific Discipline

NEC SX-ACE The NEC SX-ACE is a vector computer optimized for applications demanding vector operations and high memory bandwidth.

Electrical Engineering 0.73 % Computer Science 0.77 % Other 1.21 % Bioinformatics 1.39 %

CPU

NEC Vector CPU, 4 cores @ 1.0 GHz

Number of nodes / cores

64 / 256

Peak performance

~16 TFLOPS

Memory

4 TB

Memory BW per node

220 GB/s (single core), 256 GB/s (4 cores)

Interconnect

NEC IXS

Materials Science 0.73 % Structural Mechanics 0.01 %

 Transportation and Climate 1.77 %  Chemistry 2.18 %  Solid State Physics 2.21 %  Reacting Flows 3.2 %

CFD 53.69  %  Physics 32.12 %

NEC Cluster System Usage by State This standard PC cluster was installed in spring 2009. To meet increasing demands for compute resources, its Lower Saxony 0.65 %

configuration has been constantly adapted. The current configuration is as follows:

Berlin 1.14 %

Node type

Intel Xeon E5-2670 (SandyBridge)

124

Hessen 1.64 %

Bavaria 0.34 % Hamburg 0.29 %

Federal Research Center 4.17 %

Node type

Intel Xeon E5-2660 v3 @ 2.6 GHz (Haswell)

88

Node type

Intel Xeon E5-2680 v3 @ 2.5 GHz (Haswell)

360

Memory per node

32 / 64 / 128 / 256 GB

Interconnect

Infiniband QDR/FDR

Rheinland-Palatinate 4.49 %

North Rhine-Westphalia 45.4 %

Baden-Württemberg 41.87 %

58 / 59

HLRS ANNUAL REPORT 2018

Third-Party Funded Research Projects

In addition to providing supercomputing resources for scientists and engineers in academia and industry, HLRS

Leadning European HPC centers December 2018 - November 2021

conducts its own funded research on important topics relevant for high-performance computing. These activities,

➜ In ChEESE, leading European HPC centers, academia, hardware developers, alongside SMEs, industry and

many of which are conducted in collaboration with investigators at other institutes, are designed to address key

public governance bodies such as civil protection are working together to prepare European flagship codes for

challenges and opportunities in the field. The following is a list of funded projects that operated in 2018.

upcoming pre-exascale and exascale supercomputing systems to tackle global challenges in the domain of

EU

solid earth. For more information about our current projects, visit www.hlrs.de/about-us/research/current-projects/ DIPL-ING

April 2017 - March 2019

BMBF

➜ The project is researching solutions for efficiently managing the high amounts of data emerging from engineer-

ing education programs at the University of Stuttgart.

Project

Duration

Funded by

BEAM-ME

December 2015 - November 2018

BMBF

EOPEN

November 2017 - October 2020

EU

➜ BEAM-ME is exploiting the potential of parallel and high-performance computing using distributed memory for

➜ EOPEN is tackling technical barriers that result from massive streams of Earth observation data and seeks to

high-resolution optimization models in energy system analyses.

ensure that methods for data harmonization, standardization, fusion, and exchange are scalable.

bw Naha 2

January 2017 - December 2019

MWK

➜ This project supports implementation of an energy management system (ISO 50001) and an environmental man-

EuroLab-4-HPC 2

May 2018 - April 2020

EU

➜ EuroLab-4-HPC aims to establish a European Research Center of Excellence for HPC systems.

agement system (EMAS), which will reduce consumption, improve environmental performance, and contribute to EUXDAT

the realization of HLRS’s sustainability strategy.

November 2017 - October 2020

EU

➜ EUXDAT provides a platform that unites HPC and cloud infrastructures to manage and process high amounts of

BW Stiftung II

October 2016 - September 2019

MWK

heterogeneous data. Its focus is to support sustainable development in agriculture.

➜ BW Stiftung supports universities and other nonprofit research institutions in Baden-Württemberg in using HLRS

ExaFLOW

computers and advises them on issues related to optimization.

October 2015 - September 2018

EU

➜ ExaFLOW is addressing key algorithmic challenges in computational fluid dynamics that will need to be solved

bwHPC-S5

July 2018 - December 2020

MWK

➜ BwHPC coordinates support for HPC users in Baden-Württemberg and the implementation of related measures

to enable simulation at exascale. It is guided by use cases of industrial relevance and will provide open-source pilot implementations.

and activities, including data intensive computing and large scale scientific data management. Exasolvers bwVisu II

August 2014 - October 2020

MWK

➜ bwVisu is developing a service for remote visualization of scientific data, ensuring high scalability through cloud

May 2016 - April 2019

DFG

➜ The exascale computers of the future are characterized by extreme parallelism. Exasolvers is combining crucial

aspects of extreme scale solving, developing methods that scale perfectly and have optimal complexity.

technologies. EXCELLERAT CATALYST

October 2016 - September 2019

MWK

December 2018 - November 2021

EU

➜ EXCELLERAT’s goal is to facilitate the development of important codes for high-tech engineering, including

➜ CATALYST researches methods for analyzing modelling and simulation data with the goal of implementing a

maximizing their scalability to ever larger computing architectures and supporting the technology transfer that will

framework that unites HPC and data analytics.

enable their uptake within the industrial environment.

60 / 61

HLRS ANNUAL REPORT 2018

EXPERTISE

March 2017 - February 2020

EU

MoeWe

July 2016 - December 2020

ESF, MWK

➜ EXPERTISE is a European training network (ETN) for the next generation of mechanical and computer science

➜ To address the long-term demand for supercomputing experts, particularly in industry, MoeWe has been devel-

engineers. Its objective is to develop advanced tools for analyzing fluid dynamics in large-scale models of turbine

oping a modular, flexible training program called the Supercomputing-Akademie.

components and to eventually enable the virtual testing of an entire machine. MontBlanc 3 FocusCoE

December 2018 - November 2021

EU

➜ FocusCoE supports EU-funded Centres of Excellence to more effectively exploit the tangible benefits of extreme

October 2015 - September 2018

EU

➜ MontBlanc 3 aims to design a new type of computer architecture capable of setting future HPC standards. The

approach is based on energy efficient solutions used in embedded and mobile devices.

scale applications for addressing scientific, industrial or societal challenges by creating a platform for the CoEs to MWK CoE: Automotive

coordinate strategic collaboration and outreach.

March 2016 - June 2018

MWK

Simulation Exzellenzcluster 2 FORTISSIMO 2

November 2015 - December 2018

EU

➜ This project is establishing and strengthening concepts for using simulation and HPC in the automotive industry.

➜ FORTISSIMO 2 supports small and medium-sized enterprises (SMEs) in accessing simulation tools on super-

The center of excellence will be developed with the support of international networks and the analysis of funding

computers, promoting an expansion of their business and improvements in their competitiveness.

opportunities.

HiDALGO

OpenForecast

December 2018 - November 2021

EU

September 2019 - August 2020

EU

➜ HiDALGO enables the assessment of Global Challenges problem statements by enabling highly accurate simu-

➜ The overall goal of OpenForecast is to deliver a novel generic service of high quality and efficiency for the Public

lations, data analytics, artificial intelligence and data visualization, but also by providing knowledge on how to inte-

Open Data Digital Service Infrastructure. This generic service combines public open data sources and high-perfor-

grate the various workflows and the corresponding data.

mance computing (HPC) to establish a new generation of services.

HPC-Europa 3

May 2017 - March 2020

EU

OSCCAR

Juni 2018 - Mai 2021.

EU

➜ HPC-Europa 3 fosters transnational cooperation among EU scientists (especially junior researchers) who work

➜ The EU Horizon 2020 research project OSCCAR — Future Occupant Safety for Crashes in Cars — is developing

on HPC-related topics such as applications, tools, and middleware.

a novel, simulation-based approach to safeguard occupants involved in future vehicle accidents.

HyForPV

October 2018 - September 2021

BMWi

PetaGCS

January 2010 - December 2019

BMBF / MWK

➜ The overall aim of HyForPV is to develop and operationalize new prediction products for the system integration

➜ PetaGCS has been supporting the procurement and operation of next-generation supercomputers at HLRS from

of photovoltaics (PV) into the energy market and smart grids by delivering highly detailed simulations of PV power

2011 to 2019. Acquisitions are coordinated by the Gauss Centre for Supercomputing.

output with very high resolution in space and time. PHANTOM InHPC-DE

November 2017 - September 2021

BMBF

December 2015 - November 2018

EU

➜ This project is addressing challenges in the development of energy-efficient parallel infrastructures in domains

➜ This collaboration aims to lay the groundwork for a standardized and distributed HPC ecosystem that integrates

such as the Internet of Things and high-performance computing, using acceleratable hardware such as GPU and

Germany’s three Tier-1 supercomputing centers. It provides funding for 100 Gbit networking and opportunities for

CPU.

high-speed data management and visualization.

62 / 63

HLRS ANNUAL REPORT 2018

Pop-CoE

October 2015 - November 2018

EU

TranSim

January 2016 - December 2018

MWK

➜ POP-CoE assesses the performance of computing applications, identifying issues affecting code performance

➜ The goal of the project Transforming Society – Transforming Simulation is to explore how computer simulation

as well as the best ways to address them.

is transforming science and the worlds of work, knowledge, and values.

PRACE

May 2017 - April 2019

EU

VirMuDeKo

März 2018 - Februar 2019

MWK

➜ PRACE supports high-impact scientific discovery and engineering R&D to enhance European competitiveness

➜ The goal of this project is to develop a pilot and demonstration framework through which virtual reality and aug-

for the benefit of society.

mented reality can support scientific collaboration in spatially distributed projects.

Reallabor Stadt:quartiere 4.0

January 2016 - December 2018

MWK

Visdral

May 2016 - December 2018

BMWi

➜ Reallabor is exploring new methods and technologies that support participatory town planning and sustainable

➜ Visdral is testing the use of 3D laser scanning and simulation technologies to virtually document and investigate

development, resulting in the development of a digital twin and a planning and decision support tool.

traffic accidents.

SimTech

December 2018 - May 2022

DFG

➜ SimTech aims at developing simulation technologies towards an integrative system science based on data inte-

grated modelling. Within the project, HLRS supports the development of efficient methods for uncertainty quantification and management. Simulated Worlds

February 2011 - June 2018

MWK

➜ Despite its importance in R&D, simulation is not widely understood. Simulated Worlds aims to bridge this gap by

making it accessible to students, the next generation of scientists. SiVeGCS

January 2017 - December 2025

BMBF / MWK

➜ Coordinates and ensures the availability of HPC resources of the Gauss Centre for Supercomputing, address-

ing issues related to funding, operation, training, and user support across Germany’s national HPC infrastructure.

Funder Abbreviations BMBF

Federal Ministry of Education and Research

➜ Smart-DASH is continuing development of the C++ template library DASH, which offers distributed data struc-

BMWi

Federal Ministry for Economic Affairs and Energy

tures with flexible data partitioning schemes and a set of parallel algorithms.

DFG

German Research Foundation

ESF

European Social Fund

EU

European Union

MWK

Baden-Württemberg Ministry for Science, Research, and Art

Smart-DASH

TalPas

May 2016 - April 2019

DFG

January 2017 - December 2019

BMBF

➜ TalPas is developing a self-optimizing, task-based approach to high-performance particle simulations.

64 / 65

HLRS ANNUAL REPORT 2018

HPC Training Courses in 2018

HLRS offered 41 courses in 2018, providing continuing professional education on a wide range of topics relevant for high-performance computing. The courses took place over 129 course-days in Stuttgart and at other locations in Germany and internationally. Approximately 1,000 trainees participated in these activities. For a current listing of upcoming courses, please visit www.hlrs.de/training/.

Date

Location

Topic

Jan 18

Garching

Introduction to Hybrid Programming in HPC (MPI+X) *

Jul 10-13

Stuttgart

Advanced C++ with Focus on Software Engineering

Feb 5-7

Paderborn

Parallel Programming with MPI and OpenMP

Aug 20-23

Zürich

Parallel Programming with MPI and OpenMP

Feb 5-7

Amsterdam

Parallel Programming with MPI and OpenMP

Sep 10-11

Innsbruck

Parallel Programming with MPI and OpenMP

Feb 12-16

Dresden

Parallel Programming & Parallel Tools

Sep 10-14

Garching

Iterative Linear Solvers and Parallelization

Feb 19-23

Siegen

Computational Fluid Dynamics

Sep 10-14

Stuttgart

Computational Fluid Dynamics

Mar 5-9

Stuttgart

CFD with OpenFOAM®

Sep 12-14

Innsbruck

Parallel Programming with MPI and OpenMP

Mar 12-13

Stuttgart

OpenMP + OpenACC GPU Directives for Parallel Accelerated Super­

Sep 19

Stuttgart

Deep Learning Workshop

computers *

Sep 24-28

Siegen

CFD with OpenFOAM®

Mar 19-23

Stuttgart

Iterative Linear Solvers and Parallelization

Oct 15-19

Stuttgart

Parallel Programming and Advanced Topics in Parallel Programming *

Apr 9-12

Mainz

Parallel Programming with MPI and OpenMP

Oct 25-26

Stuttgart

Scientific Visualization

Apr 9-13

Stuttgart

Fortran for Scientific Computing *

Nov 5-6

Vienna

Parallel Programming with MPI and OpenMP

Apr 11-13

Innsbruck

Parallel Programming with MPI and OpenMP

Nov 5-9

Stuttgart

Optimization of Scaling and Node-Level Performance on Hazel Hen

Apr 23-26

Stuttgart

Optimization of Scaling and Node-Level Performance on Hazel Hen

Nov 7-9

Vienna

Parallel Programming with MPI and OpenMP

Apr 26-27

Vienna

Parallel Programming with MPI and OpenMP

Nov 19-22

Stuttgart

Advanced C++ with Focus on Software Engineering

May 7-8

Stuttgart

Scientific Visualization

Nov 26-28

Jülich

Advanced Parallel Programming with MPI and OpenMP

May 14-17

Stuttgart

Advanced C++ with Focus on Software Engineering

Nov 27 + 30

Heverlee

Parallel Programming with MPI and OpenMP

May 15-17

Vienna

Parallel Programming with MPI and OpenMP

Dec 3-7

Stuttgart

Fortran for Scientific Computing

Jun 6-7

Vienna

Introduction to Hybrid Programming in HPC (MPI+X)

Dec 13

Heverlee

Parallel Programming with MPI and OpenMP

Jun 12-13

Stuttgart

Fortran Modernization Workshop

Jun 14-15

Stuttgart

Node-Level Performance Engineering *

Jun 19

Stuttgart

Introduction to Hybrid Programming in HPC (MPI+X)

Jun 20-21

Stuttgart

Cluster Workshop

Jul 2-3

Stuttgart

Concepts of GASPI and Interoperability with Other Communication APIs *

Jul 5-6

Stuttgart

Introduction to UPC and Co-Array Fortran *

66 / 67

HLRS ANNUAL REPORT 2018

*

PRACE courses: HLRS is a member of the Gauss Centre for Supercomputing (GCS). GCS is one of ten PRACE Training Centres in the EU. The marked courses are in part sponsored by PRACE and are part of the PRACE course program.

Workshops and Conferences 2018

Date

Location

Partners

Topic

Mar 9-10

Erlangen

FAU Institute for Sociology

Simulation in the Social Sciences and the Sociology of Simulation

Apr 16-18

Stuttgart

17th HLRS/hww Workshop on Scalable Global

Structure

Institut für Höchst­leistungs­ rechnen

Director Prof. Dr.-Ing. Dr. h.c. Dr. h.c. Prof. E.h. Michael M. Resch Secretary, Assistant

Parallel File Systems Apr 23-26

Kaliningrad

Russian Academy of

Third Annual German-Russian Workshop

Manager

Sciences, Keldysh Institute June 7-8

Stuttgart

Sep 17

Stuttgart

ZIH, TU Dresden

12th International Parallel Tools Workshop

Oct 1-2

Stuttgart

Hyperion Research

HPC User Forum

Oct 4-5

Stutgart

Oct 9-10

Stuttgart

Nachhaltigkeitstage (Sustainability Days)

Dr. Bastian Koller

High-Performance Computing in Science & Engineering: 21st Results and Review Workshop Tohoku University

28th Workshop on Sustained Simulation Performance

Oct 23-24

Stuttgart

Nov 26-27

Stuttgart

Nov 28-30

Stuttgart

Dec 11

Stuttgart

Workshop on Sustainable HPC Infrastructure HPC-Europa3, SICOS-BW

1st HPC-Europa Workshop for Small and Medium

Software & Systems

Applications & Visualization

Administration & Information

Staff Unit

Thomas Beisel

Dr. Bastian Koller

Agnes Lampke

Dr. Norbert Conrad

HPCN Production

Visualization

Administration Staff, Finances, Projects

Infrastructure

Numerical Methods & Libraries

Scalable Progamming Models & Tools

Communications & Industrial Trainings

Project & User Management, Accounting

Service Management & Business Processes

Enterprises Science and Art of Simulation Workshop 2018 SICOS-BW

2nd Industrial HPC User Roundtable

BelWü

Parallel Computing, Training & Application Services

68 / 69

HLRS ANNUAL REPORT 2018

Philosophy of Simulation

Divisions and Departments

Administration and Information

of insights. The department has expertise in tools such

Responsible for the operation of all platforms in the

on parallel programming, computational fluid dynam-

➜ Leader: Agnes Lampke

as virtual reality, augmented reality, and has designed a

compute server infrastructure. This department also

ics, performance optimization, scientific visualization,

method for integrating processing steps spread across

operates the network infrastructure necessary for HPC

programming languages for scientific computing and

multiple hardware platforms into a seamless distributed

system function and is responsible for security on net-

data in HPC. The department also organizes the review

software environment.

works and provided platforms.

process for simulation projects running at the national

Numerical Methods and Libraries Leader: Dr.-Ing. Ralf Schneider

provision for industrial clients. Additionally, it pro-

ning, controlling and bookkeeping, financial project

Scalable Programming Models and Tools Leader: Dr. José Gracia

management and project controlling, legal issues,

Conducts research into parallel programming models

Provides numerical libraries and compilers for HLRS

researchers in structural mechanics and chemistry.

human resources development, personnel adminis-

and into tools to assist development of parallel appli-

computing platforms. The department has expertise

tration, procurement and inventory, and event support.

cations in HPC. Currently the focus is on transparent

in implementing algorithms on different processors

global address spaces with background data transfers,

and HPC environments, including vectorization based

Communications and Industrial Trainings Leader: Dr. Jutta Oexle

task-parallelism based on distributed data-dependen-

on the architecture of modern computers. Department

cies, collective off-loading of I/O operations, and par-

members also conduct research related to the simula-

Supervises and executes HLRS’s communication to the

allel debugging. As a service to HLRS users, the group

tion of blood flow and bone fracture in the human body,

general public and the media. It is the central point of

also maintains part of the software stack related to pro-

and are responsible for training courses focused on

Philosophy of Science and Technology of Computer Simulation Leader: Dr. Andreas Kaminski

contact for all questions regarding the center and its sci-

gramming models, debugging, and performance anal-

programming languages and numerical methods that

Examines both how computer simulation changes sci-

entific work, and promotes new findings, achievements,

ysis tools.

are important for HPC.

ence and technology development and how society and

Project and User Management, Accounting Leader: Dr.-Ing. Thomas Bönisch

standing of knowledge and how we justify scientific

and workshops for the industrial and service sectors,

Service Management and Business Processes Leader: Michael Gienger

expanding interest in and accessibility of HPC technol-

Works on the development and operation of dynamic

Responsible for user management and accounting,

tainties about the future? And how do we deal with the

ogies and solutions beyond its traditional community

and scalable cloud computing services, particularly in a

including creating and maintaining web interfaces

uncertainties of simulation itself?

of scientific users.

business context. The group conducts research focus-

necessary for (federal) project management and data

ing on performance and availability monitoring, elastic

availability for users. The department also conducts

workflow management, and energy-efficient operation

activities related to the European supercomputing infra-

Infrastructure Leader: Marcel Brodbeck

for federated cloud environments. It also works on issues

structure (PRACE) and data management. This involves

Responsible for planning and operating facilities and

related to the establishment of high-performance com-

operating and continually developing high-perfor-

infrastructure at HLRS. This division ensures reliable

puting clouds, particularly for data intensive applications.

mance storage systems as well as conceiving new

and efficient operation of the HLRS high-performance

strategies for data management for users and projects

computing systems, provides a comfortable working

working in the field of data analytics.

environment for HLRS staff, and fosters all aspects of

Administration Leader: Agnes Lampke Manages issues related to the day-to-day operation of HLRS. Areas of responsibility include financial plan-

supercomputing center and participates in service

Applications and Visualization ➜ Leader: Dr. Bastian Koller

Visualization Leader: Dr.-Ing. Uwe Wössner Supports engineers and scientists in the visual analysis of data produced by simulations on high-perfor-

Software and Systems ➜ Leader: Thomas Beisel

mance computers. By providing technologies capable of immersing users in visual representations of their data, the department enables users to interact directly with it, reducing analysis time and enabling new kinds

70 / 71

Staff Units: Related Research

politics react to it: Does simulation change our under-

and other news from around the center. In addition, the department designs and offers training courses

vides installation and software support for academic

High-Performance Computing Network – Production (HPCN Production) Leader: Thomas Beisel

HLRS ANNUAL REPORT 2018

results? How can simulation help to overcome uncer-

energy efficient HPC operation. It is also responsible

Parallel Computing, Training and Application Services Leader: Dr. Rolf Rabenseifner Organizes HLRS’s academic continuing education program in high-performance computing, with emphases

for HLRS’s sustainability program, which encourages and supports the entire HLRS staff in acting according to principles of sustainability.

© 2019

Director, HLRS: Prof. Dr.-Ing. Dr. h.c. Dr. h.c. Prof. E.h. Michael M. Resch

High-Performance Computing Center Stuttgart (HLRS)

Leader, Department of Communications and Industrial Training: Dr. Jutta Oexle

University of Stuttgart Nobelstrasse 19 | 70569 Stuttgart | Germany

Editor: Christopher M. Williams

phone ++49 (0)711 685 - 87269

Contributing Writers: Lena Bühler (LB), Eric Gedenk (EG),

fax

Amelie Liebgott (AL), Christopher M. Williams (CW)

 ++49 (0)711 685 - 87209

Translations: Samantha Siegert

email [email protected] web www.hlrs.de

Production Manager: F. Rainer Klank Photography and Images: Unless otherwise indicated, all images property of HLRS. Printing: Nino Druck GmbH Design: Zimmermann Visuelle Kommunikation www.zimmermann-online.info

This magazine is printed on paper that has been certified by FSC, the EU Ecolabel, and the Blue Angel Ecolabel.

72

HLRS ANNUAL REPORT 2018

Stuttgart

This magazine is printed on paper that has been certified by FSC, the EU Ecolabel, and the Blue Angel Ecolabel.

© Mathis Bode, Institute for Combustion Technology, RWTH Aachen

High-Performance Computing Center