Data storage solutions have been evolved with the changing needs of technological advancements. At the same time, the computing pattern is also modernized over time. To understand the context of HPC storage, first, we must explain a little bit about high-performance computing. We all heard about supercomputers, which can process data and perform complex calculations at high speed. This is an example of high-performance computing or HPC. In this case, thousands of compute nodes work together parallelly to perform one or more tasks. These supercomputers also evolved with the changing needs of complex computing models.

However, the core need for HPC storage is generated for the storage requirement of Big data. As the need for analytical processing, AI, Machine learning, Industrial IoT, etc., have risen, at the same time, it demands to deal with the massive amount of data. Not only data, but such data also needs to be processed at high speed for real-time data analytics and decision making. There comes the need for high-performance computing.

However, HPC solutions need precise analysis that demands specialized storage to handle faster and frequent data access. Traditional data storage is unable to fulfill this requirement. There comes the role of HPC storage. Moreover, today high-performance computing is a pertinent solution for every industry from oil and gas companies, genetic analytics, government research, and many more.

Related post – Hot data storage 2020 technology trends

How does HPC work?

HPC solutions work based on the three main components:

Compute servers – These are computers or nodes that networked together into a cluster. Additionally, algorithms and software programs run simultaneously in the cluster.

Network –The network connects the clusters to the data storage and supports high-speed transportation data between data storage and compute servers.

Storage –This stores massive amounts of data and feeds and ingests data as fast as possible between the compute servers.

Synchronization between these components is essential; otherwise, the entire HPC infrastructure can suffer. Hence, these components operate seamlessly to complete a diverse set of tasks.

The I/O data amount that HPC storage handles are about exabytes. This is why the storage system to I/O must be offloaded from the CPU as much as possible to maintain consistent computation. The HPC storage systems efficiently handle this by keeping CPUs busy while the data is read from or written to the disk drives uninterruptedly.

For this purpose, the HPC storage follows a different approach where a parallel file system with a global namespace is used that effectively solve the following demands of HPC storage solutions:

– It makes any data to be available to any node at any time

– The most recent data is delivered at any point in time

– Handles both small and large size data requests

– Latest in storage technology like SSDs are used

– As the capacity grows it scales with constant latencies, in the millisecond range

– Supports the most performance-oriented and most popular protocols.

HPC Storage Architecture

To obtain the best result, the HPC cluster must be architect efficiently. Several critical features must be considered in this case, and a few of them are like:

– Software installation must be as simple as possible

– For efficient I/O, complexity of orchestration must be removed while tuning must be done as much as possible.

– HPC storage software should not be complex for the system administrator or developer.

We have mentioned in the previous section that HPC storage uses a parallel file system. However, legacy parallel file systems require a significant amount of infrastructure knowledge and system expertise as part of the functional part. The tasks include in this case are:

– Installing different software on metadata servers (MDS), object storage servers (OSS), metadata targets (MDT), and object server targets (OST).

– Different services installation on different physical servers.

– Frequent and significant investment in IT resources is often applied in the case of large HPC environments.

– The legacy systems need to be updated and modified continuously.

HPC storage must be designed in a way so that it can provide an optimal HPC storage solution. Thus, it has to be the right mix of both traditional storage and cloud storage. Traditional storage involves on-prem disk drives, whereas cloud storage includes SSDs and HDDs.

The role of cloud storage will be to channelize data-intensive IOPS, while disk drives will handle usual data streaming. For performance scaling, we need an efficient combination of software-defined storage and hardware configuration. Moreover, the HPC storage must handle complex sequential I/O while supporting the parallel file systems.

Image source

Additionally, HPC applications may have “hot” data, as well as “cold data.” As we know, hot data needs to be accessed frequently. It should be kept close to the compute nodes on SSDs. Simultaneously, the “cold” data, which is usually used infrequently, can reside in an object storage array. Furthermore, data and metadata services need to be fully distributed to make the process of managing the overall environment simpler.

The modern parallel file system architecture maintains a tiered architecture for a range of HPC workloads. In such architecture, and integration between hot and cold data tiers happens with metadata servers and scalable data. These metadata services must always be available irrespective of the data is on the hot or cold tier. Hence, ideally, the critical and mostly used data is stored close to the computer nodes, while for less active data, the less expensive storage devices are used. Since the modern HPC storage system utilizes a range of storage devices, it can respond with millisecond latencies whereas can store exabytes of data.

Top HPC Storage Products

Here are references of some of the popular HPC storage solutions in the market –

ClusterStor E1000 All Flash – By Cray (An HPE Company)

E5700 – By NetApp

HPE Apollo 2000 System – By HPE

PRIMERGY RX2530 M5 – By Fujitsu

PowerSwitch Z9332F-ON – By Dell EMC

ScaTeFS – By NEC Corporation

HPC-X ScalableHPC – By Mellanox

Panasas ActiveStor-18 – By Mircorway

Final verdict

With the enormous growth of data sets, we can say it has become a modern technology trend, and this is endless. Hence, HPC storage must process data at the highest speed, which is almost equal to light speed. This will help to maintain compute efficiency at the peak. Furthermore, this needs an HPC storage capacity to climb from petascale to exascale. In addition to that, it must have flawless in-built security, fault-tolerance, modularity in design, and most importantly, seamless scalability.  Furthermore, hybrid cloud technology-based HPC storage is a crucial area that needs runtime control of its components.