What High Performance Computing Looks Like Today

High performance computing (HPC) is a collection of computing resources that solves complex problems that personal computers cannot handle. While supercomputers remain a special case of HPC systems, modern HPC typically mixes supercomputing with virtualized and bare metal servers, platform management, and integration capabilities. The Cray supercomputer family is a classic example: what began as specialized, multi-million-dollar machines has evolved into distributed processing technology that runs on legacy hardware, putting HPC within reach of far more organizations.

Why Cloud HPC Matters

Several factors drive HPC users toward the public cloud. Cloud hardware today dramatically improves on legacy on-premises infrastructure, especially for workloads requiring graphical processing units (GPUs). Cloud HPC also brings auto-scaling, orchestration, development tools, big data analytics, and management software to the table.

Simplified Sharing and Collaboration

Cloud-based HPC systems make data and metadata easy to share, enabling collaboration and supporting automated workflows. For instance, users can search for previous jobs with similar properties to avoid re-analysis, or feed HPC data directly into analytics and AI systems. Legacy HPC architectures were not designed to exchange information efficiently between remote systems, whereas cloud infrastructure opens up collaboration across corporate boundaries.

Better Management and Traceability

HPC applications run on highly distributed hardware, making continuous end-to-end monitoring difficult for engineering and IT teams. In cloud-based HPC, each part of the process runs on a specific, isolated hardware and software stack. Tiered cloud storage aligns storage and optimizes performance based on process needs. This approach reduces the number of manual scripts required and increases abstraction, aligning HPC workloads with their hardware capabilities across the entire business stack.

Efficiency Gains

HPC ecosystems promote collaboration across social, transportation, media, and supply networks. With customers and engineering centers worldwide, organizations need HPC that supports cross-site collaboration, brings IT resources closer to users, and addresses data fragmentation. Cloud-based systems handle these demands naturally, making HPC affordable and accessible to startups and research programs that local systems cannot support.

A Comparison of Cloud HPC Options

AWS: Storage and Orchestration Choices

Selecting Storage

Amazon Elastic Block Store (EBS) offers continuous block storage with high IOPS optimization, enabled by default on C5 and M5 instances. This helps reduce network contention, improve application performance, and support memory- and genomics-heavy workloads. For high I/O requirements, AWS provides Amazon FSx for Lustre, a parallel file system with fixed latency under 1 millisecond. Each TB delivers 200MB/s, scalable to hundreds of GB/s and millions of IOPS. Object storage (S3), file system storage (EFS), and file storage (Amazon Glacier) round out the options.

Automation and Scheduling

Running an AWS HPC instance typically requires both schedulers and batch management. Users can bring existing systems or build scheduling with SQS and CloudWatch, or the spot instance market. AWS Batch is a managed task scheduler that dynamically organizes IT resources for batch jobs. AWS ParallelCluster lets you run virtual HPC clusters in minutes, and CloudFormation templates automate entire HPC workloads.

Azure: Deployment Scale and Monitoring

Sizing Deployments

Azure recommends keeping HPC deployments under 500 virtual machines or 1000 cores. These are not hard limits, but exceeding them can cause loss of active instances, implementation delays, and IP swapping issues.

Remote Access and Monitoring

Connecting to the root node via remote desktop can be slow when the node is busy. The HPC Pack Client Utility is a better alternative to Remote Desktop Service (RDS) for logging into the system. For large deployments, Azure Monitor tracks deployments and integrates with SQL Server and HPC managed services. You can even disable collection of performance indicators instead of continuously checking performance.

Google Cloud: Storage, Placement, and CPU Tuning

Choosing a File System

Google Cloud offers two primary storage options:

  • NFS-based solutions are the simplest shared storage and suit applications with low I/O requirements and minimal data sharing between compute nodes.
  • Parallel file systems follow the POSIX standard, commonly used in message passing interface (MPI) applications. Open source Lustre can serve thousands of nodes and enables parallel I/O for large supercomputers. Use Lustre for I/O and data libraries like HDF5 and NetCDF.

Placement and Hyper-Threading

Location policies determine where virtual machines sit in the data center. The compact deployment strategy provides a low latency topology within the same Availability Zone, using the API to create up to twenty two physically adjacent custom C2 virtual machines. For more than twenty two instances (over 660 physical cores), set up several deployments.

Disabling hyper-threading on guest operating systems can improve MPI application performance. Google Cloud provides two virtual cores (VCPUs) per physical core. While over-processing dramatically improves many I/O-intensive tasks, it hurts performance when the two virtual cores are computationally connected.

Key Takeaways

Moving HPC to the cloud has proven highly successful. As a shared solution, HPC becomes affordable, letting more organizations and startups advance their research and development. Cloud integration adds automation and orchestration resources, while user-friendly interfaces and managed services make cloud HPC accessible to a wide range of users.