Correfoc

Correfoc combines CPU compute nodes, GPU-accelerated nodes, GPFS research storage, InfiniBand HDR data networking, service networking, tape backup for selected filesystems and Slurm partitions for scientific workloads.

At a glance

Global hardware figures

67 Compute nodes
7,968 CPU cores
68 TB Total RAM
57 NVIDIA L40S GPUs
3 PB GPFS raw storage
HDR 200 Gb/s InfiniBand data network

Compute overview

CPU and GPU hardware groups

Correfoc exposes its compute resources through Slurm partitions that map to CPU, GPU and high-capacity hardware groups.

std-cpu

General-purpose CPU compute nodes

  • Intel Xeon E5-2630 v3 nodes
  • Intel Xeon E5-2683 v4 nodes
  • Intel Xeon Gold 6248R nodes
  • Slurm partition: std-cpu

General-purpose CPU workloads, pipelines, simulations, statistical analysis and batch jobs.

std-gpu

Standard GPU and interactive nodes

  • Intel Xeon Gold 6430 nodes
  • 64 CPU cores per node
  • 2 TB RAM per node
  • NVIDIA L40S GPUs
  • 1 GPU per node
  • Slurm partition: std-gpu

Interactive, graphical and GPU-accelerated workloads such as visualization, notebooks, image analysis and AI workflows.

high-cpu

High-capacity CPU compute nodes

  • Intel Xeon 6980P nodes
  • 256 CPU cores per node
  • 1.5 TB RAM per node
  • Slurm partition: high-cpu

Large CPU workloads, parallel jobs, memory-intensive analysis and demanding scientific computing workflows.

high-gpu

Demanding GPU-accelerated nodes

  • Intel Xeon Gold 6530 nodes
  • 64 CPU cores per node
  • 1 TB RAM per node
  • NVIDIA L40S GPUs
  • 4 GPUs per node
  • Slurm partition: high-gpu

AI, deep learning, medical imaging, molecular modelling and other GPU-accelerated scientific workloads.

Storage

GPFS research storage

Correfoc uses a GPFS parallel filesystem with 3 PB of raw capacity for research data and scientific workflows.

scratch

Temporary high-performance space

Temporary high-performance space for active computations.

projects

Project data space

Project data space for research groups. Backed up to tape.

homes

Personal user space

Personal user space. Backed up to tape.

Backup policy: projects and homes are backed up to tape. Scratch is intended for temporary data and should not be used as the only copy of important research data.

Interconnect

Network and interconnect

Correfoc separates high-performance data traffic from service and management traffic.

Data network

InfiniBand HDR 200 Gb/s

  • Used for compute-to-storage traffic and high-performance data movement
Service network

10/25 GbE switches

Used for operating system services, Slurm, routing and general service traffic.

Management network

Out-of-band management / IPMI

Used for hardware management.

Architecture

Simple infrastructure view

Users Researchers and institutional workloads.
Login / visualization Service-facing entry layer for interactive and graphical sessions.
Slurm scheduler Schedules jobs into std-cpu, std-gpu, high-cpu and high-gpu.
scheduled jobs and data paths

Compute

CPU nodes General-purpose and high-capacity CPU hardware for std-cpu and high-cpu.
GPU nodes NVIDIA L40S accelerated nodes for std-gpu and high-gpu.

Network

InfiniBand HDR 200 Gb/s Compute-to-storage data network through  HDR switches.
10/25 GbE service network Operating system services, Slurm, routing and service traffic.
Management / IPMI Out-of-band hardware management.

Storage

GPFS research storage 3 PB raw parallel filesystem for research data and workflows.
scratch projects homes
Tape backup Backup coverage for projects and homes.
Control and scheduler layer Data and service networks Storage and backup layer