Systems

HPC@Mines has two main HPC systems available for researchers and students: Wendian and Mio.

Wendian

Wendian is the newest high performance computing platform at Mines. Wendian came online in the fall of 2018. It contains 87 compute nodes; Skylake Intel processors comprise the bulk of the system, with nine Nvidia GPU nodes and two OpenPower nodes. Max performance rating is over 350 teraflops. Additionally, Wendian has three administration nodes and six file system nodes holding up to 1152 terabytes of raw storage with over 10 gigabytes/second transfer speeds. Wendian currently runs CentOS 7, a community-driven, functionality-compatible computing platform to Red Hat Enterprise Linux . Simulations are managed as jobs using the SLURM scheduler.

Compute Configuration

Processor

Cores

Memory (GB)

Nodes / Cards

Cores Total

Memory Total (GB)

Skylake 6154

36

192

39

1,404

7,488

Skylake 6154

36

384

39

1,404

14,976

Skylake 5118

24

192

5

120

960

Skylake 5218

32

192

4

128

768

GPU cards for 5118’s (Volta V100)

32

20

640

GPU cards for 5218’s (Tesla A100)

40

16

640

OpenPower 8

16

256

2

32

512

OpenPower 9

16

256

2

32

512

Totals

123

3112

26496

Compute Node Details

  • 78 Relion XO1132G Server – Skylake Nodes

    • Dual Intel Xeon 6154 (18 cores, 36 threads, 3.0GHz)

      • 39 nodes with 192GB RAM, DDR4-2666MHz ECC (12 x 16GB)

      • 39 nodes with 384GB RAM, DDR4-2666MHz ECC (12 x 32GB)

    • 256 GB SSD

  • 5 Relion XO1114GTS Server GPU Nodes

    • Dual Intel Xeon Gold 5118 CPU (12 cores, 24 threads, 2.30GHz)

    • 192GB RAM, DDR4-2666MHz REG, ECC, 2R (12 x 16GB)

    • 256GB SSD, 2.5″, SATA, 6Gbps, 0.2 DWPD, 3D TLC (Micron 1100)

    • 4 x NVIDIA Tesla V100-SXM2, 32GB HBM2, 5120 CUDA, 640 Tensor, 300W

  • 4 Relion XO1114GT GPU Server Nodes

    • Dual Intel Xeon Gold 5218 (16C, 2.30GHz, 125W)

    • 192GB RAM, DDR4-2933MHz ECC (12 x 16GB)

    • 256GB SSD

    • 4 x NVIDIA A100-PCIe, 40GB HBM2

Filesystem

  • 960TB Usable Capacity @ 10GB/s

  • Relion 1900 Server – running the BeeGFS MDS

  • 2 x 150GB SSD, 2.5″, SATA, 6Gbps, 1 DWPD, 3D MLC

  • 4 x 400GB SSD, 2.5″, SATA, 6Gbps, 3 DWPD, MLC

  • IceBreaker 4936 Server – running BeeGFS OSS

  • 2 x 150GB SSD, 2.5″, SATA, 6Gbps, 1 DWPD, 3D MLC

  • 4 x 36 x 8TB HDD, 3.5″, SAS, 12Gbps – 1,152TB raw

  • Ability to create parallel file systems from local disk on the fly

Cooling

The XO1132g and X01114GTS servers have on-board water cooling for the CPUs. These are all fed water from a cooling distribution unit, a CDU. This removes about 60% of the total heat generated. The water to the compute resources is in a closed loop. The CDU has a heat exchanger with the heat emitted by the closed loop warming chilled water from central facilities. Remaining heat from the servers and heat generated by the other nodes is removed via two in-row coolers. The equipment list includes (2) APC ACRC301S In-Row Coolers and a MOTIVAIR Coolant Distribution Unit MCDU25.

Quality of Service (QoS) Queues

On Wendian, there are three types of queues:

  • group: Comprises QoSes of nodes purchased by PIs;

  • full: Comprises all nodes on machine (including purchased nodes).

  • normal: Comprises of nodes not purchased by PIs; jobs in this QoS are not preemptable. Currently limited to 1080 cores (30 nodes).

For example, to switch to the normal QoS, add the following slurm directive to your submission script:

#SBATCH --qos=normal

As a practical rule of thumb, here’s what we recommend for which QoS to use:

  • If your PI has purchased a priority QoS, your default QoS will be theirs and you will have priority access to the number of CPUs/nodes that they have bought priority access for.

    • If you see a QOSGrpCpuLimit status in your job when checking with squeue, that means that maximum number of CPUs for your group’s QoS are being used and you must wait until they’re freed. Alternatively, you can switch to either the full or normal QoS to avoid waiting for your group.

    • If your PI has not purchased a priority QoS, you have two options:

  • Use the full QoS if you want to increase the pool of possible nodes to submit jobs to, at the risk of job preemption (cancellation)

  • Use the normal QoS if you want to avoid preemption, at the risk of potentially waiting for resources if the normal QoS is full.

Mio

Mio is currently the oldest running HPC system @ Mines. It is a 120+ Tflop HPC cluster for Mines student and faculty research use. The name “Mio” is a play on words. It is a Spanish translation of the word “mine,” as in “belongs to me.” The phrase “The computer is mine” can be translated as “El ordenador es mío.”

Students

Students have already purchased some access to Mio with Tech Fee funds—usable for general research, class projects, and learning HPC techniques. Students may also at times use Mio nodes purchased by their academic advisor or other professors. The HPC Group offers assistance to students (and faculty) to get up and running on Mio. Individual consultations and workshops are available.

Faculty

Mio holds many advantages for professors:

  • There’s no need to manage their own HPC resources

  • Professors can access other professors’ resources when allowed

  • Mines supplies high-quality Infiniband network infrastructure, which greatly improves the scalability of multinode applications

Compute Configuration

As of Jun 7 2022, the available compute options are as follows:

$ /opt/utility/slurmnodes -fAvailableFeatures -fRealMemory | /opt/utility/jlines 3
 compute000    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute001    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute002    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute003    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute004    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute005    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute006    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute007    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute008    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute009    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute010    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute011    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute012    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute013    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute014    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute015    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute016    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute017    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute018    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute019    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute020    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute021    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute022    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute023    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute024    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute025    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute026    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute027    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute030    RealMemory 21600    AvailableFeatures core8,nehalem,mthca,ddr
 compute031    RealMemory 21600    AvailableFeatures core8,nehalem,mthca,ddr
 compute032    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute033    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute036    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute037    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute038    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute039    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute040    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute041    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute043    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute044    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute045    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute050    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute051    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute052    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute054    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute055    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute056    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute057    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute058    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute059    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute061    RealMemory 21600    AvailableFeatures core12,westmere,mthca,ddr
 compute062    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute064    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute065    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute066    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute067    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute068    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute069    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute070    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute071    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute072    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute073    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute074    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute075    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute076    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute078    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute079    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute080    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute081    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute083    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute084    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute085    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute086    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute087    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute088    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute089    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute090    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute091    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute092    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute093    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute094    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute095    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute096    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute097    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute098    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute099    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute100    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute101    RealMemory 21600    AvailableFeatures core12,westmere,mlx4,ddr
 compute102    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute103    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute105    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute106    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute107    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute108    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute110    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute111    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute114    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute115    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute116    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute117    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute119    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute120    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute121    RealMemory 64000    AvailableFeatures core16,sandybridge,mlx4,qdr
 compute122    RealMemory 64000    AvailableFeatures core16,sandybridge,qib,qdr
 compute123    RealMemory 64000    AvailableFeatures core16,sandybridge,qib,qdr
 compute124    RealMemory 64000    AvailableFeatures core16,sandybridge,qib,qdr
 compute125    RealMemory 64000    AvailableFeatures core16,sandybridge,qib,qdr
 compute126    RealMemory 64000    AvailableFeatures core20,ivybridge,qib,qdr
 compute128    RealMemory 64000    AvailableFeatures core20,ivybridge,qib,qdr
 compute129    RealMemory 64000    AvailableFeatures core20,ivybridge,qib,qdr
 compute130    RealMemory 64000    AvailableFeatures core20,ivybridge,qib,qdr
 compute131    RealMemory 64000    AvailableFeatures core20,ivybridge,qib,qdr
 compute132    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute133    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute134    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute135    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute136    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute137    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute138    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute139    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute140    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute141    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute142    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute143    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute144    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute145    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute146    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute147    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute148    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute149    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute150    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute151    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute152    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute153    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute154    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute155    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute156    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute157    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute158    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute159    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute160    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute161    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute162    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute163    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute164    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute165    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute166    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute167    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute168    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute169    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute170    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute171    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute172    RealMemory 64000    AvailableFeatures core24,haswell,mlx4,fdr
 compute174    RealMemory 254000    AvailableFeatures core24,haswell,mlx4,fdr
 compute175    RealMemory 254000    AvailableFeatures core24,haswell,mlx4,fdr
 compute176    RealMemory 254000    AvailableFeatures core24,haswell,mlx4,fdr
 compute177    RealMemory 254000    AvailableFeatures core24,haswell,mlx4,fdr
 compute178    RealMemory 254000    AvailableFeatures core24,haswell,mlx4,fdr
 compute179    RealMemory 254000    AvailableFeatures core24,haswell,mlx4,fdr
 compute180    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute181    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute182    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute183    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute184    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute186    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute187    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute188    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute189    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute190    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute191    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute193    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute194    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute195    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute196    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute197    RealMemory 64000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute198    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute199    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute200    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute201    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute202    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute203    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute204    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute205    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute206    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute207    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute208    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute209    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute210    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute211    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute212    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute213    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute214    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute215    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute216    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute217    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute218    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 compute219    RealMemory 254000    AvailableFeatures core28,broadwell,mlx4,fdr
 gpu004    RealMemory 190903    AvailableFeatures core16,skylake,mlx4,qdr
 ppc001    RealMemory 253000    AvailableFeatures (null)
 ppc002    RealMemory 253000    AvailableFeatures (null)