Systems
HPC@Mines has two main HPC systems available for researchers and students: Wendian and Mio.
Wendian
Wendian is the newest high performance computing platform at Mines. Wendian came online in the fall of 2018. It contains 87 compute nodes; Skylake Intel processors comprise the bulk of the system, with nine Nvidia GPU nodes and two OpenPower nodes. Max performance rating is over 350 teraflops. Additionally, Wendian has three administration nodes and six file system nodes holding up to 1152 terabytes of raw storage with over 10 gigabytes/second transfer speeds. Wendian currently runs CentOS 7, a community-driven, functionality-compatible computing platform to Red Hat Enterprise Linux . Simulations are managed as jobs using the SLURM scheduler.
Wendian Compute Configuration
Processor |
Cores |
Memory (GB) |
Nodes / Cards |
Cores Total |
Memory Total (GB) |
|---|---|---|---|---|---|
Skylake 6154 |
36 |
192 |
39 |
1,404 |
7,488 |
Skylake 6154 |
36 |
384 |
39 |
1,404 |
14,976 |
Skylake 5118 |
24 |
192 |
5 |
120 |
960 |
Skylake 5218 |
32 |
192 |
4 |
128 |
768 |
GPU cards for 5118’s (Volta V100) |
32 |
20 |
640 |
||
GPU cards for 5218’s (Tesla A100) |
40 |
16 |
640 |
||
OpenPower 8 |
16 |
256 |
2 |
32 |
512 |
OpenPower 9 |
16 |
256 |
2 |
32 |
512 |
Totals |
123 |
3112 |
26496 |
Compute Node Details
78 Relion XO1132G Server – Skylake Nodes
Dual Intel Xeon 6154 (18 cores, 36 threads, 3.0GHz)
39 nodes with 192GB RAM, DDR4-2666MHz ECC (12 x 16GB)
39 nodes with 384GB RAM, DDR4-2666MHz ECC (12 x 32GB)
256 GB SSD
5 Relion XO1114GTS Server GPU Nodes
Dual Intel Xeon Gold 5118 CPU (12 cores, 24 threads, 2.30GHz)
192GB RAM, DDR4-2666MHz REG, ECC, 2R (12 x 16GB)
256GB SSD, 2.5″, SATA, 6Gbps, 0.2 DWPD, 3D TLC (Micron 1100)
4 x NVIDIA Tesla V100-SXM2, 32GB HBM2, 5120 CUDA, 640 Tensor, 300W
4 Relion XO1114GT GPU Server Nodes
Dual Intel Xeon Gold 5218 (16C, 2.30GHz, 125W)
192GB RAM, DDR4-2933MHz ECC (12 x 16GB)
256GB SSD
4 x NVIDIA A100-PCIe, 40GB HBM2
Filesystem
960TB Usable Capacity @ 10GB/s
Relion 1900 Server – running the BeeGFS MDS
2 x 150GB SSD, 2.5″, SATA, 6Gbps, 1 DWPD, 3D MLC
4 x 400GB SSD, 2.5″, SATA, 6Gbps, 3 DWPD, MLC
IceBreaker 4936 Server – running BeeGFS OSS
2 x 150GB SSD, 2.5″, SATA, 6Gbps, 1 DWPD, 3D MLC
4 x 36 x 8TB HDD, 3.5″, SAS, 12Gbps – 1,152TB raw
Ability to create parallel file systems from local disk on the fly
Cooling
The XO1132g and X01114GTS servers have on-board water cooling for the CPUs. These are all fed water from a cooling distribution unit, a CDU. This removes about 60% of the total heat generated. The water to the compute resources is in a closed loop. The CDU has a heat exchanger with the heat emitted by the closed loop warming chilled water from central facilities. Remaining heat from the servers and heat generated by the other nodes is removed via two in-row coolers. The equipment list includes (2) APC ACRC301S In-Row Coolers and a MOTIVAIR Coolant Distribution Unit MCDU25.
Mio
Mio is currently the oldest running HPC system @ Mines. It is a 120+ Tflop HPC cluster for Mines student and faculty research use. The name “Mio” is a play on words. It is a Spanish translation of the word “mine,” as in “belongs to me.” The phrase “The computer is mine” can be translated as “El ordenador es mío.”
Students
Students have already purchased some access to Mio with Tech Fee funds—usable for general research, class projects, and learning HPC techniques. Students may also at times use Mio nodes purchased by their academic advisor or other professors. The HPC Group offers assistance to students (and faculty) to get up and running on Mio. Individual consultations and workshops are available.
Faculty
Mio holds many advantages for professors:
There’s no need to manage their own HPC resources
Professors can access other professors’ resources when allowed
Mines supplies high-quality Infiniband network infrastructure, which greatly improves the scalability of multinode applications
Mio Compute Configuration
As of Aug 1 2023, the available compute options are as follows:
$ /sw/utility/local/slurmnodes -fAvailableFeatures -fRealMemory | /sw/utility/local/jlines 3
compute000 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute001 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute002 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute003 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute004 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute005 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute006 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute007 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute008 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute009 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute010 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute011 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute012 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute013 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute014 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute015 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute016 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute017 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute018 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute019 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute020 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute021 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute022 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute023 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute024 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute025 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute026 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute027 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute102 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute103 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute105 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute106 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute107 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute108 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute110 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute111 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute114 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute115 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute116 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute117 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute119 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute120 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute121 RealMemory 64000 AvailableFeatures core16,sandybridge,mlx4,qdr
compute122 RealMemory 64000 AvailableFeatures core16,sandybridge,qib,qdr
compute123 RealMemory 64000 AvailableFeatures core16,sandybridge,qib,qdr
compute124 RealMemory 64000 AvailableFeatures core16,sandybridge,qib,qdr
compute125 RealMemory 64000 AvailableFeatures core16,sandybridge,qib,qdr
compute126 RealMemory 64000 AvailableFeatures core20,ivybridge,qib,qdr
compute128 RealMemory 64000 AvailableFeatures core20,ivybridge,qib,qdr
compute129 RealMemory 64000 AvailableFeatures core20,ivybridge,qib,qdr
compute130 RealMemory 64000 AvailableFeatures core20,ivybridge,qib,qdr
compute131 RealMemory 64000 AvailableFeatures core20,ivybridge,qib,qdr
compute132 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute133 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute134 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute135 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute136 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute137 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute138 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute139 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute140 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute141 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute142 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute143 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute144 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute145 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute146 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute147 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute148 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute149 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute150 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute151 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute152 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute153 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute154 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute155 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute156 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute157 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute158 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute159 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute160 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute161 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute162 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute163 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute164 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute165 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute166 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute167 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute168 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute169 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute170 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute171 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute172 RealMemory 64000 AvailableFeatures core24,haswell,mlx4,fdr
compute174 RealMemory 254000 AvailableFeatures core24,haswell,mlx4,fdr
compute175 RealMemory 254000 AvailableFeatures core24,haswell,mlx4,fdr
compute176 RealMemory 254000 AvailableFeatures core24,haswell,mlx4,fdr
compute177 RealMemory 254000 AvailableFeatures core24,haswell,mlx4,fdr
compute178 RealMemory 254000 AvailableFeatures core24,haswell,mlx4,fdr
compute179 RealMemory 254000 AvailableFeatures core24,haswell,mlx4,fdr
compute180 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute181 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute182 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute183 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute184 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute186 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute187 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute188 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute189 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute190 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute191 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute193 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute194 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute195 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute196 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute197 RealMemory 64000 AvailableFeatures core28,broadwell,mlx4,fdr
compute198 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute199 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute200 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute201 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute202 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute203 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute204 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute205 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute206 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute207 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute208 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute209 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute210 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute211 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute212 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute213 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute214 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute215 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute216 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute217 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute218 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
compute219 RealMemory 254000 AvailableFeatures core28,broadwell,mlx4,fdr
gpu004 RealMemory 190903 AvailableFeatures core16,skylake,mlx4,qdr