LAMMPS

The cluster provides the following versions of LAMMPS:

Available versions of LAMMPS
Version	MPI support	CPU support	GPU support	Binary name	Modulefile	Packages
20240829.2	yes	yes	yes (preferred)	`lmp_mpi_gpu`	`lammps/20240829.2-nvhpc-25.5-aocc-5.0.0`	GPU, KOKKOS, KSPACE, MANYBODY, MOLECULE, REAXFF, RIGID, OPENMP, EXTRA-MOLECULE, EXTRA-COMPUTE, EXTRA-DUMP, EXTRA-FIX

Running LAMMPS

The current LAMMPS built in the cluster is designed to run with one MPI process per GPU. Each process spawns multiple OpenMP threads, ideally bound to separate CPU cores. For example, on a node with 2 GPUs and 48 CPU cores, use:

2 MPI processes (1 per GPU)
24 OpenMP threads per process (48 cores / 2 processes)

This ensures optimal utilization of both CPU and GPU resources.

Example Slurm Job Script

#!/bin/bash
#SBATCH --job-name=lammps_gpu
#SBATCH --partition=gpu1h100
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=4
#SBATCH --gpus-per-task=1
#SBATCH --time=02:00:00
#SBATCH --output=lammps_%j.out
#SBATCH --error=lammps_%j.err

# All requirements will be automatically loaded
module load lammps/20240829.2-nvhpc-25.5-aocc-5.0.0

# Activate any environment variables or paths if needed
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export OMP_PLACES=cores
export OMP_PROC_BIND=close

# Path to your LAMMPS executable.
# NOTE: The LAMMPS_ROOT variable is set by the lammps module above
LAMMPS_EXEC=$LAMMPS_ROOT/bin/lmp_mpi_gpu

ulimit -l unlimited

# Run the simulation
mpirun -np $SLURM_NTASKS --bind-to core \
    -x OMP_NUM_THREADS \
    -x OMP_PLACES \
    -x OMP_PROC_BIND \
    $LAMMPS_EXEC -in input.lmp

Building LAMMPS

This example demonstrates how to build LAMMPS version 29Aug2024_update2 with GPU support using NVIDIA’s HPC SDK (nvhpc) 25.5 and CUDA 12.9.

LAMMPS provides a set of CMake presets to simplify configuration. These are located in: stable_29Aug2024_update2/cmake/presets/. This example uses the following presets:

basic.cmake - Enables core packages: KSPACE, MANYBODY, MOLECULE, RIGID
kokkos-cuda.cmake - Enables the KOKKOS package with CUDA and OpenMP support
nvhpc.cmake - Configures compilers and MPI settings for NVIDIA’s HPC toolkit

INSTALL_PREFIX=/path/to/lammps/install

# Download and extract the specified LAMMPS version
wget https://github.com/lammps/lammps/archive/refs/tags/stable_29Aug2024_update2.tar.gz
tar xf stable_29Aug2024_update2.tar.gz
mkdir -p stable_29Aug2024_update2/build
cd stable_29Aug2024_update2/build

# Load required modules
module load cmake/3.30.5-gcc-11.5.0-zxhlylf
module load nvhpc/25.5

# Update the Kokkos preset to match the GPU architecture (e.g., H100 = HOPPER90)
sed -i 's/^set(Kokkos_ARCH_.*/set(Kokkos_ARCH_HOPPER90 ON CACHE BOOL "" FORCE)/' ../cmake/presets/kokkos-cuda.cmake
# Update the Kokkos preset to add NATIVE architecture
sed -i -e '/set(Kokkos_ARCH_HOPPER90 ON CACHE BOOL "" FORCE)/a\' -e 'set(Kokkos_ARCH_NATIVE ON CACHE BOOL "" FORCE)' ../cmake/presets/kokkos-cuda.cmake

# Configure the build
cmake -C ../cmake/presets/basic.cmake \
      -C ../cmake/presets/kokkos-cuda.cmake \
      -C ../cmake/presets/nvhpc.cmake \
      -D BUILD_SHARED_LIBS=ON \
      -D LAMMPS_MACHINE=mpi_gpu \
      -D CMAKE_INSTALL_PREFIX=$INSTALL_PREFIX \
      -D CMAKE_PREFIX_PATH="$NVHPC_ROOT;$NVHPC_ROOT/cuda/12.9;$NVHPC_ROOT/math_libs;$NVHPC_ROOT/cuda/12.9/targets/x86_64-linux/lib/cmake" \
      -D CMAKE_BUILD_TYPE=Release \
      -D CUDAToolkit_INCLUDE_DIRECTORIES=$NVHPC_ROOT/cuda/12.9/include \
      -D CMAKE_CXX_STANDARD=17 \
      -D LAMMPS_MEMALIGN=64 \
      -D PKG_REAXFF=ON \
      -D PKG_GPU=ON \
      -D PKG_OPENMP=ON \
      ../cmake

# Build and install
make
make install