GPU Machine Learning

With modern pytorch and tensorflow frameworks, we recommend using a python virtual environment instead of the machine learning containers outlined in this article. The containers may still be useful for compatibility with old versions of ML applications.

An updated machine learning container can be accessed by loading environment module ml-gpu. For instructions on how to use these containers, please refer to https://researchit.las.iastate.edu/machine-learning-singularity-image .

Older machine learning containers built by HPC group can be downloaded from Singularity hub at https://singularity-hub.org/collections/876 . There are several containers based on various CUDA versions.

To pull(download) the container

module load singularity

singularity pull shub://ISU-HPC/machine-learning

You can download specific version based on your preference by adding the tag

singularity pull shub://ISU-HPC/machine-learning:cu80dnn6

This pulls the container based on CUDA 8.0 and CUDNN 6

For interactive access, "--nv" provides support for nvidia GPUs within the container.

singularity shell --nv <container_image>

Note: For SCSLab cluster users, your data directory is not available via the container. You need to manually bind mount it. 

singularity shell --nv -B /data <container_image>

These containers are also available on the clusters in /shared/hpc/containers/machine-learning

List of available packages in the container

• Keras
• MXNET
• scikit-learn
• tensorflow
• dlib
• Lasange

If you need to use packages that are not available in the container, you can install the missing packages in your home directory from within the container (keep an eye on the utilization as home dir has 5GB quota):

singularity exec <container_image>  pip3 install <name_of_the_package> --user