@ai-ml/nvidia-container-toolkit

Description

In addition to this COPR, the package has now been added to Fedora as golang-github-nvidia-container-toolkit. Merging the efforts is underway.

Installation Instructions

Setup the NVIDIA kmod via RPMFusion. Ensure this is working before continuing.

Install the CUDA libraries:

$ sudo dnf install xorg-x11-drv-nvidia-cuda

Setup the COPR repo:

$ sudo dnf copr enable @ai-ml/nvidia-container-toolkit

Install the NVIDIA container toolkit:

$ sudo dnf install nvidia-container-toolkit nvidia-container-toolkit-selinux

Setup the CDI configuration:

$ sudo nvidia-ctk cdi generate -output /etc/cdi/nvidia.yaml

Test everything is working:

$ podman run --device nvidia.com/gpu=all --rm fedora:latest nvidia-smi

If everything is working, you should see something like:

  Fri Mar 22 03:18:29 2024         
  +-----------------------------------------------------------------------------------------+  
  | NVIDIA-SMI 550.67                 Driver Version: 550.67         CUDA Version: 12.4     |  
  |-----------------------------------------+------------------------+----------------------+  
  | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |  
  | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |  
  |                                         |                        |               MIG M. |  
  |=========================================+========================+======================|  
  |   0  NVIDIA GeForce RTX 3080 Ti     Off |   00000000:08:00.0  On |                  N/A |  
  |  0%   46C    P8             38W /  350W |     482MiB /  12288MiB |     12%      Default |  
  |                                         |                        |                  N/A |  
  +-----------------------------------------+------------------------+----------------------+  
                                                                                             
  +-----------------------------------------------------------------------------------------+  
  | Processes:                                                                              |  
  |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |  
  |        ID   ID                                                               Usage      |  
  |=========================================================================================|  
  +-----------------------------------------------------------------------------------------+  

Test the Difference Between CPU and GPU Performance

Download a PyTorch environment and ensure it runs:

$ podman run --device nvidia.com/gpu=all --rm docker.io/pytorch/pytorch:latest

Test CPU:

$ time podman run --device nvidia.com/gpu=all --rm docker.io/pytorch/pytorch:latest bash -c 'pip install pytorch-benchmark && python -c "import torch; import json; from torchvision.models import efficientnet_b0; from pytorch_benchmark import benchmark; model = efficientnet_b0().to(\"cpu\"); sample = torch.randn(8, 3, 224, 224); results = benchmark(model, sample, num_runs=200); print(json.dumps(results, indent=4))"'

Test GPU:

$ time podman run --device nvidia.com/gpu=all --rm docker.io/pytorch/pytorch:latest bash -c 'pip install pytorch-benchmark && python -c "import torch; import json; from torchvision.models import efficientnet_b0; from pytorch_benchmark import benchmark; model = efficientnet_b0().to(\"cuda\"); sample = torch.randn(8, 3, 224, 224); results = benchmark(model, sample, num_runs=200); print(json.dumps(results, indent=4))"'

Test SELinux Policy

Test if the shipped NVIDIA DGX SELinux policy is working (the nvidia-container-toolkit-selinux package enables container_use_devices by default) :

$ sudo setsebool container_use_devices 0
$ podman run --device nvidia.com/gpu=all \
             --security-opt label=type:nvidia_container_t \
             --rm fedora:latest nvidia-smi

Re-run testing and then reset SELinux.

$ sudo setsebool container_use_devices 1

Monitoring CPU vs GPU Usage

Use the nvtop utility to monitor what load is on your CPU/GPU:

$ sudo dnf install nvtop
$ nvtop

Something Fun, InstructLab (ilab)

Use InstructLab to run a pre-trained Large Language Model (LLM) and chat with it.

Clone InstructLab and build the container:

$ git clone https://github.com/instructlab/instructlab.git ~/instructlab
$ podman build ~/instructlab/containers/cuda -t instructlab
$ podman run -it --rm --device nvidia.com/gpu=all instructlab

Initialize InstructLab inside the container and start chatting with the LLM:

$ ilab init
$ ilab download
$ ilab chat

Please note, the models, config, etc. are downloaded into the container and will be deleted after the container is stopped.

Something Fun, Image Generation Using ImaginAIry

Use imaginAIry to generate prompt based images using multiple models and the StableStudio web UI.

Build a F40 based container with imaginAIry installed:

$ mkdir ~/imaginary
$ tee ~/imaginary/Dockerfile << EOF
FROM registry.fedoraproject.org/fedora:40
RUN dnf -y install python3-pip rust cargo openssl-devel gcc-c++ libglvnd-glx
RUN pip install imaginairy && pip check
EXPOSE 8000
CMD ["aimg", "server"]
EOF
$ podman build ~/imaginary -t fedora-imaginairy

Run the server:

$ podman run --device nvidia.com/gpu=all \
             -p 8000:8000 \
             --rm fedora-imaginairy

Visit the web UI at localhost:8000 and start generating images based on prompts.

Something Fun, Image Generation Using InvokeAI

Use InvokeAI to generate prompt based images using multiple models.

Run the server:

$ podman run --device nvidia.com/gpu=all \
             -p 9090:9090 \
             --rm ghcr.io/invoke-ai/invokeai

Visit the web UI at localhost:9090 and start generating images based on prompts.

Please note, the models are downloaded into the container and will be deleted after the container is stopped.

Active Releases

The following unofficial repositories are provided as-is by owner of this project. Contact the owner directly for bugs or issues (IE: not bugzilla).

Release	Architectures	Repo Download
Fedora 38	aarch64 (132)*, x86_64 (295)*	Fedora 38 (0 downloads)
Fedora 39	aarch64 (216)*, x86_64 (368)*	Fedora 39 (0 downloads)
Fedora 40	aarch64 (95)*, x86_64 (1301)*	Fedora 40 (310 downloads)
Fedora 41	aarch64 (44)*, x86_64 (652)*	Fedora 41 (132 downloads)
Fedora rawhide	aarch64 (159)*, x86_64 (204)*	Fedora rawhide (61 downloads)

* Total number of downloaded packages.