Using AMD GPUs with minikube

This tutorial shows how to start minikube with support for AMD GPUs.

Support is provided by the AMD GPU device plugin for Kubernetes.

Prerequisites

• Linux
• Latest AMD GPU Drivers 6.2.1 or greater
• minikube v1.35.0 or later (docker driver only)

Using the docker driver

• Ensure you have an AMD driver installed, you can check if one is installed by running rocminfo, if one is not installed follow the Radeon™ Driver Installation Guide

• Delete existing minikube (optional)

  If you have an existing minikube instance, you may need to delete it if it was built before installing the AMD drivers.

  minikube delete
  

• Start minikube:

  minikube start --driver docker --container-runtime docker --gpus amd
  

Verifying the GPU is available

Test the AMD GPUs are available to the cluster.

1. Create the following Job:

   cat <<'EOF' | kubectl apply -f -
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: amd-gpu-check
     labels:
       purpose: amd-gpu-check
   spec:
     ttlSecondsAfterFinished: 100
     template:
       spec:
         restartPolicy: Never
         securityContext:
           supplementalGroups: 
           - 44
           - 110
         containers:
           - name: amd-gpu-checker
             image: rocm/rocm-terminal
             workingDir: /root
             command: ["rocminfo"]
             args: []
             resources:
               limits:
                 amd.com/gpu: 1 # requesting a GPU
   EOF
   

2. Check the Job output kubectl logs jobs/amd-gpu-check looks something like the following:

   ROCk module version 6.8.5 is loaded
   =====================    
   HSA System Attributes    
   =====================    
   Runtime Version:         1.14
   Runtime Ext Version:     1.6
   System Timestamp Freq.:  1000.000000MHz
   Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
   Machine Model:           LARGE                              
   System Endianness:       LITTLE                             
   Mwaitx:                  DISABLED
   DMAbuf Support:          YES
   
   ==========               
   HSA Agents               
   ==========               
   *******                  
   Agent 1                  
   *******                  
     Name:                    AMD Ryzen 7 7840U w/ Radeon  780M Graphics
     Uuid:                    CPU-XX                             
   ...
   

Where can I learn more about GPU passthrough?

See the excellent documentation at https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

Why does minikube not support AMD GPUs on Windows?

minikube supports Windows host through Hyper-V or VirtualBox.

• VirtualBox doesn’t support PCI passthrough for Windows host.

• Hyper-V supports DDA (discrete device assignment) but only for Windows Server 2016

Since the only possibility of supporting GPUs on minikube on Windows is on a server OS where users don’t usually run minikube, we haven’t invested time in trying to support GPUs on minikube on Windows.