Editing CUDA + NVIDIA drivers

== Installation ==
Download repo rpm from http://developer.nvidia.com/cuda-downloads (or copy the repo file over from another similar system). For further information, see the [https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html Installation Guide for Linux] and the [https://docs.nvidia.com/cuda/cuda-quick-start-guide/index.html CUDA Quick Start Guide].

=== Fedora 33 ===
Install the repo and then deploy the full package:
 sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/fedora32/x86_64/cuda-fedora33.repo
 sudo dnf clean all
 sudo dnf -y module install nvidia-driver:latest-dkms
 sudo dnf -y install cuda

=== CentOS 7 ===
It seems that the dependency for the linux kernel devel package has been lost ... and the driver install just silently fails
 yum install kernel-devel

Now install the repository
 sudo yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo
 sudo yum clean all

On regular compute nodes where the full CUDA libraries aren't needed:
 yum install nvidia-driver-latest-dkms

On a full workstation or where CUDA is needed:
 sudo yum -y install nvidia-driver-latest-dkms cuda
 sudo yum -y install cuda-drivers

There is a yum plugin that facilitates the installation and management of NVIDIA kernel modules:

 yum -y install yum-plugin-nvidia

=== Debian 10/11 ===
Install the repo and then deploy the full package:
 sudo apt-get install -y software-properties-common
 sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/debian10/x86_64/7fa2af80.pub
 sudo add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/debian10/x86_64/ /"
 sudo add-apt-repository contrib
 sudo apt-get update
 sudo apt-get -y install cuda

Alternatively, if the full CUDA distribution is not wanted/needed:
 sudo apt -y install nvidia-driver nvidia-smi

After the install completes, a reboot is required to activate the drivers.

== Updating drivers ==
The above process installs the cuda repository for yum ... and while most of the time updates will happen automatically, there are some things to consider when updating.

The command examples are for CentOS ... use the command and package set appropriate for the OS in use.

=== Reinstalling ===
Occasionally you need to just uninstall and reinstall the drivers.  Rather than uninstalling and reinstalling the whole works, you can just do one group:

 sudo yum -y remove kmod-nvidia-dkms-latest
 sudo yum -y install kmod-nvidia-dkms-latest
 reboot

Unfortunately, you cannot use the 'yum reinstall' command -- that just overlays the package with a new copy from the .rpm file.  DKMS sees that the module already exists (whether it works or not) and just does nothing.  The reboot is needed for the kernel to reload the reinstalled drivers.

=== Kernel Updates ===
Most of the time, a driver update by itself will work fine ... as will a kernel update.  When these happen at the same time, problems can occur.  It is best for the kernel update to occur first -- even if you have to do it manually:

 sudo yum -y update kernel kernel-devel

Then you need to reboot (sorry).  This way, when the driver update is applied, it is applied to the current (new) kernel version:

 sudo yum -y update kmod-nvidia-dkms-lastest

This will require another reboot to activate the new modules.

=== DKMS Issues ===
Sometimes the update process will not recognize that it can do an update for a particular kernel.  In this case, you need to completely remove the driver, reboot, and reinstall.  When it happens, it is usually in conjunction with a kernel update (see above) -- so ... do the kernel update after removing the drivers, reboot, and then reinstall the driver:

 sudo yum -y remove kmod-nvidia-dkms-latest
 <update kernel if needed>
 reboot
 
 sudo yum -y install kmod-nvidia-dkms-latest

Ignore any warnings about not finding the latest version of the kmod-nvidia-dkms package ... the process will create it.