Set up an A2 GPU instance¶
This guide shows how to create a Compute instance with an NVIDIA A2 GPU, install the NVIDIA driver, and check that the GPU works. At the end, there is an optional example of running a local language model (LLM).
Note
This guide covers NVIDIA A2 only. There is no setup guide for NVIDIA H100 NVL yet. See GPU Flavors for all available GPUs.
The examples use Ubuntu 24.04. Other operating systems also work, but the driver installation differs. See NVIDIA's driver installation guide for other distributions.
Before you start¶
You need:
- access to a Safespring Compute project in STO2, where A2 flavors are currently available
- an SSH key pair imported into the dashboard
- a security group that allows SSH (port 22)
If you have not created an SSH key pair or a security group before, follow the Preparations section of Getting Started first.
1. Launch a GPU instance¶
Follow Launch your first instance in Getting Started, with these choices:
- Dashboard: log in to the STO2 dashboard. A2 flavors are not shown at other sites.
- Source: choose the image
Ubuntu 24.04to follow this guide exactly. - Flavor: choose a flavor ending in
gA2, for exampleb2.c4r8.gA2. See GPU Flavors for all A2 flavors. - Networks: choose public, so that you can connect with SSH.
b2 or l2?
Not sure which to pick? See Flavors: What should I choose? for more information.
When the instance has status Running, connect to it:
ssh ubuntu@<ip-address>
2. Install the NVIDIA driver¶
The instance has a GPU, but Ubuntu needs a driver to use it.
Update the system and install the tool that finds the right driver:
sudo apt update && sudo apt upgrade -y
sudo apt install ubuntu-drivers-common
List the available drivers:
ubuntu-drivers devices
The output shows several drivers. Look for the one marked recommended, and choose the version ending in -server-open. Install it, for example:
sudo apt install nvidia-driver-580-server-open
sudo reboot
Replace 580 with the version recommended for your instance. The reboot closes your SSH connection. Wait a minute and connect again.
3. Verify the GPU¶
Check that the GPU is detected:
nvidia-smi
The output should show NVIDIA A2, the driver version and the CUDA version. The version numbers in your output may differ from this example:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.172.08 Driver Version: 570.172.08 CUDA Version: 12.9 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
|=========================================+========================+======================|
| 0 NVIDIA A2 On | 00000000:00:05.0 Off | 0 |
| 0% 35C P8 5W / 60W | 0MiB / 15356MiB | 0% Default |
+-----------------------------------------+------------------------+----------------------+
If you see this, the instance is ready for GPU workloads.
4. Optional: run a local LLM with Ollama¶
A common use case for A2 instances is running language models locally. This example uses Ollama to run a model and Open-WebUI for a chat interface in the browser.
Install Ollama¶
curl -fsSL https://ollama.com/install.sh | sh
ollama -v
Download and run a model¶
ollama pull llama3:8b
ollama run llama3:8b
To see that the model uses the GPU, run nvidia-smi in a second SSH session while the model is running.
Access Ollama from your computer¶
Ollama only listens on the instance itself (localhost:11434). To reach it from your computer, connect with SSH port forwarding:
ssh -L 11434:localhost:11434 ubuntu@<ip-address>
Add a chat interface with Open-WebUI¶
Install Docker on the instance:
sudo apt install -y docker.io
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"
Log out and connect again, this time forwarding port 8080 for the web interface:
ssh -L 8080:localhost:8080 ubuntu@<ip-address>
Start Open-WebUI:
docker run -d \
--name open-webui \
--network=host \
-e OLLAMA_BASE_URL=http://127.0.0.1:11434 \
-v open-webui:/app/backend/data \
--restart always \
ghcr.io/open-webui/open-webui:latest
Open http://localhost:8080 in your browser. All processing happens on your instance.