Building A Local LLM Playground 02

# LLM Playground #3: Giving Rootless Docker Access to the NVIDIA GPUs
At this point the environment had two independently working pieces:
`
Rootless Docker ✓
RTX 3060 GPUs ✓
`
Unfortunately, that did not mean Docker could use the GPUs.
The next job was to connect those two worlds without abandoning the rootless Docker setup.

/*******************/BEGIN/*******************/
## 1. Check everyone is still speaking to eachother:

nvidia-smi

if you see the Nvidia-smi app - great - keep going - almost there!
If not - perhaps it is time to turn in for the night?

## 2. Add NVIDIA's package repository and install the Container Toolkit
The NVIDIA driver lets Ubuntu communicate with the GPUs. Containers need another layer called the NVIDIA Container Toolkit.
Unlike the Ubuntu-packaged NVIDIA driver from Post #2, the Container Toolkit is installed from NVIDIA's own APT repository. That means Ubuntu first needs NVIDIA's repository signing key and repository definition.
Install the prerequisites:

sudo apt update
sudo apt install -y --no-install-recommends \
  ca-certificates \
  curl \
  gnupg2

Add NVIDIA's repository signing key and production repository:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor \
  -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Refresh APT so Ubuntu learns about the new packages:

sudo apt update

Then install the toolkit:

sudo apt install -y nvidia-container-toolkit

Confirm the utility exists:

nvidia-ctk --version

The toolkit does not install another GPU driver inside Docker. It gives the container runtime a way to expose the host's NVIDIA devices and libraries.
## 3. Remember: this Docker daemon is rootless
Rootless Docker uses:~/.config/docker/daemon.json
Create the directory if needed:

mkdir -p "$HOME/.config/docker"

Configure the NVIDIA toolkit, tell it about the rootless docker locaiton:

nvidia-ctk runtime configure \
  --runtime=docker \
  --config="$HOME/.config/docker/daemon.json"

Inspect it:

cat "$HOME/.config/docker/daemon.json"

## 4. Configure NVIDIA for rootless operation
The rootless runtime cannot rely on the same privileged cgroup behavior as a conventional Docker daemon.
The configuration used during the build was:

  sudo nvidia-ctk config \
    --set nvidia-container-cli.no-cgroups \
    --in-place

running the following command will allow you to validate the config has taken:

nvidia-ctk config

...
[nvidia-container-cli]
#debug = "/var/log/nvidia-container-toolkit.log"
environment = []
#ldcache = "/etc/ld.so.cache"
ldconfig = "@/sbin/ldconfig.real"
load-kmods = true
no-cgroups = true
#path = "/usr/bin/nvidia-container-cli"
#root = "/run/nvidia/driver"
#user = "root:video"
...

## 5. Restart the Docker daemon
For our rootless daemon:

systemctl --user restart docker

Then verify:

systemctl --user status docker
docker info | grep -Ei 'rootless|runtimes'

## 6. The test that matters
Do not install Ollama yet. First use a disposable CUDA container whose only job is to run nvidia-smi:

docker run --rm --gpus all \
  nvidia/cuda:12.6.3-base-ubuntu24.04 \
  nvidia-smi

The exact CUDA image tag will change over time. If this tag is no longer available, use a current NVIDIA CUDA base image that matches a supported Ubuntu release. The important part of the test is that nvidia-smi runs successfully from inside the container.
Once both cards appear from inside the disposable container, rootless Docker and NVIDIA are talking.
Now the host nvidia-smi works and the disposable CUDA container can also run nvidia-smi, it's time to start downloading building the docker containers that will run the local models and allow interraction between you and them via https:

## 7. Install Ollama, OpenWebUI and Nginx
Toxbox has a fast M2 drive that I selected to host ollama, openwebui, nginx and as a place to store the LLMs when they are not in use. Some of the models are quite large and thus loading them to the GPU speeds up with fast storage. Any model that you wish to interract with must be pulled from storage to the GPU in order to be operational.
The below yml file can be used to setup three components which facilitate interaction with the LLM. Ollama is what I like to think of as the orchestrator of models, it pulls them from storage, dumps them on the gpu, evicts GPU resident LLM when instructed to do so, and facilitates interraction with the models and other stuff (more on that later). OpenWebUI is the web based user interface portion which facilitates web based model interraction through a comon UI as well as agents built, configs tweaked, models downloaded and a whole bunch of other stuff I haven't figured out yet. Finally a little nginx engine to provide an encrypted https connection to openWebUI.

The below scripts create the directory structure and build the files required. They will need modify them to match the environment if the default location (~/docker) is not desired. Ensure you have plenty of space, as there are plenty of models to download it's easy to chew through a couple hundred gigs.

The first script builds the directory structure which will host ollama, openWebUI and nginx containers in the current user home directory: '~/docker'.
Change the storage locaitons as appropriate if you want it running somewhere other than the default.

mkdir -p ~/docker/ollama
cd ~/docker/ollama

mkdir -p \
  ollama_ollama/_data \
  ollama_open-webui/_data \
  nginx/conf.d \
  nginx/certs

chmod 700 ~/docker/ollama
chmod 755 \
  ollama_ollama \
  ollama_open-webui \
  nginx \
  nginx/conf.d \
  nginx/certs

chmod 700 \
  ollama_ollama/_data \
  ollama_open-webui/_data

The next script builds the compose.yml file which is a configuraiton file that informs docker of the attributes of the containers (ollama, OpenWebUI, nginx). If necessary, change the locaitons as appropriate (currently ~/docker)

cat > ~/docker/ollama/compose.yml <<'EOF'
services:

  ollama:
    image: ollama/ollama:${OLLAMA_DOCKER_TAG:?OLLAMA_DOCKER_TAG must be set}
    container_name: ollama
    restart: unless-stopped

    environment:
      OLLAMA_KEEP_ALIVE: 30m

    ports:
      - "127.0.0.1:11434:11434"

    volumes:
      - ./ollama_ollama/_data:/root/.ollama


  open-webui:
    image: ghcr.io/open-webui/open-webui:${WEBUI_DOCKER_TAG:?WEBUI_DOCKER_TAG must be set}
    container_name: open-webui
    restart: unless-stopped

    depends_on:
      - ollama

    environment:
      OLLAMA_BASE_URL: http://ollama:11434
      WEBUI_SECRET_KEY: ${WEBUI_SECRET_KEY:?WEBUI_SECRET_KEY must be set}

    volumes:
      - ./ollama_open-webui/_data:/app/backend/data

    expose:
      - "8080"

  open-webui-nginx:
    image: nginx:${NGINX_DOCKER_TAG:?NGINX_DOCKER_TAG must be set}
    container_name: open-webui-nginx
    restart: unless-stopped

    depends_on:
      - open-webui

    ports:
      - "${OPEN_WEBUI_PORT:?OPEN_WEBUI_PORT must be set}:443"

    volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d:ro
      - ./nginx/certs:/etc/nginx/certs:ro

EOF

The override.yml controls gpu stuff - which gpu to expose to ollama - there may or may not be a better reason to or not to store this in a separate file - but this is what worked for me and is part of my current setup. Change the locaitons as appropriate (currently ~/docker)

cat > ~/docker/ollama/compose.override.yml <<'EOF'
services:
  ollama:
    # GPU support
    deploy:
      resources:
        reservations:
          devices:
            - driver: ${OLLAMA_GPU_DRIVER-nvidia}
              count: ${OLLAMA_GPU_COUNT-all}
              capabilities:
                - gpu
EOF

You will also need an .env file to store environment attributes and some secrets in - best to keep these out of the yml file, also is a handy one stop shop to go to update the versions of ollama, OpenWebUI and nginx too. Change the locaitons as appropriate (currently ~/docker)
This script does four things:
1. Generates a random 256-bit secret.
2. Writes it directly into ~/docker/ollama/.env.
3. Sets .env to 600, so only the owning user can read/write it.
4. Removes the temporary shell variable afterward.

WEBUI_SECRET_KEY="$(openssl rand -hex 32)"

cat > ~/docker/ollama/.env <<EOF
WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
OPEN_WEBUI_PORT=3000
OLLAMA_DOCKER_TAG=latest
WEBUI_DOCKER_TAG=main
NGINX_DOCKER_TAG=stable-alpine
EOF

chmod 600 ~/docker/ollama/.env
unset WEBUI_SECRET_KEY

Create the certificates that nginx will use to secure the connection (they will be local certs - you will get a warning when you connect). Before Running the below command change the text 'YourComputerName' so that it matches the name of the computer that will be running ollama, also don't forget to change the storage locaitons as appropriate (currently ~/docker)

cat > ~/docker/ollama/setup-nginx-tls.sh <<'EOF'
#!/usr/bin/env bash

set -e

BASE_DIR="$HOME/docker/ollama"
CERT_DIR="$BASE_DIR/nginx/certs"
CONF_DIR="$BASE_DIR/nginx/conf.d"

echo "Creating Nginx directories..."

mkdir -p "$CERT_DIR"
mkdir -p "$CONF_DIR"

echo "Generating TLS private key and certificate..."

openssl req \
  -x509 \
  -nodes \
  -newkey rsa:4096 \
  -sha256 \
  -days 1825 \
  -keyout "$CERT_DIR/open-webui.key" \
  -out "$CERT_DIR/open-webui.crt" \
  -subj "/CN=YourComputerName" \
  -addext "subjectAltName=DNS:YourComputerName,DNS:localhost,IP:127.0.0.1"

chmod 600 "$CERT_DIR/open-webui.key"
chmod 644 "$CERT_DIR/open-webui.crt"

echo "Creating Nginx Open WebUI configuration..."

cat > "$CONF_DIR/open-webui.conf" <<'NGINX'
server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name _;

    ssl_certificate     /etc/nginx/certs/open-webui.crt;
    ssl_certificate_key /etc/nginx/certs/open-webui.key;

    ssl_protocols TLSv1.2 TLSv1.3;

    location / {
        proxy_pass http://open-webui:8080;

        proxy_http_version 1.1;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_read_timeout 3600;
        proxy_send_timeout 3600;
    }
}
NGINX

echo
echo "TLS setup complete."
echo
echo "Certificate:"
echo "  $CERT_DIR/open-webui.crt"
echo
echo "Private key:"
echo "  $CERT_DIR/open-webui.key"
echo
echo "Nginx configuration:"
echo "  $CONF_DIR/open-webui.conf"
echo
echo "Next:"
echo "  cd $BASE_DIR"
echo "  docker compose restart open-webui-nginx"
EOF

chmod 700 ~/docker/ollama/setup-nginx-tls.sh

Make the script executable (change the locations as appropriate (currently ~/docker)).

chmod +x ~/docker/ollama/setup-nginx-tls.sh

Run the script (change the locations as appropriate (currently ~/docker)).

~/docker/ollama/setup-nginx-tls.sh

and Finally - what you have been waiting for ....(change the location as appropriate (currently ~/docker)).

cd ~/docker/ollama && \
docker compose pull && \
docker compose up -d

in the tone of a cockney - 'go pull a model!'

ollama pull llama3.2

If you will be connecting to openwebui from another computer - add a firewall rule to alow inbound to port 3000:

sudo ufw allow from any to any port 3000 proto tcp comment 'OpenWebUi Inbound'

When 'ollama pull llama3.2' has finished downloading - open a web browser and point it towards 'https://yourIPAddr:3000'
and youre kinda done with the build - what happens next is up to you!

/*****************/END BEGIN/*****************/

Related Posts

Building A Local LLM Playground 01

Building A Local LLM Playground 00

Building Safer Financial Systems Starts with Operational Discipline