Instructions to run ollama in an isolated mode in docker.

isolated_ollama.md

Running Ollama in Docker with GPU Acceleration and No Internet Access

This setup runs Ollama in Docker on a Linux host with:

• NVIDIA GPU acceleration
• Ollama accessible from VS Code/Continue at 127.0.0.1:11434
• No inbound LAN access to Ollama
• No outbound internet/LAN access from the Ollama container
• Persistent model storage
• A temporary internet-enabled container for downloading models

Important: This approach uses a normal Docker bridge plus host firewall rules rather than internal: true. On your system, the internal network prevented the published localhost port from working.

1. Prerequisites

Confirm Docker, Compose and your NVIDIA GPU are working:

docker --version
docker compose version
nvidia-smi

Confirm Docker can access the GPU:

docker run --rm --gpus all \
  nvidia/cuda:12.9.0-base-ubuntu24.04 \
  nvidia-smi

If this doesn't work, configure the NVIDIA Container Toolkit before continuing.

---

2. Stop the Host Ollama Service

If Ollama is already installed directly on Linux:

sudo systemctl stop ollama
sudo systemctl disable ollama

Check that port 11434 is free:

sudo ss -ltnp | grep 11434

There should be no output.

---

3. Create compose.yml

Create a directory:

mkdir -p ~/dev/oolamadocker
cd ~/dev/oolamadocker

Create compose.yml:

services:

  # ----------------------------------------------------------
  # Normal isolated Ollama instance
  # ----------------------------------------------------------

  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped

    environment:
      OLLAMA_HOST: "0.0.0.0:11434"

    ports:
      - "127.0.0.1:11434:11434"

    networks:
      - ollama_isolated

    volumes:
      - ollama_data:/root/.ollama

    cap_drop:
      - ALL

    security_opt:
      - no-new-privileges:true

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu


  # ----------------------------------------------------------
  # Temporary internet-enabled Ollama instance
  # ----------------------------------------------------------

  ollama-online:
    image: ollama/ollama:latest
    container_name: ollama-online

    profiles:
      - online

    environment:
      OLLAMA_HOST: "0.0.0.0:11434"

    networks:
      - internet

    volumes:
      - ollama_data:/root/.ollama

    cap_drop:
      - ALL

    security_opt:
      - no-new-privileges:true

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities:
                - gpu


networks:

  ollama_isolated:
    driver: bridge

  internet:
    driver: bridge


volumes:

  ollama_data:

Notice that ollama-online does not publish port 11434. It is intended only for maintenance/model downloads using docker exec.

---

4. Validate the Compose File

Run:

docker compose config

There should be no errors.

---

5. Start Ollama

Start the normal instance:

docker compose up -d ollama

Check it:

docker compose ps

You should see something similar to:

NAME      IMAGE                  STATUS       PORTS
ollama    ollama/ollama:latest   Up           127.0.0.1:11434->11434/tcp

Also check:

docker port ollama

Expected:

11434/tcp -> 127.0.0.1:11434

---

6. Verify Ollama

Check the API:

curl http://127.0.0.1:11434/api/tags

You should receive JSON containing your installed models.

Check the logs:

docker compose logs ollama

You should see Ollama listening on:

[::]:11434

---

7. Verify GPU Acceleration

Check the container logs:

docker compose logs ollama | grep -i -E 'cuda|gpu|vram'

For example, on an RTX 3060 you might see:

library=CUDA
description="NVIDIA GeForce RTX 3060"
total="11.6 GiB"

You can also run:

docker exec ollama nvidia-smi

---

8. Find the Ollama Docker Subnet

Now determine the network Docker assigned:

docker network inspect oolamadocker_ollama_isolated

Look for:

"IPAM": {
    "Config": [
        {
            "Subnet": "172.20.0.0/16"
        }
    ]
}

Your subnet may be different.

For example:

172.20.0.0/16

Do not assume this value. Use the subnet reported on your machine.

You can get it directly with:

docker network inspect oolamadocker_ollama_isolated \
  --format '{{(index .IPAM.Config 0).Subnet}}'

---

9. Block Ollama's Outbound Network Access

Docker provides the DOCKER-USER chain for host-defined filtering of forwarded container traffic.

First store the actual subnet:

OLLAMA_SUBNET=$(docker network inspect oolamadocker_ollama_isolated \
  --format '{{(index .IPAM.Config 0).Subnet}}')

echo "$OLLAMA_SUBNET"

For example:

172.20.0.0/16

We want to permit replies to connections initiated from the host while preventing the container from initiating connections elsewhere.

Add:

sudo iptables -I DOCKER-USER 1 \
  -s "$OLLAMA_SUBNET" \
  -m conntrack --ctstate ESTABLISHED,RELATED \
  -j ACCEPT

Then block other forwarded traffic originating from that subnet:

sudo iptables -I DOCKER-USER 2 \
  -s "$OLLAMA_SUBNET" \
  -j DROP

Inspect the result:

sudo iptables -L DOCKER-USER -n -v --line-numbers

You should see rules resembling:

ACCEPT  all  --  172.20.0.0/16  0.0.0.0/0  ctstate RELATED,ESTABLISHED
DROP    all  --  172.20.0.0/16  0.0.0.0/0

Docker documents DOCKER-USER as the appropriate place for user-defined filtering ahead of Docker's forwarding rules. Docker firewall documentation

---

10. Test the Isolation

Ollama API should work

From the Linux host:

curl http://127.0.0.1:11434/api/tags

This should succeed.

Internet access should fail

The Ollama image may not contain curl, so the easiest evidence may be Ollama itself.

Check:

docker compose logs ollama

Attempts by Ollama to reach cloud services should fail.

For a stronger test, inspect network traffic or temporarily use a diagnostic container attached to the same network.

For example:

docker run --rm \
  --network oolamadocker_ollama_isolated \
  curlimages/curl \
  --connect-timeout 5 \
  https://example.com

This should fail.

---

11. Verify LAN Access Is Also Blocked

The DROP rule applies to forwarded traffic from the Ollama network, not merely TCP 80/443.

Therefore the container cannot simply bypass the restriction by:

• using a raw IP address
• changing DNS servers
• using a different TCP port
• connecting to another LAN machine

The intended architecture is:

                       INTERNET
                           ^
                           |
                           X
                    DOCKER-USER DROP
                           |
                    +------+------+
                    |             |
                    |   Ollama    |
                    |   Docker    |
                    |             |
                    +------+------+
                           |
                      :11434
                           |
                    Docker publish
                           |
                    127.0.0.1:11434
                           |
                    +------+------+
                    | Linux host  |
                    |             |
                    |   VS Code   |
                    |  Continue   |
                    +-------------+

---

12. Configure Continue

Continue should connect to the Ollama API running on the host.

For your model:

name: Local Assistant
version: 1.0.0
schema: v1

models:

  - name: Qwen3.8 27B Local

    provider: ollama

    model: smtek/Qwen3.8-27B:Q2_K_XL-12gb

    roles:
      - chat
      - edit
      - apply
      - summarize

    capabilities:
      - tool_use

    defaultCompletionOptions:
      contextLength: 8192

Continue talks to:

http://127.0.0.1:11434

The Ollama API isn't exposed to your LAN because Docker publishes it specifically on 127.0.0.1.

---

13. Download New Models

The normal Ollama container should remain isolated.

When you need a new model, stop it:

docker compose stop ollama

Start the temporary internet-enabled container:

docker compose --profile online up -d ollama-online

Check:

docker compose ps

Then pull a model:

docker exec ollama-online ollama pull MODEL_NAME

For example:

docker exec ollama-online \
  ollama pull smtek/Qwen3.8-27B:Q2_K_XL-12gb

Both containers use:

volumes:
  - ollama_data:/root/.ollama

so the model remains available after the temporary container is removed.

---

14. Return to Isolated Mode

Destroy the internet-enabled instance:

docker compose --profile online down

Then start normal Ollama:

docker compose up -d ollama

Verify:

curl http://127.0.0.1:11434/api/tags

Your newly downloaded model should appear.

---

15. Important Security Caveat: Continue / VS Code

This configuration isolates Ollama, but it does not isolate VS Code:

                         INTERNET
                             ^
                             |
                         allowed
                             |
                      +------+------+
                      |   VS Code   |
                      |  Continue   |
                      +------+------+
                             |
                      127.0.0.1
                             |
                             v
                      +------+------+
                      |   Ollama    |
                      |             |
                      |   Docker    |
                      +------+------+
                             |
                             X
                          Internet

Therefore, if the threat you're defending against is source-code exfiltration, you should also consider what VS Code extensions are installed and whether Continue or another extension is configured to use any cloud services.

Blocking Ollama's internet access does not prevent an internet-enabled VS Code extension from transmitting source code.

---

16. Firewall Persistence

One important issue remains: manually entered iptables rules generally do not survive reboot automatically.

Once you've tested the configuration thoroughly, make the firewall rules persistent using the mechanism appropriate for your Linux distribution, such as iptables-persistent, a systemd unit, or an equivalent nftables configuration.

Do this after testing, rather than making an incorrect firewall rule permanent.

---

17. Useful Verification Commands

Check Ollama:

docker compose ps

Check published ports:

docker port ollama

Check the API:

curl http://127.0.0.1:11434/api/tags

Check GPU:

docker exec ollama nvidia-smi

Check Ollama logs:

docker compose logs --tail=100 ollama

Check the isolated network:

docker network inspect oolamadocker_ollama_isolated

Check firewall rules:

sudo iptables -L DOCKER-USER -n -v --line-numbers

Test internet isolation:

docker run --rm \
  --network oolamadocker_ollama_isolated \
  curlimages/curl \
  --connect-timeout 5 \
  https://example.com

The desired final result is:

VS Code / Continue
       |
       | allowed
       v
127.0.0.1:11434
       |
       v
    Ollama
       |
       +---- NVIDIA GPU      ✓
       |
       +---- model volume    ✓
       |
       +---- Internet        ✗
       |
       +---- LAN             ✗

This gives you a useful defense-in-depth setup: Ollama has GPU access and can communicate with Continue through one localhost port, while the host firewall—not Ollama itself—is responsible for preventing outbound network access.