Docker Model Runner cannot be connected to after installing a runner

[factor3]

I have been using Docker Model Runner for gguf- based models for some time now. I have been using the following URL to
access it:

http://192.168.2.42:12434/engines/llama.cpp/v1/

This URL has been working well using WebUI as well as my own chat application.

I have recently installed the diffusers runner and stable-diffusion onto my machine. Unfortunately, since installing them no application can connect to my Docker Model Runner instance.

The only way I can access DMR is by using a curl command provided to me by @ilopezluna::

 curl -s -X POST http://localhost:12434/engines/diffusers/v1/images/generations   -H "Content-Type: application/json"   -d '{
    "model": "ai/stable-diffusion",
    "prompt": "A cat sitting on a couch",
    "size": "512x512"
  }'
  

I believe that this returns an image (it does return a lot of data), though I cannot confirm that the data I am seeing is actually an image using curl.

This doesn't matter much, because if I move to another machine and use the curl to access my machine:

 curl -s -X POST http://192.168.2.42:12434/engines/diffusers/v1/images/generations   -H "Content-Type: application/json"   -d '{
    "model": "ai/stable-diffusion",
    "prompt": "A cat sitting on a couch",
    "size": "512x512"
  }'
  

I still cannot connect to the machine.

With WebUI, I have tried placing it on my machine and using:

http://localhost:12434/engines/llama.cpp/v1/

and

http://localhost:12434/engines/diffusers/v1/images/generations

and I am still getting connection errors.

I suspect that this is a bug in DMR's runner. At this point, I cannot use it.

[doringeman]

Hey @factor3,

I believe that this returns an image

Yes, it does, see the example of how to actually see the image: https://docs.docker.com/ai/model-runner/inference-engines/#generating-images-with-diffusers.

I still cannot connect to the machine.

Because by default DMR starts on localhost. Run docker model uninstall-runner, then install-runner --backend diffusers again with --host/--port.

docker model install-runner --help | grep -e --host -e --port
      --host string         Host address to bind Docker Model Runner (default "127.0.0.1")
      --port uint16         Docker container port for Docker Model Runner (default: 12434 for Docker Engine, 12435 for Cloud mode)

and I am still getting connection errors.

You need http://localhost:12434/engines/v1, see https://docs.docker.com/ai/model-runner/openwebui-integration/#docker-engine-linux.

Regarding using Open WebUI for generating images, me and @ilopezluna will come back with some steps on this, I don't have them handy right now. 🙌

[ilopezluna]

Hello @factor3

I believe that this returns an image (it does return a lot of data), though I cannot confirm that the data I am seeing is actually an image using curl.

The model returns an image in base64, you can store it as PNG and verify it's a cata :D

curl -s -X POST http://localhost:12434/engines/diffusers/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/stable-diffusion",
    "prompt": "A cat sitting on a couch",
    "size": "512x512"
  }' | jq -r '.data[0].b64_json' | base64 -d > cat.png

I still have to check image generation support using OpenWebUI 👍

[factor3]

Okay, @doringeman:

First of all: thank you for the information -- especially for the correct WebUI container and the docker-compose.yml file. It turns out that I was using a different container and I wasn't using docker compose to start it. I had looked at the wrong instructions for using WebUI with DMR.

In any event, I have uninstalled the runner and reinstalled it:

$ docker model uninstall-runner
$ docker model install-runner --backend diffusers --host 192.168.2.42

The status, after that, actually looks good (!):

$ docker model status
Docker Model Runner is running

BACKEND    STATUS         DETAILS
diffusers  Running        diffusers v0.1.0-20260216-000000
llama.cpp  Running        llama.cpp 3191462
sglang     Error          Python not found
mlx        Not Installed  only supported on Apple Silicon
vllm       Not Installed  binary not found

I have made the appropriate changes in the docker-compose file:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "9020:8080"
    environment:
      - OPENAI_API_BASE_URL=http://192.168.2.42:12434/engines/v1
      - OPENAI_API_KEY=not-needed
      - WEBUI_AUTH=false
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui:/app/backend/data

volumes:
  open-webui:

And, of course, I started up the WebUI system in the usual manner:

$ docker compose up -d && docker compose logs -f

The WebUI system does not connect to DMR.

The only way it actually connects to DMR is if I change the URL back to localhost:

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "9020:8080"
    environment:
      - OPENAI_API_BASE_URL=http://localhost:12434/engines/v1
      - OPENAI_API_KEY=not-needed
      - WEBUI_AUTH=false
    extra_hosts:
      - "host.docker.internal:host-gateway"
    volumes:
      - open-webui:/app/backend/data

volumes:
  open-webui:

This despite the fact that I did change the host when I installed the runner.

A couple of things to note:

1. WebUI does send prompts to my GGUF LLMs and uses them without problems. Unfortunately, It generates weird errors when I attempt to send a prompt to the stable-diffusion LLM.

2. I cannot access DMR from anywhere except localhost -- which makes it impossible to (for example) test my own chatting code from another machine.

3. Even if I run my chatting code, or any other AI based software (like N8N or Openhands) on the same machine as DMR using the same localhost URL as WebUI, I am getting connection errors.

I now have limited use of DMR (through WebUI), but it is still not usable for any of my larger purposes.