sbr: Add options to configure static gateway IP and preserve default routes #1217

dagrayvid · 2025-11-27T22:23:10Z

This PR aims to solve two problems I faced using the source-based router CNI plugin for distributed LLM inference with vLLM / llm-d.

Option to configure a statically defined gateway IP to set or override the gateway if not passed in from the CNI config chain.

In my environment (IBM Cloud), the DHCP server my secondary high-speed networks does not provide the gateway IP, so I needed a way to override / statically define the gateway IPs to ensure that the custom routing tables included the default route to the gateway.

Option to "preserve default routes" in the main routing table, so that packets without a source IP are still routed correctly. For example in my environment after enabling SBR, ping -I 10.0.0.6 10.1.0.7 would work, but ping 10.1.0.7. With "preserveDefaultRoutes": true, both work.

Tested in my environment with this CNI config:

{
  "cniVersion": "0.3.1",
  "name": "dhcp-host-device-port-1",
  "plugins": [
    {
      "type": "host-device",
      "device": "enp163s0",
      "isRdma": true,
      "ipam": {
        "type": "dhcp"
      }
    },
    {
      "type": "sbr-custom",
      "gateway": "10.0.0.1",
      "preserveDefaultRoutes": true
    }
  ]
}

In a Pod, the result of the statically defined gateway:

bash-5.1# ip route show table 100
default via 10.0.0.1 dev net1
10.0.0.0/16 dev net1 proto kernel scope link src 10.0.0.5

and the result of preserve default route:

bash-5.1# ip route show table 254
default via 10.130.0.1 dev eth0
10.0.0.0/16 dev net1 scope link src 10.0.0.5
10.1.0.0/16 dev net2 scope link src 10.1.0.6
10.2.0.0/16 dev net3 scope link src 10.2.0.6
...

squeed · 2025-12-01T11:31:49Z

Looks nice!
Would there ever be a case where we wanted just source-hinted routes without the table?

dagrayvid · 2025-12-01T17:10:16Z

Thanks for the review @squeed!

Would there ever be a case where we wanted just source-hinted routes without the table?

I'm not sure I fully understand the question. If a user wanted only source-hinted routes without custom tables, I don't think they'd need SBR at all - the kernel already creates these automatically when IPs are assigned.

In my case at least, without SBR on my interface at 10.5.0.5, the source-hinted route is already in the main table:

$ ip route show
...
10.5.0.0/16 dev net6 proto kernel scope link src 10.5.0.5

The SBR plugin (without this added preserveDefaultRoutes option) deletes these after copying them to the custom tables:

plugins/plugins/meta/sbr/main.go

Lines 396 to 402 in 0d9f9b1

    
           for _, route := range routes { 
        
           	log.Printf("Deleting route %s from table %d", route.String(), route.Table) 
        
           	err := netlink.RouteDel(&route) 
        
           	if err != nil { 
        
           		return fmt.Errorf("Failed to delete route: %v", err) 
        
           	} 
        
           }

Does that make sense? Please let me know if I've misunderstood your question.

squeed · 2025-12-08T15:02:39Z

🤦 of course. I don't know what I was thinking.

plugins/meta/sbr/main.go

squeed · 2025-12-08T15:26:29Z

plugins/meta/sbr/main.go

 	Table *int `json:"table,omitempty"`
+	// Gateway allows specifying a static/hardcoded gateway IP address
+	// If set, this will be used instead of the gateway from prevResult
+	Gateway string `json:"gateway,omitempty"`


Note: This should permit dual-stack :-). Ideally this is a list.

Also could be useful for anyone simply having multiple gateways

@squeed I have updated the plugin to support dual-stack, PTAL when you get a chance.

Also could be useful for anyone simply having multiple gateways

Currently the static gateway option introduced in this PR will apply the same gateway to all IPs passed in from prevResult.

I see two potential enhancements here:

Support multiple IPs by configuring a mapping of subnets to gateways, e.g.

{ "type": "sbr", "gateways": { "192.168.1.209/24": "192.168.1.1", "192.168.101.209/24"": "192.168.101.1" }, "preserveDefaultRoutes": true }

Support multiple gateways per subnet (multiple default routes per address, ECMP load-balancing). This would be a net-new feature that goes beyond what the sbr plugin currently does with prevResult

Are you ok with the current limitation on static gateways, or would you prefer if I implement matching gateways to subnets for the mulitiple IP case in this PR?

LionelJouin · 2025-12-08T15:30:04Z

plugins/meta/sbr/main.go

 	Table *int `json:"table,omitempty"`
+	// Gateway allows specifying a static/hardcoded gateway IP address
+	// If set, this will be used instead of the gateway from prevResult
+	Gateway string `json:"gateway,omitempty"`


Also could be useful for anyone simply having multiple gateways

plugins/meta/sbr/main.go

karampok

Some comments from my side

5 commits seem too many imo, maybe 2 (one for static gateway on for preserveDefaultRoutes) or even 1 (like atomic commits)
test look a bit thin, like only ipv4 not unhappy path

(Note: I am not maintainer of the repo)

karampok · 2025-12-11T08:26:21Z

plugins/meta/sbr/main.go

+	// PreserveDefaultRoutes keeps subnet routes in the main routing table
+	// with proper source IP hints for destination-based routing.
+	// This allows packets without an explicit source IP to be routed correctly.
+	PreserveDefaultRoutes bool `json:"preserveDefaultRoutes,omitempty"`


PreserveDefaultRoutes: is not so clear to me.

The "defaultRoutes" suffix makes think that in the main table I see a default route, which is not true, I only should see the subnet via eth0 src x.y.z.k, is that correct?

thx

I think this is a good point. I renamed this option to AddSourceHints. Do you think that is more clear?

plugins/meta/sbr/main.go

karampok · 2025-12-11T08:28:25Z

plugins/meta/sbr/main.go

+			} else if ipCfg.Gateway != nil {
+				gateway = ipCfg.Gateway
+			}
+		} else {


It is more golang if we have code without else and else if

Thanks, I made this a switch statement.

karampok · 2025-12-11T08:44:00Z

plugins/meta/sbr/sbr_linux_test.go

+	"cniVersion": "0.3.0",
+	"name": "cni-plugin-sbr-test",
+	"type": "sbr",
+	"gateways": ["192.168.1.254"],


Ideally a test for ipv4, ipv6 and dualstack

squeed

Can you please squash your commits? And a v6 test would be good.

Add two new configuration options: 1. "gateways" ([]string): Static gateway IPs that override prevResult. Supports dual-stack (one IPv4 and/or one IPv6 address). 2. "addSourceHints" (bool): Preserves subnet routes in the main table with source IP hints, enabling destination-based routing to work alongside source-based routing. Example: { "type": "sbr", "gateways": ["10.0.0.1"], "addSourceHints": true } Signed-off-by: David Whyte-Gray <[email protected]>

dagrayvid · 2026-01-07T22:09:33Z

Thanks @squeed @karampok for the review. I renamed PreserveDefaultRoutes to AddSourceHints and added tests for ipv6 and dual stack. Also squashed the commits. PTAL

dagrayvid force-pushed the main branch 3 times, most recently from 78e2abb to 0d9f9b1 Compare November 28, 2025 16:49

karampok reviewed Dec 8, 2025

View reviewed changes

plugins/meta/sbr/main.go Outdated Show resolved Hide resolved

squeed reviewed Dec 8, 2025

View reviewed changes

dagrayvid force-pushed the main branch from 0d9f9b1 to 1581f99 Compare December 10, 2025 02:30

LionelJouin reviewed Dec 10, 2025

View reviewed changes

karampok reviewed Jan 5, 2026

View reviewed changes

squeed approved these changes Jan 5, 2026

View reviewed changes

squeed requested changes Jan 5, 2026

View reviewed changes

dagrayvid force-pushed the main branch from d855348 to 3e174b8 Compare January 7, 2026 21:34

dagrayvid force-pushed the main branch from 3e174b8 to 406a898 Compare January 7, 2026 21:57

sbr: Add options to configure static gateway IP and preserve default routes #1217

Are you sure you want to change the base?

sbr: Add options to configure static gateway IP and preserve default routes #1217

Conversation

dagrayvid commented Nov 27, 2025

Uh oh!

squeed commented Dec 1, 2025

Uh oh!

dagrayvid commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

squeed commented Dec 8, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dagrayvid Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

karampok left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

squeed left a comment

Choose a reason for hiding this comment

Uh oh!

dagrayvid commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dagrayvid commented Dec 1, 2025 •

edited

Loading

dagrayvid Dec 10, 2025 •

edited

Loading

dagrayvid commented Jan 7, 2026 •

edited

Loading