A gateway in front of my local LLM: breaking Kong on purpose

In my posts about language models so far, the model has been at one end and my own code at the other, talking to each other directly. That is not how most companies run language models. Between the application and the model there is usually a gateway: one front door that checks who is calling, limits how much they can use, counts the tokens, and decides which model gets the request.

I wanted to see how one of these gateways behaves when things go wrong, so I ran one myself. I put Kong Gateway and its AI Proxy plugin in front of the same Qwen model I used in the last few posts. Then I did what I spent most of my career doing on the support side: I broke requests on purpose, and looked for where the evidence of each failure ended up.

For years, much of my work went into Spark jobs that failed somewhere between the client, a gateway, Livy and YARN. My approach was to componentize the path: understand what each layer is responsible for, then probe it at different points to see which pieces were working and which were not. Once the problem was isolated to one layer, I could work on that layer alone, with everything around it out of the way. I took the same approach here: break it into layers, probe each one, and for every failure, ask which layer actually said no.

This is not a review of Kong, and it is not a benchmark. It is one short experiment, one model and two machines on my LAN. A few things surprised me, and for each one I have included the evidence that led me there: the headers, the log lines and the measurements.

The setup

For this lab, I ran Kong in a Docker container on hpsvr, another machine on my LAN (a Xeon E-2286M). Kong can also be installed from Linux packages or run on Kubernetes, but a container was the simplest way to try it. I used DB-less mode, so the whole configuration is one YAML file and there is no database. The model is Qwen 3.8 Flash Next, served by SGLang on bigai, my workstation with two RTX PRO 6000 Blackwell cards. It is the same server as in the context-engineering post, and I did not change it for this one.

SGLang already speaks OpenAI’s chat API, so to Kong my local model can look like any other OpenAI-style provider. Here is the route, close to the whole configuration:

_format_version: "3.0"
services:
  - name: qwen-local
    url: http://192.168.1.136:8000   # required, but ai-proxy
    routes:                          # uses upstream_url below
      - name: qwen-chat
        paths: ["/qwen"]
        protocols: ["http"]          # more on this line soon
        plugins:
          - name: ai-proxy
            config:
              route_type: llm/v1/chat
              model:
                provider: openai     # speak OpenAI's API
                name: qwen38-27b     # an alias, not a 27B model
                options:
                  upstream_url: http://192.168.1.136:8000/v1/chat/completions
              logging:
                log_statistics: true # token counts in the logs
plugins:
  - name: file-log                   # one JSON line per request
    config:
      path: /logs/kong-requests.log

A note on qwen38-27b: it is only the name the server answers to, left over from a model I ran on it before. The model behind it is Qwen 3.8 Flash Next.

The client sends an ordinary OpenAI-style request to http://hpsvr:8000/qwen. It does not need to know where the model is, what it is called, or which key it needs. Those are now the gateway’s problem. A response that came back through ai-proxy carries a few extra headers, and they turned out to be the most useful thing in this whole exercise:

HTTP/1.1 200 OK
server: uvicorn                     # SGLang wrote this response
X-Kong-LLM-Model: openai/qwen38-27b # provider/model ai-proxy used
X-Kong-Upstream-Latency: 123        # ms waiting for the model
X-Kong-Proxy-Latency: 1             # ms spent inside Kong
Via: 1.1 kong/3.16.0.0-enterprise-edition
X-Kong-Request-Id: 3fcd2307657b5777ddeb9764504d3f62

My first request failed

Before any of that worked, my very first request came back like this, without ever reaching the model:

HTTP/1.1 426
Server: kong/3.16.0.0-enterprise-edition
X-Kong-Response-Latency: 0

{"message":"Please use HTTPS protocol", ...}

My first draft of the route had no protocols line. Kong’s Admin API shows the configuration as Kong actually loaded it, and there the route’s protocols were ["https"]. Kong’s breaking-changes page explains why: since 3.14.0.0, “the default setting for Route protocols has changed from http,https to https.” Plain HTTP still matches the route, and Kong answers with a 426 asking the caller to upgrade. That is a sensible default for anything facing the internet. In a lab on my own LAN I set protocols: ["http"] explicitly.

It is a small thing, but it set the pattern for the rest of the experiment. The headers already told me who answered: Server: kong and X-Kong-Response-Latency mean Kong wrote the response itself. When the model answers, the server header is SGLang’s (uvicorn) and Kong adds X-Kong-Upstream-Latency instead.

What the extra hop costs

To measure the gateway, the test script ran on hpsvr and sent the same short request 40 times directly to SGLang and 40 times through Kong, alternating, with thinking off and temperature 0 so both paths produced the same answer:

For a whole answer, the gateway costs nothing I could measure; the differences are smaller than the noise between two requests. Kong reports about a millisecond of its own time per request, and I believe it.

The third row is different. With streaming, the first token arrived about 38 milliseconds later through Kong, every single time, while the whole answer did not get any slower. That looked like something worth understanding. (It turned out to be the same thing as the clumping below.)

Streamed tokens arrive in clumps

So I recorded the arrival time of every chunk of a streamed answer, and compared five paths: direct, Kong as a plain reverse proxy with no AI plugin, and Kong with ai-proxy, each with and without response buffering on the route.

Directly from SGLang, a chunk arrives about every 13 milliseconds. Through Kong as a plain proxy, exactly the same. Through ai-proxy with the route’s default settings, the same chunks arrive in clumps of about six, every 80 milliseconds. Nothing is lost and the total time is the same. But the first clump waits for the first six chunks, and to a person watching text appear, the stream stutters instead of flowing.

The fix is to add one setting, response_buffering: false, to the route. Here is the route from the configuration above with that line added. Nothing else changes:

      - name: qwen-chat
        paths: ["/qwen"]
        protocols: ["http"]
        response_buffering: false    # the new line
        plugins:                     # the same ai-proxy
          ...                        # settings as before

With it, ai-proxy delivered chunks one at a time again, with the same timing as direct. Kong documents this setting in a support article on streaming text/event-stream responses. What I did not expect is that only the AI plugin path showed the clumping. Kong as a plain proxy, with buffering on, streamed perfectly well.

What the measurements show so far: each clump held about six chunks, and the clumps arrived about 80 milliseconds apart, which is about how long the model takes to produce six chunks (6 × 13 ms). So something in the ai-proxy path collects the stream and passes it on in batches. What I don’t know yet is whether that batch is set by time, by a number of chunks or by a buffer size.

That is the next thing I want to pin down, and there is a clear way in. Kong 3.9.3, which showed the same clumping, is open source, and its ai-proxy plugin and the streaming code behind it are on GitHub. I plan to read how that code handles a streamed response, then test what I find. For example, if I rerun the stream with much larger chunks and the clumps still arrive every 80 ms, the batching is based on time; if they get smaller, it is based on buffer size. I will post an update here with the answer. Update, October 11: I found the answer. It is at the end of this section.

I also wanted to know whether this was new. Kong 3.9.3, the last open-source image, ran the same ai-proxy route and showed the same thing: about 127 ms to the first token, and the same clumps.

Update, October 11, 2026: it’s a buffer size

The batch is set by bytes: about 1,460 of them. That number is the default of an Nginx setting called postpone_output. The clumps had nothing to do with time or with a count of chunks.

ai-proxy does not hold the stream back itself. Its code takes each chunk Nginx hands it, rewrites it into the provider-neutral format, and hands it straight back. The holding happens one step later, in Nginx’s write filter. That filter does not send fewer than postpone_output bytes unless a buffer is marked as a point to flush, and two markings count: flush and recycled.

• With response buffering on, Nginx marks every piece of the upstream response recycled. That is why Kong as a plain proxy streamed perfectly well.
• With buffering off, Nginx marks every piece flush. That is why response_buffering: false fixed ai-proxy.
• When a Lua plugin replaces a chunk, the OpenResty code that builds the new buffer copies its flush marking, but not recycled. With buffering on, the new buffer has neither, so Nginx waits for 1,460 bytes. At about 270 bytes per chunk, that is six chunks, or about 80 ms of this model’s output.

I read this in the Nginx and OpenResty source that Kong 3.9.3 is built on, and then tested it. Kong passes Nginx directives through from its own configuration (here, the environment variable KONG_NGINX_PROXY_POSTPONE_OUTPUT), so I ran a few throwaway copies of the gateway next to the lab one, changing only postpone_output or the chunk size. Asking for logprobs makes each chunk about 4,400 bytes, which was the larger-chunk test I had planned.

Kong 3.9.3 gave the same numbers as 3.16 and responded to postpone_output in the same way. One more detail fits: the first clump held five chunks, not six, because the response headers pass through the same filter and count toward the 1,460 bytes. The 38 ms extra wait for the first token in the table further up is the same thing: the first clump waits until 1,460 bytes have arrived.

For a customer I would still recommend response_buffering: false on routes that stream. It is documented, and it only affects that route. Setting postpone_output to 0 also works, but it changes every route on the node, so I would use it only to confirm the diagnosis. One practical point: small chunks clump the most, so two clients on the same route can see different behavior if one of them asks for logprobs.

For 3.16, whose ai-proxy ships as compiled bytecode, this is based on behavior rather than reading the code: the same numbers, and the same response to the setting.

Breaking it on purpose

Next, the part I was most curious about. I set up extra routes that fail the way I have seen requests fail in other systems: a missing key, a rate limit, a closed port, a slow answer, a prompt that is too large. For each one I kept what a support engineer would ask for: the status, the body, the headers, Kong’s file-log line, Kong’s error log, and SGLang’s own request counter, read before and after. The counter answers a question the gateway cannot: did the request ever reach the model at all?

server: kong/3.16.0.0-enterprise-edition
x-kong-response-latency: 0
x-kong-request-id: a63a897ea7d95f7f2798bc042459a15e

{
  "message":"Please use HTTPS protocol",
  "request_id":"a63a897ea7d95f7f2798bc042459a15e"
}

server: kong/3.16.0.0-enterprise-edition
x-kong-response-latency: 0
x-kong-request-id: 0aa74b48e0c06c6dd53f7a7ae4e6835c

{
  "message":"no Route matched with those values",
  "request_id":"0aa74b48e0c06c6dd53f7a7ae4e6835c"
}

server: kong/3.16.0.0-enterprise-edition
x-kong-response-latency: 1
x-kong-request-id: 4be4f02681b9aa0750d36722cbd5f333
www-authenticate: Key

{
  "message":"No API key found in request",
  "request_id":"4be4f02681b9aa0750d36722cbd5f333"
}

server: kong/3.16.0.0-enterprise-edition
x-kong-response-latency: 1
x-kong-request-id: 484d5d2f2be76b05a9cadc4a94a74745
www-authenticate: Key

{
  "message":"Unauthorized",
  "request_id":"484d5d2f2be76b05a9cadc4a94a74745"
}

server: kong/3.16.0.0-enterprise-edition
x-kong-response-latency: 1
x-kong-request-id: c7784357dffc7561ea69406484bd3a3e
retry-after: 55
x-ratelimit-limit-minute: 3
x-ratelimit-remaining-minute: 0
ratelimit-reset: 55

{
  "message":"API rate limit exceeded",
  "request_id":"c7784357dffc7561ea69406484bd3a3e"
}

server: kong/3.16.0.0-enterprise-edition
x-kong-response-latency: 0
x-kong-request-id: 51b50d1fd5d7122f8c18d96f9c99fe45

{"error":{"message":"cannot use own model - must be: qwen38-27b"}}

server: kong/3.16.0.0-enterprise-edition
x-kong-response-latency: 1
x-kong-request-id: 5691bd36ddeb4012676ff2799fde1a66

{"error":{"message":"request body doesn't contain valid inputs"}}

# docker logs kong-ai-lab
2026/10/10 01:27:07 [warn] 2980#0: *13110 a client request body is buffered to a temporary file /usr/local/kong/client_body_temp/0000000002, client: 172.19.0.1, server: kong, request: "POST /qwen HTTP/1.1", host: "127.0.0.1:8000", request_id: "5691bd36ddeb4012676ff2799fde1a66"
2026/10/10 01:27:07 [warn] 2980#0: *13110 [kong] parse-request.lua:200 [ai-proxy] unable to parse request body, consider increase plugin's config.max_request_body_size or Kong's nginx_http_client_body_buffer_size; the request content-length is 1300157 bytes, client: 172.19.0.1, server: kong, request: "POST /qwen HTTP/1.1", host: "127.0.0.1:8000", request_id: "5691bd36ddeb4012676ff2799fde1a66"

server: kong/3.16.0.0-enterprise-edition
via: 1.1 kong/3.16.0.0-enterprise-edition
x-kong-upstream-latency: 0
x-kong-proxy-latency: 4
x-kong-request-id: bd44f76672b8973dea6dd24a085b9e31

{
  "message":"An invalid response was received from the upstream server",
  "request_id":"bd44f76672b8973dea6dd24a085b9e31"
}

# docker logs kong-ai-lab
2026/10/10 01:26:57 [error] 2980#0: *13094 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: kong, request: "POST /qwen-down HTTP/1.1", subrequest: "/kong_buffered_http", upstream: "http://192.168.1.136:8009/v1/chat/completions", host: "127.0.0.1:8000", request_id: "bd44f76672b8973dea6dd24a085b9e31"
  (the same line 6 times)

server: kong/3.16.0.0-enterprise-edition
via: 1.1 kong/3.16.0.0-enterprise-edition
x-kong-upstream-latency: 15001
x-kong-proxy-latency: 1
x-kong-request-id: 08d3ac17113acf443386622b7c5d512c

{
  "message":"The upstream server is timing out",
  "request_id":"08d3ac17113acf443386622b7c5d512c"
}

# docker logs kong-ai-lab
2026/10/10 01:27:57 [error] 2980#0: *13871 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.19.0.1

Most of these behaved the way the documentation says they would, and the messages were clear. Three taught me something.

The prompt that was “not valid”

A 1.3 MB prompt (about 260,000 words of filler) came back as a 400 with this message:

{"error":{"message":"request body doesn't contain valid inputs"}}

The body was perfectly valid. The same request sent straight to SGLang was accepted: 260,032 prompt tokens, just under the 262,144-token context window I run the model with, answered in 33 seconds. The real reason was only in Kong’s error log, as a warning:

[warn] ... [ai-proxy] unable to parse request body, consider
increase plugin's config.max_request_body_size or Kong's
nginx_http_client_body_buffer_size; the request content-length
is 1300157 bytes

ai-proxy’s max_request_body_size defaults to 1 MiB. With long-context models that is not an exotic size anymore: in my context-engineering experiment, the agent reading the raw sources built prompts of up to 214,000 tokens. If a customer reports that big requests are “invalid” and small ones work, this is the first thing I would look at, and the error log is the only place that says so.

One request, six connection attempts

With ai-proxy pointed at a port where nothing listens, the client got a 502 with “An invalid response was received from the upstream server”, in a few milliseconds. The error log had the same line six times for that one request:

[error] ... connect() failed (111: Connection refused) while
connecting to upstream, ... subrequest: "/kong_buffered_http",
upstream: "http://192.168.1.136:8009/v1/chat/completions", ...

That is one try plus Kong’s default of five retries, and the file-log entry lists all six in its tries array. With a refused connection all six fail instantly, so nobody notices. I did not test a host that does not answer at all, where each attempt would have to wait for connect_timeout (60 seconds by default) before giving up. That is a test I want to do next.

A timeout that streaming does not hit

The qwen-15s route has a 15-second read_timeout. I asked for a long answer that takes the model about 21 seconds. Not streamed, Kong gave up at exactly 15.0 seconds with a 504, “The upstream server is timing out”. The error log says upstream timed out (110: Connection timed out) while reading response header from upstream. Kong made one try, not six, so it did not retry the timed-out request.

The same request, streamed, through the same route, finished normally after 21.0 seconds with all 6,000 tokens. Kong defines read_timeout as “the timeout in milliseconds between two successive read operations”, not as a limit on the whole request. A streamed answer keeps sending bytes, so the clock keeps resetting. A non-streamed one sends nothing at all until it is finished. The same route, timeout and model can succeed or fail depending only on whether the client asked for a stream. With reasoning models that think for a minute or more before answering, the default of 60 seconds is closer than it sounds.

When the caller walks away

The timeout raised a question that matters more for GPUs than for ordinary web servers. When nobody is waiting for the answer anymore, does the model stop working on it? I sampled SGLang’s count of running requests every half second while a caller gave up after five seconds on a request that takes about 21.

Two different things are going on here, and it took me a few runs to separate them.

Not streamed, a client that hangs up on SGLang directly, or on Kong as a plain proxy, stops the work about 2.5 seconds later. Through ai-proxy, the model kept going for the full 21 seconds and wrote an answer nobody received. That happened with and without response buffering, in both runs. My reading of the evidence is this: the error-log line above shows that ai-proxy sends a non-streamed request through an internal Nginx subrequest (/kong_buffered_http), and Kong’s generated Nginx configuration comments that this “will create a new nginx request”. I suspect that inner request does not learn that the outer client has left. That is a guess from the outside, not something I have confirmed in Kong’s source.

Streamed, the model kept going after the caller left on every path, including directly to SGLang. So that part is my model server, not the gateway. I checked it with plain curl --max-time 5 to make sure it was not my test client.

When Kong itself gives up, on the other hand, it closes the connection to the model, and SGLang stopped within a second. A gateway timeout protects the GPU in a way that a client timeout, in this setup, does not.

Counting tokens

One of the reasons people put a gateway in front of models is to count usage per team, per key or per application, because tokens are the bill. With log_statistics on, ai-proxy writes the counts into every log entry. I compared them with what the model itself reported.

When the model reports usage, Kong’s numbers matched exactly: in six requests with thinking on, streamed and not, the prompt and completion tokens in Kong’s log were identical to the usage block the client received. One detail: SGLang reports how many of the completion tokens were reasoning in a top-level reasoning_tokens field, and Kong’s completion_tokens_details came back empty, so the log does not show how much of the bill was thinking.

When a streaming client does not ask for usage, ai-proxy has nothing to copy, so it estimates. In OpenAI’s API, streamed responses only include usage if the client sends stream_options: {"include_usage": true}, and many clients do not. I sent eight prompts twice each, once asking for usage and once not, and compared:

With thinking off, the estimate for the answer was in the right neighborhood, but the prompt came out at about half. My guess is that the estimate counts the words of the messages and not the tokens of the chat template that the model actually reads. With thinking on, the estimate counted only the visible answer: 191 tokens where the model generated 1,210. For “Is 391 a prime number?”, the model wrote 258 tokens of reasoning and Kong estimated an 11-token answer.

So if usage-based limits or chargeback depend on these numbers, it is worth making sure every streaming client asks for usage. One detail worth knowing: for a non-streamed request, the log’s time_to_first_token is the whole response time, because the client gets everything at once.

What I would write down for next time

If I were writing the troubleshooting guide for this setup, it would start like this:

1. Read the headers before anything else. Server: kong with X-Kong-Response-Latency means Kong answered on its own and the model never saw the request. X-Kong-Upstream-Latency means Kong tried. The model’s own server header means the answer, or the error, came from the model.
2. Grab the request ID (X-Kong-Request-Id). It is on the response, in the file-log entry, and in every Nginx error-log line that has one. The read-timeout line did not, so match those by time.
3. For 400s from ai-proxy, check the error log. The body may describe a different problem from the real one.
4. For 502s, count the tries. The file-log tries array shows every attempt, and the default is up to six.
5. For timeouts, ask whether the client streams. read_timeout is the gap between reads, not the whole request.
6. For choppy streaming, time the chunks and check response_buffering on the route. Small chunks clump the most.
7. For token counts, check stream_options.include_usage on streaming clients, and remember the reasoning tokens.
8. Ask the model server too. Its metrics say whether a request arrived, and whether it is still running after everyone else has given up.

What is in the box

The usual caveats apply. This is one gateway node, one model and a handful of requests per test, on a quiet LAN. I used the openai provider because SGLang speaks that API; Kong also has a vllm provider that I did not try. And I only used the free parts of Kong. The features that probably matter most for language models in production (load-balancing across models, semantic caching, token-based rate limits) are in plugins I have not tested yet. Those are next on my list, starting with that host that never answers.