Build a Self-Hosted Support Ticket Triage Service with Qwen3.5-4B

In late June 2026, vLLM shipped v0.21: speculative decoding support for reasoning models, KV cache offload, and Model Runner V2 becoming the default for dense Llama and Mistral models. That release is the missing half of a build a lot of teams keep postponing: a support ticket triage service that runs entirely on your own hardware. The model half has been sitting there since Alibaba released Qwen3.5-4B under Apache 2.0 earlier this year, a dense 4-billion-parameter model with a 262K context window that fits on a single mid-range GPU. In this tutorial you serve Qwen3.5-4B with vLLM, force its output into a validated triage schema, wrap it in a FastAPI service with routing rules, and then use v0.21's speculative decoding to roughly double your throughput with a draft model.

Strip the product language and ticket triage is three operations: classify (urgency, category), extract (product area, sentiment, a one-line summary), and route (which queue, which alert). Keyword rules do this badly for well-known reasons: "the export worked fine until today" contains no negative keyword, "URGENT!!!" from a user who flags everything urgent contains nothing but one, and a quarter of your tickets arrive in Spanish. Classification with a language model solves the paraphrase problem, the tone problem, and the language problem in one move. It does not require a frontier model. Deciding whether a ticket about failed payments is urgent is not graduate-level reasoning, and paying frontier per-token prices for it is how inference bills quietly triple.

vLLM is the open-source inference engine that turns a Hugging Face checkpoint into a production OpenAI-compatible API. Three of its properties carry this build. Continuous batching: concurrent requests share the GPU efficiently, so a backlog drain and live traffic coexist. Structured output: vLLM constrains generation with a grammar compiled from your JSON schema, so the model cannot emit anything that fails validation. And as of v0.21, speculative decoding: a small draft model proposes tokens the 4B verifies in batches, which is the difference between one GPU handling hundreds versus thousands of tickets per hour.

One command after one pip install. The 262K native context is overkill for tickets, and capping it saves memory for batching.

The first launch pulls the checkpoint from Hugging Face. A 4B model in bf16 wants roughly 9 GB of VRAM before KV cache, so this runs on a single RTX 4090 or an L4 instance, hardware most teams already have idling somewhere. If VRAM is tighter than that, the AWQ-quantized variant of the same checkpoint drops the footprint under half at a small accuracy cost.

The schema is the real interface of the service. Everything downstream trusts it, so make it enum-heavy: free text is where triage systems rot.

reply_language is the field teams forget. Detecting that the ticket arrived in Portuguese costs the model nothing extra and saves the agent from opening a translator to find out.

vLLM's OpenAI-compatible server accepts the schema through extra_body, and the guided decoder guarantees the response parses. Not "usually parses". Parses.

The line in TRIAGE_RULES about tone is doing more work than it looks like. Without it, every ticket written in capital letters drifts toward critical, and your on-call engineer learns to ignore the pager, which defeats the entire exercise.

The model classifies. It does not decide who gets paged. Routing stays in plain code where you can test it, audit it, and change it without touching a prompt.

This split earns its keep the first time someone asks "why did this page me at 3 am". The answer is two dictionary lines, not an archaeology dig through model outputs.

vLLM's continuous batching means the right client-side pattern is firing requests concurrently and letting the server schedule them.

Thirty-two in flight keeps a single GPU saturated without starving the live /triage endpoint. Tune it by watching vLLM's logged batch occupancy, not by guessing.

This is the section the vLLM release makes possible. Add a draft model, and Qwen3.5-0.8B is the natural drafter for its own family: same tokenizer, same conventions, one-fifth the size.

The mechanics: the 0.8B proposes five tokens, the 4B verifies them in one forward pass, and every accepted proposal is a token you generated at draft-model cost. Acceptance rates run high on triage output because guided JSON is predictable, exactly the regime speculative decoding loves. Measure it on your own tickets; structured classification workloads are close to the best case for this technique, and unmeasured performance claims are how this industry got its reputation.

Guided decoding guarantees the JSON is valid, not that it is true: a confused model produces a well-formed wrong answer, so sample a percentage of classifications into a human review queue and track disagreement. A 4B model misreads sarcasm, mixed-language tickets, and domain jargon more than a frontier model does; expect a few points of accuracy ceiling as the price of data residency. Speculative decoding gains collapse when output is unpredictable, so benchmark with the draft model on and off before promising numbers to anyone. And self-hosting means the 2 am GPU alert is now your alert. The vendor you fired was also an ops team.

This build pays off when ticket volume makes per-token pricing hurt, when support tickets contain customer data that should not transit a third party, or when triage latency needs to be local-network fast. Start with the plain 4B server, measure accuracy against two weeks of human-triaged tickets, then turn on the draft model and re-measure throughput. If accuracy comes up short, the same vLLM command serves Qwen3.5-9B or 27B with one string changed. The serving layer, the schema, and the routing code all survive the model swap untouched, which is what a well-built LLM service looks like: the model is the least permanent part of it.