Benchmark a Gemini 3.5 Flash to Gemini 3.6 Flash Migration Before You Ship
On July 21, 2026, Google shipped Gemini 3.6 Flash to general availability, alongside Gemini 3.5 Flash-Lite and a security-focused Gemini 3.5 Flash Cyber variant.
Introduction
On July 21, 2026, Google shipped Gemini 3.6 Flash to general availability, alongside Gemini 3.5 Flash-Lite and a security-focused Gemini 3.5 Flash Cyber variant. The headline from independent testing is unusual: Artificial Analysis measured the intelligence index flat against Gemini 3.5 Flash, while time per task roughly halved. A model upgrade that makes nothing smarter and everything faster is exactly the kind of release teams skip, and for agent workloads that instinct is expensive. The harness you build here runs your actual tasks against both models and answers, with numbers, whether Gemini 3.6 Flash should replace Gemini 3.5 Flash in your pipeline this week.
Why a speed-only upgrade changes agent math
A chat user feels latency once per message. An agent feels it once per step, and steps are serial: plan, call a tool, read the result, call another, synthesize. An eight-step chain at nine seconds per model call spends seventy-two seconds inside the model; halve the per-call time and the same chain returns half a minute sooner. Multiply by every run in a batch pipeline and per-task latency quietly becomes the number that decides whether your overnight job finishes overnight. Capability benchmarks do not capture this. Wall-clock on your own workload does.
What to measure, and what to refuse to measure
A migration harness needs exactly three measurements per task: did the output pass your checks, how long did the call take, and what did it cost in tokens. Resist the urge to add a subjective quality score in v1. Scores you cannot explain become scores you argue about, and the migration decision stalls in the argument. Pass/fail checks you wrote yourself, latency, and cost: all three are boring, and all three are sufficient for a swap-or-stay call.
Define the task set from production, not imagination
Pull ten to thirty real inputs from the workload you intend to migrate: real tickets, real documents, real diffs. Synthetic prompts measure the model's performance on a job you do not have.
# tasks.py
from pydantic import BaseModel
class Task(BaseModel):
task_id: str
prompt: str
# Substrings that MUST appear for a pass, e.g. a required field name.
must_contain: list[str] = []
# Substrings that must NOT appear, e.g. a hallucination tripwire.
must_not_contain: list[str] = []
require_valid_json: bool = False
TASKS = [
Task(
task_id="triage-001",
prompt=(
"Classify this support ticket into one of: billing, bug, "
"feature_request, spam. Reply as JSON with keys category and "
"reason.\n\nTicket: Payment page shows a blank screen after "
"clicking Pay Now on mobile Safari."
),
must_contain=['"category"', "bug"],
require_valid_json=True,
),
Task(
task_id="extract-001",
prompt=(
"Extract vendor_name, invoice_number, and total_due as JSON "
"from this text:\n\nInvoice INV-2291 from Meridian Hosting. "
"Amount due: $1,284.00 by August 15."
),
must_contain=["INV-2291", "Meridian"],
require_valid_json=True,
),
# Add 10-30 tasks lifted from your real pipeline here.
]The checks are deliberately mechanical. A task passes when the required substrings appear, the forbidden ones do not, and the JSON parses when it must. Anything a regex cannot adjudicate belongs in a later, slower review, not in the swap decision.
Build the timed runner
One function calls one model on one task and returns everything the decision needs. The Gemini API exposes token counts in usage_metadata, so cost capture is arithmetic, not estimation.
# runner.py
import json
import time
from google import genai
from pydantic import BaseModel
from tasks import Task
client = genai.Client() # picks up GEMINI_API_KEY from your shell
class Result(BaseModel):
task_id: str
model: str
passed: bool
latency_s: float
input_tokens: int
output_tokens: int
failure_reason: str = ""
def check(task: Task, text: str) -> str:
if task.require_valid_json:
cleaned = text.strip().removeprefix("```json").removesuffix("```").strip()
try:
json.loads(cleaned)
except json.JSONDecodeError as exc:
return f"invalid JSON: {exc}"
for needle in task.must_contain:
if needle.lower() not in text.lower():
return f"missing required content: {needle}"
for needle in task.must_not_contain:
if needle.lower() in text.lower():
return f"contains forbidden content: {needle}"
return ""
def run_one(model: str, task: Task) -> Result:
start = time.perf_counter()
response = client.models.generate_content(model=model, contents=task.prompt)
elapsed = time.perf_counter() - start
reason = check(task, response.text or "")
usage = response.usage_metadata
return Result(
task_id=task.task_id,
model=model,
passed=reason == "",
latency_s=round(elapsed, 3),
input_tokens=usage.prompt_token_count,
output_tokens=usage.candidates_token_count,
failure_reason=reason,
)Note what the timer wraps: the API call only. Network jitter is part of the number because it is part of production. Run from the same region your pipeline runs from, or you are benchmarking your laptop's wifi.
Run the A/B and repeat it
One pass per model is an anecdote. Three passes is a small sample with visible variance, which is the minimum for an infrastructure decision.
# ab_test.py
import statistics
from collections import defaultdict
from runner import run_one
from tasks import TASKS
MODELS = ["gemini-3.5-flash", "gemini-3.6-flash"]
ROUNDS = 3
def main() -> None:
results = []
for round_num in range(ROUNDS):
for model in MODELS:
for task in TASKS:
result = run_one(model, task)
results.append(result)
print(f"[round {round_num + 1}] {model} {task.task_id} "
f"{'PASS' if result.passed else 'FAIL'} "
f"{result.latency_s}s")
by_model = defaultdict(list)
for result in results:
by_model[result.model].append(result)
print("\nmodel pass% p50 lat p95 lat out-tokens")
for model, rows in by_model.items():
lats = sorted(row.latency_s for row in rows)
pass_rate = 100 * sum(row.passed for row in rows) / len(rows)
p50 = statistics.median(lats)
p95 = lats[max(0, int(len(lats) * 0.95) - 1)]
tokens = statistics.mean(row.output_tokens for row in rows)
print(f"{model:<20} {pass_rate:5.1f} {p50:7.2f} {p95:7.2f} {tokens:8.0f}")
if __name__ == "__main__":
main()Read the output like an operator, not a fan. The p95 column matters more than the median for agent chains, because a chain is as slow as its slowest step, and a pipeline of chains is as slow as its unluckiest run.
Interpret the table
| Outcome | Decision |
|---|---|
| Pass rate equal, latency halved | Migrate. This is the advertised case; take the win. |
| Pass rate equal, latency equal on your tasks | Stay. The speedup did not survive contact with your workload. |
| Pass rate drops on 3.6 Flash | Stay, and file the failing tasks as your regression set for the next release. |
| Pass rate improves | Migrate, then ask why your checks were failing on a model with a flat intelligence index. Usually the checks were sloppy, not the old model. |
One more column worth adding when the first table is close: Gemini 3.5 Flash-Lite. If your tasks pass on the Lite tier, the migration conversation was never about 3.6 Flash at all; you were overpaying a tier the whole time, and the benchmark you almost did not run just found the refund
What this harness does not do
- Multi-turn agent behavior is out of scope. Wrap your real agent loop with the same timer for that; the principle survives, the code gets longer. - It does not detect subtle quality drift that substring checks cannot see. It is a swap gate, not an eval suite. - It does not control for provider-side variance over days. A Tuesday benchmark describes Tuesday (rerun the winner for a week in shadow mode before deleting the loser from your config). - It does not decide for you. It shrinks the decision to a table small enough to argue about honestly.
The final working example
"""Complete Gemini 3.5 -> 3.6 Flash migration benchmark, single file.
Requires: pip install google-genai pydantic
Env: GEMINI_API_KEY
"""
import json
import statistics
import time
from collections import defaultdict
from google import genai
from pydantic import BaseModel
client = genai.Client()
MODELS = ["gemini-3.5-flash", "gemini-3.6-flash"]
ROUNDS = 3
class Task(BaseModel):
task_id: str
prompt: str
must_contain: list[str] = []
must_not_contain: list[str] = []
require_valid_json: bool = False
class Result(BaseModel):
task_id: str
model: str
passed: bool
latency_s: float
output_tokens: int
failure_reason: str = ""
TASKS = [
Task(task_id="triage-001",
prompt=("Classify this support ticket into one of: billing, bug, "
"feature_request, spam. Reply as JSON with keys category "
"and reason.\n\nTicket: Payment page shows a blank screen "
"after clicking Pay Now on mobile Safari."),
must_contain=['"category"', "bug"], require_valid_json=True),
Task(task_id="extract-001",
prompt=("Extract vendor_name, invoice_number, and total_due as "
"JSON from this text:\n\nInvoice INV-2291 from Meridian "
"Hosting. Amount due: $1,284.00 by August 15."),
must_contain=["INV-2291", "Meridian"], require_valid_json=True),
]
def check(task: Task, text: str) -> str:
if task.require_valid_json:
cleaned = text.strip().removeprefix("```json").removesuffix("```").strip()
try:
json.loads(cleaned)
except json.JSONDecodeError as exc:
return f"invalid JSON: {exc}"
for needle in task.must_contain:
if needle.lower() not in text.lower():
return f"missing: {needle}"
for needle in task.must_not_contain:
if needle.lower() in text.lower():
return f"forbidden: {needle}"
return ""
def run_one(model: str, task: Task) -> Result:
start = time.perf_counter()
response = client.models.generate_content(model=model, contents=task.prompt)
elapsed = time.perf_counter() - start
reason = check(task, response.text or "")
return Result(task_id=task.task_id, model=model, passed=reason == "",
latency_s=round(elapsed, 3),
output_tokens=response.usage_metadata.candidates_token_count,
failure_reason=reason)
def main() -> None:
results = [run_one(model, task)
for _ in range(ROUNDS) for model in MODELS for task in TASKS]
by_model = defaultdict(list)
for result in results:
by_model[result.model].append(result)
print("model pass% p50 lat p95 lat")
for model, rows in by_model.items():
lats = sorted(row.latency_s for row in rows)
pass_rate = 100 * sum(row.passed for row in rows) / len(rows)
p95 = lats[max(0, int(len(lats) * 0.95) - 1)]
print(f"{model:<20} {pass_rate:5.1f} {statistics.median(lats):7.2f}"
f" {p95:7.2f}")
for result in results:
if not result.passed:
print(f"FAIL {result.model} {result.task_id}: {result.failure_reason}")
if __name__ == "__main__":
main()When to run this
Run it every time a Flash-tier model ships, which lately is monthly. The harness is thirty minutes to build once and five minutes to rerun forever, and it converts every future "should we upgrade" thread into a table. Speed-only releases are the ones most worth testing precisely because they look skippable: nothing about your outputs changes, and everything about your throughput might.
You might also like
Keep reading from the journal.
August 3, 2026Cloud
The return was decided before you shipped the box
Calibrated return-risk scoring at checkout prices the fitting room your parcels became
July 27, 2026DocumentProcessing
The care you delivered but never billed
Ontology-grounded clinical NLP closes the gap between what your encounter notes document and what your claims code, with evidence attached to every finding.
July 27, 2026MachineLearning
The fraud looked normal one transaction at a time
Graph features and community detection surface coordinated fraud rings weeks before the cash-out, where per-transaction scoring sees nothing wrong.