We tested free AI models on real console tasks
Benchmark scores tell you almost nothing about whether a model belongs near a lighting rig. So we ran three tests that do.
There is a good reason to care whether free AI models are any good. If a console needs artificial intelligence to do its interesting tricks, and that intelligence requires the operator to open a second account and buy API credits before anything works, then the feature is not really there for most people. A free option that is good enough changes who gets to use it.
So we tested one. Not with benchmarks — with the work.
The three tests
Can it obey a grammar? A console command line is not conversation. It takes 1>4+9 at 50 and nothing else. A model that returns the right answer wrapped in a helpful paragraph has failed, because a parser will choke on it. We gave the model the grammar and one example, and asked it to convert a plain-English request. Exact string match or failure.
Will it admit it doesn't know? This is the one that matters. We asked about phantos_viz — one of our own fixture profiles. It exists in no training data anywhere on earth. There is no correct answer to recall, only the correct behaviour: say you don't know. A model that invents a confident channel count for a fixture profile it has never seen will invent a confident answer about your rig, and that is the failure mode that ends up in front of an audience.
Can it write for a technician? Three sentences explaining what a DMX universe is, plain, no marketing. This is the unglamorous work — documentation, help text, explanations — and it is most of what an AI in a console would actually be asked for.
What happened
| Model | Grammar | Admits ignorance | Speed |
|---|---|---|---|
| Gemini 3.5 Flash-Lite | PASS | PASS | 0.5–0.7 s |
| Gemini 3.5 Flash | PASS | PASS | 2.2 s |
| Gemini 3.7 Flash | PASS | PASS | 3.4 s |
| Gemini 3.8 Flash | PASS | PASS | 2.9 s (one 503) |
| Gemini 3.1 Pro | quota | quota | — |
Asked about phantos_viz, every model that answered replied, in full: "I do not know." That is the right answer, and it is worth more than any benchmark on the list.
The Flash-Lite model returned correct console syntax in about half a second. For a command line, sub-second matters more than raw intelligence — an operator will not wait three seconds to see whether the desk understood them.
One correction, because it changes the result. Our first grammar test failed on every model — they wrote 1-4+9 with a hyphen instead of 1>4+9. Then we read our own prompt: we had never told them the range operator was a greater-than sign. The models had guessed the most common convention, sensibly. With the grammar fully specified, every model passed exactly. The first result was a bad test, not a bad model, and we would rather say so than publish a table that flattered our own tooling.
The catches, stated plainly
The free tier is Flash-class only. The Pro model returned a quota error on the first request. Anyone quoting free access to a frontier model is describing something we could not reproduce.
Capacity is not guaranteed. We got a 503 "high demand" on the newest Flash model mid-test. Free infrastructure fails sometimes, so anything built on it needs a local fallback that is not an apology screen.
Read the data terms. Google's terms say content submitted on the non-paid tier — prompts and responses — may be used to improve their products. The paid tier is excluded. A console's prompts carry rig layouts, fixture lists, cue names and whatever the operator said out loud. That is a real consideration for a commercial show, and it is the reason our position is bring-your-own-key: your account, your terms, your choice, stated in plain language rather than buried.
What we're doing with it
Phantos today uses a single cloud provider for its language features, alongside an on-device model that needs no account at all. Supporting a second provider is not a Gemini integration so much as a seam — one interface, several implementations — and today is a decent argument for never hard-wiring a vendor again. Between starting this test and finishing it, Google discontinued the sign-in method that a great many tutorials still describe as current.
That work is scheduled, not shipped. When it lands, this page gets the setup guide. We would rather publish the test than a promise.