Prompt2Model: a vision model factory with a refusal gate

Describe the model you want in plain language. Get back a calibrated, exported, deployable vision model, or a documented refusal. Prompt2Model is a typed pipeline: a prompt becomes a dataset config, then a trained model, then a conformally calibrated model that can abstain, then a verified ONNX export, then an optional compressed variant that only ships if it clears an accuracy floor. The factory's contract is not "a model, always". It is "a model you can trust, or an explicit, logged refusal".

The problem

Picture a small operations team running a packing line. They want a camera to sort items into three bins by shape and color, or to flag when a specific object shows up in a fixed camera view. They can describe the task in one sentence. What they do not have is a labeling team, an ML engineer, or six weeks. This is exactly the kind of narrow, well-bounded vision task that a small model handles well, and exactly the kind of project that never gets built because the path from "one sentence" to "deployed model" traditionally runs through hand-labeling thousands of images and hiring someone to babysit a training pipeline.

Automated model factories exist, but most share one dangerous property: they always hand you a model. Ask for the impossible and you still get a model file back, one that is confidently wrong on inputs it has never seen, or one that was quietly quantized into uselessness to hit a size target. For an operator, a factory that silently ships an overconfident or under-accurate model is worse than no factory at all, because the failure is invisible until it costs something. The model does not know what it does not know, and neither does the person deploying it.

Prompt2Model is built around the opposite contract. The factory compiles a plain-language task description into a full training and export pipeline, and at the two points where silent failure usually happens it installs an explicit refusal: a per-input abstention gate calibrated on held-out data (the model says "I don't know" instead of guessing), and a build-time accuracy floor on compression (the factory keeps the uncompressed model rather than ship a degraded one). Both refusals are enforced in code and logged in the run report, not just documented.

How it works: the pipeline

The whole factory is one typed pipeline. Every stage consumes and produces a validated, inspectable object, so a half-parsed prompt or a missing field fails loudly at the boundary instead of surfacing three stages later as a mystery. The two red boxes below are the refusal gates.

Stage by stage

1. Prompt to config. A deterministic, dependency-free parser turns the prompt into a typed pipeline configuration: task, labels, augmentations (for example "under low light"), and constraints such as a latency target, a power budget, a parameter cap, or an accuracy floor ("keep at least 90% accuracy" becomes a hard number the compression gate must respect). An optional LLM planner can sit in front of it for indirect phrasing; it only overlays fields it actually extracted, never replaces the deterministic parse, and any planner failure falls back to the parser, so a config is always produced.

2. Data and training. Image-folder classification and COCO-style detection loaders, a small model registry (a stated constraint always pushes model choice down to a smaller tier, never up), and standard training loops with the prompt's augmentations injected. For this page's evidence the data is the factory's own generated toy shape set; on a real task you point it at your own folder.

3. Refusal gate 1: conformal abstention calibration. On the held-out validation split the factory fits a temperature (logit scaling that minimizes validation NLL, with expected calibration error recorded before and after so the effect is auditable) and a split-conformal abstention threshold: the (1 - alpha) quantile of validation nonconformity, where nonconformity is 1 minus the calibrated top-class probability. Both values are embedded in the ONNX file's metadata, so they travel with the artifact. At inference, nonconformity above the threshold means the model returns ABSTAIN instead of a guessed label.

4. Export. The model is exported to ONNX with its labels, preprocessing parameters, and calibration block injected as metadata, then re-loaded and executed in ONNX Runtime to verify the file actually runs, not just that it exists on disk.

5. Refusal gate 2: the accuracy floor on compression. Optionally the factory distills or INT8-quantizes the exported model. The compressed candidate is then re-evaluated in ONNX Runtime, the same runtime it would ship in, and compared against the stricter of a relative floor (98% of baseline accuracy by default) and any absolute floor stated in the prompt. If it cannot hold that floor, the factory refuses to ship the compressed artifact and keeps the uncompressed one, logging the refusal verbatim in the run report.

6. Deployment target. The artifact compiles to a target, not a box: ONNX Runtime by default, TensorRT when requested (built locally when trtexec is present, otherwise the factory emits a reproducible on-device build script instead of failing silently).

7. Flywheel. At inference, abstained and low-confidence frames can be captured into a bounded hard-case store, so the inputs the model could not handle become the candidate training data for the next iteration instead of being guessed at and forgotten.

The interactive demo: real per-image results

Both browsers below show real, committed example images and the real per-image output of the factory's own exported ONNX models, produced by python -m prompt2model.cli smoke-test --output-dir output/smoke_verify at commit 34103b3 this session (2026-07-09) and then run image by image. No number below is interpolated or invented; this Space is static, so what you see is a faithful record of those runs, not live compute.

Classification: the abstention gate on 36 real images

The prompt behind this run: "Classify red square, blue circle, and green triangle images under low light and prioritize speed." (the factory's own committed classification smoke-test prompt). The factory trained a 1.520931M-parameter mobilenet_v3_small from ImageNet-pretrained weights, fit a conformal threshold of 0.004888 from 7 held-out validation samples at alpha=0.1, with logits temperature-scaled by temperature=0.05 (ece_before=0.3864 to ece_after=0.0014, a real calibration improvement). Pick an image to see the model's real prediction, its real nonconformity score, and the gate's decision:

true label

predicted

score

gate

The real, honest finding: the model predicts confidently and correctly on all 36 of 36 real committed images checked this session: score 1.000000 (nonconformity 0.0) on all 24 red-square and blue-circle images, and score 0.995113 (nonconformity 0.004887) on all 12 green-triangle images, every one below the 0.004888 threshold by a razor-thin margin (the green-triangle nonconformity sits just 0.000001 under the threshold), so the gate answers on all 36. Zero real abstentions came out of this run: on this near-perfectly separable synthetic toy set, a healthy pretrained backbone simply does not find anything to hesitate on. That is disclosed as the real result, not softened into a fake abstain example.

Because the real run produced no ABSTAIN case to show, here is the gate exercised on an illustrative, synthetic, out-of-distribution input instead of one of the 36 real committed images, so the refusal mechanism is still demonstrated end to end with the same real model and the same real threshold. Feeding the exact same ONNX model a 96×96 image of pure random RGB noise (not a photo of anything, generated with numpy.random.default_rng(seed=7)) produces top-class score 0.969760, nonconformity 0.030240, above the 0.004888 threshold, and the model's own EdgeModel.run_inference returned abstained: true, rerun fresh this session:

Try the gate's decision rule yourself

The abstention decision is one comparison, computed here in your browser exactly as the deployed EdgeModel computes it: abstain when 1 - p > threshold.

Detection: ground truth vs. the model's real top detections

The detection smoke test uses the prompt "Detect squares and circles in low light images and prioritize speed." and trains an ssdlite320_mobilenet_v3_large for one toy epoch. Below, real committed COCO-style ground-truth boxes (green) versus the freshly exported detector's real top-3 scored boxes (red), run image by image this session:

Green boxes: real committed ground truth. Red boxes: the real exported model's top-3 scored boxes this session.

The real, honest finding: across all 12 committed example images, the fresh detector's top-3 boxes now vary with the image (the earlier, pre-fix export produced identical boxes on every image; that specific pathology is gone after the pretrained-backbone fix), but they still do not track the real object locations, scores sit in a low 0.3683 to 0.4504 band, and 34 of the 36 top boxes carry the raw label index 2, which falls outside the 2-class label list (logged here as the raw index rather than invented as a real class name; the remaining 2 boxes, on image_008, are labeled circle and land loosely near the real circle). This matches the freshly measured mAP@0.5 of 0.0338: a toy detector trained for one epoch on a handful of synthetic images has not learned to localize yet. Reported as measured, not smoothed over.

Measured results, with provenance for every row

Rules for this section: every table states where each number comes from (exact command and commit, or exact file). Rows rerun this session are labeled fresh; rows kept from earlier same-day runs are labeled historical and are never silently mixed with fresh ones.

Release and test suite

The plain pytest terminal summary line was swallowed by a local reporting quirk (same quirk as the earlier session), so the machine-written JUnit XML counts are quoted instead; the run's exit code was 0.

Toy smoke-test results, fresh and historical

Fresh rows: python -m prompt2model.cli smoke-test --output-dir output/smoke_verify at commit 34103b3, 2026-07-09; numbers verbatim from output/smoke_verify/classification_run/telemetry.json (latency_ms 70.59494140557945, fps 14.165320915203214) and output/smoke_verify/detection_run/telemetry.json (map@0.5 0.03375959079283888, map@[0.5:0.95] 0.013069053708439898, latency_ms 37.29286682792008, fps 26.814779475503588, gflops 0.510593344). Both fresh ONNX exports were verified runnable (edge_inference_verified: true). Historical rows are retained from the earlier same-day sessions at the commits shown; the historical detection row predates the pretrained-backbone fix and used a smaller from-scratch configuration, which is why its parameter count differs. The point of this table is that the typed pipeline runs end to end and both refusal gates fire correctly when they should, not that these toy models are competitive.

Refusal gate 1: conformal abstention, full numbers

Fresh, this session (2026-07-09, commit 34103b3): per-image EdgeModel.run_inference over all 36 committed example images and the seed-7 noise image, against output/smoke_verify/classification_run/model.onnx. The scores reproduce the earlier ba62c2f session's per-image results exactly. Note the caveat the numbers themselves carry: a threshold fit from only 7 validation samples is statistically noisy, and the green-triangle margin under the threshold is 0.000001, both disclosed rather than smoothed.

Refusal gate 2: the accuracy floor on compression, both real outcomes

The gate's promise is "a smaller model OR a refusal". Both outcomes have now actually happened, and both are shown with full numbers rather than picking the flattering one:

Historical column: retained unchanged from the earlier same-day session at commit 4e478b2, a separate custom "low-power edge camera" prompt run whose own "keep at least 60% accuracy" wording set the 0.6 absolute floor; that run's model was undertrained (from scratch, pre-fix), which is why both accuracies are 0.0, and the gate correctly refused to ship a smaller copy of a broken model. Fresh column: rerun this session, 2026-07-09, commit 34103b3, command python -m prompt2model.cli run --prompt "Classify red square, blue circle, and green triangle images, keep at least 90% accuracy" --task classification --dataset-root output/smoke_verify/classification_data --dataset-format imagefolder --quantize --pretrained --output-dir output/quant_verify. The prompt's stated 90% floor and the default 98% relative retention floor are both computed and the stricter one (0.98) applied. That fresh run picked a larger 4.205875M-parameter backbone (its prompt has no "prioritize speed" constraint) and its own calibration block reads temperature 0.05, conformal_threshold 0.389276, ece_before 0.3336, ece_after 0.1112, quoted here for completeness.

Deployment step, historical 4e478b2 run: target tensorrt, built=false, logged reason: "trtexec not found on this host, run build_tensorrt.sh on the target device". The pipeline emits a ready-to-run build script rather than failing silently when the target toolchain is absent. Not rerun this session.

Training-loss history, complete and unedited

The current report format records final metrics only, so no fresh per-epoch curve exists for the healthy 34103b3 models; the curves below are the complete recorded histories from the earlier runs, kept as labeled historical artifacts. Nothing is truncated: the toy classification run has exactly 2 epochs of history and the toy detection run exactly 1.

Verbatim from the Training History blocks of output/smoke/classification_run/evaluation_report.md and output/smoke/detection_run/evaluation_report.md (the earlier runs' committed reports). The classification curve's 0.0 validation accuracy is the pre-fix bug this project already disclosed: an unseeded, from-scratch backbone whose BatchNorm statistics collapsed on the tiny split. The fix (seeded RNGs, stratified splits, pretrained backbone) landed in PR #18, and the fresh 34103b3 rerun reaches accuracy 1.0 / macro F1 1.0; the broken curve is kept here because deleting an already published negative result would defeat the point of this page.

Detection: ground truth vs. prediction, summarized

Fresh, this session: per-image ONNX inference against output/smoke_verify/detection_run/model.onnx over all 12 committed detection examples; ground truth from the committed assets/p2m_examples/detection/annotations.json. Every per-image box drawn in the viewer above comes from this run.

Limitations, stated plainly

• Every model on this page is trained in seconds on a few dozen synthetic shape images. That is not evidence about real-world vision model quality; it exists to prove the pipeline (prompt to dataset config to train to calibrate to export to gate) runs end to end.
• The classification toy set is near-perfectly separable, so the fresh healthy model produced zero real abstentions across all 36 committed images. The ABSTAIN walkthrough above therefore uses a clearly labeled synthetic out-of-distribution input (random noise), not a real example. A gate that never fires on its own demo set is a disclosed weakness of the demo set, not evidence the gate works in the wild.
• The conformal threshold is fit from only 7 held-out samples. Split-conformal guarantees are distribution-free but a 7-sample quantile is statistically noisy, and the green-triangle margin under the threshold is 0.000001. On a real task the calibration split must be far larger.
• The fresh detection model still cannot localize: mAP@0.5 is 0.0338, top-box scores sit around 0.37 to 0.45, and 34 of its 36 top boxes carry a label index outside the 2-class list. One toy epoch on a handful of images does not produce a detector, and this page does not pretend otherwise.
• The training-loss curves shown are historical, pre-fix artifacts (the current report format records final metrics only). The classification curve documents a real bug this project shipped and then fixed: 0.0 validation accuracy from an unseeded, from-scratch backbone.
• The accuracy-floor refusal shown is from the earlier undertrained run; the fresh healthy run passed its gate. Both outcomes are shown. No fresh refusal case exists for the healthy classification model because dynamic INT8 quantization simply did not degrade it on this toy set.
• The TensorRT deployment step has not built on any host used for this evidence (no trtexec); the pipeline emitted a build script instead. No on-device (Jetson-class) latency for the emitted recipe has been measured yet.
• Calibration, the compression gate, and flywheel capture are validated for the classification path; detection follows the same interfaces where noted in code but is not yet the fully validated path for those stages.
• No trained checkpoint from these toy runs is published as if it were a usable model. When the pipeline is run on a real task with a real budget, that model would be published separately with its own real data and metrics.
• No customer or production deployment claims are made anywhere on this page.

FAQ

What data were these numbers measured on?

Entirely synthetic toy sets the factory generated for itself: procedurally drawn shape images (12 red squares, 12 blue circles, 12 green triangles committed as the classification examples; 12 images each containing one square and one circle, with COCO-style annotations, as the detection examples). They are illustrative fixtures whose job is to prove the pipeline runs end to end. They are published in full in the prompt2model-examples dataset so every per-image number here can be checked against the actual pixels.

Is any of this real-world performance?

No, and this page says so everywhere a number appears. There is no real-world benchmark result for Prompt2Model yet. The measured artifacts are a pipeline smoke test: what is demonstrated is that the factory's machinery (typed config, training, calibration, verified export, both refusal gates, deployment targets) actually executes and that its refusals actually fire. Real-task results would be published separately, labeled with their own data and metrics.

Can I run it on my own task and data?

That is the design: you point the CLI at an image-folder (classification) or COCO-style (detection) dataset with a plain-language prompt, and opt into compression, deployment targets, or the LLM planner with flags. Constraints written in the prompt ("keep at least 90% accuracy", "runs at 2 watts", a latency target) become enforced configuration, not comments. The code itself is open source under MIT, public at GitHub.

What does the refusal gate actually do when the model is not good enough? What does an operator see?

Two different things at two different moments. At inference, per input, the deployed model returns an explicit abstained: true with its score instead of a guessed label; the operator sees ABSTAIN and can route the frame to a human or to the flywheel hard-case store rather than acting on a guess. At build time, per artifact, the compression gate re-evaluates the compressed candidate in the runtime it ships in; if it falls below the floor, the factory ships the uncompressed model instead, sets compression.passed=false in the machine-readable result, and writes the refusal reason verbatim into the run report. Nothing is silently degraded and nothing is silently blocked: you always get an artifact plus the written record of the decision.

Why two separate refusal gates instead of one?

Because they catch different failure modes at different times. Gate 1 is a per-input, run-time decision: this particular image is unlike what the model was calibrated on, so do not answer. Gate 2 is a per-artifact, build-time decision: this compressed file broke the accuracy contract the prompt stated, so do not ship it. Neither subsumes the other: a perfectly calibrated model can still be ruined by quantization, and a quantized model that held its floor can still meet inputs it should refuse. The historical refusal above is exactly gate 2 catching a model that gate 1's calibration could not save.

How were these numbers produced, and can I check them?

Every table carries a provenance line with the exact command, commit, and file the numbers came from, and the page separates fresh reruns (2026-07-09, commit 34103b3) from retained historical results instead of mixing them. This Space is static, so nothing is recomputed at page load; what you read is a fixed record of those runs. The example images and annotations are public in the dataset linked below, and the v0.1.0 release post walks through the release-level numbers and the full list of open issues.