Local Distillation Experiment
The real question was not whether distillation could run, but how to make a local model improve on real code-repair tasks in a way that stayed interpretable and repeatable.
This experiment used GPT-5.4 as the teacher and covered the full loop from sampling and response cleaning to verified best-of-k selection, MLX LoRA training, and repeat-3 holdout evaluation. At this stage, I care less about any single score and more about which effects look stable versus which ones are really coming from upstream data formation.
better than non-strict at 12/24
better than strict 1200
the evaluation stage itself stayed stable
Why I Ran This
I wanted to distill GPT-5.4's code-repair behavior into a local model rather than stop at an online-only workflow.
The hard part was not getting data and training scripts to run. The hard part was understanding where improvement actually came from and whether the result could be trusted.
Repeat-3 evaluation showed the holdout result itself was stable, which pushed the real uncertainty back toward teacher sampling and dataset formation.
How I Structured the Pipeline
Sampled multiple GPT-5.4 teacher candidates per training task and kept the raw responses so formatting noise and content noise stayed visible.
Cleaned GPT-5.4 teacher outputs into executable file updates, then verified them against task checks so only real passing fixes survived.
Selected only verified passing candidates for each task instead of feeding plan text, JSON, or invalid repairs into the student.
Trained adapters locally, then repeated the same holdout evaluation three times to check whether the result was actually stable.
How the Experiment Evolved
The first goal was to improve sample quality through verified selection rather than simply increase teacher volume.
I tightened both the system prompt and the task-local output contract to suppress plans, JSON, wrapper prose, and shell commands.
I did not want to trust a single report, so I ran the exact same holdout set three times at each budget to see where the variance really lived.
What Strict vs Non-strict Means
Non-strict means a looser output contract for the GPT-5.4 teacher. It gives the model more room to elaborate, which also makes it more likely to emit plans, JSON, explanatory prose, or other wrapper text.
Strict means a tighter output contract. I explicitly push the GPT-5.4 teacher to produce content that is closer to the final repair so cleaning and verification fail less often.
The comparison is not between two models. It is the same GPT-5.4 teacher under two output contracts, distilled into the same local Qwen3.5 35B.
Latest Results
10 clean failures, 1 verify failure
0 clean failures, 0 verify failures
both runs covered the full train set
base 11/24
base 13/24
base 11/24
base 13/24
What the Numbers Mean
Strict prompting clearly helped at lower budget, which suggests cleaner teacher contracts improved low-budget reliability.
The gain was not universal. Once the budget opened up, the strict profile lost some flexibility on a subset of tasks.
What I'm Seeing
After running repeat-3, I'm fairly confident the holdout evaluation itself is stable. At least in this round, it does not look like the main source of variation.
The bigger uncertainty is still upstream: changes in teacher sampling affect cleaning quality, which affects what gets selected, which then shows up in student performance.
So if I keep pushing this experiment, I would not keep tightening the prompt. I would test a medium-constraint variant and focus on the regression at the 1200-token budget.
How My Read Changed During the Experiment
At the beginning, I leaned toward the idea that tighter prompting would simply make the distillation cleaner and more reliable.
After the repeat-3 runs and task-level comparisons, that read changed. Strict prompting clearly helped at lower budget, but once the budget opened up, the real question became whether I was over-constraining the model and suppressing useful flexibility.
So the next step now looks different to me: spend less effort tightening the prompt even further, and more effort on teacher sampling plus a medium-constraint variant.