Your project, translated into plain English
We began with a 1.7-billion-parameter language model, discovered the limits of generative classification, moved to BERT and then DeBERTa, and learned that model architecture, numerical precision, data quality and honest evaluation all matter.
“The cash machine charged me an extra fee.”
Qwen generation → encoder classification
cash_withdrawal_chargeAll three models already understood language. We changed how they were adapted to the same 77-intent decision problem.
01 · First principles
A specialist classifier—not a chatbot, not RAG, and not a model trained from zero.
“My transfer is still waiting.”
pending_transferQwen solved this by generating label tokens. BERT and DeBERTa solve it with 77 output scores called logits. The highest score becomes the predicted intent.
10,003 public training questions and 3,080 test questions across 77 banking intents.
It is reproducible, inspectable, and much cheaper than generating thousands of examples in chat. Anyone can rerun our experiment.
A generative language model adapted with LoRA. It taught us the full training lifecycle, but generation was not the best architecture for closed 77-way classification.
“1.7B” means roughly 1.7 billion learned parameters. BF16 stores each weight with 16 bits. Qwen generated text, so invalid label strings were possible.
BERT and DeBERTa use a dedicated 77-class head. We first trained the full encoder, then refined only selected upper layers.
A classifier head is a small final layer that converts the encoder's representation into 77 scores. Unlike generation, it can never produce a label outside those 77 positions.
02 · The architecture lesson
Follow the lineage from a general-purpose generator to a purpose-built encoder classifier. The switch was an engineering decision, not an admission that the first model was useless.
Generative language model
question→label tokensQwen predicts the next token repeatedly. We needed constrained decoding because a reasonable phrase could still be an invalid BANKING77 label.
A closed classification task does not need open-ended generation.
Bidirectional text encoder
question→77 logitsBERT reads the entire question in both directions and compresses it into a representation. A new classifier head maps that representation directly to 77 choices.
DeBERTa improves how content and token position are represented.
Stronger encoder classifier
question→77 logitsDeBERTa separates token content from token position inside attention. We had to force float32 on MPS because lower-precision parameter updates became non-finite.
03 · Model workbench
Pick a model state. Always read whether its score came from validation or reporting-only test data.
04 · Data discipline
Most beginner ML mistakes come from letting information leak between these piles.
The optimizer sees these examples and updates either LoRA adapters or selected encoder weights.
Ten held-out examples per intent tell us whether training is learning or becoming unstable.
Never used to update weights. It was used for reporting on finalized champions and is now locked from future selection.
It is like seeing exam answers while studying, then changing your study plan because of them. We verified zero text overlap, pinned checksums, and require every current experiment to print test_rows_loaded: 0.
05 · The flight recorder
Select a stage to see what we did, why it mattered, and what it taught us.
06 · Inside one training step
Turn a banking question into token IDs and attach its correct class number.
The model produces token probabilities or 77 class logits, depending on architecture.
Cross-entropy measures how much probability the model failed to give the correct answer.
AdamW nudges adapters, the full encoder, or only unfrozen upper layers.
LoRA microscope
Use the switch to compare the two adapter designs we tested.
07 · Failure lab
Open each incident. Learn to separate a code bug, a data problem, numerical instability, and an infrastructure limitation.
08 · Reading the instruments
How surprising the correct output tokens were during training. Useful for stability and convergence—not a final product score.
The fraction of test questions with the exact correct label. Easy to explain, but frequent classes can dominate.
Compute F1 separately for each intent, then average. Every one of the 77 intents gets equal weight.
Relevant to Qwen generation. Encoder classifier heads always choose one of their 77 output positions.
Think before revealing
Optimization worked—the model learned its training objective better. Generalization did not. It may have overfit, learned noisy labels, or optimized an objective that does not improve held-out decisions. This is why validation selects experiments and the test set remains outside current development.
09 · What Qwen taught us
Allowed token paths
At every generated token, we mask paths that cannot finish as one of the 77 labels.
get_virtual_cardMeaningful, but invalidgetting_virtual_cardCanonical taxonomy labelThe mask changes logits before greedy selection. It never looks at the expected answer. Our matched batch-size experiment exists because changing batch shape can introduce tiny floating-point differences, so a fair A/B comparison must hold batch size constant.
10 · Current investigation
Exp012–015 changed our question from “How can we push the model harder?” to “Should the model trust every training label equally?”
Five-fold out-of-fold predictions examined 9,233 training rows without reading validation or test data.
1,186 disagreementsWe retained original labels and pushed against likely rival labels. Both experiments lost one validation answer.
Hypothesis rejectedRemoved 1,078 confidently suspicious training rows and rewrote no labels. The stable run reached only 90.78% validation.
17 fewer correct than Exp011The OOF model disagrees and gives the supplied label less than 25% probability.
A prediction is evidence of suspicion, not proof of the correct replacement label.
All preparation and probing report zero validation/test rows loaded.
Out-of-fold means each training example is scored by a model that did not train on that example. This reduces self-confirmation, but it does not make the predictions ground truth. Exp015 proved that our threshold removed useful training signal along with suspected noise.
11 · Interview room
Say your answer aloud, then reveal the model answer. The phrasing is based on what we actually did.
12 · Knowledge check
This is not trivia. Each question tests a decision you will face in another training project.
13 · Pocket glossary
Current lab checkpoint
The noise-pruned child reached 0.907792 validation accuracy versus Exp011's 0.929870. We reject the child, preserve Exp011, and do not open the locked test for a model that already lost on validation.
champion = exp-011 · next hypothesis not yet selectedScientific decision: a stable training run is not automatically a successful model. Exp015 completed correctly and still failed its promotion rule.