The Model Wasn't Random — It Was Backwards

The Setup

The task: given two dating profile texts, predict whether the people are compatible. A classic embedding problem — encode both profiles, compute cosine similarity, threshold at 0.5.

I started with all-MiniLM-L6-v2, a popular sentence transformer. It's trained on general text similarity — semantic relatedness, paraphrase detection, that kind of thing. Not dating. But surely it would be neutral, right? Two random profiles should score around 0.5. Compatible and incompatible pairs should be mixed.

I ran the baseline evaluation on 1,469 held-out pairs.

Wait, Worse Than Random?

An AUC of 0.50 means "no better than guessing." An AUC of 0.40 means the model is systematically wrong — if you flipped its predictions, you'd do better than chance.

The negative Cohen's d confirmed it. On average, incompatible pairs had higher similarity scores than compatible ones. The model wasn't confused — it was confidently backwards.

Why?

Textual Similarity ≠ Compatibility

Here's the insight that made everything click: incompatible people often talk about the same topics.

The pre-trained model was doing exactly what it was trained to do: find texts that discuss similar topics. It had no concept of compatibility. Shared vocabulary + opposite positions = high similarity + zero compatibility.

Fine-Tuning Doesn't Just Add Signal — It Corrects It

This reframed what fine-tuning was actually doing. I wasn't teaching a blank model something new. I was correcting a model that had learned the wrong signal for this domain.

The training process (CosineSimilarityLoss on 6,000 labeled pairs) pushed compatible pairs together and incompatible pairs apart. After 4 epochs:

Cohen's d went from -0.36 to +2.17 — a "huge" effect size by statistical convention. The model didn't just become neutral; it learned what compatibility actually means.

Why This Matters

If the baseline had been random (AUC ~0.50), fine-tuning would have been about adding signal where none existed. That's a harder problem — you're starting from nothing.

Because the baseline was backwards, fine-tuning had a clearer job: invert the existing signal and amplify it. The model already "knew" something about compatibility — it just had the sign wrong. Correcting a systematic error is often easier than creating signal from scratch.

The Limit: Data Quality

The fine-tuned model plateaued at 83% accuracy. I ran 7 hyperparameter iterations — epochs, learning rate, batch size, warmup — and they all landed in the same range. The ceiling wasn't training configuration; it was data quality.

The training data used generic language: "I value honesty", "I enjoy spending time outdoors." When profiles are vague platitudes instead of specific preferences, there's only so much signal to extract.

That's a real finding too. The model learned everything it could from the data. Further improvement requires better data, not more engineering.

The Takeaway

Before fine-tuning an embedding model, run a baseline evaluation. Don't assume the pre-trained model is neutral. It might be:

• → Random: No signal. You're teaching from scratch.
• → Weak but correct: Some signal. Fine-tuning amplifies it.
• → Backwards: Strong signal, wrong direction. Fine-tuning corrects it.

Each scenario tells you something different about the problem. A backwards baseline isn't a failure — it's information about how the domain differs from the model's training data.

And sometimes, backwards is exactly what you want to find.