Too accurate to be comfortable

Picture the last time a store knew something about you that you hadn't told it. Not a lucky guess. The real thing. You looked at one pair of running shoes, and the next page knew your size, knew you'd been marathon training, knew you'd buy the recovery sandals too, and laid all of it out before you'd decided you wanted any of it. For a second it is convenient. For the second after that, it is unsettling, and you close the tab.

A team I know built exactly that, and watched conversions fall.

The recommendation engine had gotten good. Genuinely good, the kind of good you put in a board deck. It moved from "people who bought this also bought" to predicting the next purchase before the customer had reached for it, stitching browsing, timing, and past orders into a profile that was usually right. The metric they'd optimised, prediction accuracy, climbed beautifully. The metric they cared about, purchases, did not follow. It dipped.

Past a certain point, accuracy stops feeling like service and starts feeling like surveillance.

What they had missed is that a shopper is not only deciding whether to buy. They are deciding whether they are comfortable, and comfort has a ceiling on how much a stranger is allowed to know about them out loud. When the site surfaced a prediction that was too specific, too soon, it didn't read as helpful. It read as being watched. The customer hadn't told the store they were marathon training. The store knowing anyway broke something that no amount of relevance could pay back.

This is the part that makes it hard to fix with more model. The engine was not wrong. Its predictions were landing. The problem was not accuracy, so improving accuracy made it worse, which is a deeply confusing place to stand when every instinct you have says a better model is a better product. Here, a better model was a creepier one.

The teams that get this right treat how-much-to-reveal as its own decision, separate from how-much-they-know. The model can know you're marathon training. The interface can choose to simply show you more running shoes, good ones, without announcing the inference behind them. Same intelligence, different manners. You let the relevance show up in what gets ranked, not in a caption that brags about how well you've been profiled. The customer feels understood instead of exposed, and those are very different feelings sitting behind the same click.

Restraint is a strange thing to engineer on purpose. Every part of the build is pushing the other way, toward showing the customer how clever the system is, because the team is proud of it and the demo lands harder when the prediction is spelled out. But the demo is being watched by your colleagues, who are delighted to be profiled. The product is being used by a stranger, who is not.

The right amount of personalization is not the maximum amount. It is the most you can offer before the person on the other end remembers they are being measured.

TensorLabs has had to dial a model back more than once. It is an odd recommendation to make and a profitable one: the accuracy was never the product. The way it felt was.