Seeing the Shopper: A Retail Digital Twin Built on What Customers Say, Feel, and Buy

This post is adapted from my invited talk at CV-S3 2026, the International Conference on Computer Vision for Smart Sustainable Systems. The argument fits in one sentence: retail can build a digital twin of the shopper from three layers of evidence — what customers say, how they feel, and what they buy. Computer vision supplies the middle layer. Retail is the one industry that already has the third, and that third layer is what lets us prove the twin is right.

The Problem: Stated Intent Is Not Behavior

In a survey, a customer tells us: “I would definitely buy this.” In the store, the product stays on the shelf.

That gap isn’t dishonesty. Memory is imperfect, and people want to be agreeable — answers get shaped by how the question was asked and who is asking it. But the traditional tools we use to close the gap, surveys and focus groups, are slow and expensive: studies that run for weeks or months, physical samples produced only to be tested, and results that still lean on what people say.

For a retailer, a wrong answer here isn’t an academic error. It becomes real inventory in real stores.

Why a Digital Twin: Test the Copy, Not the Customer

Engineers don’t build a jet engine and hope. They simulate it and stress-test it in software first. A digital twin of the shopper applies the same principle: model how a customer evaluates and decides, then test a product, a package, or a price against that model before committing to a launch.

But a twin is only as good as the evidence behind it. If it learns only from survey answers, it inherits exactly the problem above — it becomes a very fast way to be wrong in the same direction.

Vision Adds the “Feel” Layer

More and more consumer research is now recorded on camera. With consent, that video becomes raw material for a second layer of evidence. Vision models can read three things from it:

• Expression — whether the reaction is positive, neutral, or uncertain
• Attention — where gaze lands on the package, and the moment it drifts
• Hesitation — the pause before the answer

Picture a participant looking at a new product. At four seconds they smile. At nine seconds their gaze drifts away from the package. At twelve seconds they pause — and then say yes. A transcript records only the yes. The video records the whole story.

Inside the Vision Pipeline

For readers from the computer vision side, here’s how those signals are produced:

1. Detect — faces and landmarks, frame by frame.
2. Encode — Facial Action Units (FACS), not coarse emotion labels. Action Units describe observable muscle movements, so they travel better across people and contexts than “happy” or “sad.”
3. Attend — gaze and head pose mapped to regions of the stimulus, like the claim or the price.
4. Sequence — reactions modeled over time (onset, peak, latency) rather than as single frames.
5. Fuse — video aligned with the transcript and voice tone on one timeline.
6. Validate — every signal checked against human labels, and scored separately for each demographic group.

That last step is the one I’d underline. I’ll come back to why.

The Retail Advantage: Three Layers, One Twin

Here’s where retail is different from almost every other industry building twins:

• Say — survey responses
• Feel — vision-read reactions
• Buy — actual transactions and orders

Most companies working on digital twins have to estimate or license that third layer. A retailer generates it every day. That changes what the twin is. Sales data isn’t just another input — it’s ground truth.

In production, that becomes one platform with one loop: capture consented video, analyze it with the vision pipeline, model the twin using those signals plus anonymized sales, and help teams decide what to launch and for whom. Post-launch sales flow back in automatically — the recalibration isn’t a quarterly manual review, it’s part of the system.

Why Now: The Field Is Moving Fast

This isn’t hypothetical, and it isn’t early. Digital-twin research is attracting serious investment:

• Simile builds twins from long-form interviews and raised $200M at a reported $2B valuation.
• Brox has built 60,000 twins of real, consenting people that can be surveyed in hours rather than months.
• Aaru simulates synthetic populations; EY used it to recreate a global survey study that originally took six months in a single day.

What strikes me is that all three learn mostly from what people say. That’s the gap retail can close — by adding what people feel and what they actually buy.

The Sustainability Case

In retail, a bad guess is physical. That’s why this belongs in a conversation about sustainable systems:

• Fewer prototypes — concepts that fail can fail in software, not in sample runs and reprinted packaging.
• Less overstock — forecasts closer to true demand mean fewer markdowns and less product written off.
• Faster cycles — fewer in-person studies, and less travel to run them.

At retail scale, a forecast corrected in simulation is inventory that never becomes waste.

Three Non-Negotiables

Reading faces is sensitive work, and it has to be done on firm ground:

• Consent — participants opt in and know how their video is used.
• Privacy — sales data is anonymized and aggregated. The twin learns patterns, not people.
• Fairness — this is the hard one. Expressions don’t carry identical meaning across cultures, and these models can perform unevenly across skin tones and age groups. That’s why the pipeline validates per group, and why people stay in the loop. Regulators are converging on the same point, from the EU AI Act to state rules in California.

And one boundary I hold to above all: the twin informs decisions; it does not replace real customers.

Open Questions I’m Still Working On

I don’t have all of this solved. Three questions I’d love to hear from others on:

1. How do you measure a twin’s fidelity when the three layers disagree?
2. How much does vision actually add beyond transaction history alone?
3. How do we make expression models fair across cultures?

If you work on any of these problems, I’d genuinely welcome the conversation.