From demo to production: Generative search lessons from Alexa+

Generative, LLM-powered search demos beautifully, then collides with production: hard latency deadlines, a bill that scales with traffic, and near-zero tolerance for failure. This experience report argues that the hidden specification for production AI search is the customer's expectation of difficulty—how hard a request looks to them—which sets both the quality they demand and the latency they tolerate. Drawing on generative video search built for the launch of Alexa+, Amazon's next-generation assistant, across Fire TV and Echo Show serving millions of customers, we report six lessons: why queries that look trivial are the ones that must never fail and how we measure them with an LLM-as-judge framework; why perceived difficulty, not model capability, must set the latency and cost budget; and what this forces in engineering—speculative execution, predictable degradation, catalog-grounded safety, and small fine-tuned understanding models that beat a larger general LLM. Finally, the query distribution is non-stationary: we optimize for the demand we will have, not yesterday's logs.