By: Jason Chan
The race to build more powerful artificial intelligence models often overshadows a quieter, equally critical challenge: making those models efficient enough to run at a global scale without consuming prohibitive amounts of computing resources. As AI deployment accelerates across every sector of the economy, the gap between research breakthroughs and production-ready infrastructure has become a bottleneck for the entire industry.
Making AI Infrastructure More Efficient
One software engineer is helping to close that gap through systematic work on model-serving infrastructure at a major US technology company. Ke Shao, who focuses on the interface between machine learning research and production engineering, has spent the past several years tackling the problem of AI efficiency, not by building larger models but by making existing infrastructure radically more efficient. His most significant contribution lies in optimizing how AI accelerators host speech foundation models, the specialized large models that power speech-to-text and text-to-speech features used by millions daily. Through a combination of smarter autoscaling, load-signaling improvements, and regional grouping algorithms, Shao has enabled the company to avoid using more than 2,200 high-performance AI accelerators globally, enough computing capacity to simultaneously support dozens of large AI models or free up resources to train multiple state-of-the-art systems from scratch. In an era of AI chips being both a strategic asset and a major capital expense, this level of efficiency translates directly into faster deployment, lower operational costs, and reduced energy consumption.
How Smarter Autoscaling Reduces Hardware Use
The largest single efficiency gain came from a project that enabled and optimized autoscaling for the company’s speech foundation model. Shao designed a system that automatically adjusts the number of accelerators based on real-time traffic, ensuring that the model uses the minimum chips necessary without harming latency. To make this work, he developed a way to extract “batch fullness” as a reliable load signal, a metric that later became the foundation for multiple other efficiency initiatives. Beyond autoscaling, Shao tackled the problem of model density. Traditionally, one accelerator could serve only one speech model. He developed a regional grouping algorithm that allows up to 16 models to share a single accelerator, depending on regional traffic patterns. For high-traffic regions like Asia, models are kept separate to ensure performance. For lower-traffic regions like South America, the same models can be safely grouped together. This approach, combined with aggressive grouping of low-traffic models and deprecation of unused recognizers, generated substantial additional savings.
Building Voice Technology That Includes Everyone
Shao’s work is not measured only in hardware. He also made a direct contribution to accessibility, a tangible public benefit with real human impact. He supported the launch of a dysarthric recognizer, a feature specifically designed for users with speech difficulties. Dysarthria, a condition often associated with neurological disorders such as Parkinson’s disease, cerebral palsy, or stroke recovery, affects millions of Americans. Conventional speech recognition systems frequently fail to understand dysarthric speech, effectively excluding this population from using voice-enabled technologies. Shao helped build the speech pipeline that connects the recognizer to the backend model hosting server. The feature enables critical voice interaction capabilities, from voice commands to dictation, for people who cannot reliably use conventional speech recognition. For veterans recovering from strokes, for individuals living with cerebral palsy, or for elderly users whose speech has been affected by age-related conditions, this technology represents more than convenience. It can mean independence and access. The dysarthric recognizer is now deployed in production, making voice technology more inclusive for millions of potential users in a country where accessible technology has become an increasing priority across both public and private sectors.
What Efficient AI Infrastructure Means for the Future
Taken together, Shao’s work addresses two fundamental challenges facing the future of AI in the United States. The first is the need to deploy advanced models efficiently enough to maintain global competitiveness without runaway computing costs. The second is the imperative to ensure that these technologies serve all Americans, including those with disabilities. Industry experts believe that such infrastructure-level ingenuity, often invisible alongside high-profile model breakthroughs, will ultimately determine which companies and nations can sustain AI leadership responsibly in the coming decade.
Looking ahead, Shao shows no signs of slowing down. He is currently focused on further refining the automated release workflow for foundation models, aiming to reduce the deployment cycle from days to hours. He also sees potential to apply his grouping and autoscaling techniques to other types of large models beyond speech, including multimodal systems that combine text, image, and audio understanding. “Infrastructure is never ‘done’,” he notes. “As models grow larger and use cases multiply, the need for efficiency only becomes more urgent. I want to keep building systems that allow AI to scale without waste, both for the companies that deploy it and for the people who rely on it.” For Ke Shao, the work is not just about engineering. It is about creating lasting infrastructure that will support the next generation of AI applications.
