China's BenZhi Activation Leads Charge in Full-Stack On-Device AI Innovation

AsianFin -- BenZhi Activation, an AI startup incubated by the Institute of Parallel and Distributed Systems (IPADS) at Shanghai Jiao Tong University, is pioneering a full-stack native on-device AI solution designed to transform personal computing.

Founded and led by Associate Professor Ze-Yu Mi, the team draws on IPADS’s global leadership in operating and distributed systems research, backed by world-class expertise in large-scale on-device AI models and infrastructure.

Their portfolio includes internationally recognized open-source projects like PowerInfer and SmallThinker, demonstrating strength in both cutting-edge AI development and engineering implementation.

As AI technology reshapes industries, BenZhi Activation aims to revolutionize how billions of devices—including PCs, smartphones, and smart terminals—process data by embedding advanced AI capabilities locally, rather than relying on cloud-based models. Cloud AI currently faces significant challenges around privacy, latency, and personalization.

Uploading sensitive personal data to cloud servers raises security concerns and discomfort among users reluctant to entrust their “digital lives” to third parties. Additionally, cloud interactions incur high costs and latency that limit frequent, in-depth AI use. Finally, generalized cloud models struggle to deeply personalize experiences, lacking the ability to learn continuously from individual user data while maintaining privacy.

In response, BenZhi Activation introduces a disruptive, native on-device approach that rebuilds the AI software and hardware stack from the ground up. This strategy bypasses traditional model compression methods, enabling powerful AI models to run efficiently and securely on end-user hardware without sacrificing performance.

The company has achieved collaborative breakthroughs across on-device large model algorithms, infrastructure systems, and hardware optimization, delivering AI that is both highly capable and fully private.

BenZhi Activation has already delivered world-leading innovations. In December 2023, they released PowerInfer, an on-device AI infrastructure capable of running tens-of-billions-parameter models on consumer NVIDIA GTX 4090 GPUs with performance near that of data-center A100 GPUs, boasting inference speeds over 11 times faster than prior methods. This open-source project quickly rose to the top of GitHub’s global trending list. By June 2024, their PowerInfer-2 system, incorporating a proprietary TurboSparse sparsification method, enabled a 4.7-billion-parameter model to run smoothly on smartphones, outpacing the international benchmark llama.cpp by 29 times and marking a leap from desktop to mobile deployment at scale.

Looking ahead, BenZhi Activation will collaborate with Shanghai Jiao Tong University to release the world’s first batch of native large AI models pre-trained and architected explicitly for edge deployment. These models address the tight computational, memory, and storage constraints of low-cost hardware, allowing seamless operation of tens-of-billions-parameter models on devices costing only a few hundred yuan. This milestone exemplifies the team’s full-stack capabilities, from foundational algorithm design to on-device infrastructure. Their earlier release of SmallThinker, a 3-billion-parameter reasoning model optimized for on-device use, achieved over 100,000 downloads on HuggingFace within a week and ranked second globally among trending AI models.

Leading venture capitalists praise BenZhi Activation’s innovative approach. Chen Yu, Partner at Yunqi Partners, noted that the company effectively bridges the gap between powerful AI models and mainstream device computing capabilities, accelerating low-cost and efficient AI deployment. Liu Shui, Managing Director at Baidu Ventures, emphasized the significance of coordinated optimization across models, systems, and hardware to reduce AI costs and deliver real-time, private AI experiences on smart devices. Huang Xinxin from Lighthouse Capital highlighted BenZhi Activation’s rare combination of top-tier R&D and production expertise, positioning the startup as a global leader in edge AI innovation.

BenZhi Activation’s pioneering native on-device AI technology signals a fundamental shift in the industry, empowering billions of users with secure, responsive, and deeply personalized AI experiences directly on their devices. This approach is set to redefine the future of personal intelligence, putting advanced AI capabilities squarely in users’ hands.