Edge AI: Small Is the New Large
embedUR CEO Rajesh Subramaniam Says NPUs Will Make Edge Devices Intelligent
The computing power in today's smartphones and tablets rivals laptops from two years ago. By next year, smartphones will contain AI chips, paving the way for similar advancements in edge computing devices, said Rajesh Subramaniam, CEO at embedUR.
The plan is set in motion: Semiconductor manufacturers are designing low-power chips and neural processing units, NPUs; embedded software stack and operating systems are already available; and large language models will be "stripped down" to small or micro language models to enable intelligent processing at the edge. Custom applications can be developed using the chip manufacturer's software development kit, or SDK.
"[Edge] devices are going to get compute power that is unprecedented," Subramaniam said. "Today, all the semiconductor manufacturers are developing NPUs for the edge. Until now, NPUs were only used in laptops and servers."
Use Cases Across Industries
Edge AI devices offer advancements across industries. In sports, for example, players' movements on the field can be tracked through smart badges pinned to their shirts, providing data for analysis. In the healthcare industry, smart badges equipped with sensors can monitor the vital signs of patients, alerting doctors or relatives in cases of emergencies. For example, an alarm on the badge can remind a diabetes patient to take an insulin shot.
The logistics and manufacturing industries could benefit from autonomous decision-making by intelligent AI-powered devices, eliminating the need to send data back to the cloud for analysis. Intelligent warehouse management systems could autonomously control the movements of robots and packaging machines, manage inventory and workflows, and streamline product flows.
There are applications in the automotive industry too. Sensors and computer vision technology are already used in driverless vehicles. Other potential applications are anti-collision systems, predictive maintenance, fuel efficiency and monitoring driver behavior.
The technologies driving these advancements include AI-enabled chips, NPUs, embedded operating systems, the software stack and pre-trained models. Collectively, they form a SoC - system on chip.
Software, hardware and applications are key to enabling an intelligent device at the edge. The embedded software stack in the chip brings it all together and makes it work. Silicon Valley-based embedUR specializes in creating software stacks for bespoke edge devices, acting as a "software integrator" that collaborates closely with chip manufacturers to build custom solutions.
"We have the ability to build managed software, as well as build individual software stacks for small, medium and large devices. You can think of us as a virtual R&D team," Subramaniam said.
Advances in NPUs
NPUs can perform AI computations more efficiently than GPUs, and before long, will become standard in PCs, smartphones and edge computing devices.
In May, this year, Microsoft announced that its Copilot+ PCs will use NPUs to bring AI capabilities to endpoint devices such as laptops. An NPU embedded in Qualcomm's Snapdragon X series processor will process AI-centric workloads and work with the CPU and GPU in the PC. It will enable generative AI processing in the device itself, without the need for the PC to connect to the cloud for processing AI workloads on a cluster of servers.
Qualcomm's competitors are not far behind in NPU development. According to an article by Tom's Hardware, AMD will offer its Ryzen Hawk Point platform, with its NPU sporting 16 TOPS of compute.
ISMG found that NXP Semiconductor offers the eIQ Neutron NLP for machine learning acceleration, while Cypress Semiconductor, now an Infineon company, builds application-specific integrated circuits for customer-specific applications. And Synaptics has already launched its Astra SL- and SR-series chips for edge AI.
Several operating systems also cater to embedded systems, including Linux, BlackBerry QNX, Integrity OS, VxWorks, FreeRTOS, LiteOS, and Zephyr. Each OS offers unique features suited to specific applications.
But, integrating NPUs into edge computing devices poses challenges, including managing power consumption and memory footprint.
Small Language Models for Edge AI
Unlike LLMs that are trained on massive datasets for general purposes, intelligent edge devices will be powered by small language models, SLMs, that are trained for specific tasks.
According to an article in the Wall Street Journal, OpenAI, Google and startups Anthropic, Mistral AI and Cohere have released smaller models this year. Microsoft led the way when it launched its family of SLMs named Phi, which it said was 1/100th the size of the model behind ChatGPT at the time.
OpenAI released a smaller version of the ChatGPT language model called GPT-4o mini, set to be 60% cheaper than GPT-3.5. But smaller does not mean less powerful, in terms of AI processing.
Despite their smaller size, SMLs possess substantial reasoning and language understanding capabilities. For instance, Phi-2 has 2.7 billion parameters, Phi-3 has 7 billion, and Phi-3 mini has 3.8 billion.
The next series of language models could be called "micro language models" and will be created especially for low-power edge AI devices.
While it is still early days for edge AI, industry experts expect many product announcements in the next 18 to 24 months. To quote another article from the Journal, "... tech giants and startups are thinking smaller as they slim down AI software to make it cheaper, faster and more specialized."
Subramaniam said the applications and use cases will drive wider industry adoption of edge AI.
