Cloud to edge AI: If and when, to what degree, and how and why

AI processing, inference specifically, remains mostly a cloud-centric function, but not for long. Whether it shifts to client devices in tandem or completely, who will win?

Way back in December 2022, in the introduction to a teardown of an autonomous object-tracking accessory for a smartphone’s integrated camera, I more broadly positioned the trade-off between cloud- and edge-based AI processing, with the latter exemplified by the widget.

One common way to reduce the device BOM is to offload as much of the total required processing, memory and other required resources to other connected devices. A “cloud” server is one common approach, but it has notable downsides that also beg for consideration from the device supplier and purchaser alike, such as:

  • Sending raw data up to the “cloud” for processing, with the server subsequently sending results back to the device, can involve substantial roundtrip latency. There’s a reason why self-driving vehicles do all their processing locally, for example!
  • Sending data up to the “cloud” can also engender privacy concerns, depending on exactly what that data is (consider a “baby cam”, for example) and how well (or not) the data is encrypted and otherwise protected from unintended access by others.
  • Taking latency to the extreme, if the “cloud” connection goes down, the device can turn into a paperweight, and
  • You’re trading a one-time fixed BOM cost for ongoing variable “cloud” costs, encompassing both server usage fees (think AWS, for example) and connectivity bandwidth expenses. Both of those costs also scale with both the number of customers and the per-customer amount of use (both of each device and cumulatively for all devices owned by each customer).

Those points all remain valid nearly four years later, IMHO at least, although an added cloud-based set of downsides now also begs for inclusion in the list. Back then, the AI data center (also called a hyperscaler) build-out phenomenon was still in its comparative infancy, with much of the compute capability then coming online intended for cryptocurrency mining (which ironically has subsequently been redirected to deep learning processing). Nowadays, conversely, the data center phenomenon is in fuller stride, with understandable backlash from the broad populace.

  • Speculative investment deals, often “circular” in nature (a hardware supplier invests in a data center developer, for example, with the latter promising to use the former’s chips and software in exchange), result in frequent stock market volatility and broader growing “bubble market” concern.
  • Governmental jurisdictions provide lucrative tax breaks and other incentives to data center developers, even though any tangible increase in employment for the local populace will be time-restricted to only the subsequent duration of construction.
  • Data center energy demands are substantially incremental, necessitating in-parallel built-outs of additional power plant facilities whose costs are assumed by all.
  • The incremental energy predominantly comes from hydrocarbon-based power sources such as coal and natural gas, with substantive pollution and noise impacts.
  • Water demands for cooling purposes are equally sizeable, redirecting and draining aquifer reserves no matter whether the sources are fresh, recycled, or a mix of these.
  • And the facilities are massive in size, gobbling up available open space as well as fields formerly used for various agricultural purposes.

Edge demands

And here’s what I wrote about the edge processing alternative back in late 2022.

Another popular BOM-slimming approach involves leveraging a wired or (more commonly) wireless tethered local device with abundant processing, storage, imaging, and other resources, such as a smartphone or tablet. This technique has the convenient advantage of employing a device already in the consumer’s possession, which he or she has already paid for, and for which any remaining “cloud” processing bandwidth involved in implementing the complete solution he or she will also bankroll. The latency is also notably less than with the pure “cloud” approach, privacy worries are lessened if not fully alleviated, and although the smartphone’s connection to the “cloud” may periodically go down, the connection between it and the device generally remains intact.

All still valid, too, IMHO. And add to the list the facts that the alluded-to enhanced privacy not only applies in a general sense but also specific to keeping personal data out of the “cloud” provider’s training data set, and that the cessation of ongoing-cost outflows involves not only hardware but also monthly expenses for AI service subscriptions (if you’re a consumer) or tokens (for developers). However, the only downside I mentioned back then was the following.

The notable requirement for the manufacturer here is that whatever device-offloading application is running on the tethered mobile device must comprehend all common mobile operating systems (predominantly Android and iOS, along with the related iPadOS for tablets, nowadays) and the various devices that run on them, and it must also evolve as both operating systems and hardware evolve. Otherwise, the application will sooner-or-later stop meaningfully working, as will the device it’s tethered to.

Admittedly, in my defense, the device I was showcasing in that teardown heavily depended on a smartphone-based app, so it made at least some sense to focus on that nuance. But although I noted that the app leveraged the smartphone’s built-in image capture and processing resources, thereby obviating the need for costly redundancy in the accessory, I overlooked the not-insignificant and growing burden of that processing, along with the associated necessity for adequate memory—both nonvolatile for storage and volatile for code execution and temporary data—and how those memory capacity and speed needs might expand in the future.

Now we know. Not only are the not-unlimited supplies of SSDs (along with HDDs, interestingly, trading off raw performance and energy consumption for raw capacity and cost-per-bit) and DRAM being redirected to data centers, the required capacities of each in PCs, smartphones and the like for today’s edge AI processing functions are increasing in lockstep. Take, for example, the Gemini AI models embedded in latest-generation Chrome browser installs, which take up at least 4 GBytes of storage space. Or the Apple Intelligence included with iOS 27, now required vs. optional with the version 26 O/S predecessor, and using up to 14 GBytes of capacity. The similarly required-by-default AI model suite included with MacOS 27? 12-30 GBytes in size, although open source can come to the rescue in this case.

While a company called PrismML, which specializes in quantizing and otherwise compressing deep learning models for use in resource-constrained systems, was able to shrink Alibaba’s Qwen3.8 27B open-source model by between 9-10x while still matching 98% of Qwen’s aggregate benchmark scores, the resultant model was still 5.9 GBytes. Perplexity’s Portable Computer for local on-computer model storage and processing? 20 GBytes. And of course, storage needs in both cloud and edge scenarios will further increase in the future.

Hybrid intermediaries

One of many ironies here is that a not-insignificant percentage of the processing resources in data centers find use, at least on a part-time basis, not for inference but for training—initial training, periodic retraining to counteract performance decay, iterative training to encompass new and expanded datasets, and transfer learning to adapt frontier models for specific tasks. OpenAI recently reported, for example, that the training run for its latest-generation “Astra” GPT-6 fully consumed more than 100,000 GPUs. These particular operations are compute-intensive and highly parallelizable, which explains why GPU-derived architectures have come to the fore in this particular application.

For inference, conversely, only the prefill portion of the total task is parallelizable, therefore GPU-amenable (albeit not necessarily GPU-optimal). The subsequent decode portion of the total inference pipeline is conversely memory-intensive, from both capacity and bandwidth standpoints, the latter rationalizing the ascendance of advanced HBM approaches.

That all said, any claim that inference is instead going to transition fully to the edge in all cases is, once again IMHO, a pipe dream. Autonomous vehicles and other specific scenarios? Sure. But there are plenty of other applications where latency and privacy (which can alternatively be handled via encryption, anyway) aren’t an issue but the need for access to sizeable trained models is, and servers rule in such cases, as I noted in late-2024 follow-up coverage.

Hybrid approaches, in which a base level of inference occurs locally with cloud resources tapped as-needed for beefier-function requirements, will be increasingly common going forward. Alternatively, other local-ish (or, if you prefer, local networked cloud) scenarios are also beginning to emerge. Take, for example, NVIDIA’s Personal AI Router (PAIR), which enables spreading the overall inference (including agents) processing task across multiple LAN-connected computers (including non-NVIDIA-based Macs). Or instead consider storing the models on a LAN-shared NAS for common access, although RAM limitations will likely preclude full inference processing there.

Revenge of the commodity (and the proprietary)

The obvious beneficiaries of these trends include memory and broader mass storage suppliers. I admittedly find it ironic that after countless commodity boom-and-bust cycles, several of which I personally lived through from the supplier side, the semiconductor memory industry may have finally found a path to a predictable and otherwise sustainable business model.

The other potential winners are inference chip and IP core suppliers. As mentioned earlier, whereas GPU-based architectures will likely continue to reign into the future from a training standpoint (if for no other reason than proprietary software tools prevalence), an ever-increasing percentage of total growth in the AI processing space will instead be in inference, where toolset lock-ins are more fungible and other parallel-processing, not to mention more memory-centric architectures may indeed end up more optimal for the necessary tasks.

What are your thoughts on the trends I’ve identified, their root causes and beneficiaries, and their short- and long-term outcomes? Let me and your fellow readers know in the comments!

—Brian Dipert is the associate editor, as well as a contributing editor, at EDN.

Related Content

 

The post Cloud to edge AI: If and when, to what degree, and how and why appeared first on EDN.

Amazon, Apple and Microsoft: New mobile computing platforms take off