Toronto Global Forum: The Foundations of Intelligence

The physical infrastructure supporting artificial intelligence (AI) is facing new demands as the industry shifts from targeted usage to scaled deployment, including deploying agentic AI at scale. “The Foundations of Intelligence,” a session atthe20th annual Toronto Global Forum, organized by the International Economic Forum of the Americas (IEFA), explored how data center architecture needs to evolve to support this transition.

The one-on-one conversation featured Mathew Mehrotra, Group Head, Canadian Personal & Business Banking, and Co-Head, Canadian Personal & Commercial Banking at BMO, speaking with Dr. Tony Chan Carusone, Senior Vice President of Technology at Qualcomm.

Listen to an episode of Markets Plus based on insights from this conversation:

Here are the highlights.

“The moment that we’re in right now is this transition from initially research, then into experimentation,” noted Mathew Mehrotra. “And now this very high expectation around value creation in businesses, value creation for clients.”

To meet those expectations, AI companies are increasingly shifting their focus from model training to inference and commercialization, as companies look to demonstrate the real-world value of the technology, said Dr. Tony Chan Carusone.

This shift is happening because of the rapid rise of agentic AI workloads. As Chan Carusone explained, it’s a different kind of workload, where models and prompts are transformed into tokens to generate value. That process is creating new opportunities for differentiated hardware systems that are better suited to these workloads, he explained.

“That’s why you’re seeing a lot of excitement and interest in hardware,” he said. “When it comes to Canada in particular, it's almost it feels almost like we saw this coming for a long time, because we have this unique confluence of skills and expertise.”

The agentic workflows are at an inflection point, with each agentic call spawning dozens of inference tasks. Each prompt can generate between 50 and 100 times more compute activity than pre-agentic workloads, explained Chan Carusone.

Previously, AI compute worked by taking commands from the internet and the CPUs would do the work, supported by GPUs and other accelerators. That model is changing, he explained, noting that now GPUs are doing the planning through inference to figure out what needs to happen, generating a ton of code for CPUs to run. That’s creating the new opportunities for hardware, he said.

The rapid growth in the size of agentic workloads, along with the rising in physical and energy requirements as the industry scales has led to three memory walls, according to Chan Carusone: the supply chain limits on high-bandwidth memory (HBM), the connectivity memory wall, where the compute has grown faster than the ability to move that data, and the capacity needs of memory have outpaced the physical memory capacity in HBMs.

The industry is facing power consumption bottlenecks. “There’s this desire to take as much compute as you can and pack it within as small volume as you can,” Chan Carusone said. “A megawatt of power being dissipated in the volume of a large refrigerator. I mean, that’s astounding.”

Packing high-density compute into compact footprints creates significant engineering and power-management challenges. Chan Carusone pointed to advances in custom silicon, connectivity and memory architecture as potential ways to improve the performance and energy efficiency of AI infrastructure.

Both Chan Carusone and Mehrotra also examined how Canada could build on its strengths in talent, energy, and technology to capture more of the economic value created by AI. The cool climates and availability of low-carbon power lend themselves to data center development, and there is a wealth of intellectual talent, from academia to commercialization, that gives Canada the building blocks of AI infrastructure.

Chan Carusone also stressed the importance of talent development, particularly in semiconductor engineering and related disciplines. While concerns about brain drain persist, he argued that Canada should remain focused on attracting and retaining highly skilled global talent through its universities and research programs, viewing that as a critical competitive advantage.

In Mehrotra’s view, the opportunity for Canada lies in building on these strengths to capture greater value. Chan Carusone echoed that sentiment.

Instead of only being a destination for data centers, Canada should focus on high-value hardware and intellectual property. “We have these strategic ingredients that we can double down on to be a key part of global supply chains,” said Chan Carusone. “By developing the chips that are the brains of this new age, we can help author this new digital age instead of just being a player in it.”