SoC design: When a network-on-chip meets cache coherency

Many people have heard the term cache coherency without fully understanding the considerations in the context of system-on-chip (SoC) devices, especially those using a network-on-chip (NoC). To understand the issues at hand, it’s first necessary to understand the role of cache in the memory hierarchy.

Cache in the memory hierarchy

Inside a CPU are a relatively small number of registers with extremely high speed. These registers can be accessed by the CPU in a single clock cycle. However, their storage capacity is minimal. In contrast, accessing the main memory for reading or writing data takes up many clock cycles. This often results in the CPU being idle most of the time.

In 1965, a British computer scientist, Maurice Wilkes, introduced the concepts of cache memory and memory caching. This involved having a small amount of fast memory called a cache adjoining the CPU. The word “cache” itself comes from the French word “cacher,” meaning “to hide” or “to conceal,” the idea being that the cache hides the main memory from the CPU.

This process operates based on two key points. First, when a program running on the CPU does something involving one location in the main memory, it typically performs operations on several nearby locations. Consequently, when the CPU requests a single piece of data from the main memory, the system brings in data from nearby locations.

A high-level view of a memory hierarchy involving a simple cache is illustrated in Figure 1.

Figure 1 High-level view shows where cache stands in the memory hierarchy. Source: Arteris

This approach ensures that related data is readily available if needed. Second, programs usually conduct numerous operations on the same data sets. Therefore, storing the actively used data in the cache closest to the CPU is beneficial. This proximity allows quicker access and processing of the data.

Cache in the context of an SoC

In the case of an SoC, the cache is implemented on-chip in high-speed, high-power, and low-capacity SRAM. Meanwhile, the main memory is implemented off-chip on the PCB, typically in the form of low-speed, low-power, and high-capacity DRAM.

To minimize latency, designers have added multiple levels of cache in many of today’s SoCs. These typically include three levels: L1, L2 and L3. The L1 cache is closest to the CPU and has the smallest capacity but the fastest access times, usually within 1-2 clock cycles. The L2 cache is a bit further from the CPU and offers higher capacity but slower access times, generally between 4-10 clock cycles. The L3 cache is still further from the CPU and provides the largest capacity among the three, but has the slowest access times, ranging from 10-30 clock cycles.

Multiple cache levels maximize performance while minimizing off-chip accesses to the main memory. Accessing this main memory can consume hundreds of clock cycles. By using multiple cache levels, data can be retrieved more quickly from these caches rather than the slower main memory, enhancing overall system efficiency.

The complexity of all this increases when multiple CPU cores are involved. Consider a common scenario with a cluster of four CPU cores, labeled as cores 0 to 3, each with its own dedicated L1 cache. In some implementations, each core will also have its own dedicated L2 cache, while all four cores share a common L3 cache. In other designs, cores 0 and 1 share one L2 cache, cores 2 and 3 share another L2 cache, and all four collectively use the same L3 cache. These varying configurations impact how data is stored and accessed across different cache levels.

Typically, all processor cores within a single cluster are homogeneous, meaning they are the same type. However, having multiple clusters of processor cores is becoming more common. In many cases, the cores in different clusters are heterogeneous, or of different types. For example, with Arm’s big.LITTLE technology, the “big” cores are designed for maximum performance but are used less frequently.

The “LITTLE” cores are optimized for power efficiency with lower performance and are used most of the time. For instance, in an Arm-based smartphone, the “big” cores might be activated for tasks like Zoom calls, which are relatively infrequent. In contrast, the “LITTLE” cores could handle more common, less demanding tasks like playing music and sending text messages.

Maintaining cache coherency

In systems where multiple processing elements with individual caches share the same main memory, it’s possible to have multiple copies of the shared data. For example, one copy could be in the main memory and more in each processor’s local cache. Maintaining cache coherency requires that any changes to one copy of the data are reflected across all copies. This can be achieved by updating all copies with the new data or marking the other copies invalid.

Cache coherency can be maintained under software control. However, software-managed coherency is complex and challenging to debug. Still, it can be achieved using techniques such as cache cleaning, whereby modified data stored in a cache is marked as dirty, meaning it must be written back to the main memory. Cache cleaning can be performed on the whole cache or with specific addresses, but it is costly in CPU cycles and must be performed on all CPUs holding a copy of the data.

The preferred way to maintain cache coherency is with special hardware built to manage the caches invisibly from software. For example, the caches associated with the cores in a processor cluster typically include hardware required to maintain cache coherence.

To use or not to use

SoCs are composed of large numbers of functional blocks called intellectual property (IP) blocks. A processor cluster would be one such IP block. A common way to connect the IP blocks is to use a NoC.

In many SoC designs, coherence isn’t needed outside of the processor cluster, allowing a non-coherent or IO-coherent AXI5 or AXI5-Lite NoC, such as NI from Arm or FlexNoC from Arteris. However, for SoC designs with multiple processor clusters lacking inherent cache coherence or when integrating third-party IPs or custom accelerator IPs that require cache coherence, a coherent NoC is needed. Examples include CMN from Arm using the AMBA CHI protocol or Ncore from Arteris using AMBA ACE and/or CHI.

Figure 2 In the above example, the main system employs a coherent NoC in conjunction with a safety island employing a non-coherent NoC. Source: Arteris

Applying cache coherency universally across the entire chip can be resource-intensive and unnecessary for specific components. Therefore, isolating cache coherency to a subset of the chip, such as CPU clusters and specific accelerator IPs, allows for more efficient use of resources and reduces complexity, as shown in Figure 2. Coherent NoCs like Ncore excel in scenarios where stringent synchronization is necessary. Meanwhile, non-coherent interconnects, such as FlexNoC, are ideal in scenarios where strict synchronization is unnecessary.

Designers can strategically balance the need for data consistency in specific areas while benefiting from more streamlined communication channels where strict coherence is unnecessary. In today’s sophisticated heterogeneous SoCs, the synergy between coherent and non-coherent interconnects becomes a strategic advantage, enhancing the overall efficiency and adaptability of the system.

Andy Nightingale, VP of product management and marketing at Arteris, has over 36 years of experience in the high-tech industry, including 23 years in various engineering and product management positions at Arm.

<!–
googletag.cmd.push(function() { googletag.display(‘div-gpt-ad-native’); });
–>

The post SoC design: When a network-on-chip meets cache coherency appeared first on EDN.

SoC design: When a network-on-chip meets cache coherency

Become a member

Become a member

Become a subscriber

Become a sponsor