PCIe 7.0 fundamentals: Baseline ordering rules

Adding more compute is no longer enough to maximize AI training and inference performance in today’s AI factories. The real challenge is how efficiently data flows through AI systems, not raw processing power.

Training remains the foundation of AI development, and maximizing throughput across large clusters is critical as models internalize structure, learn statistical relationships, and establish a baseline for downstream workloads. Inference shifts the focus, demanding ultra-low latency and high-reliability token generation. Both of these phases are characterized by exponential scale.

Training trillion-parameter models and performing inference that requires enormous amounts of contextual information places significant pressure on the “plumbing” of modern computing platforms. In this environment, the efficiency, predictability, and speed of data movement across the CPUs, accelerators, memory, and I/Os that compose these AI systems have become the true bottleneck.

Eliminating data bottlenecks is now dependent on optimizing interconnect bandwidth. The interconnects themselves are crucial to system success, with PCI Express (PCIe), specifically PCIe 7.0, being a prime example.

PCIe’s role in multi-GPU scale-up systems

For more than a decade, PCIe has been the backbone of multi-GPU server systems due to its universal and extensible interconnect that ties all compute and I/O devices together inside a node, such as GPU-to-DPU/NIC, or GPU-to-switch. Even as newer, high-bandwidth GPU-to-GPU fabrics and proprietary accelerator meshes have emerged, systems continue to rely on PCIe for baseline connectivity, system bring-up, and data movement across system components.

Announced in June 2025, PCIe 7.0 doubles link bandwidth to 128 GT/s, delivering up to 512 GB/s of bi-directional throughput per x16 connection. While this increased bandwidth helps alleviate I/O bottlenecks for AI computing, fully utilizing PCIe 7.0 for inference workloads also requires minimizing latency across the fabric.

Multiple inference streams sharing the same PCIe path may cause head-of-the-line blocking due to unnecessary serialization. This results in delays in unrelated traffic, which impacts overall system efficiency.

To maintain low latency and fully utilize PCIe 7.0 bandwidth under parallel workloads, a more flexible ordering model is required.

Baseline PCIe ordering rules: Why serialization exists

First, it’s helpful to understand PCIe’s baseline ordering rules. Most systems using early PCIe generations—from 2.5 GT/s in PCIe 1.0 to 8 GT/s in PCIe 3.0—relied on simple point‑to‑point connections supporting a single application or device context. As a result, the PCIe protocol strictly enforced baseline ordering rules to ensure that the results of memory operations are presented in an order that matches software expectations.

Within a single traffic class, PCIe groups transaction-layer packets (TLPs) into posted, non-posted, and completion categories, each governed by defined ordering constraints. Posted requests are memory writes (MWR) and messages (MSG) that operate without needing a completion, while non‑posted requests include memory reads (MRD) and configuration transactions that must receive a completion. To simplify the discussion, the focus is on the main traffic. Only MRD requests, MWR requests, and read completions (CPL) are described in the ordering rules.

Table 1 Baseline ordering rules highlight the relationship between the current and previous requests. Source: Cadence Design Systems

Table 1 shows the relationship between the current requests in rows A, B, and C and the previous requests in columns 2, 3, and 4. The “Y/N” in the table is the abbreviation of “Yes/No” that implies the row’s request/completion “may pass” the column’s request/completion type.

To understand the A3 deadlock scenario detailed in Figure 1, assume the root complex (RC) issues an MRD request (1) followed by an MWR request (2) toward the endpoint (EP) device. A deadlock can occur when the RC exhausts the completion credits, and its completion buffer becomes full (3). The RC is unable to accept new completions (4) associated with the outstanding non-posted MRD (1).

Figure 1 In A3 deadlock, the RC completion queue (CQ) is full, preventing it from returning completion to release the MRD from blocking the MWR request. Source: Cadence Design Systems

Because strict ordering prevents the newer MWR (2) from bypassing the unresolved MRD (1) until its completion (4) is received, the RC’s transmit request path is also blocked. This prevents the issuance of MWR (2) from propagating to the EP link (5). This head-of-the-line blocking creates circular dependency, which stalls internal request queue draining and completion acceptance. Unless the MWR is allowed to bypass the MRD, a deadlock results.

For the C3 deadlock scenario illustrated in Figure 2, assume both the RC and EP issue many non-posted read requests (1), which aggressively fill both the RX and TX request queues (RQ) (2) and prevent them from accepting any new MRD requests (3). Meanwhile, the completions (4) are returned for pending MRD requests (1) in the opposite direction, but they can’t be forwarded to fulfill the previous pending MRD request (1). This is because they arrived behind the new MRD request (3). If the completion is not allowed to bypass the previous MRD requests in the same direction, it will result in a deadlock.

Figure 2 A C3 deadlock occurs if both the RX and TX request queues are full, and the completion is not allowed to pass the previous MRD request. Source: Cadence Design Systems

For A2 (Row A, column 2), B2, and C2, MWR, MRD, and CPL requests cannot pass MWR requests to maintain correctness. These three scenarios are illustrated in Figure 3, Figure 4, and Figure 5, respectively.

Figure 3 In A2, current MWR requests cannot pass previous MWR requests. Source: Cadence Design Systems

Figure 4 In B2, current MRD requests cannot pass previous MWR requests. Source: Cadence Design Systems

Figure 5 In C2, current completion requests cannot pass previous MWR requests. Source: Cadence Design Systems

However, A3 and C3 illustrate that both MWR requests and completions can pass an earlier MRD request to avoid a deadlock. This is shown in Figure 6 and Figure 7.

Figure 6 In A3, MWR requests are allowed to bypass previous MRD requests to avoid a deadlock. Source: Cadence Design Systems

Figure 7 In C3, completions are allowed to bypass previous MRD requests to avoid a deadlock. Source: Cadence Design Systems

For B3, the current MRD request might be bypassed or blocked by the previous MRD request. For A4 and B4, the current MWR or MRD request is permitted to pass the previous completion or be blocked by the completion.

Figure 8 describes both the C4a and C4b scenarios. The yellow and green completions belong to the pending yellow (RD1) and green (RD0) MRD requests, respectively. If current completions belong to different MRD requests, they can pass each other as CPL10 passes CPL01 (C4a scenario). However, if they belong to the same MRD request as in the C4b scenario, they must follow the order and cannot pass previous completions (CPL00 followed by CPL01, and CPL10 followed by CPL11).

Figure 8 In C4, completions of the same MRD request can pass completions of the previous MRD, and completions of the same MRD request must follow in order. Source: Cadence Design Systems

Strict ordering: A safe but conservative baseline

PCIe’s default strict ordering rules ensure safe producer-consumer software behavior. Under strict ordering, the system observes transactions issued by a requester in program order. Posted writes must be completed before subsequent read completions of subsequent pending MRD requests.

However, this global ordering discipline is conservative. It causes unrelated transactions to wait for one another, even when there is no true data dependency. For instance, this can occur when different functions access data from different memory segments in the host or local memory. As PCIe link speeds increase, this approach becomes a scalability bottleneck because it causes head-of-the-line blocking of unrelated serialized traffic and underutilizes the available bandwidth.

Why relaxed ordering is needed

Relaxed ordering loosens these global constraints. When a transaction is marked as relaxed, it tells the PCIe fabric that this MRD or MWR request does not need to participate in the default system‑wide ordering guarantees. Relaxed ordering improves throughput and reduces latency by enabling certain transactions to be reordered. The key point is that relaxed ordering removes unnecessary ordering barriers between independent operations.

However, it still preserves most of the transactional correctness, such as ensuring the completion order of the same MRD requests. This is especially valuable for workloads such as prefetching, polling reads, or accelerator traffic, where software already explicitly manages synchronization. Relaxed ordering addresses the performance loss caused by overly strict global rules. However, it treats ordering as a binary choice, either fully ordered or globally relaxed.

Why ID‑based ordering is also necessary

Relaxed ordering alone is too coarse‑grained for modern devices. High‑performance endpoints, such as GPUs, NICs, and NVMe controllers, generate traffic from many independent sources, including queues, processes, virtual machines, or process address space IDs (PASIDs).

These sources often require ordering within themselves, but not between each other. With ID‑based ordering, PCIe maintains ordering among transactions that share the same requester ID or PASID, while permitting reordering across different IDs. In effect, it scopes ordering guarantees to a logical context rather than imposing them system‑wide.

This allows transactions of the same function or context to maintain the correct program semantics, while the fabric freely parallelizes traffic across independent functions or contexts. Without ID-based ordering, systems would be forced to choose between full serialization for safety or full relaxation with no per‑context guarantees.

Attribute-based ordering: Relaxed and ID-based ordering

Because the PCIe fabric already enforces ordering semantics for both relaxed ordering and ID‑based ordering, system software and device logic influence these behaviors by setting attributes in the TLP headers rather than redefining the rules themselves. Relaxed ordering and ID‑based ordering address different dimensions of the same problem, which is why both are required to meaningfully relax PCIe’s strict ordering rules.

Relaxed ordering removes unnecessary global ordering constraints between different classes of traffic, enabling better scheduling and reducing head-of-the-line blocking. In contrast, ID‑based ordering refines ordering to the level of a requester or context, preserving correctness where the associated software expects it while eliminating artificial dependencies elsewhere.

Together, they allow PCIe to scale with modern parallel workloads. While strict ordering provides a safe default, relaxed ordering removes global bottlenecks, and ID‑based ordering preserves local semantics without sacrificing concurrency. This combination allows PCIe to support today’s accelerators, virtualized I/Os, and high‑throughput devices without breaking the programming models that software relies on.

Table 2 Here is a highlight of the ordering rules for relaxed ordering and ID-based ordering. Source: Cadence Design Systems

Table 2 specifies the ordering rules for relaxed ordering and ID-based ordering. The “Y/N” in the table is the abbreviation for “yes/no”, indicating whether the row’s request/completion “may pass” the column’s request/completion type.

The key differences between the relaxed ordering and ID-based ordering rules detailed in Table 2 and the baseline rules shown earlier in Table 1 are A2, B2, and C2 vs. D2, E2, and F2, respectively. For baseline rules, current MWR, MRD, or completions are not allowed to pass previous MWR requests. However, relaxed ordering and ID-based ordering allow them to pass previous MWR requests if their request IDs—bus, device, function, and PASID—are different.

Vanessa Do is a senior product marketing manager for PCIe IP at Cadence with over 20 years of experience in PCIe design, system validation, and customer engagement. Her background spans PCIe protocol development, FPGA-based customer support, and leading cross‑functional teams to debug complex PCIe issues at the system level.

Editor’s Note

This is Part 1 of the article series about PCIe 7.0 fundamentals. Part 2 will explain why PCIe 7.0 bandwidth alone isn’t enough while highlighting the importance of addressing legacy ordering limitations with UIO.

The post PCIe 7.0 fundamentals: Baseline ordering rules appeared first on EDN.