CXL Type 3: Tooling and boot path from power-on to usable memory

Part 1 of this min-series established why CXL Type 3 memory expanders matter for capacity-bound workloads and where expander memory sits in the latency–capacity pyramid relative to local DRAM. It also explained what must align across the stack before memory becomes usable—CPU and BIOS enablement, kernel CXL support, device firmware, RAS paths, and the NUMA topology that Linux exposes through cxl_pci. Next, it delved into why many CXL problems surface, such as placement or bandwidth imbalance rather than obvious enumeration failures.

This part builds on that foundation with the tools and timeline you need day-to-day to navigate system bring-up. You will learn which user-space utilities reveal what the OS actually sees on the CXL fabric, how to differentiate between “device present” to “memory consumable,” and how to walk the boot sequence from slot power and DRAM training through DVSEC discovery, CEDT read, CDAT delivery, ACPI handoff, and finally driver bind-framing each stage as a validation checkpoint with recognizable failure signatures.

Let’s get to work.

Kernel drivers establish whether a CXL Type 3 device is present, configured, and represented as memory, but validation engineers spend much of their time during bring-up reconciling what firmware advertised, what the driver registered, and how user-visible policy (NUMA placement, DAX/region modes, namespace layout) matches the intended deployment. That reconciliation is difficult from dmesg and kernel logs alone. Practical programs rely on a small set of user-space utilities that expose sysfs and kernel abstractions in forms suitable for automation and field triage.

Essential user-space tooling

cxl/libcxl (often shipped with ndctl sources as “CXL tools”)

The cxl command-line interface and libcxl library walk the CXL sysfs hierarchy—ports, endpoints, memdevs, and decoders—and print structured output, commonly JSON. This is the closest thing to a standard “show me what the OS thinks on the CXL fabric” tool; serial numbers, capacity hints, which PCI function hosts a memdev, and whether decode topology looks sane before debugging performance or NUMA. It’s usually the first stop after dmesg when firmware and driver disagree about what should be visible.

ndctl

ndctl provides user-space administration for the LIBNVDIMM/regions/namespaces model that Linux also uses for some persistent-memory-class bring-up paths. Depending on kernel and platform integration, CXL-attached memory may surface as a separate NUMA node or through PMEM-style abstractions. ndctl lists regions, creates or destroys namespaces, and clarifies whether capacity is in a state software can consume, not merely whether a PCI device exists.

daxctl

daxctl manages direct-access (DAX) devices and related system-RAM or devdax configuration knobs exposed by the kernel. Some deployments expose memory through DAX-oriented paths, especially when treating capacity like PMEM/DAX rather than only anonymous DRAM. daxctl helps verify mode, online/offline behavior, and whether the system matches workload expectations. Misconfiguration here often looks like “memory is there but unusable, wrong interface, or wrong policy.”

numactl and numastat

numactl controls NUMA placement policy; numastat reports per-node memory statistics. Expander memory frequently lands as a separate NUMA node or as far memory relative to a socket. These tools prove placement hypotheses during bring-up, bind threads and allocations, measure local versus remote behavior, and catch cases where OS defaults silently place hot pages on CXL. Many “CXL is slow” bugs are NUMA policy bugs, not link bugs.

acpica-tools

This suite provides a useful utility, acpidump, which extracts ACPI tables from the kernel and dumps the raw values of the specified ACPI table. While a user should not need this during regular bring-up, it can be very useful in sticky situations when the DDR memory enumerates but does not show up either as a NUMA node or as a PMEM device. In such cases, it might be useful to dump certain acpi tables and parse raw values via a debug script.

lspci and setpci (pciutils)

Since CXL Type 3 memory expanders attach over a PCIe/CXL link, pciutils belongs in every bring-up kit alongside CXL-specific tools. lspci lists PCI functions on the bus, reports vendor and device IDs, class codes, negotiated link speed and width, and—when invoked with verbose flags—the extended capability chains that expose CXL and DVSEC registers. It’s often the fastest way to confirm that the endpoint is visible at the transport layer, that link training reached the expected generation and lane count, and that the kernel bound the intended driver (for example cxl_pci).

setpci reads and writes configuration-space dwords for targeted experiments during debug—checking capability offsets, toggling test bits where platform policy allows, or verifying that firmware left key control fields in the expected state. Used together, lspci answers “what does the bus see?” (peek) and setpci supports “can we inspect or adjust a specific config field?” (poke) before diverting attention to higher-level CXL utilities or firmware logs.

Topology and observability helpers

lstopo/hwloc produce human-readable CPU–memory topology maps—useful to confirm how the OS labels CXL memory relative to sockets. lspci/setpci (pciutils) confirm the PCI/CXL function at the bus level when debugging binding (cxl_pci versus overrides) and link issues. A verbose lspci dump reveals device capabilities that Part 3 decodes in detail.

Another, currently open-source, tool for viewing the PCIe hierarchy is pcicrawler, which shows the PCIe topology similar to lspci but in a nicer format.

From power-on to usable memory

The end-to-end path for a CXL Type 3 memory expander runs from first application of host and slot power to the point the operating system can issue CXL.mem accesses to host-managed device memory (HDM). Exact timing and responsibility splits vary by CPU, root complex, memory expander ASIC, and BIOS, but the dependencies recur. In other words, power and clocks before reset release; DDR readiness before credible capacity reporting; configuration-space discovery before decode programming; and table exchange before stable OS topology.

Boot sequence is shown for a system with CXL memory expander. Source: Author

  1. Power, clocks, and ASIC bring-up

The sequence begins when host platform and slot power are applied. The expander ASIC must reach an internally consistent state: regulators settle, oscillators stabilize, PLLs lock, and on-chip reset completes so an embedded control processor can execute first-stage firmware. The host must provide a stable PCIe reference clock and manage PERST# deassertion per PCIe/CXL electrical requirements, so the endpoint is not expected to train before clocks and power are valid. During this phase, the device is not yet advertising complete HDM metadata.

  1. On-device DRAM: controller release, training, and SPD

The ASIC releases reset to the DDR controller and run DRAM initialization and training for attached DIMMs. Firmware discovers configuration and capacity through serial presence detect (SPD) reads. In parallel, the ASIC initializes high-speed SerDes and the PCIe/CXL controller. There is a critical interval where HDM must not be treated as authoritative.

Firmware clears or gates HDM metadata until DRAM discovery completes—conceptually mem_info_valid = 0. Only after capacity and layout are known does firmware program HDM-related fields in PCIe extended configuration space via CXL-designated vendor-specific extended capability (DVSEC) structures and assert mem_info_valid = 1.

  1. PCIe link training, DVSEC, and HDM registration

As link training toward the host begins, firmware populates HDM capability structures through DVSEC containers—HDM instance count, per-region sizing, and validity flags. Setting “memory info valid” is the device’s contract that subsequent host reads from HDM descriptors consistent with trained DRAM.

  1. PCIe/CXL link up and configuration-space discovery

When the physical link reaches DL_Up at negotiated width and speed, the host enumerates the endpoint as a PCI function, parsing capability lists to discover CXL entries, DVSEC registers, and HDM decoders.

  1. Decode programming and mem_enable

Platform firmware must program host-side address decoding, so HDM contributes to the system physical address map. A common milestone is establishing the system physical address window and asserting memory enable (mem_enable). When mem_enable is recognized, device firmware may finalize the coherent device attribute table (CDAT) for OS/firmware NUMA heuristics.

  1. CDAT delivery via DOE and mailbox exchange

CDAT is typically transported using data object exchange (DOE) over CXL.io. The CXL mailbox command interface supports diagnostics and device management. Treat DOE/CDAT success and mailbox responsiveness as separate health checks.

  1. Firmware table construction and OS handoff

Host firmware synthesizes ACPI tables, including CXL Early Discovery Table (CEDT), System Resource Affinity Table (SRAT), and Heterogeneous Memory Attribute Table (HMAT), exposing HDM ranges as distinct memory affinity domains—often NUMA nodes.

  1. OS driver binding

On Linux, cxl_pci binds to the PCI/CXL function, exposes memdev objects, and enables memory to be onlined. Once complete, the host can issue CXL.mem loads, stores, and DMA through the programmed decode window.

Read the boot flow as a chain of implied tests—power/clock/PERST, DDR training, valid HDM, stable link, decode/mem_enable ordering, CDAT/DOE liveness, ACPI coherence, driver bind, and memory online. Failures produce characteristic signatures at each stage.

Part 3 applies this framework to hands-on test and debug: lspci field interpretation, NUMA verification with numactl, memory-mode configuration with daxctl, and workload tools for bandwidth and stress validation.

Ameet Sanghavi works in post-silicon validation for PCIe and CXL at Nvidia with a focus on interface bring-up and validation on shipping products. He has worked on PCIe since 2005 (from PCIe 1.1 onward) and on CXL since 2020 (from CXL 1.1 onward).

Editor’s Note

Part 1 of this mini-series on CXL Type 3 memory technology explains why AI and data-intensive workloads are driving interest in memory expanders and how CXL Type 3 devices differ from local DIMMs even when they appear as ordinary RAM. Part 3 covers integration modes and when boot parameters apply.

The views and content of the article are author’s own and not affiliated to any of his current or previous employers.

Related Content

The post CXL Type 3: Tooling and boot path from power-on to usable memory appeared first on EDN.

Making noise with a BANG, part 1: Concept and hardwareUnderstanding all costs involved in analog ASIC development