Developing high-performance compute and storage systems
Peripheral Component Interconnect Express, or PCI Express (PCIe), is a widely used bus interconnect interface, found in servers and, increasingly, as a storage and GPU interconnect solution. The first version of PCIe was introduced in 2003 by the Peripheral Component Interconnect Special Interest Group (PCI-SIG) as PCIe Gen 1.1. It was created to replace the original parallel communications bus, PCI.
With PCI, data is transmitted at the same time across many wires. The host and all connected devices share the same signals for communication; thus, multiple devices share a common set of address, data, and control lines and clearly vie for the same bandwidth.
With PCIe, however, communication is serial point-to-point, with data being sent over dedicated lines to devices, enabling bigger bandwidths and faster data transfer. Signals are transferred over connection pairs, known as lanes—one for transmitting data, the other for receiving it. Most systems normally use 16 lanes, but PCIe is scalable, allowing up to 64 lanes or more in a system.
The PCIe standard continues to evolve, doubling the data transfer rate per generation. The latest is PCIe Gen 7, with 128 GT/s per lane in each direction, meeting the needs of data-intensive applications such as hyperscale data centers, high-performance computing (HPC), AI/ML, and cloud computing.
PCIe clocking
Another differentiating factor between PCI and PCIe is the clocking. PCI uses LVCMOS clocks, whereas PCIe uses differential high-speed current-steering logic (HCSL) and low-power HCSL clocks. These are configured with the spread-spectrum clocking (SSC) scheme.
SSC works by spreading the clock signal across several frequencies rather than concentrating it at a single peak frequency. This eliminates large electromagnetic spikes at specific frequencies that could cause interference (EMI) to other components. The spreading over several frequencies reduces EMI, which protects signal integrity.
On the flip side, SSC introduces jitter (timing variations in the clock signal) due to frequency-modulating the clock signal in this scheme. To preserve signal integrity, PCIe devices are permitted a degree of jitter.
In PCIe systems, there’s a reference clock (REFCLK), typically a 100-MHz HCSL clock, as a common timing reference for all devices on the bus. In the SSC scheme, the REFCLK signal is modulated with a low-frequency (30- to 33-kHz) sinusoidal signal.
To best balance design performance, complexity, and cost, but also to ensure best functionality, different PCIe clocking architectures are used. These are common clock (CC), common clock with spread (CCS), separate reference clock with no spread (SRNS), and separate reference clock with independent spread (SRIS). Both CC and separate reference architectures can use SSC but with a differing amount of modulation (for the spread).
Each clocking scheme has its own advantages and disadvantages when it comes to jitter and transmission complexity. The CC is the simplest and cheapest option to use in designs. Here, both the transmitter and receiver share the same REFCLK and are clocked by the same PLL, which multiplies the REFCLK frequency to create the high-speed clock signals needed for the data transmission. With the separate clocking scheme, each PCIe endpoint and root complex have their own independent clock source.
PCIe switches
To manage the data traffic on the lanes between different components within a system, PCIe switches are used. They allow multiple PCIe devices, from network and storage cards to graphics and sound cards, to communicate with one another and the CPU simultaneously, optimizing system performance. In cloud computing and AI data centers, PCIe switches connect multiple NICs, GPUs, CPUs, NPUs, and other processors in the servers, all of which require a robust and improved PCIe infrastructure.
PCIe switches play a key role in next-generation open, hyperscale data center specifications now being worked on rapidly by a growing developer contingent around the world. This is particularly needed with the advent of ML- and AI-centric data centers, underpinned by HPC systems. PCIe switches are also instrumental in many industrial setups, wired networking, communications systems, and where many high-speed devices must be connected with data traffic managed effectively.
A well-known brand of PCIe switches is the Switchtec family from Microchip Technology. The Switchtec PCIe switch IP manages the data flow and peer-to-peer transfers between ports, providing flexibility, scalability, and configurability in connecting multiple devices.
The Switchtec Gen 5.0 PCIe high-performance product lineup delivers very low system latency, as well as advanced diagnostics and debugging tools for troubleshooting and fast product development, making it highly suitable for next-generation data center, ML, automotive, communications, defense, and industrial applications, as well as other sectors. Tier 1 data center providers are relying on Switchtec PCIe switches to enable highly flexible compute and storage rack architectures.
Reference design and evaluation kit for PCIe switches
To enable rapid, PCIe-based system development, Microchip has created a validation board reference design, shown in Figure 1, using the Switchtec Gen 5 PCIe Switch Evaluation Kit, known as the PM52100-KIT.
The reference design helps developers implement the Switchtec Gen 5 PCIe switch into their own systems. The guidelines show designers how to connect and configure the switch and how to reach the best balance for signal integrity and power, as well as meet other critical design aspects of their application.
Fully tested and validated, the reference design streamlines development and speeds time to market. The solution optimizes performance, costs, and board footprint and reduces design risk, enabling fast market entry with a finished product. See the solutions diagram in Figure 2.
As with all Switchtec PCIe switch designs, full access to the Microchip ChipLink diagnostics tool is included, which allows parameter configuration, functional debug, and signal integrity analysis.
As per all PCIe integrations, clock and timing are important aspects of the design. Clock solutions must be highly reliable for demanding end applications, and the Microchip reference design includes complete PCIe Gen 1–5 timing solutions that include the clock generators, buffers, oscillators, and crystals.
Microchip’s ClockWorks Configurator and product selection tool allow easy customization of the timing devices for any application. The tool is used to configure oscillators and clock generators with specific frequencies, among other parameters, for use within the reference design.
The Microchip PM52100-KIT
For firsthand experience of the Switchtec Gen 5 PCIe switch, Microchip provides the PM52100-KIT (Figure 3), a physical board with 52 ports. The kit enables users to experiment with and evaluate the Switchtec family of Gen 5 PCIe switches in real-life projects. The kit was built with the guidance provided by the Microchip reference design.
The kit contains an evaluation board with the necessary firmware and cables. Users can download the ChipLink diagnostic tool by requesting access via a myMicrochip account.
With the ChipLink GUI, which is suitable for Windows, Mac, or Linux systems, the board’s hardware functions can easily be accessed and system status information monitored. It also allows access to the registers in the PCIe switch and configuration of the high-speed analog settings for signal integrity evaluation. The ChipLink diagnostic tool features advanced debug capabilities that will simplify overall system development.
The kit operates with a PCIe host and supports the connection of multiple host entities to multiple endpoint devices.
The PCIe interface contains an edge connector for linking to the host, several PCIe Amphenol Mini Cool Edge I/O connectors to connect the host to endpoints, and connectors for add-in cards. The 0.60-mm Amphenol connector allows high-speed signals to Gen 6 PCIe and 64G PAM4/PCIe, but also Gen 5 and Gen 4 PCIe. This connector maintains signal integrity, as its design minimizes signal loss and reflections at higher frequencies.
The board’s PCIe clock interface consists of a common reference clock (with or without SSC), SRNS, and SRIS. A single stable clock, with low jitter, is shared by both endpoints. The second most common clocking scheme is SRNS, where an independent clock is supplied to each end of the PCIe link; this is also supported by the Microchip kit.
Among the kit’s peripherals are two-wire (TWI)/SMBus interfaces; TWI bus access and connectivity to the temperature sensor, fan controller, voltage monitor, GPIO, and TWI expanders; SEEPROM for storage and PCIe switch configuration; and 100 M/GE Ethernet. The kit also includes GPIOs for TWI, SPI, SGPIO, Ethernet, and UART interfaces. There is UART access with a USB Type-B and three-pin connector header.
The included PSX Software Development Kit (integrating GHS’s MULTI development environment) enables the development and testing of the custom PCIe switch functionalities. An EJTAG debugger supports test and debug of custom PSX firmware; a 14-pin EJTAG connector header allows PSX probe connectivity.
Switchtec Gen 5 52-lane PCIe switch reference design
Microchip also offers a Switchtec 52-lane Gen 5 PCIe Switch Reference Design (Figure 4). As with the other reference design, it is fully validated and tested, and it provides the components and tools necessary to thoroughly assess and integrate this design into your systems.
This board includes a 32-bit microcontroller (ATSAME54, based on the Arm Cortex-M4 processor with a floating-point unit) to be configured with Microchip’s MPLAB Harmony software, as well as a CEC1736 root-of-trust controller. The CEC1736 is a 96-MHz Arm Cortex-M4F controller that is used to detect and provide protection for the PCIe system against failure or malicious attacks.
Microchip and the PCIe standard
Microchip continues to be actively involved in the advancement of the PCIe standard, and it regularly participates in PCI-SIG compliance and interoperability events. With its turnkey PCIe reference designs and field-proven interoperable solutions, a high-performance design can be streamlined and brought to market very quickly.
To view the full details of this reference design, bill of materials, and to download the design files, visit https://www.microchip.com/en-us/tools-resources/reference-designs/switchtec-gen-5-pcie-switch-reference-design.
The post Developing high-performance compute and storage systems appeared first on EDN.
