Chapter 0 — Overview
Reaching Production Readiness: A Practical Guide to Building FPGA-Based PCIe DMA Engines.
Reaching Production Readiness: A Practical Guide to Building FPGA-Based PCIe DMA Engines
Chapter 0 — Overview
________________________________
/ Reaching Production Readiness: \
| A Practical Guide to Building |
| FPGA-Based PCIe DMA Engines |
\________________________________/
| - - - - - - - - |
| (c) 2026 Alex Gutierrez |
| @ HF Labs |
| [https://hflabs.dev] |
| All rights reserved. |
\________________________________/While this book uses the Stratix V GS as an example platform, the design and patterns are broadly applicable to FPGA-based DMA engines across various hardware environments.
A little background about myself and what to expect in this book:
I’m a full-stack developer who has been programming as a hobbyist for roughly 20 years. Over that time, I’ve worked across 50+ programming languages well enough to drop into their development flow and build useful things on-the-fly.
Most of my accelerator infrastructure experience has been on GP-GPU and ASIC platforms (including Iris Xe and Poplar environments). This project.. FPGAs are different: it started as a curiosity-driven dive into FPGA development with Verilog/SystemVerilog and the surprisingly accessible Catapult v2 boards you can find online (often around the price of lunch), built around Stratix V GS FPGAs and onboard RAM.
What this book is
This book is a descriptive compilation of FPGA-oriented systems I’ve been building in Verilog/SystemVerilog as part of learning the domain. The emphasis is on:
- Documenting architecture and design tradeoffs,
- Highlighting safety and convenience considerations,
- Showing concrete modules and integration patterns,
- Capturing simulation-oriented validation paths,
- Highlighting efficient coding practices and design patterns,
- Recording what worked, what broke, and what changed.
Important context
I have no hands-on production FPGA hardware experience.. yet. This is a research-and-build journal: real code, real design attempts, and simulator-driven validation.
If you run any of these systems on actual hardware YOU DO SO AT YOUR OWN RISK, as this code has never been tested on real hardware. That being said, feedback would be hugely valuable—especially confirmation (or correction) of behavior, timing assumptions, and integration viability.
Tone and intent
Think of this as engineering notes turned into a practical field guide: candid, implementation-heavy, and iteratively improved. If you’re exploring FPGA development from software-first instincts, you’re in the right place.
Features that can be implemented through this book
Core Functionality
- Multi-Channel DMA: 16 independent DMA channels (8 read, 8 write)
- Descriptor-Based Operation: Ring-buffer based descriptor fetching
- Deterministic Scheduling: Priority-based deterministic scheduler
- Zero-Copy Transfers: Direct memory-to-memory data movement
- Completion Queues: Hardware-managed completion notifications
- Scatter-Gather Support: Non-contiguous memory transfers with chained descriptors
Agent System
- Agent Message Router: 16-agent message routing with priority queuing
- Inter-Agent Communication: 256-bit message width with 8-deep FIFOs per agent
- Priority-Based Routing: 3-level priority system for message handling
- Statistics Tracking: Total messages routed and dropped message counters
Telemetry & Monitoring
- Performance Counters: Per-channel and global transfer statistics
- Real-Time Throughput: Average and peak throughput measurement in Mbps
- Error Tracking: Per-channel and global error counters
- 64-bit Counters: Support for large-scale deployments
- Memory-Mapped Readout: Accessible via telemetry register interface
P2P DMA Infrastructure
- Address Decoder: Routes P2P traffic to GPU/IPU address windows
- TLP Generator: Produces simplified PCIe TLPs for peer transfers
- GPUDirect RDMA: Handshake hooks for NVIDIA GPU memory access
- IPU Gateway Shim: Bridge for Graphcore IPU ingress/egress
Management Plane
- BMC Mailbox: Sideband control plane register block
- I2C & GPIO: Platform management interfaces for sensors and control
- Thermal Hooks: Temperature monitoring with over-temp alerts
PCIe Interface
- Gen3 x8 Support: Up to 8 GT/s per lane (theoretical 64 Gbps)
- BAR Configuration: Memory-mapped control registers
- MSI-X Interrupts: 16 configurable interrupt vectors
- 64-bit Addressing: Full 64-bit address space support
NVMe Interface
- NVMe 1.4 Compatible: Standard NVMe command/completion queues
- MMIO Registers: Full NVMe register set implementation
- Admin Queue Support: Controller initialization and configuration
- I/O Queue Management: Submission and completion queue handling
Performance Features
- 512-bit Data Path: 64-byte data transfers per cycle
- Burst Transfers: Up to 4KB bursts for efficiency
- Pipeline Architecture: Overlapped fetch, schedule, execute stages
- Low Latency: Minimal cycle counts through critical paths
Project layout skeleton (seed structure)
Use this as a baseline project shape for the examples in this book. The idea is to keep module ownership and integration boundaries obvious from day one.
catapult_ptp/
├── fpga_top.v # Top-level integration (clock/reset, subsystem wiring)
├── pcie_endpoint.v # PCIe ingress/egress boundary
├── dma_engine/ # DMA pipeline and scheduling components
├── util/
│ ├── width_converter.v # Core width conversion logic
│ ├── width_adapters.v # Convenience wrappers for common width paths
│ └── cdc_modules.v # CDC safety primitives (sync, pulse, handshake)
├── timing/ # PTP/timestamp-related logic and maps
├── interrupt/ # MSI/MSI-X and related signaling
├── testbench/
│ └── tb_dma_engine.v # Simulation harness and smoke scenarios
└── book/ # This documentation setand a fully implemented project will look like this:
fpga/
├── fpga_top.v # Top-level FPGA integration
├── pcie_endpoint.v # PCIe Gen3 x8 endpoint wrapper
├── bar_control.v # BAR register control module
│
├── dma_engine/
│ ├── dma_top.v # DMA engine top module
│ ├── dma_desc_fetch.v # Descriptor fetch from host memory
│ ├── dma_scheduler.v # Multi-channel deterministic scheduler (16 channels)
│ ├── dma_scheduler_worksteal.v # Enhanced scheduler with work stealing support
│ ├── dma_addr_gen.v # Address generation for transfers
│ ├── dma_read_engine.v # Read data from NVMe to host
│ ├── dma_write_engine.v # Write data from host to NVMe
│ ├── dma_completion.v # Completion notification handler (16 channels)
│ ├── dma_fifos.v # Data and descriptor FIFOs
│ ├── agent_message_router.v # Agent message routing with zero-copy buffer transfers
│ ├── scatter_gather_engine.v # Scatter-gather DMA operations
│ ├── telemetry_collector.v # Performance monitoring and statistics
│ ├── inline_transform_engine.v # Programmable data transformation pipeline
│ ├── qos_bandwidth_controller.v # Adaptive QoS and bandwidth allocation
│ └── ptp_clock.v # IEEE 1588 PTP timestamping
│
├── nvme_iface/
│ ├── nvme_mmio.v # NVMe MMIO register interface
│ └── nvme_queue.v # NVMe queue management (SQ/CQ)
│
├── interrupt/
│ └── msix_ctrl.v # MSI-X interrupt controller
│
├── p2p/
│ ├── p2p_address_decoder.v # P2P address decoding
│ ├── p2p_tlp_generator.v # Simplified TLP generator
│ ├── gpudirect_interface.v # NVIDIA GPUDirect handshake
│ ├── ipu_gateway_shim.v # Graphcore IPU gateway shim
│ └── p2p_dma_controller.v # P2P DMA orchestration
│
├── ipu/
│ ├── ipu_gateway_pcie.v # IPU PCIe gateway bridge
│ ├── ipu_sync_controller.v # IPU sync pulse controller
│ ├── ipu_tile_router.v # Tile routing fabric
│ └── ipu_exchange_buffer.v # Tile exchange buffer
│
├── bmc/
│ ├── i2c_master.v # I2C master controller
│ ├── gpio_controller.v # GPIO controller
│ ├── bmc_mailbox.v # Management mailbox
│ ├── ipmi_bridge.v # IPMI bridge
│ ├── thermal_interface.v # Thermal monitoring
│ └── ncsi_controller.v # Optional NC-SI support
│
└── util/
├── skid_buffer.v # AXI-Stream skid buffer
├── fifo_sync.v # Synchronous FIFO
├── fifo_async.v # Asynchronous FIFO (CDC)
└── arbiter_rr.v # Round-robin arbiterEverybody loves block diagrams, so here’s one showing the major components and data/control flow in the fully implemented design:
┌─────────────────────────────────────────────────────────────────┐
│ fpga_top.v │
│ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ pcie_endpoint │ │ bar_control │ │ msix_ctrl │ │
│ │ (Gen3 x8) │ │ (Config) │ │ (Interrupts) │ │
│ └────────┬───────┘ └──────┬───────┘ └────────┬─────────┘ │
│ │ │ │ │
│ ┌────────┴──────────────────┴────────────────────┴─────────┐ │
│ │ dma_top.v │ │
│ │ ┌──────────────┐ ┌─────────────┐ ┌────────────────┐ │ │
│ │ │ desc_fetch │→ │ scheduler │→ │ read_engine │ │ │
│ │ └──────────────┘ └─────────────┘ └────────────────┘ │ │
│ │ ┌──────────────┐ ┌─────────────┐ ┌────────────────┐ │ │
│ │ │ dma_fifos │ │ addr_gen │ │ write_engine │ │ │
│ │ └──────────────┘ └─────────────┘ └────────────────┘ │ │
│ │ ┌──────────────────────────────────────────────────┐ │ │
│ │ │ dma_completion │ │ │
│ │ └──────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │ │ │
│ ┌────────┴─────────┐ ┌───────────┴────────┐ │
│ │ nvme_mmio.v │ │ nvme_queue.v │ │
│ │ (Registers) │ │ (SQ/CQ Handler) │ │
│ └──────────────────┘ └────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘In this section (Production Readiness)
This section captures production-readiness improvements made to address known limitations in standard everyday FPGA-based DMA engine implementations:
- Data-path width conversion moved from padding/truncation to proper adapters.
- Safe clock-domain crossing modules were added for multi-clock operation.
- Basic verification infrastructure was introduced for repeatable validation.
- A staged roadmap was defined for production deployment.
Performance and architecture highlights
- PCIe Gen3 x8 (theoretical 8 GB/s; Catapult v2 boards feature dual x8 / x16 bifurcated paths)
- 256-bit PCIe edge with 512-bit internal DMA datapath
- Descriptor/completion queue model with parameterized depth
- Multi-channel scheduling and interrupt-backed completions