SigmanticAI published Apex, an open-source FPGA inference chip that runs Qwen2.5-0.5B at 0.56 tokens per second on silicon

SigmanticAI published Apex, an open-source LLM inference chip design implemented in RTL that runs Qwen2.5-0.5B on FPGA silicon. The repository at SigmanticAI/apex-inference-chip was first pushed on 17 August 2026 and has 616 GitHub stars as of 8 September. It is Apache-2.0 licensed and covers one full transformer decoder layer.

What is in the repo

The design targets two boards. The first is a Lattice ECP5-85F built with an open-source toolchain, with a committed bitstream and a KV-compression engine at its full shipping configuration. The second is an AWS F2 instance with a VU47P part built through Vivado. Both are described in the README as independent existence proofs, each with its own place-and-route reports.

The RTL covers an INT8 systolic GEMM array called MXE, time-multiplexed across seven of the operations in a transformer layer. The KV cache uses per-channel INT4 keys and per-token INT4 values, with an FP16 outlier lane. Online softmax, RoPE, RMSNorm and SwiGLU are all in the tree. The README notes that the contribution being claimed is the integrated, verified whole, and disclaims any originality on the KV-quantisation approach itself.

The measured numbers

The README quotes a throughput ladder. The baseline is 0.004 tokens per second in host-driven mode, where the host makes a round trip per job. The fastest registered image, called A0 and running at 62.5 MHz, reaches 0.56 tokens per second and a steady 1.78 seconds per token. A reference image labelled A2 runs at 15.625 MHz and 0.25 tokens per second. The 140 times figure quoted in the description is the climb from the 0.004 baseline to the 0.56 measured rate.

SigmanticAI says every RTL block is verified against an executable NumPy reference and is bit-identical to it, with no tolerance. The testbenches are mutation-tested, and the repository states anti-fabrication rules for any published claim. Qwen2.5-0.5B is what runs on silicon. Qwen2.5-7B has been carried through the software golden pipeline but not put on hardware.

What is still open

The README is honest about the state of the bring-up. Host-mode attention is proven bit-exact on FPGA, the norm and residual chains are walked in silicon, and DDR weight streaming works. Walked attention on the FPGA, meaning the full layer walker cycle, is still in active bring-up, and the README describes root-causing a synthesis-versus-simulation defect on that path. Getting the full layer walker running in silicon is on the roadmap as the next milestone.

The organisation behind the repository, SigmanticAI, sells autonomous verification agents that take hardware from a natural-language specification to verified RTL, using an in-house Polaris Engine. Apex is presented as a demonstration of what that verification pipeline can produce end to end. There is no plan in the README to tape it out as a commercial chip.

The audience for this is narrower than a coding-agent tool. It is teams working on custom inference silicon, edge-inference research, and anyone trying to build a hardware LLM design that can be audited line by line. What is unusual is the shape: a public, license-permissive, verified reference of a decoder layer on real FPGA silicon, with the evidence trail attached. Reading someone else's working RTL is the standard way to learn how these blocks fit together, and there are not many public examples with this level of verification.

Source

Primary source: SigmanticAI/apex-inference-chip on GitHub.