Alex Medick

AMD’s MI300X, MI325X, and MI355X GPUs deliver breakthrough performance and efficiency, redefining AI inference economics. See the TensorWave advantage.

Benchmark Breakdown: How AMD’s MI300X, MI325X, and MI355X Are Redefining AI Inference Economics

In the fast-paced world of AI infrastructure, staying ahead means not just keeping up with hardware innovations but also tracking the relentless evolution of software stacks. 

That's why the recent launch of InferenceMAX™ by SemiAnalysis is a game-changer. This open-source, automated benchmarking initiative runs nightly tests on leading GPUs and inference frameworks, providing real-time insights into key metrics like throughput, latency, performance per dollar, and energy efficiency. And the re

In the fast-paced world of AI infrastructure, staying ahead means not just keeping up with hardware innovations but also tracking the relentless evolution of software stacks. 

That's why the recent launch of InferenceMAX™ by SemiAnalysis is a game-changer. This open-source, automated benchmarking initiative runs nightly tests on leading GPUs and inference frameworks, providing real-time insights into key metrics like throughput, latency, performance per dollar, and energy efficiency. And the results? They shine a bright light on AMD's Instinct™ GPUs (MI300X, MI325X, and MI355X) as powerhouse performers for diverse AI workloads.

At TensorWave, we're building the next-generation AI cloud exclusively on AMD GPUs because we believe in delivering unmatched value, scalability, and efficiency without vendor lock-in. The InferenceMAX™ report validates this vision, showcasing how AMD's hardware and rapidly improving ROCm software stack deliver competitive, cost-effective inference performance. Let's dive into the highlights and explore why this matters for enterprises and developers choosing TensorWave.

Read the full SemiAnalysis InferenceMAX Benchmark report here: Read Report


The InferenceMAX™ Benchmark: A Neutral Lens on AI Performance

InferenceMAX™ isn't your typical static benchmark. It's designed to evolve with the ecosystem. It tests popular models like Llama 3 70B, DeepSeek R1 670B (a proxy for frontier models), and GPT-OSS 120B across real-world scenarios: chat (1K in/1K out), reasoning (1K in/8K out), and summarization (8K in/1K out). Frameworks include vLLM, SGLang, and TensorRT-LLM, with sweeps over concurrency, parallelism, and precisions like FP8, FP4, and MX4.

The report emphasizes the fundamental trade-off between throughput (tokens/second/GPU) and interactivity (tokens/second/user), normalizing results for total cost of ownership (TCO) per million tokens and tokens per megawatt. This holistic view cuts through hype, revealing where AMD excels, often in practical, cost-sensitive ranges that align perfectly with enterprise needs.


AMD Is Closing (and Surpassing) the Gap

The benchmarks position AMD's GPUs as highly competitive, frequently outperforming or matching NVIDIA equivalents in TCO and efficiency. 

Here's a breakdown of the positives for the MI300X, MI325X, and MI355X 👇


MI300X: Memory and Bandwidth Advantages Shine

 * Strong in Low-Interactivity Regimes: On Llama 3 70B FP8 (reasoning workload), the MI300X outperforms the H100 at 20-30 tokens/second/user, leveraging superior memory bandwidth and capacity at tensor parallelism 1 (TP1).
 * Exceptional TCO Value: For GPT-OSS 120B with MX4 weights, it delivers "very strong performance per TCO" across the entire interactivity range, beating the H100 handily.
 * Power Efficiency Baseline: At 90 tokens/second/user on GPT-OSS 120B MX4, it generates 750,000 tokens/second per all-in provisioned megawatt (MW), setting a strong foundation for generational leaps.


MI325X: Competitive Across the Board

 * Edges Out H200 in Key Ranges: On vLLM with GPT-OSS 120B MX4 (summarization), the MI325X holds an advantage below 110 tokens/second/user and remains competitive above it.
 * TCO Leadership: It beats the H200 on vLLM across all interactivity levels for the same model. Surprisingly, open-source vLLM on MI325X even outperforms NVIDIA's "mostly" open-source TRT-LLM on H200 above 135 tokens/second/user—highlighting ROCm's optimizations.
 * Real-World Relevance: These wins come in scenarios mimicking enterprise tasks like summarization, where cost predictability is key.


MI355X: Next-Gen Powerhouse with Massive Gains

 * Tight Race with B200: On GPT-OSS 120B, the MI355X is "never more than ~15 seconds slower" in latency for given throughput, staying competitive in the practical 150-200 tokens/second/user range.
 * TCO Wins in Practical Scenarios: It delivers lower TCO per million tokens than the B200 on vLLM for Llama 3 70B FP4 (reasoning) and beats B200 TRT-LLM below 225 tokens/second/user on GPT-OSS 120B FP4 (summarization).
 * 3x Power Efficiency Jump: Compared to MI300X, the MI355X (CDNA4 architecture) achieves ~3x better tokens/second per MW - up to 2,550,000 on GPT-OSS 120B MX4 at 90 tokens/second/user. While slightly behind Blackwell overall (due to TDP differences), it's a testament to AMD's rapid advancements.
 * Future-Proof Potential: With upcoming ROCm integrations into SGLang and support for multi-node expert parallelism/disaggregated prefill, expect even stronger results for large MoE models like DeepSeek 670B.


Performance Where It Matters

While raw benchmark numbers can tell one story, total cost of ownership (TCO) and energy efficiency tell the real one.

According to the benchmark:

 * MI300X: ~750,000 tokens/sec per provisioned megawatt
 * MI355X: ~2.55 million tokens/sec per megawatt

That’s an extraordinary improvement - one that translates directly into lower inference costs and higher sustainability for production-scale AI workloads.

And when normalized for cost per million tokens, AMD’s GPUs consistently landed on or near the Pareto frontier, meaning there’s no “better-for-cheaper” option across most operational scenarios.


Software Maturity: The Silent Win

Perhaps one of the most underappreciated points from the report: the software experience.

SemiAnalysis reported that AMD’s inference stack (ROCm + vLLM) presented fewer bugs and easier fixes compared to competing hardware. We believe this is a clear sign that the ROCm software ecosystem has matured rapidly.

At TensorWave, we see the same story daily. Our customers deploying on AMD Instinct GPUs experience robust, stable environments with the flexibility to scale seamlessly across frameworks like vLLM, SGLang, TGI, and Modular’s MAX Engine.


Energy Efficiency: A Competitive Edge That Compounds

Data centers don’t just compete on performance anymore.They compete on power.

With a growing emphasis on watts per token, the MI355X represents a pivotal shift for hyperscale and enterprise inference. AMD’s efficiency improvements enable not just faster inference, but greener and more cost-efficient operations.

When scaled across clusters, such as TensorWave’s multi-megawatt MI325X and MI355X deployments, those power efficiency gains directly reduce operational costs, making large-scale AI deployments both economically and environmentally viable.


In Summary

Across the board, AMD GPUs show "lots of room for improvement" through weekly ROCm updates, with fewer bugs encountered during development compared to competitors. The report praises AMD's "2.0 sense of urgency" in fixing issues and contributing to stability. This is echoed by customers like xAI.

As Dr. Lisa Su, AMD's Chair and CEO, notes in the report: 

“The open-source InferenceMAX benchmark gives the community transparent, nightly results that inspire trust and accelerate progress. It highlights the competitive TCO performance of our AMD Instinct MI300, MI325X, and MI355X GPUs across diverse workloads, underscoring the strength of our platform and our commitment to giving developers real-time visibility into our software progress.”

And from our own Darrick Horton, CEO of TensorWave: 

“At TensorWave, we’re building a next-generation cloud on AMD GPUs because we believe innovation thrives when customers have strong alternatives. InferenceMAX™ reinforces that vision by providing open source, reproducible benchmarks that track throughput, efficiency, and cost across the latest hardware and software. By cutting through synthetic numbers and highlighting real-world inference performance, it helps customers see the full potential of AMD platforms for AI at scale.”


Why TensorWave is Your AMD-Powered AI Cloud Partner

To us, these results aren't just numbers. They're proof that AMD's ecosystem is maturing rapidly, offering enterprises a path to high-performance AI without the premiums of traditional providers. At TensorWave, we've optimized our cloud for AMD Instinct™ GPUs, including the cutting-edge MI355X, to deliver:

 * 40% Cost Savings: Compared to legacy clouds, with no lock-in and portable code.
 * Scalable Inference: Seamless support for frameworks like vLLM and SGLang, ensuring you capture the latest optimizations.
 * Enterprise-Ready Features: From high-throughput reasoning to low-latency chat, our platform handles it all with AMD's memory advantages.
 * Sustainability Focus: With AMD's generational efficiency gains, we're helping you maximize tokens per megawatt in power-constrained environments.

Whether you're migrating workloads or building new AI applications, TensorWave makes it easy to harness AMD's strengths. 

Our recent partnerships, like with Modular and AMD for the MI355X, further enhance portability and performance - no more rewriting code when shifting between providers.


Looking Ahead: Join the AMD Revolution

For AMD, the trajectory is clear: weekly improvements and ecosystem momentum position it as a leader in open, efficient AI infrastructure.

Ready to experience AMD's edge? Head to TensorWave.com to access our AI cloud, powered by MI325X and MI355X GPUs. Contact our team for free POC for your AI workloads and let's build the future of AI together.


About TensorWave

TensorWave is the AI neocloud purpose-built for performance. Powered exclusively by AMD Instinct™ Series GPUs, we deliver high-bandwidth, memory-optimized infrastructure that scales with your most demanding models—training or inference.

Ready to get started? Connect with a Sales Engineer.

AMD Instinct™ MI355X GPUs Now Available

Benchmark Breakdown: How AMD’s MI300X, MI325X, and MI355X Are Redefining AI Inference Economics

Subscribe to our Blog

Stay ahead of the curve with the latest in AI, AMD accelerators, and all things TensorWave.

Product

Resources

Company

© 2025 TensorWave Inc. - All rights reserved.