Home / Solutions / Technology Advisory
01 · Technology Advisory
The right AI infrastructure starts with the model you will run and the users you will serve. RAICOSOL sizes CPU, GPU, memory, storage, network, server and rack as one system, so capital goes into capacity you will actually use.
Why architecture decides ROI
Peak FLOPS rarely decide the outcome. A GPU waiting on a slow CPU, a saturated network or an overheated rack is an expensive idle asset. Each layer has to be sized against the others.
The host feeds the accelerators: data loading, tokenization, orchestration and pre- and post-processing.
Memory capacity and bandwidth decide which models fit and how fast tokens stream, often more than raw compute.
Training checkpoints and dataset reads must keep pace with the GPUs, or utilization collapses.
Three networks, three jobs: scale-up inside the server, scale-out across the cluster, and scale-across between sites.
The chassis sets GPU count, PCIe topology, cooling method and serviceability for the life of the asset.
Power and cooling per rack now decide where a cluster can live, and how much of it fits.
Inside an AI server
A typical 8-GPU node pairs two host CPUs with eight accelerators, one high-speed network port per GPU, and local NVMe. The bottleneck moves to whichever path was undersized.
There is no universal CPU-to-GPU ratio. We profile the input pipeline and the p95 latency of your service, then size cores, memory and storage to match.
CPU or GPU?
CPUs excel at serial, branching work and fast responses to unpredictable requests. GPUs run thousands of lightweight threads at once, which is what the matrix math in deep learning needs. Most production systems use both.
A GPU generation stays competitive for roughly 18 months, while servers are usually depreciated over five years. We model ownership, leasing and cloud capacity side by side, including egress, support and exit costs, before you commit.
Start from the workload
Training is bursty and bound by memory bandwidth and interconnect. Inference is continuous, latency-sensitive and judged on throughput per dollar and proximity to users.
| Workload | What limits it | Architecture we typically recommend |
|---|---|---|
| Foundation model training | HBM bandwidth, GPU-to-GPU and cluster interconnect | 8-GPU OAM / SXM nodes (MI355X, B200, B300), rail-optimized 800G fabric, liquid cooling |
| Fine-tuning | Memory capacity, FP8 / BF16 support | High-memory 8-GPU nodes (MI325X, MI355X, H200) |
| Large-model inference | Memory per GPU, tokens per second per dollar | High-HBM GPUs so the model fits in fewer devices; FP8 / FP4 serving |
| Enterprise inference in existing racks | Rack power, air cooling, standard servers | PCIe accelerators (MI350P, RTX PRO 6000) in air-cooled servers |
| Vision, video and graphics | Mixed compute and graphics, media engines | RTX PRO 6000-class PCIe GPUs |
| Scientific and HPC | FP64 throughput, memory bandwidth | FP64-strong parts (MI325X, MI355X); verify FP64 before choosing B300 |
Accelerator comparison
Published per-GPU specifications for the accelerators we deploy most often. Compute figures are dense (without sparsity) so the two vendors compare fairly.
GB of on-package memory. More memory means larger models on fewer GPUs.
PFLOPS. The main precision for modern training and inference.
| Specification | MI325X | MI350P | MI355X |
|---|---|---|---|
| Memory | 256 GB HBM3E | 144 GB HBM3E | 288 GB HBM3E |
| Memory bandwidth | 6 TB/s | 4 TB/s | 8 TB/s |
| FP8 (dense) | 2.61 PFLOPS | 2.3 PFLOPS | 5 PFLOPS |
| FP4 (dense) | Not supported | 4.6 PFLOPS | 10.1 PFLOPS |
| FP64 | 81.7 TFLOPS | 36 TFLOPS | 78.6 TFLOPS |
| Board power | 1,000 W | 600 W (450 W cfg.) | 1,400 W |
| Form factor | OAM module | PCIe 5.0 card, dual-slot, air-cooled | OAM module, liquid or air |
| GPU-to-GPU link | Infinity Fabric, 8 × 128 GB/s | PCIe (up to 8 per server) | Infinity Fabric, 7 × 153 GB/s |
| Specification | H200 | RTX PRO 6000 | B200 | B300 |
|---|---|---|---|---|
| Memory | 141 GB HBM3e | 96 GB GDDR7 | 180 GB HBM3e | 288 GB HBM3e |
| Memory bandwidth | 4.8 TB/s | 1.6 TB/s | 8 TB/s | 8 TB/s |
| FP8 (dense) | 1.98 PFLOPS | 1 PFLOPS* | 4.5 PFLOPS | 4.5 PFLOPS |
| FP4 (dense) | Not supported | 2 PFLOPS* | 9 PFLOPS | 13.5 PFLOPS |
| FP64 | 34 TFLOPS | Not published | 37 TFLOPS | 1.25 TFLOPS |
| Board power | 700 W | 600 W | 1,000 W | ~1,100 W+ |
| Form factor | SXM module | PCIe 5.0 card, dual-slot | SXM module | SXM module, liquid-cooled |
| GPU-to-GPU link | NVLink 900 GB/s | PCIe (no NVLink) | NVLink 1.8 TB/s | NVLink 1.8 TB/s |
Figures are vendor-published peak theoretical values, per GPU, without sparsity. AMD FP4 uses MXFP4. NVIDIA HGX B200 / B300 values are the 8-GPU baseboard totals divided by eight; B300 has 288 GB nominal (about 270 GB usable). *RTX PRO 6000 FP8 and FP4 are estimated at half of NVIDIA's published figures, which we read as with sparsity. Board power for B300 varies by system vendor. Real-world throughput depends on the model, software stack and cluster design; we benchmark on your workload before you buy.
Spotlight · AMD Instinct MI350P
Most enterprises cannot rebuild a data hall for liquid-cooled GPU racks. The MI350P puts CDNA 4 and 144 GB of HBM3E on a standard dual-slot PCIe card, so generative and agentic AI can run in mainstream air-cooled servers, without a rack redesign.
CDNA 4 on TSMC 3 nm and 6 nm: four compute dies of 32 CUs, 4 MB shared L2 and 128 MB AMD Infinity Cache, with a 4096-bit memory interface.
AMD Enterprise AI reference stack with the GPU Operator for Kubernetes, AMD Inference Microservices for pre-validated LLM hosting, and day-0 support for leading models.
Enterprise assistants across sales, HR, legal and coding; retrieval and document AI; video and image pipelines with on-card media decode; drug discovery and materials science.
Specifications from AMD's MI350P product brochure and product page; performance figures are AMD estimates of peak theoretical values.
Open-weight models
Open-weight models such as Kimi, Gemma, Mistral, Qwen, Llama, GLM and DeepSeek now rival closed models for many enterprise tasks. You can run them on your own infrastructure, in your own jurisdiction, with predictable cost.
Most of the largest models use mixture-of-experts (MoE) designs: only a fraction of the parameters is active per token, but every parameter still has to sit in GPU memory. That makes memory capacity the first sizing question.
| Model | Developer | Parameters (total / active) | Context | License | Strength |
|---|---|---|---|---|---|
| Gemma 4 31B | 31B dense | 256K | Apache 2.0 | Single-GPU and air-gapped deployments | |
| Qwen3.5-397B-A17B | Alibaba Qwen | 397B / 17B MoE | 262K | Apache 2.0 | Multilingual, coding, general assistant |
| Llama 4 Maverick | Meta | 400B / 17B MoE | 1M | Llama 4 Community | Long-context enterprise chat |
| Mistral Large 3 | Mistral AI | 675B / 41B MoE | 256K | Apache 2.0 | EU-sovereign, multimodal general use |
| GLM-5.2 | Z.ai | 744B / 40B MoE | 1M | MIT | Agentic coding, long-horizon tasks |
| Kimi K2.6 | Moonshot AI | 1T / 32B MoE | 256K | Modified MIT | Agentic coding, tool use, multimodal input |
| DeepSeek V4 Pro | DeepSeek | 1.6T / 49B MoE | 1M | MIT | Reasoning, coding, agents |
Open-weight does not always mean open-source: licenses differ, and some restrict commercial use at scale. New versions ship monthly, so we confirm the current release and license during each engagement.
Minimum GPUs needed just to hold each model's weights at FP8 (about 1 GB per billion parameters, plus 20% headroom). Real deployments add room for the KV cache and round up to 2, 4 or 8 GPUs.
| Model (FP8 weights + 20%) | MI325X256 GB | MI350P144 GB | MI355X288 GB | H200141 GB | RTX PRO 600096 GB | B200180 GB | B300288 GB |
|---|---|---|---|---|---|---|---|
| Gemma 4 31B≈37 GB | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
| Qwen3.5-397B-A17B≈476 GB | 2 | 4 | 2 | 4 | 5 | 3 | 2 |
| Llama 4 Maverick≈480 GB | 2 | 4 | 2 | 4 | 5 | 3 | 2 |
| Mistral Large 3≈810 GB | 4 | 6 | 3 | 6 | 9 | 5 | 3 |
| GLM-5.2≈893 GB | 4 | 7 | 4 | 7 | 10 | 5 | 4 |
| Kimi K2.6≈1,200 GB | 5 | 9 | 5 | 9 | 13 | 7 | 5 |
| DeepSeek V4 Pro≈1,920 GB | 8 | 14 | 7 | 14 | 20 | 11 | 7 |
RAICOSOL estimate for planning only. FP4 roughly halves the weight footprint; long context and high concurrency increase the KV-cache requirement.
Network architecture
Distributed training exchanges gradients after every step. Across a million steps, each extra second of network delay can add more than ten days of idle compute. Power availability and data-sovereignty rules are also spreading AI capacity across more sites.
We design all three tiers together, from the top-of-rack switch to the links between data centers.
How we engage
Models, users, latency targets, data residency, budget and timeline.
GPU platform and count, CPU and memory per node, storage tiers and fabric.
Benchmarks on candidate hardware against your models and SLAs.
Rack elevations, power and cooling, top-of-rack design and OEM sourcing.
Tell us the models and the scale. We will come back with a sized design, options across vendors, and the cost to own or lease it.
Product names and trademarks belong to their owners. RAICOSOL is vendor-neutral and is not affiliated with AMD or NVIDIA.
