Rust for ML Infrastructure
Exploring Rust as a systems language for building ML tooling — data pipelines, model serving, and beyond.
Python dominates ML research, but production ML infrastructure often demands better performance, safety, and concurrency. Enter Rust.
Where Rust Shines
Rust isn't replacing Python for model development — it's complementing it at the infrastructure layer:
| Layer | Python | Rust |
|---|---|---|
| Research / prototyping | Primary | — |
| Data preprocessing | Pandas / Spark | Polars, DataFusion |
| Model serving | FastAPI + ONNX | Actix + ONNX / candle |
| Training orchestration | Hydra / Ray | Custom schedulers |
The pattern: Python at the top, Rust at the bottom.
A Minimal Example
Here's a simple tensor operation using the candle crate:
use candle_core::{Device, Tensor};
fn main() -> candle_core::Result<()> {
let device = Device::Cpu;
let a = Tensor::randn(0f32, 1.0, (2, 3), &device)?;
let b = Tensor::randn(0f32, 1.0, (3, 2), &device)?;
let c = a.matmul(&b)?;
println!("{c}");
Ok(())
}No garbage collector, no GIL, predictable latency. For serving models at scale, this matters.
The Trade-off
Rust's learning curve is real. The borrow checker will fight you. But for infrastructure that needs to run 24/7 at scale, the compile-time guarantees are worth the upfront investment.
Write the model in Python. Ship the inference in Rust.
That's the pattern I'm converging on.