Skip to main content
Search Documentation & Spec
Type to search all 11 pages

OmniBuf
for Rust.

Supports plain Rust structs via derive macros, existing buffa and prost Protobuf messages, and Buf code generation. Borrowed packet views read nested fields with zero heap allocation and splice updates in place. All Protobuf crate dependencies are optional.

Add Rust Crate
$ cargo add omnibuf
Cargo.toml Features
# Default: derive + buffa + prost
omnibuf = "0.1"

# buffa + #[derive(OmniBuf)] only
omnibuf = { version = "0.1", default-features = false, features = ["derive", "buffa"] }

# Pure wire, opaque splicing & blind decode
omnibuf = { version = "0.1", default-features = false }

Ways to use OmniBuf in Rust.

Every integration path produces identical wire bytes.

Procedural #[derive(OmniBuf)] structs

#[derive(OmniBuf)] expands straight into single-pass Prefix-Varint writes and unrolled tag-switch loops at compile time. Zero .proto build scripts are needed. Field attributes declare wire IDs, ZigZag or fixed-width overrides (wire = "sint", "fixed64"), oneof variant tags, and Field #0 names.

#[derive(OmniBuf)]
use omnibuf::{Message, OmniBuf};

#[derive(Debug, Default, PartialEq, OmniBuf)]
#[omnibuf(name = "PeerUpdate")]
pub struct PeerUpdate {
    #[omnibuf(id = 1, name = "asn")]
    pub asn: u32,
    #[omnibuf(id = 2, name = "router_id")]
    pub router_id: String,
    #[omnibuf(id = 3, name = "active")]
    pub active: bool,
    #[omnibuf(id = 5, name = "rssi_dbm", wire = "sint")]
    pub rssi_dbm: i32,
    #[omnibuf(id = 7, name = "tags")]
    pub tags: Vec<String>,
}

let pkt = peer.encode_omnibuf()?;
let out = PeerUpdate::from_omnibuf(&pkt)?;

Protobuf messages with buffa

First-class support for anthropics/buffa (features = ["buffa"], active by default) covers both static co-derivation and runtime reflection. Attaching #[derive(::omnibuf::OmniBuf)] via buffa_build::Config::message_attribute and oneof_attribute parses /// Field N: doc comments, unwraps buffa::MessageField<T>, buffa::EnumValue<E>, and Option<...> oneofs, and emits an unrolled zero-reflection codec.

Pre-compiled buffa crates keep their build.rs untouched through omnibuf::buffa_codec. encode_buffa_reflect walks Reflectable vtables directly in memory with zero intermediate Protobuf buffers, while encode_buffa and decode_buffa transcode against any Arc<buffa_descriptor::DescriptorPool>.

build.rs · Co-Derive with buffa-build
buffa_build::Config::new()
    .files(&["omnibuf/bench/peer.proto"])
    .includes(&["proto"])
    .generate_views(true)
    .generate_reflection(true)
    .message_attribute(".", "#[derive(::omnibuf::OmniBuf)]")
    .oneof_attribute(".", "#[derive(::omnibuf::OmniBuf)]")
    .compile()?;
buffa Co-Derive & buffa_codec
use omnibuf::{EncodeOptions, Message};
use omnibuf::buffa_codec::{decode_buffa, encode_buffa, encode_buffa_reflect};

// Co-derived #[derive(OmniBuf)] on buffa structs
let pkt = buffa_peer.encode_omnibuf()?;
let direct = buffa_pb::PeerUpdate::from_omnibuf(&pkt)?;

// Pre-compiled buffa crates via Reflectable vtables or DescriptorPool
let pkt = encode_buffa_reflect(&buffa_peer, EncodeOptions::default())?;
let pkt_pool = encode_buffa(&buffa_peer, &pool, EncodeOptions::default())?;
let reflected: buffa_pb::PeerUpdate = decode_buffa(&pkt, &pool)?;

Protobuf messages with prost

Workspaces built around prost and tonic pair static co-derivation with runtime descriptors (features = ["prost"], active by default). Passing #[derive(::omnibuf::OmniBuf)] to prost_build::Config::type_attribute compiles #[prost(...)] field tags into monomorphized Message and Oneof impls. Upstream crates implementing prost_reflect::ReflectMessage or DynamicMessage pass directly into encode_prost and decode_prost.

prost Co-Derive & prost_codec
use omnibuf::{EncodeOptions, Message};
use omnibuf::prost_codec::{decode_prost, encode_prost};

// Co-derived #[derive(prost::Message, OmniBuf)] via prost_build::Config::type_attribute
let pkt = prost_peer.encode_omnibuf()?;
let direct = pb::PeerUpdate::from_omnibuf(&pkt)?;

// Pre-compiled prost + prost-reflect crates
let pkt = encode_prost(&prost_peer, EncodeOptions::default())?;
let mut reflected = pb::PeerUpdate::default();
decode_prost(&pkt, &mut reflected)?;

Generated views with protoc-gen-omnibuf-rs

Pointing buf.gen.yaml at protoc-gen-omnibuf-rs generates owned Rust structs paired with lifetime-bound Opaque<'a> views. Getters hop across preceding field headers by byte length and borrow &str and &[u8] straight from the packet slice. try_new and try_marshal validate untrusted buffers up front, while splice_fields rewrites dirty fields in place and copies clean spans verbatim.

buf.gen.yaml
version: v2
plugins:
  - local: protoc-gen-omnibuf-rs
    out: src/gen
    strategy: all
Generate with Buf
$ cargo install protoc-gen-omnibuf-rs
$ buf generate
Zero-Copy Opaque<'a> Lookup & Splice
use crate::gen::omnibuf::bench::{OpaquePeerUpdate, PeerUpdate};
use omnibuf::Message;

let pkt = peer.encode_omnibuf()?;

let mut view = OpaquePeerUpdate::try_new(&pkt)?;
if view.get_active() {
    let city: &str = view.get_location().get_city();
    view.set_sequence(view.get_sequence() + 1);
    let updated: Vec<u8> = view.try_marshal()?;
}

Schema-less inspection with decode_blind

Every OmniBuf tag carries an explicit 4-bit wire type. decode_blind and decode_blind_json inspect arbitrary wire slices with zero compile-time schemas, emitting synthetic field_1, field_2, ... keys with exact integer signs, booleans, IEEE-754 -0.0, UTF-8 strings, and Base64 byte slices. Passing EncodeOptions::self_describing() prepends Field #0 (SchemaMeta) so keys resolve to declared field names.

decode_blind & Field #0 SchemaMeta
use std::collections::BTreeMap;
use omnibuf::{BlindValue, EncodeOptions, Message, decode_blind, decode_blind_json};

let pkt = peer.encode_omnibuf_with_options(EncodeOptions::self_describing())?;
let fields: BTreeMap<String, BlindValue> = decode_blind(&pkt)?;
let json: String = decode_blind_json(&pkt)?;

Rust benchmarks.

11-field PeerUpdate struct on Apple M1 Pro (rustc 1.94.0) across omnibuf, buffa, prost, rmp-serde, and serde_json. Shorter bars = Faster.

Full Message Encode Speed

Serializing PeerUpdate into a freshly allocated Vec<u8> across every library.

OmniBuf (#[derive(OmniBuf)])
1.0x faster (237 ns)
OmniBuf (Generated)
1.0x (base) (240 ns)
buffa (Protobuf)
1.1x slower (259 ns)
prost (Protobuf)
1.2x slower (298 ns)
rmp-serde (MessagePack)
3.5x slower (831 ns)
serde_json
4.3x slower (1,039 ns)
Full Message Decode Speed

Decoding &[u8] into a fresh owned PeerUpdate struct and reading all 11 fields.

OmniBuf (Generated)
1.0x (base) (549 ns)
OmniBuf (#[derive(OmniBuf)])
1.0x slower (553 ns)
buffa (Protobuf)
1.2x slower (679 ns)
prost (Protobuf)
1.6x slower (872 ns)
rmp-serde (MessagePack)
2.1x slower (1,139 ns)
serde_json
3.0x slower (1,664 ns)
Read a Single Nested Field

Borrowing a nested &str directly from &[u8] via OpaquePeerUpdate vs buffa PeerUpdateView vs full decode.

OmniBuf (Generated)
1.0x (base) (90 ns)
buffa (Protobuf)
5.3x slower (476 ns)
prost (Protobuf)
9.5x slower (859 ns)
rmp-serde (MessagePack)
12.5x slower (1,129 ns)
serde_json
18.4x slower (1,656 ns)
Selective Single-Field Mutation

Checking a nested city string and splicing an incremented 64-bit sequence counter into &[u8] vs decode and re-encode.

OmniBuf (Generated)
1.0x (base) (260 ns)
buffa (Protobuf)
3.6x slower (936 ns)
prost (Protobuf)
4.6x slower (1,188 ns)
rmp-serde (MessagePack)
7.7x slower (1,998 ns)
serde_json
10.8x slower (2,807 ns)
Large Embedded Message Encode

Serializing a 39 KiB packet into Vec::with_capacity(128). Two-pass encoded_len() pre-sizing in buffa and prost avoids single-pass 2B-to-4B header shifts.

buffa (Protobuf)
2.6x faster (1,086 ns)
prost (Protobuf)
1.5x faster (1,848 ns)
rmp-serde (MessagePack)
1.4x faster (2,045 ns)
OmniBuf (Generated)
1.0x (base) (2,832 ns)
OmniBuf (#[derive(OmniBuf)])
1.0x slower (2,918 ns)
serde_json
6.6x slower (18,620 ns)
Embedded Message Growth

Growing a nested string field past 16 KiB ahead of 32 trailing Neighbor entries via Opaque splicing vs full decode and re-encode.

OmniBuf (Generated)
1.0x (base) (2,411 ns)
buffa (Protobuf)
1.8x slower (4,443 ns)
prost (Protobuf)
2.1x slower (5,154 ns)
rmp-serde (MessagePack)
3.4x slower (8,196 ns)
serde_json
8.2x slower (19,840 ns)

Benchmark Environment

Hardware & Runtime
CPU: Apple M1 Pro (darwin/arm64)
Compiler: rustc 1.94.0 (--release)
Command: just bench

Compiled with rustc 1.94.0 in release mode on an Apple M1 Pro Mac. Encode benchmarks start from identical Vec::with_capacity(128) buffers, and decode benchmarks read all 11 fields into a black_box checksum sink.

Benchmark source lives in crates/omnibuf-bench/src/main.rs, with recorded output in testdata/benchmarks.json.

Rust questions.

How do buffa *View<'a> types compare with OmniBuf Opaque<'a> views?

buffa generates PeerUpdateView<'a> structs that parse a Protobuf buffer in one upfront pass, populating borrowed &'a str and &'a [u8] slices across every field. OpaquePeerUpdate<'a> wraps the raw OmniBuf slice as a Cow<'a, [u8]> and defers decoding until individual getter calls, stepping over earlier length-prefixed containers in O(1) time per field.

Pipelines that read most fields of an incoming Protobuf message benefit from buffa's single-pass *View<'a> borrow. Routing planes and telemetry proxies that inspect one nested header or overwrite a single sequence counter skip full-struct materialization with Opaque<'a> and splice_fields.

What differs under the hood between buffa and prost integration?

prost encodes schema metadata inside #[prost(...)] field attributes, boxes recursive messages in Option<Box<T>>, and stores enums as bare i32 values. buffa emits /// Field N: `name` doc attributes, wraps singular sub-messages in buffa::MessageField<T>, preserves unknown enum discriminants via buffa::EnumValue<E>, and exposes zero-allocation Reflectable vtables.

#[derive(OmniBuf)] recognizes both field layouts natively at macro expansion time. At runtime, buffa_codec::encode_buffa_reflect reads buffa structs in place through their Reflectable vtable, whereas prost_codec::encode_prost converts through prost_reflect::DynamicMessage.

How are untrusted network payloads bounded before allocation?

DecodeOptions caps packet byte length (max_message_bytes), cumulative array and map element counts (max_elements), total heap reservation (max_alloc_bytes), and recursion depth (max_depth, capped at 32). All four checks run ahead of owned value allocation in from_slice_with_options, decode_buffa_with_options, decode_prost_with_options, and decode_blind_with_options, verified across 758/758 conformance cases.

Can crates depend on omnibuf with zero Protobuf dependencies?

Setting default-features = false disables derive, buffa, and prost. Only the core Prefix-Varint wire encoder, zero-copy decoder, Field #0 schema reader, opaque::splice_fields, and decode_blind compile, leaving itoa and ryu as the sole dependencies. Adding features = ["derive"] re-enables #[derive(OmniBuf)] for standalone Rust structs.

Language implementations.

v0.1 Draft Wire Format · GitHub Feedback →
Architecture, Tooling & Wire Specification