Skip to main content
Search Documentation & Spec
Type to search all 11 pages

OmniBuf
for Python.

Ships prebuilt native wheels for Python 3.11+ with full static type annotations. Encode and decode Protobuf messages, typed dataclasses, or Buf-generated slotted classes, complete with lazy views over raw byte buffers and schema-less dictionary decoding.

Install with uv
$ uv add omnibuf protobuf-py protobuf-py-ext

Three ways to use OmniBuf in Python.

All three produce identical wire bytes and include full static type hints.

Native protobuf-py messages

Buf's protobuf-py generates clean typed Python classes (_pb.py) via protoc-gen-py. omnibuf.marshal and omnibuf.unmarshal compile a cached field plan from each DescMessage descriptor on first use.

protobuf-py Integration
import omnibuf
from app.gen import peer_pb

msg = peer_pb.PeerUpdate(
    asn=64512,
    router_id="edge-rtr-fra01",
    active=True,
    rssi_dbm=-62,
    location=peer_pb.Location(city="Frankfurt", rack=42),
)

pkt = omnibuf.marshal(msg)
out = omnibuf.unmarshal(pkt, peer_pb.PeerUpdate)

Annotated Python dataclasses

@omnibuf_message and omnibuf_field map standard Python type hints directly onto OmniBuf wire tags. Explicit width aliases (Uint32, Sint32, Fixed64, Float32) keep mypy and Pyright happy while pinning exact wire semantics.

@omnibuf_message Dataclass
import omnibuf
from omnibuf import Float32, Sint32, Uint32, omnibuf_field, omnibuf_message

@omnibuf_message("PeerSummary")
class PeerSummary:
    asn: Uint32 = omnibuf_field(1, default=0)
    router_id: str = omnibuf_field(2, default="")
    rssi_dbm: Sint32 = omnibuf_field(5, default=0)
    load: Float32 = omnibuf_field(6, default=0.0)

pkt = omnibuf.marshal(PeerSummary(asn=64512, router_id="rtr-01", rssi_dbm=-62))
out = omnibuf.unmarshal(pkt, PeerSummary)

Generated code with protoc-gen-omnibuf-py

Pairing protoc-gen-py and protoc-gen-omnibuf-py inside buf.gen.yaml emits slotted dataclasses alongside Opaque<Message> wrappers. Routing services can check a nested header or patch a sequence counter straight inside a bytes or memoryview buffer.

buf.gen.yaml
version: v2
plugins:
  - local: ["uv", "run", "--no-sync", "protoc-gen-py"]
    out: src/gen
  - local: ["uv", "run", "--no-sync", "protoc-gen-omnibuf-py"]
    out: src/gen
Generate with Buf
$ uv add --dev protoc-gen-py
$ buf generate
Lazy Opaque Lookup & Splice
from app.gen import peer_omnibuf_pb

view = peer_omnibuf_pb.OpaquePeerUpdate(pkt)
loc = view.opaque_location()
if loc is not None and loc.get_city() == "Frankfurt":
    view.set_sequence(9_876_543_211)
    forwarded = view.marshal()

Python benchmarks.

11-field PeerUpdate workload on Python 3.14.5 across omnibuf (compiled Rust extension), protobuf-py, upb, msgspec, msgpack, and stdlib json. Shorter bars = Faster.

Full Message Encode Speed

Serializing PeerUpdate into fresh bytes across native Rust and C extensions (omnibuf, upb, protobuf-py, msgspec, msgpack, stdlib json).

OmniBuf (Generated)
1.0x (base) (484 ns)
OmniBuf (@dataclass)
1.0x slower (494 ns)
google.protobuf (upb)
1.2x slower (564 ns)
msgspec
1.7x slower (815 ns)
protobuf-py
2.5x slower (1,222 ns)
msgpack
4.5x slower (2,173 ns)
json (stdlib)
12.8x slower (6,202 ns)
Full Message Decode Speed

Decoding bytes and reading all 11 fields in Python. Accessing every field forces upb to materialize Python strings and sub-messages out of its C arena.

OmniBuf (@dataclass)
1.0x faster (2,754 ns)
OmniBuf (Generated)
1.0x (base) (2,757 ns)
msgspec
1.0x slower (2,869 ns)
protobuf-py
1.5x slower (4,145 ns)
msgpack
1.9x slower (5,218 ns)
google.protobuf (upb)
2.1x slower (5,888 ns)
json (stdlib)
2.9x slower (7,914 ns)
Read a Single Nested Field

OpaquePeerUpdate header-hop lookup vs native C and Rust extension decode.

OmniBuf (Generated)
1.0x (base) (525 ns)
google.protobuf (upb)
1.9x slower (1,016 ns)
msgspec
2.9x slower (1,518 ns)
protobuf-py
5.5x slower (2,869 ns)
msgpack
6.9x slower (3,620 ns)
json (stdlib)
12.1x slower (6,350 ns)
Selective Single-Field Mutation

Checking a nested city string and splicing an updated sequence number vs full decode and re-encode.

OmniBuf (Generated)
1.0x (base) (1,301 ns)
google.protobuf (upb)
1.4x slower (1,758 ns)
msgspec
1.8x slower (2,374 ns)
protobuf-py
3.2x slower (4,200 ns)
msgpack
4.5x slower (5,841 ns)
json (stdlib)
10.0x slower (12,953 ns)
Large Embedded Message Encode

Serializing a 39 KiB packet whose 20 KiB and 18 KiB nested strings trigger 2B-to-4B container header expansion.

msgspec
1.4x faster (2,239 ns)
google.protobuf (upb)
1.2x faster (2,726 ns)
OmniBuf (@dataclass)
1.0x faster (3,102 ns)
OmniBuf (Generated)
1.0x (base) (3,221 ns)
msgpack
1.2x slower (3,902 ns)
protobuf-py
1.3x slower (4,073 ns)
json (stdlib)
26.5x slower (85,390 ns)
Embedded Message Growth

Growing a nested string field past 16 KiB ahead of 32 trailing Neighbor entries via Opaque splicing vs full decode and re-encode.

OmniBuf (Generated)
1.0x (base) (4,291 ns)
google.protobuf (upb)
1.4x slower (6,003 ns)
msgspec
2.1x slower (9,196 ns)
protobuf-py
4.1x slower (17,490 ns)
msgpack
5.6x slower (24,198 ns)
json (stdlib)
19.1x slower (82,162 ns)

Benchmark Environment

Hardware & Runtime
CPU: Apple M1 Pro (darwin/arm64)
Runtime: Python 3.14.5 + rustc 1.94.0
Command: just bench

Measured on CPython 3.14.5 on an Apple M1 Pro Mac. Every serializer in this chart runs compiled native code: omnibuf (Rust), protobuf-py (Rust), Google upb (C), msgspec (C), msgpack (C), and stdlib json (C). Full decode reads all 11 fields in Python so lazy C-arena wrappers materialize the exact same Python strings and sub-messages as eager dataclasses.

Benchmark source lives in benchmarks/bench.py, with recorded output in testdata/benchmarks.json.

Python questions.

How does omnibuf compare against upb, protobuf-py, and msgspec under the hood?

omnibuf ships a compiled Rust extension atop omnibuf-rs. It reads and populates Python object slots directly from native code, encoding ahead of Google's C upb backend, C msgspec, and protobuf-py, and decoding full messages over 2x faster than upb. Lazy Opaque views read nested fields and splice single-field mutations nearly 2x faster than upb.

Does omnibuf support legacy google.protobuf _pb2 modules too?

protobuf-py (_pb.py classes generated by protoc-gen-py) is the primary Python Protobuf integration. Existing codebases that still import google.protobuf _pb2.py classes can pass those messages straight to omnibuf.marshal and omnibuf.unmarshal as well.

How does schema-less inspection handle Field #0?

omnibuf.decode_blind(pkt) returns a dict with synthetic field_1, field_2, ... keys on compact packets, or resolves nested field names automatically when self_describing=True embedded Field #0.

Example
import omnibuf

pkt = omnibuf.marshal(msg, self_describing=True)
fields = omnibuf.decode_blind(pkt)

Language implementations.

v0.1 Draft Wire Format · GitHub Feedback →
Architecture, Tooling & Wire Specification