OmniBuf
for Python.
Ships prebuilt native wheels for Python 3.11+ with full static type annotations. Encode and decode Protobuf messages, typed dataclasses, or Buf-generated slotted classes, complete with lazy views over raw byte buffers and schema-less dictionary decoding.
$ uv add omnibuf protobuf-py protobuf-py-extThree ways to use OmniBuf in Python.
All three produce identical wire bytes and include full static type hints.
Native protobuf-py messages
Buf's protobuf-py generates clean typed Python classes (_pb.py) via protoc-gen-py. omnibuf.marshal and omnibuf.unmarshal compile a cached field plan from each DescMessage descriptor on first use.
import omnibuf
from app.gen import peer_pb
msg = peer_pb.PeerUpdate(
asn=64512,
router_id="edge-rtr-fra01",
active=True,
rssi_dbm=-62,
location=peer_pb.Location(city="Frankfurt", rack=42),
)
pkt = omnibuf.marshal(msg)
out = omnibuf.unmarshal(pkt, peer_pb.PeerUpdate)Annotated Python dataclasses
@omnibuf_message and omnibuf_field map standard Python type hints directly onto OmniBuf wire tags. Explicit width aliases (Uint32, Sint32, Fixed64, Float32) keep mypy and Pyright happy while pinning exact wire semantics.
import omnibuf
from omnibuf import Float32, Sint32, Uint32, omnibuf_field, omnibuf_message
@omnibuf_message("PeerSummary")
class PeerSummary:
asn: Uint32 = omnibuf_field(1, default=0)
router_id: str = omnibuf_field(2, default="")
rssi_dbm: Sint32 = omnibuf_field(5, default=0)
load: Float32 = omnibuf_field(6, default=0.0)
pkt = omnibuf.marshal(PeerSummary(asn=64512, router_id="rtr-01", rssi_dbm=-62))
out = omnibuf.unmarshal(pkt, PeerSummary)Generated code with protoc-gen-omnibuf-py
Pairing protoc-gen-py and protoc-gen-omnibuf-py inside buf.gen.yaml emits slotted dataclasses alongside Opaque<Message> wrappers. Routing services can check a nested header or patch a sequence counter straight inside a bytes or memoryview buffer.
version: v2
plugins:
- local: ["uv", "run", "--no-sync", "protoc-gen-py"]
out: src/gen
- local: ["uv", "run", "--no-sync", "protoc-gen-omnibuf-py"]
out: src/gen$ uv add --dev protoc-gen-py
$ buf generatefrom app.gen import peer_omnibuf_pb
view = peer_omnibuf_pb.OpaquePeerUpdate(pkt)
loc = view.opaque_location()
if loc is not None and loc.get_city() == "Frankfurt":
view.set_sequence(9_876_543_211)
forwarded = view.marshal()Python benchmarks.
11-field PeerUpdate workload on Python 3.14.5 across omnibuf (compiled Rust extension), protobuf-py, upb, msgspec, msgpack, and stdlib json. Shorter bars = Faster.
Serializing PeerUpdate into fresh bytes across native Rust and C extensions (omnibuf, upb, protobuf-py, msgspec, msgpack, stdlib json).
Decoding bytes and reading all 11 fields in Python. Accessing every field forces upb to materialize Python strings and sub-messages out of its C arena.
OpaquePeerUpdate header-hop lookup vs native C and Rust extension decode.
Checking a nested city string and splicing an updated sequence number vs full decode and re-encode.
Serializing a 39 KiB packet whose 20 KiB and 18 KiB nested strings trigger 2B-to-4B container header expansion.
Growing a nested string field past 16 KiB ahead of 32 trailing Neighbor entries via Opaque splicing vs full decode and re-encode.
Benchmark Environment
Measured on CPython 3.14.5 on an Apple M1 Pro Mac. Every serializer in this chart runs compiled native code: omnibuf (Rust), protobuf-py (Rust), Google upb (C), msgspec (C), msgpack (C), and stdlib json (C). Full decode reads all 11 fields in Python so lazy C-arena wrappers materialize the exact same Python strings and sub-messages as eager dataclasses.
Benchmark source lives in benchmarks/bench.py, with recorded output in testdata/benchmarks.json.
Python questions.
How does omnibuf compare against upb, protobuf-py, and msgspec under the hood?
omnibuf ships a compiled Rust extension atop omnibuf-rs. It reads and populates Python object slots directly from native code, encoding ahead of Google's C upb backend, C msgspec, and protobuf-py, and decoding full messages over 2x faster than upb. Lazy Opaque views read nested fields and splice single-field mutations nearly 2x faster than upb.
Does omnibuf support legacy google.protobuf _pb2 modules too?
protobuf-py (_pb.py classes generated by protoc-gen-py) is the primary Python Protobuf integration. Existing codebases that still import google.protobuf _pb2.py classes can pass those messages straight to omnibuf.marshal and omnibuf.unmarshal as well.
How does schema-less inspection handle Field #0?
omnibuf.decode_blind(pkt) returns a dict with synthetic field_1, field_2, ... keys on compact packets, or resolves nested field names automatically when self_describing=True embedded Field #0.
import omnibuf
pkt = omnibuf.marshal(msg, self_describing=True)
fields = omnibuf.decode_blind(pkt)