How It Works | OmniBuf

What you can do with OmniBuf that you can't do with Protobuf.

1. Schema-Less Type Inspection

Every wire type is explicit

In Protobuf, strings, bytes, nested messages, maps, and arrays all share Wire Type 2. OmniBuf's 4-bit tag identifies every type right on the wire so tools can inspect packets without the original schema.

Protobuf Raw

#3: 1 // int or bool?
#9: "\x1a\x03FRA\x20*"

OmniBuf Raw

#3 (bool): true
#9 (submsg): {#3:"FRA"}

2. Optional Inline Field Names (Field #0)

Named JSON when you want it, compact tags when you don't

Send only field numbers in production RPCs for compact binary packets, or include Field #0 (omnibuf.v1.SchemaMeta mapping field numbers to names and hoisting nested child schemas once per array) so logs and queues decode directly into named JSON without a schema registry.

Field #0: {1: "id", 3: "active", 9: "location"} → Named JSON

3. Header-Hopping & Raw Concatenation

Jump over containers in one addition & merge by appending

Every container states its total byte length up front so readers jump over arrays or sub-messages in one addition. And because packets are pure [Tag][Value] streams, concatenating two messages (append(msgA, msgB...)) merges them automatically.

Skip: pos += byteLen • Merge: append(a, b...) 0-Parse Merge

4. Native Maps & Strict Validation

Zero-allocation maps without synthetic MapEntry wrappers

Maps (WireMap = 9) encode key-value pairs directly inside a single length-prefixed container without Protobuf's synthetic MapEntry sub-messages. Strict container bounds and a 32-level nesting cap prevent allocation attacks on untrusted inputs.

Direct WireMap + verified container bounds Zero OOM Risk

Three simple changes that make OmniBuf faster and smarter.

Change 1: Prefix-Varints (TZCNT)

Decode ints in a single load

Instead of checking continuation bits on every byte in a loop, the trailing zeros in Byte 0 tell the CPU the exact byte width (1, 2, 4 , or 9 bytes) right away via TZCNT . Signed integers use ZigZag encoding .

vvvvvvv 1 1B (7b)

vvvvvv 10 2B (14b)

vvvvv 100 4B (29b)

00000000 9B (64b)

Change 2: Single-Pass Encoding

Reserve 2 bytes & patch after

Protobuf requires shortest-length varints, so encoders must measure every child object before writing. In OmniBuf, a 2-byte Prefix-Varint (TZCNT=1) is valid for any length under 16 KB, so the encoder reserves 2 bytes, writes forward in one pass, and fills in the length at the end (automatically expanding in place to 4 or 9 bytes if a container exceeds 16 KB).

1. Reserve 2-byte blank slot

Tag #9 [00 00] → write

2. Backpatch 2B length at end

Tag #9 Len=7B [7B]

Change 3: 4-Bit Self-Describing Tags

0-byte bools & explicit types

Widening wire types from 3 bits to 4 bits (0..15) puts booleans directly inside the tag byte (0 value bytes), distinguishes unsigned ( WireUint, WireUintArr) from ZigZag-signed ( WireSint, WireSintArr) integers on the wire, wraps all arrays and maps in length-prefixed containers for O(1) 1-step skipping, and makes every wire type unambiguous.

Field #

4-Bit Type

Type 1, 2 (Bool): 0 Value Bytes Type 3..9: Str ≠ SubMsg ≠ Map

Varint Microbenchmarks: LEB128 vs. TZCNT

Why 12 lines of TZCNT beats 100 lines of unrolled LEB128

In Go's official Protobuf library (protowire.ConsumeVarint), decoding a single integer requires an unrolled 10-step function with 10 sequential bounds checks, 10 continuation-bit branches (if y < 0x80), and byte-by-byte shifts. Because it is 100 lines long, the Go compiler cannot inline it. Even code generators like vtprotobuf that inline a for shift := uint(0); ; shift += 7 loop at every field still suffer branch mispredictions and byte-by-byte dependencies. OmniBuf's ReadVarintAt is 12 lines, inlines everywhere, and loads 1, 2, 4, or 8 bytes in a single unaligned hardware instruction.

Protobuf: protowire.ConsumeVarint 100 Lines • Cannot Inline

func ConsumeVarint(b []byte) (v uint64, n int) {
    var y uint64
    if len(b) <= 0 { return 0, errCodeTruncated }
    v = uint64(b[0])
    if v < 0x80 { return v, 1 }
    v -= 0x80

    if len(b) <= 1 { return 0, errCodeTruncated }
    y = uint64(b[1]); v += y << 7
    if y < 0x80 { return v, 2 }
    v -= 0x80 << 7

    if len(b) <= 2 { return 0, errCodeTruncated }
    y = uint64(b[2]); v += y << 14
    if y < 0x80 { return v, 3 }
    v -= 0x80 << 14
    // ... repeats 7 more times up to b[9] ...
}

OmniBuf: ReadVarintAt 12 Lines • Inlines Everywhere

func ReadVarintAt(b []byte, pos int) (uint64, int) {
    b0 := b[pos]
    if (b0 & 1) != 0 {
        return uint64(b0 >> 1), pos + 1
    }
    if (b0 & 2) != 0 {
        return uint64(binary.LittleEndian.Uint16(b[pos:]) >> 2), pos + 2
    }
    if (b0 & 4) != 0 {
        return uint64(binary.LittleEndian.Uint32(b[pos:]) >> 3), pos + 4
    }
    return binary.LittleEndian.Uint64(b[pos+1:]), pos + 9
}

Operation	Protobuf (`protowire`)	`vtprotobuf`	OmniBuf `TZCNT`	Speedup
Decode 1B Int	1.27 ns	0.79 ns	0.35 ns	2x to 4x faster
Decode 2B Int	1.26 ns	1.26 ns	0.78 ns	~40% faster
Decode 4B Int	1.81 ns	2.28 ns	0.97 ns	~2x faster
Decode 8B Int	4.83 ns	5.04 ns	1.54 ns	Over 3x faster
Decode 16 Mixed Ints	38.1 ns	39.7 ns	25.2 ns	~35% faster
Skip 16 Mixed Ints	38.3 ns	39.7 ns	24.7 ns	~35% faster
Encode 16 Mixed Ints	60.4 ns	59.4 ns	37.6 ns	~35% faster

Header Hopping: Skipping Fields to Reach a Target

message Node {
  uint32 id = 16;  // 5,000,000 (4B int)
  bool active = 3; // true
  string city = 9; // "FRA" (Target)
}

To skip past id (a 2-byte tag + 4-byte integer) and active (a boolean) to read city, Protobuf has to bounce across every single varint byte to check continuation bits. OmniBuf reads the width from the first byte and leaps over the entire tag or value in one go.

Tag Byte Value Byte Target Field

How OmniBuf Works on the wire.