Wire Format
Specification.

1. Design Goals & Overview

1.1 Design Goals

OmniBuf is designed around four core goals:

1. Stay Compatible with .proto File Semantics and Types: OmniBuf uses existing .proto definitions, field numbers (1..N ), scalar types, nested messages, repeated lists, and maps without requiring a new IDL, custom memory layouts, or manual alignment rules. Existing Protocol Buffers schemas and generated struct types can target the OmniBuf wire format directly.
2. Faster Encoding and Decoding on Modern Hardware: Protocol Buffers (designed in 2001) relies on 7-bit LEB128 continuation loops ( 0x80 branch per byte), two-pass Size() calculations for nested sub-messages, and byte-by-byte varint parsing when skipping fields. OmniBuf aligns the wire format with modern 64-bit little-endian CPUs:
   • Single-instruction trailing-zero count (TZCNT / CTZ) varints load 1, 2, 4, or 8 bytes in a single unaligned little-endian memory access.
   • Single-pass forward encoding reserves and backpatches 2-byte length prefixes in place without a pre-pass.
   • Explicit 4-bit wire types and outer container byte-length prefixes allow decoders to skip entire fields or nested containers in a single branchless TZCNT header hop without parsing inside strings, sub-messages, or arrays.
3. Schema-Less Value Decoding (with Optional Inline Field Names): In Protocol Buffers, Wire Type 2 (LEN) is overloaded for strings, raw bytes, nested messages, and packed repeated scalars, making it impossible to decode a packet without the .proto schema. OmniBuf uses explicit 4-bit wire types (0..15) so any decoder can losslessly inspect and decode every field's exact type and structure (DecodeBlind, including distinguishing unsigned WireUint / WireUintArr integers from ZigZag-encoded signed WireSint / WireSintArr integers, fixed-width float arrays, and arbitrary typed maps) with zero schema knowledge. When human-readable keys are desired on the wire, an optional inline schema sub-message ( Field #0 , omnibuf.v1.SchemaMeta) embeds field-number-keyed field names and nested child schemas at byte 0 of the root message (or any nested sub-message).
4. Advanced Wire Techniques with a Flat Stream Layout: Rather than requiring vtables, footer offset tables, relative pointer graphs, or arena allocators, OmniBuf achieves single-pass encoding, fast zero-copy unmarshaling, raw byte concatenation (append(msgA, msgB...)), and rapid header-hop field/array/map traversal using a single contiguous stream of [Tag][Value] fields with 4 integer widths and 16 explicit wire types.

1.2 Core Wire Primitives

To achieve these goals, OmniBuf replaces Protobuf's Tag-Length-Value (TLV) + LEB128 with four wire primitives:

1. Branchless Prefix-Varints (TZCNT): The trailing zero bit count (TZCNT) of the first byte determines the total byte width (1, 2, 4, or 9 bytes) in little-endian order, eliminating byte-by-byte continuation-bit loops.
2. Explicit 4-Bit Wire Types (0..15): Every field tag encodes (FieldNumber << 4) | WireType, unambiguously distinguishing unsigned integers (WireUint, WireUintArr), ZigZag-encoded signed integers (WireSint, WireSintArr), strings, bytes, sub-messages, string arrays, length-prefixed sub-message arrays, typed maps, fixed-width floats/ints (and packed arrays WireFixedArr), zero-byte booleans (true/false encoded directly in the tag), and a length-prefixed extension escape hatch (WireExtended = 15).
3. Single-Pass Overlong Length Backpatching: Variable-length containers ( WireSubMsg, WireStringArr, WireUintArr, WireSintArr, WireSubMsgArr, WireMap) reserve a 2-byte Prefix-Varint length slot (TZCNT = 1) before writing their body in a single forward pass and backpatch the 14-bit length upon completion, eliminating two-pass Size() pre-computation.
4. Concat-Compatible Field Stream with Outer Container Lengths: Because every scalar width is encoded in b[0] via TZCNT and every container (WireString, WireBytes, WireSubMsg, WireStringArr, WireUintArr, WireSubMsgArr, WireMap, WireSintArr, WireFixedArr, WireExtended) begins with its total ByteLen, a decoder can skip any field in a single O(1) arithmetic jump without parsing inside sub-messages or arrays, and concatenating two encoded OmniBuf messages ( append(msgA, msgB...)) produces a valid merged OmniBuf message.

2. Message Packet Layout & Concatenation Semantics

Every OmniBuf root message (and every nested sub-message in WireSubMsg or WireSubMsgArr) is a contiguous sequence of encoded [Tag][Value] fields:

 0                                                                                N
 +--------------------+--------------------+--------------------+-----------------+
 | Field #0 (opt)     | Field #f_1         | Field #f_2         | ... Field #f_k  |
 | [Tag=0x0B][Schema] | [Tag][Value]       | [Tag][Value]       | [Tag][Value]    |
 +--------------------+--------------------+--------------------+-----------------+
 |<---------------------------------- N bytes ----------------------------------->|

1. Field Ordering & Raw Message Concatenation (append(msgA, msgB...)):
   • Encoders SHOULD emit fields in ascending order of FieldNumber for deterministic output.
   • Decoders MUST accept fields in any order and MUST support duplicate field numbers so that concatenating two encoded messages (append(msgA, msgB...)) or appending an updated [Tag][Value] to the end of an existing packet is valid and follows Protocol Buffers merge semantics:
     • Singular Scalar Fields (WireUint, WireSint, WireBoolFalse, WireBoolTrue, WireString , WireBytes, WireFixed32, WireFixed64): Last occurrence wins ("last write wins").
     • Repeated Fields (WireStringArr, WireUintArr, WireSubMsgArr, WireSintArr, WireFixedArr): Elements from subsequent occurrences are appended in order to elements from earlier occurrences.
     • Nested Sub-Messages (WireSubMsg): Subsequent occurrences of the same sub-message field are merged recursively into the sub-message decoded from earlier occurrences.
     • Maps (WireMap): Entries from subsequent occurrences of the same map field are merged into the map (with later keys overriding earlier keys).
2. Zero-Byte Empty Message: An empty message with zero present fields is 0 bytes (N = 0).
3. No Trailing Footer or Padding: The message ends immediately after the last byte of the final present field.

3. Prefix-Varint Encoding (TZCNT)

All variable-length unsigned integers (field tags, WireUint values, string/byte/container lengths, and array/map counts) are encoded as Prefix-Varints.

3.1 Bit-Level Layout

The number of trailing zero bits (TZCNT) in Byte 0 (b[0]) specifies the total byte length of the integer (1, 2, 4, or 9 bytes). The remaining high bits of the little-endian word store the unsigned value:

1-Byte Prefix-Varint (TZCNT = 0, 7-bit value):
  Byte 0
  +---+---+---+---+---+---+---+---+
  | v6| v5| v4| v3| v2| v1| v0| 1 |
  +---+---+---+---+---+---+---+---+

2-Byte Prefix-Varint (TZCNT = 1, 14-bit value, Little-Endian):
  Byte 0 (LSB)                      Byte 1 (MSB)
  +---+---+---+---+---+---+---+---+ +---+---+---+---+---+---+---+---+
  | v5| v4| v3| v2| v1| v0| 1 | 0 | |v13|v12|v11|v10| v9| v8| v7| v6|
  +---+---+---+---+---+---+---+---+ +---+---+---+---+---+---+---+---+

4-Byte Prefix-Varint (TZCNT = 2, 29-bit value, Little-Endian):
  Byte 0 (LSB)                      Byte 1          Byte 2          Byte 3 (MSB)
  +---+---+---+---+---+---+---+---+ +-------------+ +-------------+ +-------------+
  | v4| v3| v2| v1| v0| 1 | 0 | 0 | |  v12 .. v5  | |  v20 .. v13 | |  v28 .. v21 |
  +---+---+---+---+---+---+---+---+ +-------------+ +-------------+ +-------------+

9-Byte Prefix-Varint (Byte 0 == 0x00, 64-bit value, Little-Endian):
  Byte 0                            Bytes 1..8
  +---+---+---+---+---+---+---+---+ +---------------------------------------------+
  | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |       uint64 little-endian (v63 .. v0)      |
  +---+---+---+---+---+---+---+---+ +---------------------------------------------+

3.2 Prefix-Varint Binary & Hex Examples

Example 1: v = 0 (1 Byte, TZCNT = 0)

• Formula: (0 << 1) | 1 = 1
• Binary: 00000001
• Hex: 01

Example 2: v = 127 (Maximum 1-Byte Value, TZCNT = 0)

• Formula: (127 << 1) | 1 = 255
• Binary: 11111111
• Hex: ff

Example 3: v = 128 (Minimum Canonical 2-Byte Value, TZCNT = 1)

• Formula: uint16(128 << 2) | 2 = 512 | 2 = 514 = 0x0202
• Little-Endian Bytes: b[0] = 0x02 (00000010, TZCNT = 1), b[1] = 0x02 (00000010)
• Hex: 02 02

Example 4: v = 300 (2 Bytes, TZCNT = 1)

• Binary of 300: 00000001 00101100 (0x012C)
• Shift left by 2 bits and OR with 0b10 (2): (300 << 2) | 2 = 1200 | 2 = 1202 = 0x04B2
• Little-Endian Bytes: b[0] = 0xB2 (10110010, low 2 bits are 10 → TZCNT = 1), b[1] = 0x04 ( 00000100)
• Hex: b2 04
• Decoding check: LE16(b2 04) >> 2 = 0x04B2 >> 2 = 1202 >> 2 = 300

Example 5: v = 16,384 (Minimum 4-Byte Value, TZCNT = 2)

• Formula: uint32(16384 << 3) | 4 = 131072 | 4 = 131076 = 0x00020004
• Little-Endian Bytes: b[0] = 0x04 (00000100, TZCNT = 2), b[1] = 0x00, b[2] = 0x02, b[3] = 0x00
• Hex: 04 00 02 00

Example 6: v = 536,870,912 (2²⁹, Minimum 9-Byte Value, TZCNT >= 3)

• Since v >= 2^29, Byte 0 is 0x00 followed by uint64(536870912) = 0x0000000020000000 in little-endian:
• Hex: 00 00 00 00 20 00 00 00 00

3.3 Overlong 2-Byte Prefix-Varints (Single-Pass Length Backpatching)

Decoders MUST dispatch strictly on TZCNT(b[0]) without rejecting non-minimal ("overlong") encodings. Specifically, a value v < 128 encoded using the 2-byte TZCNT = 1 formula (uint16(v << 2) | 2) is valid on the wire.

• Fast Path (byteLen <= 16,383): When encoding a length-prefixed container (WireSubMsg, WireStringArr, WireUintArr, WireSintArr, WireSubMsgArr, WireMap) in a single forward pass, the encoder reserves 2 bytes at lenPos, writes the container body directly into the output buffer, and backpatches the 2-byte length slot in place: LE16(buf[lenPos : lenPos+2]) = (byteLen << 2) | 2.
• Large Container Expansion (byteLen > 16,383): If a container exceeds 16,383 bytes, BackpatchContainerLen shifts the container body right by 2 bytes (for 4-byte Prefix-Varint up to 512 MB) or 7 bytes (for 9-byte Prefix-Varint) and writes the wider Prefix-Varint length header. Because containers close from innermost to outermost, outer containers automatically include any expanded inner header in their byte length.
• Example (byteLen = 42 encoded as Overlong 2-Byte Prefix-Varint): (42 << 2) | 2 = 168 | 2 = 170 = 0x00AA → Hex: aa 00 . Decoder reads TZCNT(0xAA) == 1 (2 bytes) and computes 0x00AA >> 2 = 42.

4. Signed Integers (ZigZag Encoding, WireSint = 12)

Signed integers (int32, sint32, int64, sint64) are mapped to unsigned uint64 values via ZigZag encoding before being written as a WireSint ( 12) Prefix-Varint. Giving signed integers their own explicit 4-bit wire type (WireSint = 12, distinct from WireUint = 0) allows schema-less decoders (DecodeBlind) to losslessly ZigZag-decode signed integers back to negative or positive int64 values directly from wire bytes without a .proto schema.

4.1 ZigZag Formulas

// Encoding (int64 -> uint64):
u := uint64((n << 1) ^ (n >> 63)) // arithmetic right shift by 63 bits

// Decoding (uint64 -> int64):
n := int64(u >> 1) ^ -int64(u & 1)

4.2 ZigZag Binary & Hex Examples

5. Field Tags & Explicit 4-Bit Wire Types

Every field in the message begins with a Tag encoded as a Prefix-Varint:

Tag = (FieldNumber << 4) | WireType

• FieldNumber is the non-negative schema field number (0 for the inline schema table, 1..N for data fields).
• WireType is a 4-bit unsigned integer (0..15).

5.1 Tag Width for Field Numbers 0..7 vs 8..1023

Fields #0 through #7 have Tag = (FieldNumber << 4) | WireType <= 127. They ALWAYS encode as a 1-byte tag:

1-Byte Tag Layout (Fields #0..#7):
  Bit 7   Bit 6   Bit 5   Bit 4   Bit 3   Bit 2   Bit 1   Bit 0
+-------+-------+-------+-------+-------+-------+-------+-------+
|   FieldNumber (3 bits: 0..7)  |     WireType (4 bits) |   1   |
+-------+-------+-------+-------+-------+-------+-------+-------+
Byte 0 = uint8(FieldNumber << 5) | uint8(WireType << 1) | 0x01

Fields #8 through #1023 have 128 <= Tag <= 16383. They ALWAYS encode as a 2-byte little-endian tag:

2-Byte Tag Layout (Fields #8..#1023):
  Byte 0 (LSB)                                    Byte 1 (MSB)
+-------+-------+-------+-------+-------+-------+ +-------------------------------+
| fn[1] | fn[0] |     WireType (4 bits) | 1 | 0 | |     FieldNumber[9..2]         |
+-------+-------+-------+-------+-------+-------+ +-------------------------------+
LE16(Byte 0..1) = uint16(FieldNumber << 6) | uint16(WireType << 2) | 0x02

6. Wire Type Specifications (0..15)

6.1 WireUint (WireType = 0) & WireSint ( WireType = 12)

• WireUint (0): Encodes an unsigned integer (uint32, uint64, enum ) as a single Prefix-Varint immediately after the tag.
• WireSint (12): Encodes a signed integer (int32, int64, sint32, sint64) mapped via ZigZag (EncodeZigZag(n)) as a single Prefix-Varint immediately after the tag.

Example 1: Field #1 (asn = 64512, WireUint = 0):

1. Tag: (1 << 4) | 0 = 16 → 1-byte Prefix-Varint (16 << 1) | 1 = 0x21.
2. Value: 64512 (> 16383, 4-byte Prefix-Varint TZCNT = 2): (64512 << 3) | 4 = 0x0007E004 → little-endian 04 e0 07 00.

21  04 e0 07 00
^^  ^^^^^^^^^^^
Tag Value = 64512 (LE32: 0x0007E004 >> 3)

Example 2: Field #5 (rssi_dbm = -62, WireSint = 12):

1. Tag: (5 << 4) | 12 = 92 → 1-byte Prefix-Varint (92 << 1) | 1 = 185 = 0xB9.
2. ZigZag of -62: EncodeZigZag(-62) = 123 → 1-byte Prefix-Varint (123 << 1) | 1 = 247 = 0xF7.

b9  f7
^^  ^^
Tag Value = -62 (ZigZag 123 -> DecodeZigZag(0xF7 >> 1) = -62)

6.2 WireBoolFalse (WireType = 1) & WireBoolTrue (WireType = 2)

Boolean values consume 0 payload bytes after the tag; the boolean state is encoded directly in the 4-bit WireType.

65
^^
Field #3, WireBoolTrue (true) -- (if false, tag is 0x63)

6.3 WireString (WireType = 3) & WireBytes ( WireType = 4)

Followed by [ByteLen (Prefix-Varint)] and [ByteLen raw bytes]. Strings are UTF-8 encoded without a trailing NUL terminator.

47  09  72 74 72 31
^^  ^^  ^^^^^^^^^^^
Tag Len "rtr1" (Field #2, ByteLen = 4)

6.4 WireSubMsg (WireType = 5)

Encodes a nested message as [ByteLen (Prefix-Varint)][Inner Message Bytes]. Encoders typically emit ByteLen as a 2-byte Prefix-Varint ( TZCNT = 1) via single-pass backpatching, though decoders MUST accept any valid Prefix-Varint width for ByteLen.

Inner Sub-Message (7 bytes):
  Offset 0x00: 67 07 46 52 41   -> Inner Field #3 (WireString): tag=0x67, len=3 (0x07), "FRA"
  Offset 0x05: 81 55            -> Inner Field #4 (WireUint):   tag=0x81, val=42 (0x55 = 42<<1|1)

Outer Field #9 Wrapping (2B Tag + 2B Overlong ByteLen + 7B Inner Message = 11 bytes):
  56 02  1e 00  67 07 46 52 41 81 55
  ^^^^^  ^^^^^  ^^^^^^^^^^^^^^^^^^^^
  Tag#9  Len=7  Inner Sub-Message (7 bytes)

6.5 WireStringArr (WireType = 6)

Encodes a list of length-prefixed strings or byte slices (repeated string or repeated bytes) with an outer ByteLen prefix so decoders and skippers can jump over the entire array in a single O(1) addition without walking individual elements:

+-----+---------------------------+----------------------------+-------------------------------+
| Tag | ByteLen (Prefix-Varint)   | ElemCount (Prefix-Varint)  | [StrLen (Varint)][UTF-8]...   |
+-----+---------------------------+----------------------------+-------------------------------+
                                  |<---------------------- ByteLen bytes --------------------->|

Example: Field #7 (tags = ["a", "bc"]):
  ed  1a 00  05   03 61   05 62 63
  ^^  ^^^^^  ^^^  ^^^^^   ^^^^^^^^
  Tag Len=6  N=2  "a"     "bc"

Sharing WireStringArr (6) between repeated string and repeated bytes keeps all 16 wire types (0..15) packed into 4 bits; schema-aware decoders know the target slice type ([]string vs. [][]byte), while schema-less text/JSON formatters can check utf8.Valid(elem) when rendering elements.

6.6 WireUintArr (WireType = 7) & WireSintArr ( WireType = 13)

Encodes a packed array of unsigned (WireUintArr) or ZigZag-encoded signed ( WireSintArr) integers with an outer byte-length prefix so decoders and skippers can jump over the entire array in a single addition without decoding individual varints, while preserving signedness in schema-less decoding (DecodeBlind):

+-----+---------------------------+----------------------------+--------------------------+
| Tag | ByteLen (Prefix-Varint)   | ElemCount (Prefix-Varint)  | Elem_0 ... Elem_N-1      |
+-----+---------------------------+----------------------------+--------------------------+
                                  |<------------------- ByteLen bytes ------------------->|

Example: Field #8 (prefixes = [10, 300]):
  1e 02  12 00  05   15  b2 04
  ^^^^^  ^^^^^  ^^^  ^^  ^^^^^
  Tag#8  Len=4  N=2  10  300

6.7 WireSubMsgArr (WireType = 8)

Encodes repeated message as a single contiguous container with an outer ByteLen prefix followed by length-prefixed sub-message elements:

+-----+---------+-----------------------------+-----+-----------------------------------+
| Tag | ByteLen | ElemLen_0 (Varint) | Elem_0 | ... | ElemLen_{N-1} (Varint) | Elem_N-1 |
+-----+---------+-----------------------------+-----+-----------------------------------+
                |<--------------------------- ByteLen bytes --------------------------->|

• ByteLen >= 0. If ByteLen == 0, the array contains 0 elements.
• Each element i is preceded by its byte length ElemLen_i encoded as a Prefix-Varint, followed by ElemLen_i bytes of the element's OmniBuf message.
• The sum of all (len(ElemLen_i) + ElemLen_i) across all elements MUST equal ByteLen exactly.

6.8 WireMap (WireType = 9)

Encodes any Protobuf map<K, V> as a length-prefixed container with a 1-byte MapType header followed by EntryCount key-value pairs:

+-----+---------+-------------------------------+----------------------------+-------------------+
| Tag | ByteLen | MapType (KeyWT<<4 | ValWT)    | EntryCount (Prefix-Varint) | [Key][Value]...   |
+-----+---------+-------------------------------+----------------------------+-------------------+
                |<-------------------------------- ByteLen bytes ------------------------------->|

• 1-Byte MapType Header: Immediately after ByteLen, a single byte encodes (KeyWireType << 4) | ValWireType:
  • KeyWireType: WireUint (0, for uint32/uint64/bool), WireString (3, for string), WireFixed32 (10), WireFixed64 (11), or WireSint ( 12, for int32/int64/sint32/sint64).
  • ValWireType: WireUint (0, for uint32/uint64/enum/bool), WireString (3), WireBytes (4), WireSubMsg (5), WireFixed32 (10), WireFixed64 ( 11), or WireSint (12).
• Minimum Container Length (ByteLen >= 2): Every WireMap container includes the 1-byte MapType header followed by EntryCount (Prefix-Varint, at least 1 byte), so ByteLen MUST be >= 2 (an empty map with EntryCount = 0 has ByteLen = 2).
• Untagged Self-Framing Entries: Each of the EntryCount entries consists of the untagged payload of KeyWireType followed immediately by the untagged payload of ValWireType (for example, when ValWireType is WireSubMsg (5), the value payload is [ByteLen (Prefix-Varint)][Inner Message Bytes]).
• Unordered Keys & "Last Write Wins": Entries may appear in any key order, allowing encoders to iterate native hash maps directly in O(N) time with zero sorting or temporary slice allocations. If a key appears multiple times (within a container or across concatenated messages), the last occurrence wins.
• The end of the last entry (Entry_{EntryCount-1}) MUST align exactly with ByteLen.

6.9 WireFixed32 (WireType = 10) & WireFixed64 ( WireType = 11)

• WireFixed32 (10): Followed by exactly 4 bytes in little-endian order (uint32 , int32, or IEEE-754 binary32 float32).
• WireFixed64 (11): Followed by exactly 8 bytes in little-endian order (uint64 , int64, or IEEE-754 binary64 float64).

Example: Field #6 (cpu_load = 0.5 as float64):
  d7  00 00 00 00 00 00 e0 3f
  ^^  ^^^^^^^^^^^^^^^^^^^^^^^
  Tag IEEE-754 0.5 (LE64)

6.10 WireFixedArr (WireType = 14)

Encodes a packed array of fixed-width 32-bit or 64-bit little-endian elements with an outer ByteLen prefix followed by a 1-byte ElemType header:

+-----+---------+-------------------------------------+------------------------------+
| Tag | ByteLen | ElemType (1B: WireFixed32=10 or 11) | Little-Endian Elements...    |
+-----+---------+-------------------------------------+------------------------------+
                |<-------------------------- ByteLen bytes ------------------------->|

• ByteLen >= 1: Every WireFixedArr container includes the 1-byte ElemType header, so ByteLen MUST be >= 1.
• ElemType = WireFixed32 (10): Followed by (ByteLen - 1) / 4 little-endian 4-byte elements ( repeated float, repeated fixed32, repeated sfixed32). (ByteLen - 1) MUST be a multiple of 4.
• ElemType = WireFixed64 (11): Followed by (ByteLen - 1) / 8 little-endian 8-byte elements ( repeated double, repeated fixed64, repeated sfixed64). (ByteLen - 1) MUST be a multiple of 8.

6.11 WireExtended (WireType = 15)

Reserves WireType = 15 as a forward-compatible, length-prefixed extension escape hatch capable of expressing up to 256 extended wire types while remaining skippable in O(1) time by existing decoders:

+-----+---------------------------+-----------------------+-----------------------------------+
| Tag | ByteLen (Prefix-Varint)   | ExtType (1B: 0..255)  | ExtPayload (ByteLen - 1 bytes)    |
+-----+---------------------------+-----------------------+-----------------------------------+
                                  |<--------------------- ByteLen bytes --------------------->|

• ByteLen >= 1: Every WireExtended container starts with ByteLen followed by a 1-byte extended type identifier ExtType (0..255) and ByteLen - 1 bytes of extended payload.
• O(1) Forward-Compatible Skipping: Because WireExtended begins with ByteLen just like every other variable-length container, SkipValue skips any unknown future WireExtended field in a single arithmetic jump ( pos + tn + byteLen) without needing to understand ExtType.
• In Version 1.0, no ExtType values are assigned; schema-less decoders ( DecodeBlind) MUST reject unrecognized ExtType values, while schema-aware decoders MUST skip unknown fields carrying WireExtended.

7. Branchless TZCNT Header-Hop Field Lookup (LocateField)

In Protocol Buffers, skipping a preceding varint field requires a byte-by-byte continuation-bit loop. In OmniBuf, every scalar width is determined from b[0] via TZCNT, booleans are 0 bytes after the tag, and every container starts with ByteLen, enabling an O(1) arithmetic jump per field without parsing inside containers:

// LocateField scans packet via branchless TZCNT header hops (O(1) per-field skip)
// and returns the value slice of the last occurrence of targetField.
pos := 0
var valSlice []byte
var wireType uint8
found := false

for pos < len(packet) {
    tag, tn := DecodeVarint(packet[pos:])
    if tn == 0 {
        return absent
    }
    pos += tn
    fnum  := uint16(tag >> 4)
    wtype := uint8(tag & 0x0F)

    if fnum == targetField {
        valSlice = packet[pos:]
        wireType = wtype
        found = true
    }
    pos = SkipValue(packet, pos, wtype)
}
return valSlice, wireType, found

8. Optional Inline Schema (Field #0, omnibuf.v1.SchemaMeta)

When self-describing mode is enabled, Field #0 (Tag = 0x0B, WireSubMsg = 5) is written at byte 0 of the root message (or at the beginning of a nested sub-message or concatenated message segment) as a regular OmniBuf sub-message encoding omnibuf.v1.SchemaMeta.

• Zero-Cost Skipping for Schema-Aware Decoders: Because Field #0 is a standard WireSubMsg (5) starting with ByteLen, a schema-aware decoder sees tag 0x0B, reads ByteLen, and jumps over the entire metadata block in a single O(1) addition.
• Dogfooding & Extensibility: Because Field #0 is defined by its own .proto schema (schemas/meta.proto), future optional metadata fields (such as schema hashes or enum value tables) can be added to SchemaMeta without changing the OmniBuf wire version.

8.1 omnibuf.v1.SchemaMeta Definition (meta.proto)

syntax = "proto3";
package omnibuf.v1;

// Encoded as Field #0 (Tag = 0x0B, WireSubMsg = 5) at byte 0 of a message.
message SchemaMeta {
  // Optional protobuf message type name (e.g., "PeerUpdate" or "acme.v1.Node").
  string type_name = 1;

  // Maps FieldNumber -> FieldName (e.g., 1 -> "id", 3 -> "active", 9 -> "location").
  // Encoded as WireMap (9) with MapType = 0x03 (KeyWT = WireUint(0), ValWT = WireString(3)).
  map<uint32, string> fields = 2;

  // Maps a parent FieldNumber (for a WireSubMsg, WireSubMsgArr, or WireMap value)
  // to its nested SchemaMeta so repeated sub-messages only encode their schema ONCE.
  // Encoded as WireMap (9) with MapType = 0x05 (KeyWT = WireUint(0), ValWT = WireSubMsg(5)).
  map<uint32, SchemaMeta> children = 3;
}

• Keyed by FieldNumber (map<uint32, string> fields = 2): Each entry in fields maps a FieldNumber (1B WireUint varint for fields 1..127) to its FieldName (WireString). Unlike positional string lists, keying by FieldNumber turns SchemaMeta into a reusable type descriptor that works across every instance of a message regardless of which optional fields are present or omitted.
• Hoisted Child Schemas (map<uint32, SchemaMeta> children = 3): Maps a parent field number (for a nested WireSubMsg, a WireSubMsgArr repeated sub-message array, or a WireMap sub-message value) to its child SchemaMeta.

8.2 Single Message Wire Encoding with Field #0

Consider a Location message (city = "FRA" at Field #3, rack = 42 at Field #4) encoded with Field #0 (type_name = "Location", fields = {3: "city", 4: "rack"}):

8.3 repeated message (WireSubMsgArr) Without Field-Name Duplication

In formats like JSON, an array of 10,000 sub-messages ( [{"city":"FRA","rack":42}, ...]) repeats the field name strings "city" and "rack" 10,000 times. In OmniBuf, the parent message's Field #0 hoists the child schema into children[fieldNum] once (or Element 0 of a standalone array carries Field #0), and all N elements in the WireSubMsgArr omit Field #0 completely:

Parent Message Field #0 (SchemaMeta):
  fields   = { 1: "id", 10: "neighbors" }
  children = {
    10: SchemaMeta {
      type_name: "Neighbor",
      fields: { 1: "peer_asn", 2: "router", 3: "latency_us", 4: "established" }
    }
  }

Data Field #10 (WireSubMsgArr, N = 10,000 elements):
  [ElemLen_0][Elem_0 (pure compact tags+values, 0 bytes of repeated field names)]
  [ElemLen_1][Elem_1 (pure compact tags+values, even if optional fields vary)]
  ...
  [ElemLen_9999][Elem_9999]

• When DecodeBlind decodes Field #10 (WireSubMsgArr , WireSubMsg, or a WireMap whose values are WireSubMsg), it passes activeSchema.children[10] down as the initial active schema for every child element.
• If an individual child element also includes its own inline Field #0, its SchemaMeta is merged with the inherited child schema.

8.4 Raw Concatenation (append(msgA, msgB...)) with Field #0

Because Field #0 is a standard WireSubMsg (5) whose maps use WireMap (9), concatenating two self-describing messages (append(msgA, msgB...)) works seamlessly without special framing: when DecodeBlind encounters a second Field #0 later in the stream, it recursively merges the new SchemaMeta (type_name, fields, and children) into the active SchemaMeta.

9. Complete End-to-End Annotated Wire Example

message Location {
  string city = 3; // "FRA"
  uint32 rack = 4; // 42
}

message Node {
  uint32   id       = 1; // 300
  bool     active   = 3; // true
  Location location = 9; // { city: "FRA", rack: 42 }
}

10. Wire Validity & Malformed Packet Rejection Rules

A conforming OmniBuf decoder MUST validate untrusted wire input and reject malformed or adversarial packets without panicking or reading out of bounds:

1. Empty Message: A 0-byte slice ( len(packet) == 0) is a valid empty message with zero present fields.
2. Field #0 Wire Type & Structure: Any occurrence of Field #0 MUST have WireType == WireSubMsg ( 5) and encode a valid omnibuf.v1.SchemaMeta sub-message (where fields is a WireMap with MapType = 0x03 and children is a WireMap with MapType = 0x05).
3. Defined 4-Bit Wire Types: Wire types 0..14 are core wire types and WireExtended (15) is the length-prefixed extension escape hatch. Any tag whose payload violates its wire type's structural rules MUST be rejected.
4. Payload & Container Bounds:
   • No Prefix-Varint or fixed-width value (WireFixed32, WireFixed64) may read past len(packet).
   • Length-prefixed fields (WireString, WireBytes, WireSubMsg, WireStringArr, WireUintArr, WireSubMsgArr, WireMap, WireSintArr, WireFixedArr, WireExtended) MUST satisfy byteLen <= uint64(len(packet) - pos).
   • WireStringArr, WireUintArr, and WireSintArr elements MUST end on an exact element boundary at pos + byteLen.
   • WireFixedArr MUST satisfy byteLen >= 1, with ElemType == WireFixed32 (10) and (byteLen - 1) % 4 == 0, or ElemType == WireFixed64 ( 11) and (byteLen - 1) % 8 == 0.
   • WireExtended MUST satisfy byteLen >= 1 (at least 1 byte for ExtType); schema-less decoders MUST reject unrecognized ExtType values, while schema-aware decoders skip unknown fields with WireExtended in O(1) time.
   • WireSubMsgArr elements MUST each satisfy elemLen <= uint64(len(arrPayload) - pos) and the final element MUST end exactly at len(arrPayload).
   • WireMap MUST satisfy byteLen >= 2, contain a valid 1-byte MapType header, satisfy entryCount*2 <= uint64(len(mapPayload) - pos) before allocating entries (preventing OOM pre-allocation attacks), and the final entry MUST end exactly at len(mapPayload). Duplicate keys follow "last write wins" semantics.
5. Resource & Message Size Limits:
   • Maximum Nesting Depth (MaxNestingDepth = 32): By default, sub-message recursion deeper than 32 levels MUST be rejected. Implementations MAY provide an option to configure a custom recursion limit.
   • Message & Container Byte-Size Limits: At the wire level, Prefix-Varints can represent container lengths up to 229 - 1 bytes (536,870,911 bytes, ~512 MiB) using a 4-byte header and up to 264 - 1 bytes using a 9-byte header. To guard against memory exhaustion on untrusted inputs, implementations and transport layers MAY enforce configurable maximum message size limits:
     • RPC and framing layers MAY enforce a default per-message limit of 4 MiB (4,194,304 bytes, matching gRPC's default receive limit), and general-purpose decoders MAY enforce a default total message limit of 64 MiB (67,108,864 bytes, matching Protocol Buffers' default limit) unless overridden by the caller.
     • Implementations MAY reject individual container or string lengths exceeding 536,870,911 bytes (4-byte Prefix-Varint maximum) unless explicitly configured for >512 MiB payloads.