Serialization Binary

Table of Contents

• Dependencies
• The decision this library makes
• A representative reflected model
• What versioned loading can and cannot absorb
• Storage, allocation, and lifetime
• Binary layout and the packed fast path
• Fit with neighboring libraries
• Current limits
• API reference
• Statistics

🟨 Reflection-driven binary persistence with exact and schema-aware reads

SaneCppSerializationBinary.h turns a reflected C++ object graph into a compact sequence of bytes and reconstructs it later. It is aimed at local state, caches, and other controlled persistence where the producer and consumer share the same basic C++ data model.

This is deliberately not a general-purpose wire format. Primitive values and packed objects are copied in their native representation, and an embedded schema is itself a binary array of Reflection metadata. Endianness, primitive sizes, compiler ABI, and schema representation therefore matter. Choose a format with an explicit portable encoding when data must cross unrelated architectures, toolchains, or languages.

Dependencies

• Dependencies: Reflection
• All dependencies: Reflection

The decision this library makes

Serialization Binary separates two concerns that are often conflated:

• write emits only object data. loadExact reads it with the current reflected layout and is the direct, fast path.
• loadVersioned receives the source Reflection flat schema separately and maps fields by stable memberTag values into the destination type.
• writeWithSchema prepends that schema, while loadVersionedWithSchema selects loadExact when the embedded schema equals the current schema and otherwise falls back to the versioned reader.

The useful mental model is therefore bytes plus a schema identity, not a self-describing archive in the usual sense. If the schema identity is already stored in a file header, database record, or protocol envelope, the separate write/loadExact/loadVersioned APIs avoid repeating it. For a small, self-contained local state file, writeWithSchema and loadVersionedWithSchema are the more convenient pair.

A representative reflected model

Types are described with Reflection and can be nested from primitives, fixed arrays, reflected structs, and supported container adapters. This definition is compiled as part of SerializationBinaryTest:

 
struct SC::SerializationSuiteTest::PrimitiveStruct
{
    uint8_t arrayValue[4] = {0, 1, 2, 3};
    float   floatValue    = 1.5f;
    int64_t int64Value    = -13;
 
    bool operator!=(const PrimitiveStruct& other) const
    {
        for (size_t i = 0; i < SizeOfArray(arrayValue); ++i)
        {
            if (arrayValue[i] != other.arrayValue[i])
                return true;
        }
        if (floatValue != other.floatValue)
            return true;
        if (int64Value != other.int64Value)
            return true;
        return false;
    }
};
 
SC_REFLECT_STRUCT_VISIT(SC::SerializationSuiteTest::PrimitiveStruct)
SC_REFLECT_STRUCT_FIELD(0, arrayValue)
SC_REFLECT_STRUCT_FIELD(1, floatValue)
SC_REFLECT_STRUCT_FIELD(2, int64Value)
SC_REFLECT_STRUCT_LEAVE()
 
struct SC::SerializationSuiteTest::NestedStruct
{
    int16_t         int16Value = 244;
    PrimitiveStruct structsArray[2];
    double          doubleVal = -1.24;
    Array<int, 7>   arrayInt  = {1, 2, 3, 4, 5, 6};
 
    bool operator!=(const NestedStruct& other) const
    {
        if (int16Value != other.int16Value)
            return true;
        for (size_t i = 0; i < SizeOfArray(structsArray); ++i)
            if (structsArray[i] != other.structsArray[i])
                return true;
        if (doubleVal != other.doubleVal)
            return true;
        return false;
    }
};
 
SC_REFLECT_STRUCT_VISIT(SC::SerializationSuiteTest::NestedStruct)
SC_REFLECT_STRUCT_FIELD(0, int16Value)
SC_REFLECT_STRUCT_FIELD(1, structsArray)
SC_REFLECT_STRUCT_FIELD(2, doubleVal)
SC_REFLECT_STRUCT_LEAVE()
 
struct SC::SerializationSuiteTest::TopLevelStruct
{
    NestedStruct nestedStruct;
 
    bool operator!=(const TopLevelStruct& other) const { return nestedStruct != other.nestedStruct; }
};
 
SC_REFLECT_STRUCT_VISIT(SC::SerializationSuiteTest::TopLevelStruct)
SC_REFLECT_STRUCT_FIELD(0, nestedStruct)
SC_REFLECT_STRUCT_LEAVE()

For an unchanged TopLevelStruct, pass a caller-owned appendable buffer to SC::SerializationBinary::write, then pass its Span<const char> to SC::SerializationBinary::loadExact. Both calls return bool; a failed append, a short input, an unsupported resize, or trailing bytes makes the operation fail.

For durable local state, the repository's Serialization example uses writeWithSchema to fill an SC::Buffer, writes that buffer with File, reads the file back into another buffer, and calls loadVersionedWithSchema. This keeps file I/O out of the serializer and makes buffer ownership explicit.

What versioned loading can and cannot absorb

The versioned reader uses member tags rather than member declaration order. With stable, unique tags it can match moved fields, ignore source fields absent from the destination, leave newly added destination fields at their existing values, shorten arrays, and perform supported primitive conversions. The test suite exercises a source and destination whose fields are reordered and removed:

struct SC::SerializationSuiteTest::VersionedStruct1
{
    float          floatValue     = 1.5f;
    int64_t        fieldToRemove  = 12;
    Vector<String> field2ToRemove = {"ASD1", "ASD2", "ASD3"};
    int64_t        int64Value     = -13;
};
 
SC_REFLECT_STRUCT_VISIT(SC::SerializationSuiteTest::VersionedStruct1)
SC_REFLECT_STRUCT_FIELD(2, field2ToRemove)
SC_REFLECT_STRUCT_FIELD(0, floatValue)
SC_REFLECT_STRUCT_FIELD(1, fieldToRemove)
SC_REFLECT_STRUCT_FIELD(3, int64Value)
SC_REFLECT_STRUCT_LEAVE()
 
struct SC::SerializationSuiteTest::VersionedStruct2
{
    int64_t int64Value = 55;
    float   floatValue = -2.9f;
 
    bool operator!=(const VersionedStruct1& other) const
    {
        if (floatValue != other.floatValue)
            return true;
        if (int64Value != other.int64Value)
            return true;
        return false;
    }
};
 
SC_REFLECT_STRUCT_VISIT(SC::SerializationSuiteTest::VersionedStruct2)
SC_REFLECT_STRUCT_FIELD(3, int64Value)
SC_REFLECT_STRUCT_FIELD(0, floatValue)
SC_REFLECT_STRUCT_LEAVE()
 

SerializationBinaryOptions controls the intentionally lossy cases. Its defaults currently allow float-to-integer truncation, dropping excess array items, dropping unmatched struct members, and conversions between bool and other primitive types. Tighten those flags when silent loss would be worse than rejecting old data.

Versioned loading is best-effort migration, not arbitrary data evolution. Changing the meaning of a tag, reusing a retired tag, or changing to an unsupported shape still needs an application-level migration. It is also the caller's job to retain or embed the exact source schema; compiling only today's schema cannot describe yesterday's byte stream.

Storage, allocation, and lifetime

The serializer itself does not own an archive and does not allocate hidden working storage:

• Writers append to the buffer supplied by the caller. A fixed-capacity buffer gives a bounded, allocation-free write; a growable Buffer or Vector may allocate according to that container's policy.
• Readers borrow the input Span<const char> for the duration of the call and copy values into the destination. The byte span and a separately supplied schema span only need to remain valid until the call returns.
• Deserializing a variable-length container resizes the destination. Whether that allocates, and whether it can fail for insufficient capacity, is determined by the container and its Reflection adapter.
• The entire encoded value must currently be present in memory. There is no incremental reader or writer, so file I/O commonly requires both the serialized buffer and destination object to coexist.

To serialize SC::Vector, SC::Array, SC::String, or SC::Buffer, also include the appropriate headers from Serialization Adapters. Those adapters live outside this library so Serialization Binary depends only on Reflection; support for a container is not implied merely because it is an SC type.

Binary layout and the packed fast path

The format is intentionally close to memory:

• Primitive values are copied using their native representation and endianness.
• A non-packed struct writes its reflected members in visit order; member tags are metadata for versioned matching, not field positions in the byte stream.
• A fixed C array writes its elements in sequence.
• A vector-like value writes its payload size in bytes as a uint64_t, followed by its elements. This is a byte count, not an element count.
• A recursively Packed struct, array, or container payload can be transferred in one operation. Otherwise the walker descends into its fields or elements.

Here Packed is Reflection's stronger property: the complete value can be copied without padding or omitted state. It is not the same as applying a compiler packing pragma. A static_assert on Reflection's packed trait can protect code that relies on the one-operation path, but changing packing or native representation still changes the stored bytes.

numberOfWrites and numberOfReads expose the number of byte-transfer operations and are useful for checking this optimization in tests. They are not a stable part of the file format and should not be used as a compatibility marker.

Fit with neighboring libraries

Reflection supplies the type graph, field tags, packed analysis, and flat schemas; this library supplies the binary traversal and compatibility logic. [Serialization Adapters](Containers Reflection) connects container and memory types without introducing library-to-library dependencies.

Serialization Text is the neighboring choice when inspectability and a conventional text representation matter more than compact native bytes. Binary serialization avoids tokenization and can collapse packed data into large copies, while text is easier to inspect, edit, diff, and exchange. Neither choice removes the need for deliberate field tags and migration policy.

Current limits

The library is marked MVP: the exact and versioned paths are exercised for primitives, nested structs, arrays, vectors, strings, conversions, dropped data, and packed structs, but container coverage and compatibility testing are not yet complete. There is no streaming mode, integrity check, authentication, compression, object identity, pointer graph, or endianness normalization. Treat input as trusted unless the surrounding application validates its size and provenance; lengths in untrusted archives can drive destination-container growth.

The Serialization example shows binary and JSON persistence side by side. The July 2024 update also discusses the serializer's development.

API reference

This is a versioned binary serializer / deserializer built on top of Reflection.
It uses struct member iterators on Reflection schema serialize all members and the recursively Packed property for optimizations, reducing the number of read / writes (or memcpy) needed.

Conversion options for the binary versioned deserializer.

Statistics

LOC counts exclude comments. Library counts files physically under Libraries/SerializationBinary. Single File counts SaneCppSerializationBinary.h. Standalone counts SaneCppSerializationBinaryStandalone.h and intentionally includes dependency payloads.

Metric	Header	Source	Sum
Library	593	0	593
Single File	1013	0	1013
Standalone	1943	0	1943