Serialization Text

Table of Contents

• Features
  • JSON Serializer
• Status
• JSON Serializer
• Architecture
  • Json Tokenizer
• Roadmap

🟨 Serialize to / from text formats (JSON) using Reflection

Serialization Text uses Reflection, to serialize C++ objects to text based formats that currently includes a JSON serializer / deserializer.

Features

JSON Serializer

• Serialize primitive types
• Serialize SC::Vector, SC::Array, SC::String
• Serialize T[N] arrays
• Serialize structs made of above types or other structs

Status

🟨 MVP
Only JSON has been implemented and it needs additional testing.

JSON Serializer

SC::SerializationJson reads or writes C++ structures to / from json using Reflection information.
Let's consider the following structure described by Reflection:

struct SC::Test
{
    int    x      = 2;
    float  y      = 1.5f;
    int    xy[2]  = {1, 3};
    String myTest = "asdf"_a8;
 
    Vector<String> myVector = {"Str1"_a8, "Str2"_a8};
 
    bool operator==(const Test& other) const
    {
        return x == other.x and y == other.y and                   //
               xy[0] == other.xy[0] and xy[1] == other.xy[1] and   //
               myTest == other.myTest and myVector.size() == 2 and //
               myVector.size() == other.myVector.size() and        //
               myVector[0] == other.myVector[0] and myVector[1] == other.myVector[1];
    }
};
SC_REFLECT_STRUCT_VISIT(SC::Test)
SC_REFLECT_STRUCT_FIELD(0, x)
SC_REFLECT_STRUCT_FIELD(1, y)
SC_REFLECT_STRUCT_FIELD(2, xy)
SC_REFLECT_STRUCT_FIELD(3, myTest)
SC_REFLECT_STRUCT_FIELD(4, myVector)
SC_REFLECT_STRUCT_LEAVE()

This is how you can serialize the class to JSON

    constexpr StringView   testJSON = R"({"x":2,"y":1.50,"xy":[1,3],"myTest":"asdf","myVector":["Str1","Str2"]})"_a8;
    Test                   test;
    SmallVector<char, 256> buffer;
    StringFormatOutput     output(StringEncoding::Ascii, buffer);
 
    SC_TEST_EXPECT(SerializationJson::write(test, output));
    const StringView serializedJSON({buffer.data(), buffer.size() - 1}, false, StringEncoding::Ascii);
    SC_TEST_EXPECT(serializedJSON == testJSON);

This is how you can de-serialize the class from JSON, matching fields by their label name, even if they come in different order than the original class or even if there are missing field.

    constexpr StringView scrambledJson =
        R"({"y"  :  1.50, "x": 2.0, "myVector"  :  ["Str1","Str2"], "myTest":"asdf"})"_a8;
    Test test;
    test.x = 0;
    test.y = 0;
    (void)test.myVector.resize(1);
    (void)test.myTest.assign("FDFSA"_a8);
    SC_TEST_EXPECT(SerializationJson::loadVersioned(test, scrambledJson));
    SC_TEST_EXPECT(test == Test());

This is a special loader to deserialize the class from the exact same JSON that was output by the serializer itself. Whitespace changes are fine, but changing the order in which two fields exists or removing one will make deserialization fail. If these limitations are fine for the usage (for example the generated json files are not meant to be manually edited by users) than maybe it could be worth using it, as this code path is a lot simpler (*) than SC::SerializationJson::loadVersioned.

    constexpr StringView testJSON = R"({"x":2,"y":1.50,"xy":[1,3],"myTest":"asdf","myVector":["Str1","Str2"]})"_a8;
 
    Test test;
    test.x        = 1;
    test.y        = 3.22f;
    test.xy[0]    = 4;
    test.xy[1]    = 4;
    test.myTest   = "KFDOK";
    test.myVector = {"LPDFSOK", "DSAFKO"};
    SC_TEST_EXPECT(SerializationJson::loadExact(test, testJSON));
    SC_TEST_EXPECT(test == Test());

Note

(*) simpler code probably means faster code, even if it has not been properly benchmarked yet so the hypothetical performance gain is yet to be defined.

Architecture

SC::detail::SerializationTextReadVersioned provides common framework for all text / structured formats, walking the data structure using reflection information.
Every writer or reader needs to implement methods like startObject / endObject and startArray / endArray, so that only a minimal amount of work is needed to support a new output format.
So far only JSON format has been implemented, but it would be easily possible adding XML or other formats like YAML if needed.

Also this serializer has an exact and a versioned variant.
The exact json deserializer (SC::detail::SerializationTextReadWriteExact) must receive as input a file with fields in the exact same order as output by the json writer, so it makes it a little bit inflexible but maybe it could provide some performance boost as it's clearly doing less work (even if it should always be measured to verify it's actually faster in the specific use case).

Json Tokenizer

SC::JsonTokenizer verifies the correctness of JSON elements but it will not validate Strings or parse numbers. The design is a stateful streaming tokenizer, so it can ingest documents of arbitrary size. It allocates just a single enum value for every nesting level of json objects. This small allocation can be controlled by the caller through a SC::Vector. The allocation can be eliminated by passing a SC::SmallVector with an bounded level of json nesting. The design of being a tokenizer implies that we are not building any tree / hierarchy / DOM, even if it can be trivially done by just pushing the outputs of the tokenizer into a hierarchical data structure.

Roadmap

🟩 Usable

• JSON Escape Strings
• JSON UTF Escapes
• SC::TaggedUnion serialization
• SC::ArenaMap serialization
• SC::SmallVector serialization
• SC::SmallString serialization

🟦 Complete Features:

• Streaming serializer

💡 Unplanned Features:

• XML Serializer
• YAML Serializer