DataSource / DataSink as Encoder–Decoder Components — Design Plan¶

Status: Stage 3 implemented (§9 Phases 0–3, the generic tier of Phase 6, and the port half of Phase 5). Phase 4 and Phase 7 are still plan only, as are Phase 6's typed tier and its protobuf half. This document describes a refactor of DataSource and DataSink, built on the uprops IEncoder / IDecoder abstraction, on schema-describing archives, and on runtime-loadable protocol schemas (FlatBuffers, Protocol Buffers). It records the target architecture, the reasoning, the migration order and the open decisions.

What exists today is listed in §0. Everything else is unbuilt; sections describing it are written in the future tense and should be read as intent, not as documentation of behaviour.

0. What is implemented¶

Stage 1 put the whole conversion-node concept in place and made it work for files, other processes and remote nodes, using the framework's existing binary encoding as the one protocol.

Stage 2 adds the two things that make the design's central claim true rather than merely intended: the schema archives, which derive a class's layout from its own serialize() with no hand-written .fbs anywhere, and the FlatBuffers generic tier, which carries that layout to a peer that was never compiled against the class. Protocol Buffers and the typed tier remain absent, and adding either is still a new class and a registration.

Landed in Stage 1¶

Area	Files	What it does
Type identity	`src/Core/TypeId.{h,cpp}`	Stable 128-bit class identity, UUIDv5 over the registered factory name under a frozen namespace UUID. Reproducible across processes, rebuilds and compilers.
Factory binding	`src/Core/ObjectFactory.{h,cpp}`	`Registration::id`; `GetTypeId(name)`, `GetTypeId(const Object*)`, `GetRegisteredName(TypeId)`, `Create(TypeId)`, `IsRegistered(TypeId)`.
Encoding identity	`src/Core/uprops/backend/DataFrame.h`	`EncodingId{protocol, typeId}` and `DataFrame` with its three-level `Level` enum.
Codec interfaces	`src/Core/uprops/backend/IObjectEncoder.h`, `IObjectDecoder.h`	Object-level siblings of `IEncoder`/`IDecoder`; nothing existing was widened.
Registry	`src/Core/uprops/backend/CodecRegistry.{h,cpp}`	Separate encoder/decoder tables keyed on `EncodingId`, with type-agnostic fallback, `EncodingsFor`, `TypesFor`, `CanConvert`, `Probe`.
Native codec	`src/Core/ObjectCodec.{h,cpp}`	`native-bin` encoder/decoder auto-registered for every class the factory knows, plus the `Object` ↔ `DataFrame` adapters.
Envelope	`src/Core/DataEnvelope.{h,cpp}`	Self-describing header (`uLBS`, version, protocol, TypeId, length) with non-consuming `Peek`.
Conversion node	`src/Core/DataCodec.{h,cpp}`	The (decoder, encoder) pair; `Convert`, `ReadEnveloped`, `WriteEnveloped`.
Source / sink	`src/Core/DataSource.{h,cpp}`, `DataSink.{h,cpp}`	Reseated on `DataCodec`. `DecodeInto`, `DecodeNew`, `PeekEnvelope`, `HasEnvelope`; `EncodeFrom`, `Enveloped`.
Transport	`src/Core/uprops/backend/BinaryCodec.{h,cpp}`	`MessageKind::DataFrame` plus `EncodeDataFrame`/`DecodeDataFrame`, so the framed transport already used for property events carries whole objects.
Snapshot identity	`src/Core/uprops/Metadata.h`	`ObjectSnapshot::typeId`.
Tests	`src/Core/testing/ObjectCodecTest.cpp`	Eight cases, including object → pipe → separate process → object.

Landed in Stage 2¶

Area	Files	What it does
Schema types	`src/Core/uprops/Schema.{h,cpp}`	`SchemaKind`, `SchemaValue`, `SchemaField`, `SchemaTable`, `Schema`, `SchemaId`. `SchemaId` is UUIDv5 over the canonical, value-free field list under a frozen namespace UUID, so two peers derive the same id independently and a fetched schema is checkable against it.
Schema archives	`src/Core/uprops/Archives.h` (classes), `SchemaArchive.h` (entry points)	`schema_oarchive` walks `serialize()` and builds a `SchemaTable`; `schema_iarchive` reads one back. The load side keeps a per-level cursor and resolves names from the cursor forward, which is what reconciles Boost's positional input archives with a name-keyed table.
Schema registry	`src/Core/uprops/SchemaRegistry.{h,cpp}`	Compiled-in, lazily derived, and verified-from-elsewhere resolution of a `SchemaId`. `RegisterVerified` recomputes the content hash and refuses a mismatch.
Property plumbing	`uprops/PropertyBase.h`, `uprops/Property.h`, `Core/Property.h`	The two new archives added to the `PropertyBase::serialize` virtual set and to `is_standard_archive`, so dynamic properties are walked too (§6.1's trap, closed).
Level 2 on the pivot	`backend/DataFrame.h`	`DataFrame::table`, serialized alongside `payload`, so the framed transport already carries level-2 frames.
Base schema	`backend/flatbuffers/ulib_base.fbs`	The one hand-written FlatBuffers schema: `Kind`, `Value`, `Field`, `ObjectFrame`, file identifier `uLBF`. Compiled by `flatc` at build time rather than checked in.
FlatBuffers codec	`backend/flatbuffers/FlatBuffersCodec.{h,cpp}`	Generic-tier `IObjectEncoder`/`IObjectDecoder`, registered type-agnostically (one registration serves every class). Verifies every incoming buffer before walking it, probes by file identifier without consuming, and registers the layout it learned — verified against the id the sender claimed.
Build	`src/Core/uprops/CMakeLists.txt`	`UPROPS_WITH_FLATBUFFERS`, defaulting on but degrading to off when the package is absent. Everything in §§3–6 works without it.
Tests	`uprops/testing/SchemaArchiveTest.cpp`, `FlatBuffersCodecTest.cpp`	45 + 46 checks, both standalone (no `Core`). Includes a version-gated field list, a reordered table with an unknown extra field, and a reader that decodes and renders a class it was never compiled against.

Landed in Stage 3¶

Core now reaches the schema layer, and there is a node you can wire.

Area	Files	What it does
Table codecs per class	`src/Core/ObjectFactory.{h,cpp}`	`Registration::saveTable`/`loadTable`, filled by `MakeRegistration<T>`, plus `SaveObjectTable`/`LoadObjectTable`/`CanSerializeTable`. Needed for the same reason the binary pair is: a schema walk needs the concrete class's `serialize()`, and only the registration knows what that is.
Object ↔ table	`src/Core/ObjectCodec.{h,cpp}`	`ObjectToTable`, `TableToObject` (lenient, reporting unresolved fields), `ObjectToFrame(…, Level)`, and `PromoteFrame`, which raises an opaque native frame to a table through a scratch instance. `FrameToObject` now dispatches on level.
Level-aware sink	`src/Core/DataSink.cpp`	`EncodeFrom` asks the object for exactly the level its encoder declared, and `WriteFrame` promotes rather than refusing.
Codec routing on the ports	`DataSource::CanDecodeInto`, `DataSink::CanEncodeFrom`, used by `TessellatedMesh`, `Abstract::VoxImage`, `Geant::TessellatedSolid`	A source or sink that declares a protocol the registry knows is routed through the codec; everything else still takes the domain parser. This is the port half of Phase 5, done without deleting the legacy path — the two coexist until Phase 4 moves `stl`/`vtk`/`vti` into codecs of their own.
The node	`src/Core/DataBuffer.{h,cpp}`	`Data/DataBuffer` in the Add menu. Derives `DataSource`, so it plugs into every existing `Source` port unchanged; adds an untyped `Input` port that re-encodes whatever object it is given whenever that object updates. `Backend` is a dropdown built from `CodecRegistry::GetProtocols()`, native first.
Update signalling	`TessellatedMesh.h`, `TriangleMesh.h`, `QuadMesh.h`, `DataSource.h`, `DataSink.h`	Five `Updated()` overrides emitted only their own class's signal. Signals are keyed by member-function pointer, so nothing connected to `Object::Updated` ever heard from them — which is what every generic observer in the framework uses (`uLibVtkInterface` view refresh, `Algorithm`'s AutoRun input, now `DataBuffer`). They now emit both.
Tests	`src/Math/testing/DataBufferChainTest.cpp`	The chain end to end, once per registered backend, so adding a codec adds a case for free. Plus `Object → SchemaTable → Object`, the wrong-class guard, and frame reshaping in both directions.

Landed in Stage 4¶

Area	Files	What it does
STL as a codec	`src/Math/Codecs/StlCodec.{h,cpp}`	`stl` is now a registry protocol for `TessellatedMesh`, working at `Level::Table` in both directions and delegating to the existing parser. Phase 4's useful half; moving the parser bodies out of `TessellatedMesh.cpp` can follow, duplicating them would not.
Registration order	`src/Math/MathRegistrations.cpp`	The codec is registered there, below the class registrations, because order is only guaranteed within one translation unit.
The trap, closed	`backend/CodecRegistry.cpp`	A codec whose `typeId` is the all-zero UUID is refused. That is not a valid key, it is a codec that ran before the class it names was registered — previously it filed itself under a key nothing would ever look up, silently.
Stream transcode	`src/Core/DataBuffer.{h,cpp}`	`Input` now accepts a `DataSource` as well as a plain object, and transcodes what that source carries. Wiring a file source in used to serialise the source node's own FilePath and Format instead, which the consumer then correctly refused.
Bidirectional reshaping	`ObjectCodec::ConformFrame` (was `PromoteFrame`)	The levels are ordered by cost, not containment: a Table frame carries no `payload`, so it does not satisfy an Opaque request however "high" it ranks. Handing one to the native encoder used to write zero bytes.

TessellatedMesh A --[Input]--> DataBuffer --[Source]--> TessellatedMesh B
                               Backend: flatbuffers

With Backend = native-bin the payload is 265 opaque bytes; with flatbuffers it is 2768 bytes of self-describing uLib.wire.ObjectFrame carrying 5 named fields, applied to B by name. Mutating A propagates through the whole chain.

How it is used¶

// Object -> file, self-describing
DataSink sink("Out");
sink.SetFilePath("/tmp/mesh.ulbs");
sink.SetFormat(kNativeProtocol);
sink.SetEnveloped(true);          // header names the class
sink.EncodeFrom(mesh);

// ... and back, in another process, with nothing told in advance
DataSource src("In");
src.SetFilePath("/tmp/mesh.ulbs");
DataEnvelope env;
src.PeekEnvelope(env);                        // what is this?
Object *obj = src.DecodeNew(context);         // create it and fill it

// Over the existing framed transport
uprops::DataFrame frame;
ObjectToFrame(mesh, frame);
uprops::BinaryEncoder enc;  enc.EncodeDataFrame(frame);
socket.send(enc.TakeBuffer());
// receiver
decoder.Feed(bytes);
uprops::DataFrame in;  decoder.DecodeDataFrame(in);
FrameToObject(in, targetMesh);

Deliberately still absent¶

No typed tier. ObjectFrame::typed_payload is in the base schema and nothing writes it. The generic tier is verbose — every field carries its own name — and the compact path needs reflection::Schema and the SchemaRequest/SchemaOffer exchange of §6.4, neither of which is built.
No protobuf, so the abstraction has been exercised against one wire protocol rather than two.
The domain formats have not moved. stl, vtk and vti still live in TessellatedMesh.cpp and VoxImage.cpp and are still reached through the existing Format path. That is Phase 4, and it is independent of everything above.
The ports still call the old methods. SetSource/SetSink on the domain objects have not been rewritten onto DecodeInto/EncodeFrom (Phase 5), so DataSourceTest, DataSinkTest and VoxImageDataPortTest pass unchanged.

Two decisions worth knowing¶

DataCodec declares no serialized properties. Inserting a base class into an already-archived hierarchy would shift every payload written so far, so its state is transient and DataSource/DataSink keep serializing base_object<Object>. Existing .gcx documents stay readable. InputProtocol and OutputProtocol from §7 were therefore not added; the protocol is derived from the existing Format property instead.

A type-agnostic native codec is registered alongside the per-type ones. Moving a native payload is the same work whatever class it describes, so a bare file or a raw buffer — which announces no type — still resolves a codec, and DecodeInto supplies the target object's identity as the hint. Without this a source had to be told its type before it could be read, which defeats the point.

1. The problem with the current design¶

Today the domain object owns the format knowledge and the DataSource only supplies bytes:

// Abstract::VoxImage — the object dispatches on a string it got from the source
bool VoxImage::ReadFromStream(std::istream &s, const std::string &format, bool dt) {
    const std::string fmt = NormalizeFormat(format);
    if (fmt == "vti" || fmt == "xml" || fmt == "vtu") return ReadFromVtiStream(s, dt);
    return ReadFromVtkStream(s, dt);
}

Five concrete costs:

Every format lands in the domain class. VoxImage.cpp carries the legacy .vtk parser, the .vti VTK-XML path and the cell-data XML writer; TessellatedMesh.cpp carries the binary and ASCII STL readers plus the STL writer. Adding a format means editing the geometry class.
The format string is an untyped side channel. Each object invents its own default and its own spelling. There is no registry, no validation, and no way to ask which encodings can this object be written as.
Objects cannot be matched to streams. A DataSource holding a payload has no idea what class it belongs to, so a port must be wired by hand and a mismatch surfaces only as an empty parse.
No object-to-object transfer. There is no way to pipe one object's data into another without a file in between.
No wire protocol. Boost binary archives are a C++-only, version-fragile, positional encoding. Nothing outside a matching build can read one, and there is no way for a peer to learn the layout of a class it was not compiled against.

2. The target model¶

The object speaks only its own serialize(). Everything else is a composition of independently identified components:

IDecoder — attaches to an input encoding and produces the pivot form.
IEncoder — consumes the pivot form and produces an output encoding.

A conversion node is a (decoder, encoder) pair. DataSource and DataSink are both instances of that one concept; they differ only in which end is bound to bytes and which to an object.

graph LR
    subgraph "Input side (IDecoder)"
        IN1["File (.stl, .vti)"]
        IN2["Protocol stream\n(bin / JSON / FlatBuffers / protobuf)"]
        IN3["Live Object\n(serialize())"]
    end

    PIVOT["DataFrame\nopaque payload | SchemaTable | live Object"]

    subgraph "Output side (IEncoder)"
        OUT1["File (.stl, .vti)"]
        OUT2["Protocol stream\n(bin / JSON / FlatBuffers / protobuf)"]
        OUT3["Live Object\n(serialize())"]
    end

    IN1 --> PIVOT
    IN2 --> PIVOT
    IN3 --> PIVOT
    PIVOT --> OUT1
    PIVOT --> OUT2
    PIVOT --> OUT3

Two inversions follow. The source drives the object, not the reverse: mesh.ReadFromDataSource(ds) becomes ds.DecodeInto(mesh). And a node's two ends are independent: a DataSink fed FlatBuffers can write .stl, or write bin into another node's DataSource, without either end knowing the other.

2.1 Producer–Consumer conversion pipelines (`uprops` vs `uLib::Core`)¶

The data transfer pipeline operates across two complementary layers of abstraction: 1. Low-level generic tier (uprops): Handles generic serialization, property reflection (ObjectSnapshot), and raw/opaque payloads via DataFrame. 2. Domain application tier (uLib::Core): Manages continuous streams, typed object codecs, and zero-copy packet inspection via DataEnvelope and PeekEnvelope.

Dual-layer architecture overview¶

flowchart TB
    subgraph S_UPROPS["Low-level generic tier (uprops / backend)"]
        direction TB
        P_UP["Producer (PropertyContainer / Object)"] -->|1. CreateSnapshot / Raw Payload| DF_PROD["Build DataFrame\n(level, encoding, payload, properties)"]
        DF_PROD -->|2. Boost Archive / Stream Serialization| WIRE1["Transport / Channel / Storage"]
        WIRE1 -->|3. Frame Deserialization| DF_CONS["Receive DataFrame"]
        DF_CONS -->|4. ApplySnapshot / Codec Extract| C_UP["Consumer (PropertyContainer / Target)"]
    end

    subgraph S_ULIB["Domain application tier (uLib::Core)"]
        direction TB
        P_ULIB["Domain Producer\n(Hit, Track, VoxImage, uLib::Object)"] -->|1. Encode| ENC["DataEncoder / Codec"]
        ENC -->|2. Wrap Payload + Header| DE_PROD["DataEnvelope (Magic, TypeId, Payload)"]
        DE_PROD -->|3. Stream Transmission| WIRE2["Network / File / Hardware Stream"]
        WIRE2 -->|4. Header Inspection| PEEK["PeekEnvelope (Non-consuming header inspection)"]
        PEEK -->|5. Match Codec & Allocate| DEC["Selected DataDecoder"]
        WIRE2 -->|6. Ingest Full Packet| DE_CONS["Complete DataEnvelope"]
        DE_CONS --> DEC
        DEC -->|7. Decode & Materialize| C_ULIB["Domain Consumer\n(ObjectsContext, Viewport, Analyzer)"]
    end

Detailed flow: `uprops` tier (`DataFrame`)¶

sequenceDiagram
    autonumber
    participant Prod as Producer (PropertyContainer)
    participant DF as DataFrame
    participant Wire as Serializer / Archive (Boost NVP)
    participant DecDF as DataFrame (Received)
    participant Cons as Consumer (PropertyContainer)

    Note over Prod,DF: 1. Frame Generation
    alt Property / Metadata Level
        Prod->>DF: properties = CreateSnapshot()
    else Bulk Payload Level (Voxels, Buffers)
        Prod->>DF: payload = raw_bytes; level = Level::Opaque; encoding = EncodingId(...)
    end

    Note over DF,Wire: 2. Serialization & Transport
    DF->>Wire: serialize(ArchiveT &ar) (encoding, level, version, payload, properties)

    Note over Wire,DecDF: 3. Deserialization
    Wire->>DecDF: ar & make_nvp(...)

    Note over DecDF,Cons: 4. Application to Target
    alt Property Level
        DecDF->>Cons: ApplySnapshot(properties)
    else Payload Level
        DecDF->>Cons: Codec/Handler extracts payload based on encoding & version
    end

Detailed flow: `uLib::Core` tier (`DataEnvelope` & `DataCodec`)¶

sequenceDiagram
    autonumber
    participant Prod as Domain Producer (e.g. Detector / File / Simulator)
    participant Enc as DataEncoder (DataCodec)
    participant EnvOut as DataEnvelope (Outbound)
    participant Transport as Channel / DataSource
    participant Peek as PeekEnvelope
    participant Dec as DataDecoder
    participant Cons as Domain Consumer (ObjectsContext / Pipeline)

    Note over Prod,Enc: 1. Encode Domain Object
    Prod->>Enc: Encode(DomainObject)
    Enc->>EnvOut: Build envelope (Magic `uLBS`, TypeId, protocol, payload)

    Note over EnvOut,Transport: 2. Transmit Packet Stream
    EnvOut->>Transport: Push to stream / socket / file

    Note over Transport,Peek: 3. Non-consuming Inspection (Peek)
    Transport->>Peek: Peek header bytes (magic number, protocol, TypeId)
    Note over Peek,Dec: No copy: PeekEnvelope routes to appropriate decoder
    Peek->>Dec: Identify & resolve matching DataDecoder

    Note over Transport,Dec: 4. Delivery & Materialization
    Transport->>Dec: Decode(DataEnvelope)
    Dec->>Cons: Materialize / Update uLib::Object in Context

Abstraction tier comparison¶

Feature	`uprops` tier (`DataFrame`)	`uLib::Core` tier (`DataEnvelope` + `DataCodec`)
Primary Scope	Generic object reflection, property diffs, neutral multi-level pivot	Physical data streaming, hardware/file acquisition, domain object parsing
Bulk Data Carrier	`DataFrame::payload` (`std::string` buffer)	`DataEnvelope` binary stream buffer with length header
Properties / Metadata	`ObjectSnapshot` (`PropertyContainer::CreateSnapshot()`)	Envelope transmission metadata (`uLBS` magic, `TypeId`, protocol, version)
Stream Routing	Dispatched via `EncodingId` (`protocol` + `TypeId`)	`PeekEnvelope` non-consuming header probe for dynamic stream dispatching
Target Entities	`PropertyContainer` / backend adapters	`uLib::Object`, `ObjectsContext`, detector data pipelines

3. Identity: encoding = serialization kind × type¶

An EncodingId uniquely identifies a byte layout as the pair (what protocol, which class):

Axis	Meaning	Examples
`protocol` (serialization kind)	how bytes are laid out	`native-bin`, `uprops-json`, `flatbuffers`, `protobuf`, `stl`, `vti`, `vtk`
`typeId`	which class the bytes describe	`TessellatedMesh`, `VoxImage`, a plugin class

Both halves are identified: the object type has an identity and so does the encoder. Their combination collapses to one 128-bit value — the unique encoded data identification — which the registry, the stream envelope and the port compatibility check are all keyed on.

namespace uprops {

/** @brief Identifies a byte layout: which protocol, carrying which type. */
struct EncodingId {
  std::string protocol;   // serialization kind
  std::string typeId;     // stable class identity; empty = type-agnostic

  /** @brief UUIDv5 over "protocol:typeId" — the single encoded-data id. */
  std::array<std::uint8_t,16> Uuid() const;
  std::string ToString() const;                     // "flatbuffers:VoxImage"
  bool operator==(const EncodingId &) const;
  bool operator<(const EncodingId &) const;         // usable as a map key
};

}

3.1 Where the type identity comes from¶

uprops is a standalone library — it builds on its own and must not depend on Core (src/Core/uprops/CMakeLists.txt declares its own project()). So EncodingId::typeId is an opaque std::string in uprops, and Core binds it to a class.

Three pieces already exist in Core and need joining up:

ObjectFactory::Registration already carries create, save, load, category and const std::type_info *type; GetRegisteredName(const Object*) already resolves a live object to its registered name by dynamic type. This is the native codec, already built.
SceneDocument's ObjectRecord already keys payloads by className and already refuses to decode a payload with the wrong codec — the same matching problem, solved for files only.
uLib::type_id<T> and TypeRegister in src/Core/Uuid.h already generate a UUID per type, but are unused.

type_id<T> hashes typeid(T).name(), which is not usable as a persisted or wire identity: the mangled name is compiler- and ABI-dependent, differs between GCC and Clang builds, and would change under an unrelated template refactor. It also prints to stdout from its constructor.

uLib::TypeId is instead derived from the registered factory name (the string already used in .gcx), so it is stable across compilers, machines and library versions; stable for templates registered under a collapsed name (VoxImage<Voxel> registers as "VoxImage"); and declarable by a plugin without linking against the host's type system. Derived as UUIDv5 over a fixed, documented namespace UUID — note that uLib::uLib_dns_uuid in src/Core/Uuid.cpp is currently default-constructed to all zero and must become a constant literal, because the seed becomes part of every persisted identifier.

ObjectFactory then gains GetTypeId(className), GetRegisteredName(TypeId), Create(TypeId) and GetTypeId(const Object*); Registration gains a TypeId field filled in Register().

4. The pivot: `DataFrame`, at three levels of fidelity¶

A decoder and an encoder that never met must agree on something. That something is a DataFrame, and it carries the object's state at whichever of three levels the conversion actually needs:

Level	Form	Cost	Converts across	Requires
1	opaque `payload` (bytes in some encoding)	cheapest — a copy	same typeId, same layout version	nothing
2	`SchemaTable` (named, typed fields)	one archive walk	protocols, layout versions, related types	a schema
3	live `Object`	materialization	anything a codec can express	the class registered

namespace uprops {

/** @brief The neutral form every codec converts to and from. */
struct DataFrame {
  EncodingId     encoding;    /**< what `payload` currently is */
  std::string    payload;     /**< level 1: opaque bytes */
  SchemaTable    table;       /**< level 2: named typed fields (§6) */
  ObjectSnapshot properties;  /**< property-level state; may be empty */
  std::uint32_t  version = 0;

  enum class Level : std::uint8_t { Opaque, Table, Object };
  Level level = Level::Opaque;
};

}

properties is the existing uprops::ObjectSnapshot, already produced by PropertyContainer::CreateSnapshot() and consumed by ApplySnapshot(), already encodable by BinaryEncoder::EncodeSnapshot and ProtocolCodec::EncodeObjectSnapshot. It stores values as strings (PropertyBase::GetValueAsString), which is right for a GUI or a remote property edit and wrong for a 64³ voxel block — which is exactly why levels 1 and 2 exist alongside it. Do not try to express bulk data as properties.

ObjectSnapshot::typeId closed the gap this section used to describe: a snapshot now says what class its values belong to, so one can be routed to a codec rather than merely carried.

Rule: use the cheapest level that satisfies the conversion. Copying a VoxImage to another VoxImage over native-bin never leaves level 1; shipping it as FlatBuffers to a peer that has never heard of VoxImage needs level 2; decoding an .stl needs level 3.

5. Extending `IEncoder` / `IDecoder`¶

The existing interfaces in src/Core/uprops/backend/ are property-oriented (EncodeProperty, EncodePayload / DecodeProperty, DecodePayload). BinaryEncoder / BinaryDecoder implement them with a MessageFrame envelope (kind byte, 4-byte big-endian length, body) and a Feed()-based incremental reader — already the streaming model this design needs.

Extend by sibling interfaces, not by widening. Adding pure virtuals to IEncoder would break BinaryEncoder, PropertyServer and any external implementer. Instead:

namespace uprops {

class UPROPS_EXPORT IObjectEncoder : public IEncoder {
public:
  virtual EncodingId GetEncodingId() const = 0;
  /** @brief Lowest DataFrame level this encoder can consume. */
  virtual DataFrame::Level RequiredLevel() const = 0;
  virtual bool EncodeFrame(const DataFrame &, std::ostream &,
                           std::string *error = nullptr) = 0;
};

class UPROPS_EXPORT IObjectDecoder : public IDecoder {
public:
  virtual EncodingId GetEncodingId() const = 0;
  virtual DataFrame::Level ProducedLevel() const = 0;
  virtual bool DecodeFrame(std::istream &, DataFrame &,
                           std::string *error = nullptr) = 0;
  /** @brief Recognises this encoding without consuming the stream. */
  virtual bool Probe(std::istream &, EncodingId &) const { return false; }
};

}

BinaryCodec gains a MessageKind::DataFrame = 4 and a frame encode/decode pair — one new enum value, backward compatible — so the existing framed transport carries whole objects with no new wire machinery.

5.1 Registry¶

class UPROPS_EXPORT CodecRegistry {
public:
  static CodecRegistry &Instance();

  void RegisterEncoder(std::shared_ptr<IObjectEncoder>);
  void RegisterDecoder(std::shared_ptr<IObjectDecoder>);

  const IObjectEncoder *FindEncoder(const EncodingId &) const;
  const IObjectDecoder *FindDecoder(const EncodingId &) const;

  std::vector<EncodingId> EncodingsFor(const std::string &typeId) const;
  std::vector<EncodingId> TypesFor(const std::string &protocol) const;
  bool CanConvert(const EncodingId &in, const EncodingId &out) const;
};

Encoders and decoders register separately, which is the point: a read-only importer (a vendor format we can parse but never write) registers a decoder alone and is immediately usable as an input, with every existing encoder as its output.

5.2 The native codec is free¶

Every class registered through ULIB_REGISTER_OBJECT* already has a working binary codec in Registration::save/load. ObjectFactory::Register() will additionally register a NativeEncoder/NativeDecoder pair under (native-bin, id) forwarding to them. So from the first phase every object in the framework is already transcodable at level 1.

6. Schema-describing archives¶

This is the layer that makes protocol transport possible without hand-writing a schema per class.

6.1 An archive that emits a table instead of bytes¶

The framework already does this. uLib::Archive::property_register_archive (src/Core/Property.h:635) is a boost::archive::detail::common_oarchive that walks an object's serialize() and, instead of writing bytes, builds a structure — it registers properties with names, units, ranges, defaults, descriptions, port directions, visibility and a group stack for nesting. hrt_oarchive and log_archive in uprops/Archives.h do the same on a smaller scale, intercepting save_override(const hrp<T> &).

The interception point is already rich. boost::serialization::hrp<T> (uprops/Serializable.h) is not a bare name/value pair — it carries name, units, range, default, description, port_direction, visible, serialized and read_only. That is a field descriptor. Every HRP(...) call in every serialize() in the codebase is already declaring schema.

So:

namespace uprops { namespace Archive {

/** @brief Walks serialize() and records named, typed fields with metadata. */
class schema_oarchive
    : public boost::archive::detail::common_oarchive<schema_oarchive> {
public:
  explicit schema_oarchive(SchemaTable &out, bool structureOnly = false);

  template <class T> void save_override(const boost::serialization::hrp<T> &);
  template <class T> void save_override(const boost::serialization::nvp<T> &);
  void save_override(const boost::serialization::binary_object &);   // bulk blobs
  // ... boost bookkeeping overloads suppressed, as in hrt_oarchive
};

/** @brief Reads a SchemaTable back into an object through its serialize(). */
class schema_iarchive
    : public boost::archive::detail::common_iarchive<schema_iarchive> { /* ... */ };

}}

Two products come out of one walk:

a Schema — structure only: the ordered field list with kinds and metadata, derivable from a default-constructed instance;
a SchemaTable — structure and values: the actual data (level 2 above).

The schema is derived from serialize() automatically. No hand-written .fbs per class and no codegen step on the normal path.

Honest limitation. Boost serialization is positional and imperative: a serialize() with an if (version >= 4) branch produces a different field list per version, and a loop over a container produces repeated fields. The derived schema therefore describes one particular walk, not a static type declaration. That is workable — both FlatBuffers and protobuf model optional fields natively — but it forces two rules: the schema is derived per (class, layout version), and field ids are assigned once when a schema version is first published and never renumbered.

Trap. PropertyContainer::SerializeValues is gated on uprops::Archive::is_standard_archive_v<ArchiveT>. A new archive that is not added to that trait list will silently serialize nothing. The same applies to the UPROPS_SERIALIZATION_REGISTER_ARCHIVE macro.

6.2 Schema identity¶

struct SchemaId {   // 16 bytes, UUIDv5
  // over: protocol + typeId + layoutVersion + canonical field list
};

Deriving the id from the schema content rather than assigning it means two peers that independently derive the same schema agree on its id without a central authority — and a receiver that fetches a schema can verify it against the id it was given. That property is what makes decentralised schema discovery work.

6.3 The base schema, one per protocol¶

Each supported protocol gets one small, hand-written, compiled-in schema that carries primitives plus the type ids. This is the generic tier: always available, fully self-describing, and usable for a class the receiver has never heard of.

// ulib_base.fbs — the only schema every FlatBuffers peer is compiled against
namespace uLib.wire;

enum Kind : ubyte { Null, Bool, I8, I16, I32, I64, U8, U16, U32, U64,
                    F32, F64, Str, Bytes, Table, Vector }

table Value {
  kind:  Kind;
  i:     long;        // integral and bool
  d:     double;      // floating point
  s:     string;
  b:     [ubyte];     // binary_object blobs (voxel blocks, point arrays)
  items: [Field];     // Table / Vector children
}

table Field { name: string; units: string; value: Value; }

table ObjectFrame {
  wire_version:   ushort;
  type_id:        [ubyte];   // 16 — which class
  schema_id:      [ubyte];   // 16 — which layout
  schema:         [ubyte];   // optional inline .bfbs, for peers that lack it
  fields:         [Field];   // generic tier
  typed_payload:  [ubyte];   // typed tier: a buffer following schema_id
}
root_type ObjectFrame;

The protobuf equivalent is the same shape as a .proto. Both are small enough to maintain by hand and change only when the wire envelope changes, not when a domain class does.

6.4 Typed tier and runtime schema loading¶

The generic tier is verbose — every field carries its name. For compact transfer the sender writes typed_payload following the per-class schema named by schema_id, and the receiver resolves it at runtime:

sequenceDiagram
    participant S as Sender (DataSink)
    participant R as Receiver (DataSource)
    participant REG as SchemaRegistry

    S->>S: schema_oarchive walks serialize() -> SchemaTable
    S->>REG: Derive/lookup Schema, SchemaId
    S->>R: ObjectFrame{type_id, schema_id, typed_payload}
    R->>REG: FindSchema(schema_id)
    alt schema known
        REG-->>R: reflection::Schema
    else unknown
        R->>S: SchemaRequest(schema_id)
        S-->>R: SchemaOffer(.bfbs bytes)
        R->>REG: Register (verify content hash == schema_id)
    end
    R->>R: reflective walk -> SchemaTable
    R->>R: schema_iarchive -> target object

Both protocols support this without generated code:

FlatBuffers — flatbuffers::Parser::Deserialize(bfbs) yields a reflection::Schema; flatbuffers::GetAnyFieldI/F/S walk an arbitrary buffer against it. flatbuffers::Verifier validates before the walk.
Protocol Buffers — a FileDescriptorProto into a DescriptorPool, then DynamicMessageFactory produces a Message readable by reflection.

So a listener finds the proper deserializer purely from schema_id, with no compile-time knowledge of the class. This is the same mechanism that makes plugin classes work: a plugin's types are just types the receiver was not compiled against.

6.5 Schema distribution¶

SchemaRegistry resolves a SchemaId from, in order:

Compiled in — registered at static init alongside the class.
Derived on demand — from a live registered class, via schema_oarchive over a default-constructed instance.
Inline in the frame — ObjectFrame::schema, for one-shot transfers.
From the peer — a SchemaRequest / SchemaOffer message pair on the protocol.
From a schema directory — a configured path of .bfbs / .desc files, which is also how a plugin ships its schemas.

Every path except (1) and (2) verifies the fetched schema against the content hash before registering it.

6.6 Re-encoding for the target object¶

Once a frame is at level 2, delivering it to a different class is a field mapping problem, and the framework already has an answer for the property half: PropertyContainer::ApplySnapshot looks fields up by qualified name, checks metadata.typeId matches, and skips what it cannot place rather than failing. SchemaTable → object adopts the same leniency rules:

match by field name within the same group/path;
require the field kind to match (or be losslessly widenable);
skip unmatched fields, and report them, rather than aborting the transfer.

Same-type transfer is exact. Cross-type transfer is explicitly best-effort — it is what lets a VoxImage feed a differently-versioned VoxImage, not what turns a mesh into an image. A transform between unrelated types is an Algorithm, not a codec, and this design deliberately does not blur the two.

7. What `DataSource` and `DataSink` become¶

Both become the same thing: a node holding a decoder and an encoder.

/** @brief A (decoder, encoder) pair: converts one encoding into another. */
class DataCodec : public Object {
public:
  ULIB_PROPERTY(std::string, InputProtocol,  "")   // empty = probe
  ULIB_PROPERTY(std::string, OutputProtocol, "")   // empty = from FilePath ext

  EncodingId GetInputEncoding() const;
  EncodingId GetOutputEncoding() const;
  const IObjectDecoder *GetDecoder() const;
  const IObjectEncoder *GetEncoder() const;

  /** @brief Byte-to-byte transcode at the cheapest sufficient level. */
  bool Convert(std::istream &in, std::ostream &out, std::string *error = nullptr);

  /** @brief Object-facing ends. */
  bool DecodeInto(Object &object, std::string *error = nullptr) const;
  bool EncodeFrom(const Object &object, std::string *error = nullptr);
  SmartPointer<Object> DecodeNew(ObjectsContext &into, std::string *error = nullptr) const;
};

class DataSource : public DataCodec { /* input bound to file/buffer/stream */ };
class DataSink   : public DataCodec { /* output bound to file/buffer/stream */ };

FilePath, Format, Embedded, Append, Size, OpenStream() and GetData() survive unchanged — Format simply becomes the protocol half of the node's outward-facing EncodingId instead of a string passed to the object.

The domain objects lose all format code and keep one generic port pair:

// on TessellatedMesh, Abstract::VoxImage, Geant::TessellatedSolid — identical
void SetSource(DataSource *);   // ds->DecodeInto(*this) on Updated
void SetSink(DataSink *);       // sink->EncodeFrom(*this) on Updated

7.1 Object-to-object transfer over a protocol¶

Because a node's ends are independent, chains fall out for free. The Sink output port of one node connects to the Source input port of the next, and the registry validates the join by comparing EncodingIds:

graph LR
    A["VoxImage A"] -->|serialize()| S1["DataSink\ndec: native · enc: flatbuffers"]
    S1 -->|ObjectFrame over socket| S2["DataSource\ndec: flatbuffers · enc: native"]
    S2 -->|serialize()| B["VoxImage B\n(different build, plugin type, or remote peer)"]
    S1 --> F["DataSink\ndec: flatbuffers · enc: vti"]
    F --> FILE["volume.vti"]

The same chain serves cases that are separate code paths today: copying one object into another, shipping it over a transport, and exporting to a file — and the middle node never needs to know a VoxImage exists.

Chain length is capped at one decoder plus one encoder for now. A general shortest-path search over a conversion graph is speculative until a case needs it.

8. Module boundaries¶

This matters more than usual, because the natural implementation violates the dependency direction:

Layer	Owns	Must not know about
`uprops`	`EncodingId`, `DataFrame`, `Schema`/`SchemaTable`/`SchemaId`, `schema_oarchive`/`schema_iarchive`, `IObjectEncoder`/`IObjectDecoder`, `CodecRegistry`, `SchemaRegistry`	`uLib::Object`, `ObjectFactory`, `uLib::TypeId`
`uprops/backend/flatbuffers`, `.../protobuf`	base schema, reflective codecs	domain classes
`Core`	`uLib::TypeId`, `ObjectFactory` binding, `NativeCodec`, `DataCodec`/`DataSource`/`DataSink`	domain formats
`Math` / `HEP`	`StlCodec`, `VtkLegacyCodec`, `VtiCodec`, …	each other

In uprops, typeId is an opaque string and a codec works on DataFrame alone; Core supplies the uLib::Object ↔ DataFrame adapter. That keeps uprops standalone-buildable, which it currently is and should not lose.

New dependencies. FlatBuffers is now in conanfile.txt; protobuf is not. Both are optional (UPROPS_WITH_FLATBUFFERS, UPROPS_WITH_PROTOBUF): the schema archives, the registries and native-bin all work with no new dependency, so a build that declines both still gets everything in §§3–5. The FlatBuffers option defaults on and turns itself off when the package is absent, rather than failing the configure.

9. Migration plan¶

Each phase is independently buildable and testable and ends with the full suite green. Phases 0–3 change no observable behaviour.

Phase 0 — Type identity ✅ done¶

[ ] src/Core/TypeId.{h,cpp}; fix uLib_dns_uuid to a literal constant.
[ ] Retire type_id<T> / TypeRegister in Uuid.h (unused; remove the std::cout in the constructor either way).
[ ] TypeId on ObjectFactory::Registration plus the four lookups (§3.1).
[ ] Test: name → TypeId → name round-trips for every registered class; no collisions; byte-identical across a rebuild and across GCC/Clang.

Phase 1 — `uprops` codec layer ✅ done¶

[ ] EncodingId, DataFrame in uprops/Metadata.h; typeId on ObjectSnapshot.
[ ] backend/IObjectEncoder.h, backend/IObjectDecoder.h, backend/CodecRegistry.{h,cpp}.
[ ] BinaryCodec: MessageKind::DataFrame plus frame encode/decode.
[ ] Test standalone (no Core): register a dummy codec pair, round-trip a DataFrame, confirm registry lookup by EncodingId.

Phase 2 — Core binding and native codecs ✅ done¶

[ ] NativeEncoder/NativeDecoder auto-registered from ObjectFactory::Register.
[ ] DataEnvelope (src/Core/DataEnvelope.{h,cpp}) and PeekEnvelope.
[ ] Test: encode a TessellatedMesh native → decode into a fresh one → same points and polygons; PeekEnvelope identifies the class without consuming.

Phase 3 — Schema archives ✅ done¶

[x] Schema, SchemaTable, SchemaId, SchemaRegistry in uprops.
[x] schema_oarchive; added to is_standard_archive, which meant adding two overloads to the PropertyBase::serialize virtual set (§6.1 trap).
[x] schema_iarchive. It resolves the positional/name-keyed mismatch with a per-level cursor: a name is looked for from the cursor forward first, so a load walk matching the save walk costs one comparison per field and a container's repeated item fields stay in order, while a walk that has gained, lost or reordered fields still resolves what it can and reports the rest through GetUnresolved(). binary_object works because the size is agreed by a preceding count field, exactly as the byte archives do it.
[x] Test: round-trips for scalars, a nested table, a container, a bulk blob, a version-gated field list, and a reordered table with an unknown extra field; SchemaId stable across runs and independent of values.

Two things worth recording. UPROPS_SERIALIZATION_REGISTER_ARCHIVE is not applied to these archives, contrary to what this section originally called for: that macro instantiates the polymorphic pointer serializers, and pointers are the one thing a SchemaTable does not represent. Registering them asked the linker for machinery neither archive can honour. And class versions travel in SchemaTable::classVersions in walk order, because a serialize() with an if (version >= 4) branch produces a different field list per version and the load walk has to replay the save walk's versions to match it.

Phase 4 — Move the existing formats into codecs (stl landed in Stage 4)¶

[~] src/Math/Codecs/StlCodec.{h,cpp} and VoxImageCodec.{h,cpp} — stl, vtk and vti are registered and resolvable as protocols, delegating to the existing parsers. This stopped being optional when the ports were inverted: a DataSource decodes through the registry, so without them opening a .stl or .vtk had nothing to open it with. The parser bodies have not moved out of TessellatedMesh.cpp / VoxImage.cpp yet; that refactor has no behavioural content and is what finally lets ReadFromDataSource and ReadFromStream be deleted.
[ ] VtkXmlCodec for the cell-data XML writer; byte-identical regression test against the current writers.
[ ] src/Math/Codecs/VtkLegacyCodec, VtiCodec, VtkXmlCodec — move the corresponding bodies out of VoxImage.cpp; the ToVtkImageData / FromVtkImageData file-static helpers move with them.
[ ] Keep ExportToVtk / ImportFromVti / ReadFromStlFile as thin wrappers resolving a codec, so the Python bindings, app/ and existing tests work.
[ ] Test: new codecs produce byte-identical output to the current writers for a fixed input. This is the regression gate for the move.

Phase 5 — Invert the ports ✅ done¶

[x] Add DataCodec; reseat DataSource/DataSink on it.
[x] The ports are inverted. TessellatedMesh, Abstract::VoxImage and Geant::TessellatedSolid no longer carry Source/Sink reference ports. DataSource gained Output (the object it decodes into) and DataSink gained Input (the object it encodes). The wire in the node editor runs DataSource.Output -> object.(object) and object.(object) -> DataSink.Input.
[x] The SetSource/SetSink/InputFile/OutputFile C++ API is unchanged and still exported to Python: it is the port that moved, not the API.
[x] ReadFromDataSource / WriteToDataSink survive as the implementation the domain formats are still reached through; they go when the parser bodies move out (the rest of Phase 4).
[ ] Sink→Source port chaining with EncodingId compatibility at connect time.
[ ] Resolve the DensityType question (§11.2).
[ ] Test: DataSourceTest, DataSinkTest, VoxImageDataPortTest and the SolidTest port section pass unmodified apart from renamed calls — they are the behavioural contract. Add an object-to-object chain test.

Phase 6 — Protocol transport (generic tier landed in Stage 2)¶

[x] Optional FlatBuffers dependency (UPROPS_WITH_FLATBUFFERS, on when the package is present, and everything else builds without it); ulib_base.fbs, compiled by flatc at build time so the checked-in .fbs stays the single source of truth; generic-tier encoder and decoder.
[x] Test: a reader with no compiled-in knowledge of the class enumerates its fields, reads its values, keeps its units, ranges and descriptions, and ends up holding the layout — registered under an id verified against the frame's own content.
[x] Join Core to the schema layer: ObjectToFrame at Level::Table, and PromoteFrame for a frame that arrives lower than its encoder needs.
[ ] Runtime schema loading via reflection::Schema; SchemaRequest / SchemaOffer on the framed protocol.
[ ] Typed tier for one class end to end, measured against the generic tier.
[ ] Protobuf as a second protocol, to prove the abstraction holds for two.

Phase 7 — Plugins and persistence¶

[ ] .gcx: add typeId to ObjectRecord at a format-version bump, keeping className as the fallback for older documents (§11.1).
[ ] Plugin loading: dlopen plus a uLibRegisterPlugin(ObjectFactory&, CodecRegistry&, SchemaRegistry&) entry point.

10. What this buys¶

Adding .obj mesh support is one file and one registration line; no geometry class is touched.
A GUI populates its export dropdown from EncodingsFor(typeId).
A read-only importer needs only a decoder and inherits every existing output.
Objects transfer to each other through a codec chain — same machinery as file export, no special case.
A schema comes from serialize() for free; there is no .fbs to keep in sync with the C++ class, which is the usual failure mode of schema-first designs.
A peer decodes a class it was never compiled against, which makes both remote transport and plugin objects work through one mechanism.

11. Open decisions¶

To answer before Phase 5 lands; none block Phases 0–2.

.gcx payloads must stay native-bin. Domain codecs are lossy — an STL has no instance name, no transform, no property values — so the document must never route payloads through them. This deserves an assertion in CaptureObjectState, not just a convention: the new model makes it easy to get wrong.
Where does DensityType live? A property of Abstract::VoxImage today, but genuinely a codec parameter — it scales values only through .vtk/.vti. Moving it onto the node is more faithful but changes the Python API.
Codec parameters in general (compressed for .vti, ASCII vs binary STL). A PropertyContainer per codec would surface them in the Properties panel for free and fits the framework better than a string map.
Ambiguous protocol names. "xml" currently means two different things depending on direction in VoxImage. Pin the spellings down (proposal: vtk, vti, vtk-xml-cells).
Field id assignment. §6.1 requires ids assigned once per schema version and never renumbered. Where is that ledger kept — derived from field order and frozen by SchemaId, or an explicit checked-in file? The first is less process; the second catches accidental reordering at review time.
Schema trust. A schema fetched from a peer describes how to walk a buffer. Content-hash verification (§6.5) proves it is the schema for that id, not that the id was legitimate. Decide whether peer-supplied schemas are accepted at all outside a trusted deployment.
DecodeNew ownership. It should take the target ObjectsContext so the object is parented on creation, matching how the factory is used elsewhere.
Level-1 vs level-2 default for native-bin. Staying opaque is far cheaper but silently fails across layout versions; going through a table always works but costs a walk. Probably: opaque when SchemaIds match, table otherwise.

12. Explicit non-goals¶

Not changing the .gcx on-disk layout before Phase 7, and not before a version bump with a documented fallback.
Not making Object itself aware of codecs; the registry is looked up by the node, never by the object.
Not replacing Boost.Serialization. native-bin stays boost-based; FlatBuffers and protobuf are additional protocols, not a migration.
Not making uprops depend on Core (§8), and not making the core build depend on FlatBuffers or protobuf.
Not hand-writing a .fbs per domain class. The only hand-written schemas are the per-protocol base schemas (§6.3).
Not a general conversion-graph search (§7.1) until a case demands it.
Not converting between unrelated types (§6.6). That is an Algorithm.
Not implementing plugin sandboxing, versioning or ABI policy — a separate design.