DataSink Architecture and Output Packaging

This document describes uLib::DataSink, the write counterpart of uLib::DataSource. Where a DataSource turns a file, an embedded payload or a memory buffer into an input stream, a DataSink accepts an output stream and stores whatever was written to it — identified by a Format — either into an external file or into the context itself.

graph LR
    subgraph Domain Objects
        TM["TessellatedMesh / VoxImage / TessellatedSolid\n(Sink FunctionalReferenceProperty)"]
    end

    subgraph Core Layer
        Stream["std::ostream (OpenStream())"]
        DK["DataSink Node (Category: 'Data')\n- FilePath\n- Format\n- Embedded\n- Append\n- Size\n- m_Buffer"]
    end

    subgraph External Disk
        File["Exported File (.stl, .vtk, .vti, ...)"]
    end

    subgraph Scene Archive (.gcx)
        GCX["SceneDocument"]
    end

    TM -->|WriteToStream()| Stream
    Stream -->|committed on destruction| DK
    DK -->|Flush() / not Embedded| File
    DK -->|Embedded: m_Buffer payload| GCX
    DK -.->|Emits Updated / DataWritten| TM

---

1. Symmetry with DataSource

The two classes are deliberate mirror images, so that a node graph reads the same in both directions:

DataSource (read)	DataSink (write)
OpenStream() -> std::istream	OpenStream() -> std::ostream
LoadFile()	Flush()
SaveToFile(dest)	SaveToFile(dest)
SetData() / GetData()	SetData() / GetData()
Embed(bool)	SetEmbedded(bool)
Updated / DataChanged	Updated / DataWritten
EmbedAll(context)	EmbedAll(context)
UnpackAll(context, dir)	FlushAll(context, dir)
Port direction Input	Port direction Output

Both are registered in ObjectFactory under the category "Data", so both appear in the scene hierarchy and the node graph as first-class, connectable nodes.

---

2. Properties

Property	Type	Default	Group	Description
FilePath	std::string	""	"Sink"	Path of the external file the payload is written through to. Unit metadata is "file".
Format	std::string	""	"Sink"	Data format identifier (e.g. "stl", "vtk", "vti"). Auto-inferred from FilePath if not explicitly set; this is what tells the producer how to encode.
Embedded	bool	false	"Sink"	When true, the payload stays in the context (and is archived with it) instead of being written to disk.
Append	bool	false	"Sink"	When true, a commit grows the payload instead of replacing it.
Size	size_t	0	"Sink"	Read-only size in bytes of the payload (or of the external file).

---

3. How the output stream works

OpenStream() returns a std::unique_ptr<std::ostream> that buffers in memory and commits when it is destroyed. A producer only has to write and let the stream go out of scope:

uLib::DataSink sink("MeshOut");
sink.SetFilePath("/tmp/out.stl");     // Format auto-detected as "stl"
{
  auto out = sink.OpenStream();
  mesh.WriteToStream(*out, sink.GetFormat());
}                                     // committed here

On commit:

1. The bytes land in the internal buffer (replacing it, or appended when Append is set).
2. If the sink is not Embedded and FilePath is set, the buffer is written through to that file — creating parent directories as needed.
3. Size is updated and Updated() / DataWritten() are emitted.

Producers that cannot write to a std::ostream — a library that only takes a filename, or one that hands back a string — call Commit(data, size) directly and get exactly the same behaviour. WriteData(), WriteStream(istream&) and WriteSource(const DataSource*) are convenience wrappers over it.

Turning Embedded off on a sink that is holding a payload materialises it: the buffer is written to FilePath at that moment.

---

4. Core API

namespace uLib {

class DataSink : public Object {
public:
  // Path and format configuration
  void SetFilePath(const std::string &path);
  const std::string &GetFilePath() const;
  void SetFormat(const std::string &format);
  const std::string &GetFormat() const;
  void SetEmbedded(bool embedded);
  bool IsEmbedded() const;
  void SetAppend(bool append);
  bool IsAppend() const;
  size_t GetSize() const;

  // Stream access
  std::unique_ptr<std::ostream> OpenStream();

  // Direct payload writing
  bool Commit(const void *data, size_t size);
  bool WriteData(const std::vector<uint8_t> &data);
  bool WriteData(const void *data, size_t size);
  bool WriteStream(std::istream &stream);
  bool WriteSource(const DataSource *source);

  // File IO
  bool Flush();
  bool SaveToFile(const std::string &destPath) const;

  // In-memory buffer manipulation
  void SetData(const std::vector<uint8_t> &data);
  void SetData(const void *data, size_t size);
  const std::vector<uint8_t> &GetData() const;
  void ClearData();

  // Context-wide operations
  static void EmbedAll(ObjectsContext &context, bool embed = true);
  static bool FlushAll(ObjectsContext &context,
                       const std::string &destDirectory = "");
};

}

---

5. Domain object integration

TessellatedMesh, Abstract::VoxImage and Geant::TessellatedSolid each expose the mirrored pair of ports:

Object	Input port (DataSource)	Output port (DataSink)	Formats
TessellatedMesh	"Source", InputFile	"Sink", OutputFile	stl
Abstract::VoxImage	"Source", InputFile	"Sink", OutputFile	vtk, vti, xml
Geant::TessellatedSolid	"Source", InputFile	"Sink", OutputFile	stl

The output side is a FunctionalReferenceProperty<DataSink> declared with uprops::PortDirection::Output, so the node graph draws it as an outgoing port. The accessors follow the same shape as the source side:

void SetSink(DataSink *sink);
void SetSink(const SmartPointer<DataSink> &sink);
DataSink *GetSink() const;
bool WriteToDataSink(DataSink *sink = nullptr);
bool WriteToStream(std::ostream &stream, const std::string &format) const;

const std::string &GetOutputFile() const;   // falls back to the sink's FilePath
void SetOutputFile(const std::string &file);

Attaching a sink writes immediately. Re-writing on every change is opt-in through the AutoWrite property (default false), because a re-write costs a full re-serialisation of the mesh or the voxel block:

uLib::VoxImage<uLib::Voxel> image(Vector3i(64, 64, 64));
uLib::DataSink sink("Volume");
sink.SetFilePath("/tmp/volume.vti");   // Format -> "vti"
image.AutoWrite.Set(true);
image.SetSink(&sink);                  // written now, and on every Updated()

Geant::TessellatedSolid writes through the mesh it is representing, so the STL that comes out is the tessellation that was actually handed to G4TessellatedSolid.

---

6. Stream IO added on the domain side

Making the sink useful required the read-side file IO to grow stream twins, so nothing on the write path needs a file:

• TessellatedMesh::WriteToStlStream() emits a binary STL; polygons with more than three vertices are fanned from their first vertex, the same decomposition TessellatedSolid uses to build G4 facets.
• Abstract::VoxImage::WriteToVtkStream() / ReadFromVtkStream() handle the legacy ASCII .vtk, and are what ExportToVtk() / ImportFromVtk() now call.
• Abstract::VoxImage::WriteToVtiStream() / ReadFromVtiStream() drive vtkXMLImageDataWriter / vtkXMLImageDataReader in string mode (SetWriteToOutputString / SetReadFromInputString), so a .vti payload can live entirely inside a buffer.
• Abstract::VoxImage::WriteToVtkXmlStream() emits the hand-written cell-data XML variant.

---

7. Context-wide packaging

// Keep every exported payload inside the scene archive
uLib::DataSink::EmbedAll(context, true);

// ... or write them all out to a directory, naming unnamed sinks after the node
uLib::DataSink::FlushAll(context, "./exported_assets/");

FlushAll mirrors DataSource::UnpackAll: with a destination directory it re-points each sink at <dir>/<name>.<format>, writes the payload there and clears the embedded flag; with no directory it simply flushes each sink to the path it already has.

---

8. Python bindings

DataSink is exposed in Python under uLib_python.Core:

import uLib_python as u

sink = u.Core.DataSink("MeshOut")
sink.file_path = "/tmp/out.stl"      # format auto-detected as "stl"
sink.write_data(b"raw binary payload")
print(sink.get_size(), sink.get_data())

# Pipe a DataSource straight into a DataSink
src = u.Core.DataSource("In")
src.file_path = "models/chassis.stl"
sink.write_source(src)

# Domain objects
img = u.Math.VoxImage()
img.SetOutputFile("/tmp/volume.vtk")  # creates and attaches a DataSink
print(img.GetSink().get_size())

# Context packaging
u.Core.DataSink.embed_all(context, True)
u.Core.DataSink.flush_all(context, "./exported_assets/")