DataSource Architecture and Embedded Storage System

This document describes the design and usage of the uLib::DataSource class, which manages serialized and raw data sources (e.g. 3D geometry files, STL meshes, calibration tables, detector assets) and supports self-contained scene packaging (similar to .blend file packing in Blender).

graph LR
    subgraph External Disk
        File["Mesh / Asset File (.stl, etc.)"]
    end

    subgraph Core Layer
        DS["DataSource Node (Category: 'Data')\n- FilePath\n- Format\n- Embedded\n- Size\n- m_Buffer"]
        Stream["std::istream (OpenStream())"]
    end

    subgraph Domain Objects
        TM["TessellatedMesh / VoxImage / TessellatedSolid\n(Source functional input port)"]
    end

    subgraph Scene Archive (.gcx)
        GCX["SceneDocument\n(CaptureContext with embedDataSources=true)"]
    end

    File -->|LoadFile() / Embed()| DS
    DS -->|OpenStream()| Stream
    Stream -->|ReadFromStream()| TM
    DS -.->|Emits Updated / DataChanged| TM
    DS -->|Serialized with m_Buffer payload| GCX
    GCX -->|RestoreContext()| DS

---

1. Overview and Motivation

Domain objects in uLib (such as TessellatedMesh, TessellatedSolid, or future voxel image loaders) often require external data files for their geometry or parameters.

Historically, objects directly stored a file path string (e.g. InputFile). This had several limitations: 1. Broken Links: Moving or sharing a scene document (.gcx) broke external file paths if the corresponding files were not distributed in the exact same relative/absolute locations. 2. Opaque Sources: External files were not first-class nodes in the ObjectsContext or scene node graph, making it impossible to introspect, change formats, or share one loaded source between multiple meshes. 3. No In-Memory Generation: Feeding in-memory or streamed geometry directly from network or Python buffers required writing temporary files to disk.

DataSource resolves this by acting as a first-class object that wraps external files, in-memory buffers, and embedded payloads with a unified stream interface.

---

2. Key Features

• First-Class Object: Registered in ObjectFactory under category "Data". It appears in the scene hierarchy and node graph as an introspectable, connectable node.
• Unified Binary Stream (OpenStream()): Transparently opens an std::istream regardless of whether the source is a file on disk or an in-memory buffer.
• Embedded Mode (Blender-like .blend Packing): Converts linked external files into binary memory buffers stored directly within the .gcx scene archive upon saving.
• Automatic Signal Propagation: Modifying a DataSource emits DataSource::Updated and DataSource::DataChanged signals, which downstream objects (like TessellatedMesh) listen to for automatic rebuilding.
• Format Reflection & Auto-Detection: Auto-detects extensions (e.g. .stl) or allows explicit format overrides.
• Context-Wide Utilities: One-call helpers DataSource::EmbedAll(context) and DataSource::UnpackAll(context, destDir).

---

3. Properties and API Reference

Properties

Property	Type	Default	Group	Description
FilePath	std::string	""	"Source"	Path to the external file on disk. Unit metadata is set to "file".
Format	std::string	""	"Source"	Data format identifier (e.g. "stl", "raw"). Auto-inferred from FilePath if not explicitly set.
Embedded	bool	false	"Source"	When true, data is stored in the internal memory buffer and written to archives.
Size	size_t	0	"Source"	Read-only size in bytes of the in-memory buffer or external file.

Core Methods

namespace uLib {

class DataSource : public Object {
public:
  // Path and format configuration
  void SetFilePath(const std::string &path);
  const std::string &GetFilePath() const;
  void SetFormat(const std::string &format);
  const std::string &GetFormat() const;

  // Embedding and file IO
  bool LoadFile(const std::string &path = "");
  bool Embed(bool embed = true);
  bool SaveToFile(const std::string &destPath) const;

  // Stream access
  std::unique_ptr<std::istream> OpenStream() const;

  // In-memory buffer manipulation
  void SetData(const std::vector<uint8_t> &data);
  void SetData(const void *data, size_t size);
  const std::vector<uint8_t> &GetData() const;
  void ClearData();

  // Context-wide operations
  static void EmbedAll(ObjectsContext &context, bool embed = true);
  static bool UnpackAll(ObjectsContext &context, const std::string &destDirectory);
};

}

---

4. How Stream Access and Fallback Work

When OpenStream() is called on a DataSource:

1. Embedded / In-Memory Priority: If IsEmbedded() is true and the internal buffer is non-empty, a std::unique_ptr<std::istream> wrapping a memory buffer stream over m_Buffer is returned.
2. External File Stream: If not embedded (or buffer is empty), it attempts to open FilePath on disk via std::ifstream in binary mode.
3. Fallback Resiliency: If opening FilePath from disk fails (e.g. the external file was deleted or moved), but m_Buffer is non-empty, it automatically falls back to streaming from m_Buffer.

---

5. Domain Object Integration

Classes needing input data expose a functional input port FunctionalReferenceProperty<DataSource> and functional property FunctionalProperty<std::string>. Three of them do today:

Object	Port group	Formats read
TessellatedMesh	"Geometry"	stl (binary and ASCII)
Abstract::VoxImage	"Source"	vtk (legacy ASCII), vti / xml
Geant::TessellatedSolid	"Geometry"	stl, through the mesh it owns

Each of them also carries the mirrored DataSink output port described in DataSink.

TessellatedMesh

class TessellatedMesh : public TRS {
public:
  // Functional Property & Input Port
  FunctionalProperty<std::string> InputFile;

  // Direct DataSource attachment & Input Port Accessors
  void SetSource(DataSource *source);
  void SetSource(const SmartPointer<DataSource> &source);
  DataSource *GetSource() const;

  // Reading from streams and data sources
  bool ReadFromDataSource(DataSource *source);
  bool ReadFromStream(std::istream &stream, const std::string &format = "stl");

  // File setter & getter with DataSource fallback
  const std::string &GetInputFile() const;
  void SetInputFile(const std::string &file);
};

When a DataSource is attached via SetSource(ds) or wired into the "Source" functional input port in NodeGraph: - TessellatedMesh connects to ds->Updated: any modification to FilePath, memory buffer, or embedded state triggers an immediate re-read of geometry without manual reload calls. - InputFile Fallback: If InputFile has not been explicitly set to a local filename, querying GetInputFile() / InputFile.Get() automatically falls back to GetSource()->GetFilePath(). - Backward-Compatible File Setter: Setting SetInputFile(path) automatically creates or updates an underlying DataSource and loads the mesh.

Abstract::VoxImage

The voxel image side is the same port pair, sitting on the non-template Abstract::VoxImage base so every VoxImage<T> inherits it:

class VoxImage : public uLib::StructuredGrid {
public:
  FunctionalProperty<std::string> InputFile;   // falls back to the source's path
  ULIB_PROPERTY(bool, DensityType, false)      // density units for the payload

  void SetSource(DataSource *source);
  DataSource *GetSource() const;
  bool ReadFromDataSource(DataSource *source);

  bool ReadFromStream(std::istream &stream, const std::string &format = "vtk",
                      bool density_type = 0);
  bool ReadFromVtkStream(std::istream &stream, bool density_type = 0);
  bool ReadFromVtiStream(std::istream &stream, bool density_type = 0);
};

ReadFromStream() dispatches on the source's Format: "vti", "xml" and "vtu" go through vtkXMLImageDataReader in string mode, anything else through the legacy ASCII reader. The file entry points (ImportFromVtk, ImportFromVti, ExportToVtk, ExportToVtkXml) are now thin wrappers over the stream versions, so an embedded .vti in a .gcx loads exactly as a file on disk would.

uLib::DataSource ds("Volume");
ds.SetFilePath("recon/volume.vti");
ds.Embed(true);                       // payload now travels inside the scene

uLib::VoxImage<uLib::Voxel> image;
image.SetSource(&ds);                 // reads now, and again on every ds change

Geant::TessellatedSolid

The Geant solid delegates to the TessellatedMesh it owns: attaching a DataSource wires it to that mesh, and the solid rebuilds its G4TessellatedSolid from the mesh's Updated signal as it already did.

uLib::Geant::TessellatedSolid solid("chassis");
solid.SetSource(&ds);                 // owned mesh reads the stl, facets rebuilt
solid.SetInputFile("models/chassis.stl");   // equivalent, creates the DataSource

Because the read now goes through the source's stream rather than ReadFromStlFile(), a scene saved with embedDataSources = true reconstructs its Geant geometry on a machine where the original STL never existed.

---

6. Context Saving & Packaging (.gcx)

To save scenes as completely self-contained bundles:

#include "Core/SceneDocument.h"
#include "Core/DataSource.h"

uLib::ObjectsContext context;
// ... populate context with DataSources and TessellatedMeshes ...

uLib::SceneDocument doc;
uLib::StampDocumentHeader(doc, "MyApp");

// Set embedDataSources = true to pack all files into the archive
bool ok = uLib::CaptureContext(context, doc, nullptr, nullptr, /*embedDataSources=*/true);

uLib::WriteDocument("scene_bundle.gcx", doc);

When scene_bundle.gcx is distributed and restored on another machine (even where the original STL files do not exist): - RestoreContext(doc, context) reconstructs the DataSource nodes with their embedded memory buffers. - Downstream TessellatedMesh nodes automatically read directly from the embedded streams. - If users want to extract the files back to disk, they can call DataSource::UnpackAll(context, "./extracted_assets/").

---

7. Python Bindings Example

DataSource is fully exposed in Python under uLib_python.Core:

import uLib_python.Core as Core

# 1. Create a DataSource linking to an external mesh
ds = Core.DataSource("CadSource")
ds.file_path = "models/chassis.stl"
print(f"Format: {ds.format}, Size: {ds.get_size()} bytes")

# 2. Embed into memory for scene packing
ds.embed(True)
print(f"Embedded: {ds.embedded}")

# 3. Direct binary buffer access
payload = ds.get_data()
ds.set_data(b"raw binary payload...")

# 4. Context packaging
context = Core.ObjectsContext()
context.AddObject(ds)
Core.DataSource.embed_all(context, True)