Add generic RT/NRT shared resource system (#147)

CHANGELOG.md

@@ -19,6 +19,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `METHCLA_BUILD_TESTS` CMake option, defaults to on when building as top-level project (#122)
 - CMake presets (`debug`, `release`) and GitHub Actions CI replacing ad-hoc configuration and Travis CI (#118)
 - Project documentation: CONTRIBUTING guide, architecture overview with Mermaid diagram, OSC API reference, examples README
+- Generic RT/NRT shared resource system (#147): `Methcla_ResourceDef` plugin API, client-side `ResourceId` API, OSC commands `/resource/new` and `/resource/free`, notifications `/resource/ready`, `/resource/error`, and `/resource/destroyed`, RT-side `methcla_world_resource_acquire`/`methcla_world_resource_release` primitives, NRT bracketed access via `methcla_world_perform_with_resources`, and `ResourceRef<T>` / `ResourceDef<R,O>` C++ wrappers
 
 ### Changed
 

CONTEXT.md

@@ -24,7 +24,15 @@ _Avoid_: voice, instance, node (overloaded with the Node base class)
 
 **AudioBus**:
 A multi-channel buffer used for audio routing between Synths inside the Engine.
-_Avoid_: channel, buffer, bus (acceptable shorthand in code)
+_Avoid_: channel, bus (acceptable shorthand in code)
+
+**Resource**:
+A Plugin-registered shared data object managed by the Engine, identified by a `Methcla_ResourceId` integer and typed by URI. Allocated/freed via OSC, acquired/released by Synths on the RT thread, accessed from the NRT context via `perform_with_resources`.
+_Avoid_: instance (a Synth is an instance; a Resource is the thing the Plugin allocates), buffer (overloaded)
+
+**ResourceDef**:
+A Resource type registered by a Plugin. Describes the configure, construct, destroy callbacks and the interface URI for one kind of shared Resource.
+_Avoid_: resource class, resource type
 
 **Soundfile API**:
 A Plugin category that provides file I/O capabilities to other Plugins (disksampler, sampler). Multiple soundfile API Plugins may be registered; the Engine selects by capability. Platform implementations: ExtAudioFile (macOS), libsndfile (Linux).

docs/architecture.md

@@ -33,4 +33,75 @@ graph TB
 - **Internal AudioBuses** route audio between Synths within the Engine.
 - The **Root Group** is the top of the node tree. Groups nest arbitrarily; Synths are leaves.
 - Each Synth maps its audio inputs and outputs to buses via `/synth/map/input` and `/synth/map/output` (see [`osc-api.md`](osc-api.md)).
+
+## Resource system
+
+The Engine owns a fixed-size pool of typed, reference-counted Resource slots, indexed by a client-assigned `Methcla_ResourceId`. Capacity is set at startup via `EngineOptions::maxNumResources` (default 256). A Resource is a Plugin-allocated object — its layout is declared in a Plugin-supplied header and identified by a URI — that can be shared between the RT and NRT contexts and across Synths.
+
+### Resource state machine
+
+Each slot is in one of four states; all transitions run on the RT thread.
+
+```
+                 /resource/new (validated, id reserved)
+        Free ──────────────────────────────────────────→ Constructing
+         ▲                                                    │
+         │                                                    │  NRT construct returns
+         │                                          ┌─────────┴─────────┐
+         │                                          │                   │
+         │                                       error                 ok
+         │                                          │                   │
+         │ NRT destroy returns                      ▼                   ▼
+         │ (state → Free)                       (notify              Live
+         │                                       /resource/error)     │
+         │                                          │                  │ /resource/free
+         │                                          │                  │   refCount==0 → Destroying
+         │                                          ▼                  │   refCount>0  → freePending=true,
+         └──────────────────────────────────────  Free                 │                  drain via release path
+                                                                       ▼
+                                                                  Destroying
+                                                                       │
+                                                                       │ NRT destroy returns
+                                                                       └────────→ Free
 ```
+
+`freePending` records that `/resource/free` arrived while the slot was Constructing or Live-with-refCount>0. The state stays Live until the last `resource_release` drains the refcount; that release then transitions the slot to Destroying and schedules the NRT destroy.
+
+### Acquire / release and refcount
+
+`resource_acquire(world, id, expected_uri)` on `Methcla_World`:
+
+1. Bounds-check `id`; require the slot to be Live.
+2. Compare the slot's URI to `expected_uri` (string equality); reject on mismatch.
+3. Increment `refCount` and return the resource's data pointer.
+
+`resource_release(world, id)` decrements `refCount`. If the count reaches zero and `freePending` is set, the slot transitions Live → Destroying and an NRT destroy is scheduled.
+
+Both primitives mutate state from the RT thread only; no atomics are required.
+
+### NRT access
+
+`perform_with_resources(world, ids, num_ids, perform, user_data)` brackets a worker-thread callback with RT-side acquire/release of the listed Resources. The engine acquires each id on RT, dispatches the callback to a worker thread with the array of data pointers, and releases the Resources back on RT after the callback returns. The pointers are valid only for the duration of the callback. If any acquire fails the already-acquired Resources are released and the callback is not invoked.
+
+### Mutability
+
+`Methcla_ResourceMutability` (`kMethcla_Mutable` / `kMethcla_Immutable`) is informational metadata declared by the Resource type. The Engine does not enforce read/write access; the Resource class's contract documents it for consumers and Plugin authors.
+
+### Threading summary
+
+| Event                             | Thread   | Touches                                                                       |
+| --------------------------------- | -------- | ----------------------------------------------------------------------------- |
+| `/resource/new` arrives           | RT       | slot Free → Constructing; schedule NRT construct                              |
+| NRT construct returns             | NRT → RT | RT: slot → Live; emit `/resource/ready` (or Free + `/resource/error`)         |
+| `resource_acquire`                | RT       | `refCount++`                                                                  |
+| `resource_release`                | RT       | `refCount--`; if 0 && freePending: → Destroying + schedule destroy            |
+| `/resource/free` arrives          | RT       | refCount==0 && Live: → Destroying + schedule destroy; else `freePending=true` |
+| `perform_with_resources` dispatch | RT       | acquire all; schedule NRT callback                                            |
+| NRT callback returns              | NRT → RT | RT auto-releases all                                                          |
+| NRT destroy returns               | NRT → RT | RT: slot → Free; emit `/resource/destroyed`                                   |
+
+Lifecycle notifications (`/resource/ready`, `/resource/error`, `/resource/destroyed`) originate from RT after the state transition and are dispatched via NRT — mirrors `/node/done`. Sending notifications directly from NRT would let a client's follow-up `/synth/new` race the publish.
+
+### Multi-worker caveat
+
+Resource-def callbacks (`construct`, `destroy`, and the `perform_with_resources` callback) run on the worker pool and **can execute concurrently across different worker threads** for different Resources. The engine's NRT-side APIs are thread-safe, but Plugin authors that share state across Resource instances must synchronize that state themselves.

docs/osc-api.md

@@ -131,6 +131,30 @@ The engine sends these messages to the host without a corresponding request. The
 
   Sent unconditionally when any node is freed (by any means, including **`/node/free`**, done-action flags, or parent group teardown).
 
+## Resource commands
+
+* **`/resource/new`** `i:resource-id s:uri [options...]`
+
+  Allocate a resource of type `uri` with the given client-assigned `resource-id`. Any trailing arguments are passed to the resource definition's `configure` function. On success the engine sends `/resource/ready` asynchronously once construction on the NRT thread completes. On `configure` error the engine replies with `/error`; on `construct` error it sends `/resource/error`.
+
+* **`/resource/free`** `i:resource-id`
+
+  Release a live resource. If the resource is still being constructed, or held by one or more consumers (refcount > 0), the free is deferred (`freePending`); the engine destroys it once the refcount drains and construction has completed. The engine sends `/resource/destroyed` once destruction is complete. See [architecture.md](architecture.md#resource-system) for the full state machine.
+
+## Resource notifications
+
+* **`/resource/ready`** `i:resource-id`
+
+  Sent when a resource has been successfully constructed and is ready for use.
+
+* **`/resource/error`** `i:resource-id i:error-code s:message`
+
+  Sent when a resource's `construct` function returns an error. The resource entry is freed and the id is reusable. `error-code` is a `Methcla_ErrorCode` value.
+
+* **`/resource/destroyed`** `i:resource-id`
+
+  Sent when a resource has been destroyed in response to `/resource/free`.
+
 ## Error responses
 
 * **`/error`** `i:error-code s:message`

include/methcla/common.h

@@ -68,6 +68,7 @@ typedef enum
     kMethcla_SynthDefNotFoundError = 1000,
     kMethcla_NodeIdError,
     kMethcla_NodeTypeError,
+    kMethcla_UnsupportedResourceTypeError,
 
     /* File errors */
     kMethcla_FileNotFoundError = 2000,
@@ -109,6 +110,8 @@ static inline const char* methcla_error_code_description(Methcla_ErrorCode code)
             return "Invalid node id";
         case kMethcla_NodeTypeError:
             return "Invalid node type";
+        case kMethcla_UnsupportedResourceTypeError:
+            return "Unsupported resource type";
 
         /* File errors */
         case kMethcla_FileNotFoundError:

include/methcla/detail.hpp

@@ -6,6 +6,7 @@
 #pragma once
 
 #include <memory>
+#include <ostream>
 #include <stdexcept>
 #include <string>
 
@@ -35,6 +36,17 @@ namespace Methcla { namespace detail {
             return m_id != other.m_id;
         }
 
+        operator bool() const
+        {
+            return m_id != static_cast<T>(-1);
+        }
+
+        friend std::ostream& operator<<(std::ostream& out, const D& id)
+        {
+            out << id.m_id;
+            return out;
+        }
+
     private:
         T m_id;
     };

include/methcla/engine.hpp

@@ -49,20 +49,8 @@ namespace Methcla {
         NodeId()
         : NodeId(-1)
         {}
-
-        operator bool() const
-        {
-            return *this != NodeId();
-        }
     };
 
-    inline static std::ostream& operator<<(std::ostream& out,
-                                           const NodeId& nodeId)
-    {
-        out << nodeId.id();
-        return out;
-    }
-
     class GroupId : public NodeId
     {
     public:
@@ -86,6 +74,17 @@ namespace Methcla {
         {}
     };
 
+    class ResourceId : public detail::Id<ResourceId, int32_t>
+    {
+    public:
+        explicit ResourceId(int32_t id)
+        : Id<ResourceId, int32_t>(id)
+        {}
+        ResourceId()
+        : ResourceId(-1)
+        {}
+    };
+
     // Node placement specification given a target.
     class NodePlacement
     {
@@ -509,6 +508,7 @@ namespace Methcla {
         size_t                     realtimeMemorySize = 1024 * 1024;
         size_t                     maxNumNodes = 1024;
         size_t                     maxNumAudioBuses = 128;
+        size_t                     maxNumResources = 256;
         size_t                     maxNumControlBuses = 4096;
         size_t                     sampleRate = 44100;
         size_t                     blockSize = 64;
@@ -590,6 +590,7 @@ namespace Methcla {
 
     typedef IdAllocator<NodeId, int32_t>     NodeIdAllocator;
     typedef IdAllocator<AudioBusId, int32_t> AudioBusIdAllocator;
+    typedef IdAllocator<ResourceId, int32_t> ResourceIdAllocator;
 
     class Request;
 
@@ -604,7 +605,8 @@ namespace Methcla {
             return GroupId(0);
         }
 
-        virtual NodeIdAllocator& nodeIdAllocator() = 0;
+        virtual NodeIdAllocator&     nodeIdAllocator() = 0;
+        virtual ResourceIdAllocator& resourceIdAllocator() = 0;
 
         virtual std::unique_ptr<Packet> allocPacket() = 0;
         virtual void sendPacket(const std::unique_ptr<Packet>& packet) = 0;
@@ -625,6 +627,10 @@ namespace Methcla {
                               BusMappingFlags flags = kBusMappingInternal);
         inline void set(NodeId node, size_t index, double value);
         inline void free(NodeId node);
+        inline ResourceId
+                    resource(const char*             uri,
+                             const std::list<Value>& options = std::list<Value>());
+        inline void free(ResourceId id);
     };
 
     class Request
@@ -863,6 +869,39 @@ namespace Methcla {
                 .int32(flags)
                 .closeMessage();
         }
+
+        ResourceId
+        resource(const char*             uri,
+                 const std::list<Value>& options = std::list<Value>())
+        {
+            beginMessage();
+
+            const ResourceId resourceId(
+                m_engine->resourceIdAllocator().alloc());
+
+            oscPacket()
+                .openMessage("/resource/new", 2 + options.size())
+                .int32(resourceId.id())
+                .string(uri);
+
+            for (const auto& x : options)
+                x.put(oscPacket());
+
+            oscPacket().closeMessage();
+
+            return resourceId;
+        }
+
+        void free(ResourceId id)
+        {
+            beginMessage();
+
+            oscPacket()
+                .openMessage("/resource/free", 1)
+                .int32(id.id())
+                .closeMessage();
+            m_engine->resourceIdAllocator().free(id);
+        }
     };
 
     void EngineInterface::bundle(Methcla_Time                  time,
@@ -936,6 +975,22 @@ namespace Methcla {
         request.send();
     }
 
+    ResourceId EngineInterface::resource(const char*             uri,
+                                         const std::list<Value>& options)
+    {
+        Request    request(this);
+        ResourceId result = request.resource(uri, options);
+        request.send();
+        return result;
+    }
+
+    void EngineInterface::free(ResourceId id)
+    {
+        Request request(this);
+        request.free(id);
+        request.send();
+    }
+
     class Engine : public EngineInterface
     {
     public:
@@ -944,6 +999,7 @@ namespace Methcla {
         : m_logHandler(inOptions.logHandler)
         , m_nodeIds(1, inOptions.maxNumNodes - 1)
         , m_audioBusIds(0, inOptions.maxNumAudioBuses)
+        , m_resourceIds(0, inOptions.maxNumResources)
         , m_requestId(kMethcla_Notification + 1)
         , m_notificationHandlerId(0)
         , m_packets(8192)
@@ -1023,11 +1079,16 @@ namespace Methcla {
             return m_nodeIds;
         }
 
-        AudioBusIdAllocator& audioBusId()
+        AudioBusIdAllocator& audioBusIdAllocator()
         {
             return m_audioBusIds;
         }
 
+        ResourceIdAllocator& resourceIdAllocator() override
+        {
+            return m_resourceIds;
+        }
+
         std::unique_ptr<Packet> allocPacket() override
         {
             return std::make_unique<Packet>(m_packets);
@@ -1311,6 +1372,7 @@ namespace Methcla {
         LogHandler            m_logHandler;
         NodeIdAllocator       m_nodeIds;
         AudioBusIdAllocator   m_audioBusIds;
+        ResourceIdAllocator   m_resourceIds;
         Methcla_RequestId     m_requestId;
         std::mutex            m_requestIdMutex;
         ResponseHandlers      m_responseHandlers;