fix(datalog): reject forged dl-program handles instead of dereferencing them

[belowzeroff]

How it looks from the user's side

A single line — a typo, or a stray value from an IPC client — takes the whole process down.

Before (unpatched dev), a user types (dl-free 1):

src/ops/datalog.c:66:31: runtime error: member access within misaligned address
0x000000000001 for type 'dl_program_t', which requires 8 byte alignment

=== rayforce fatal SIGSEGV at fault addr 0x0000000000003a01 ===
2   rayforce   dl_program_free + 192
3   rayforce   ray_dl_free_fn + 416
4   rayforce   ray_eval + 16932
5   rayforce   ray_repl_run_file + 2672
6   rayforce   main + 14000
=== end backtrace ===

The value 1 is dereferenced as a dl_program_t*. Every dl-* builtin unwraps its handle argument, so (dl-stratify 1), (dl-query 1 'r), (dl-eval 1), (dl-add-edb 1 …) crash the same way. On a server evaluating client input over IPC (-U/-i) this is remotely triggerable — one client kills every connected session — and since the integer is used directly as a pointer to free()/dereference, an attacker-chosen value is an arbitrary-pointer free/deref (memory corruption), not merely a DoS.

After (this PR) — the same inputs return a clean type error and the process keeps running:

    (dl-free 1)                error: type: dl-free: not a dl-program handle   (exit 1, no crash)

  A user (or IPC client) passes stray / forged values:
    (dl-free 1)        ->  type error   (before: SIGSEGV)
    (dl-stratify 1)    ->  type error   (before: SIGSEGV)
    (dl-query 1 'rel)  ->  type error   (before: SIGSEGV)

  A copy of a real handle is also refused (no double-free):
    (dl-free (+ P 0))  ->  type error   (copy is not a handle)

  Normal Datalog is unaffected:
    (dl-eval P)        ->  true
    (dl-free P)        ->  true
    (dl-free P) again  ->  false   (idempotent)

  >>> process stayed alive the entire time — no crash.

Fix

Tag the handle atom with RAY_ATTR_DLPROG, mirroring the existing RAY_ATTR_GRAPH / RAY_ATTR_HNSW handle scheme, and verify it in dl_unwrap_program:

static ray_t* dl_wrap_program(dl_program_t* prog) {
    ...
    obj->i64   = (int64_t)(uintptr_t)prog;
    obj->attrs |= RAY_ATTR_DLPROG;      // tag
    return obj;
}
static dl_program_t* dl_unwrap_program(ray_t* obj) {
    if (!obj || obj->type != -RAY_I64 || !(obj->attrs & RAY_ATTR_DLPROG))
        return NULL;                     // forged/plain integer → rejected
    return (dl_program_t*)(uintptr_t)obj->i64;
}

dl-free clears the tag and zeroes the pointer on free, so it keeps its idempotent-false contract for a genuine double-free (datalog_coverage.rfl line 542) while rejecting a forged handle with a type error.

Bit choice: RAY_ATTR_DLPROG reuses 0x20 (same value as RAY_ATTR_SORTED on vectors / ATTR_QUOTED on -RAY_SYM). Those meanings are vector- / -RAY_SYM-scoped and are never read on a -RAY_I64 atom, and — unlike HAS_INDEX (0x08) / SLICE (0x10) / ARENA (0x80) — no heap free-path or finalizer reads 0x20, so there is no collision on GC. This is the same context-reuse the codebase already does (RAY_ATTR_HNSW and RAY_ATTR_HAS_LINK share 0x04).

Also fixed by the same tag

• Arithmetic-copy double-free: (+ h 0) produces a fresh i64 without the tag, so a copy can no longer be freed (which would double-free the original). The original still frees exactly once.
• Snapshot / ser→de restored stale handle: serialization strips atom attrs, so a restored handle loses the tag and is rejected instead of dereferencing an address from the previous process. Verified: (dl-free (de (ser (dl-program)))) → type error, no crash.

Out of scope (separate follow-ups): a dropped handle still leaks (needs a heap finalizer entry), and restricted-IPC clients can still call dl-free (needs RAY_FN_RESTRICTED).

Tests

test/rfl/datalog/datalog_coverage.rfl: forged-handle rejection for dl-free / dl-stratify / dl-eval / dl-query / dl-add-edb, and the arithmetic-copy double-free case. The existing handle-lifecycle claims (free → idempotent-false → use-after-free type error) are unchanged.

Full ASan+UBSan suite green: 3708 of 3709 passed (1 skipped, 0 failed).

[singaraiona]

Reviewed with an adversarial runtime pass (ASan/UBSan, repros run on both head and pre-PR master). The fix is correct and lands a real security win — approving. Pre-PR, (dl-free 1) (a forged integer handle from one line of client input) hits a UBSan "misaligned address 0x1" and SIGSEGVs — an arbitrary-pointer deref. Post-PR, (dl-free 1), (dl-stratify 1), (dl-query 1 'x), (dl-eval 1) are all cleanly rejected with type errors. That's the whole point of the PR and it works.

I did chase down four concerns hard (the tag reuses attr bit 0x20, which is also RAY_ATTR_SORTED, and for a -RAY_I64 atom len aliases i64 in the union — both confirmed). None is a blocker after runtime testing:

• Double-free via block copy (the scary one): does NOT reproduce. The aliasing mechanism is real (ray_retain_owned_refs has HNSW/GRAPH branches but no DLPROG one), but no rfl surface produces such a copy: (set G H) shares the atom (rc bump, not a copy) so the first dl-free clears the shared tag and the second returns false; (alter 'H set 0 0) type-errors on the ray_is_vec/LIST guard before ray_cow and before the bounds check, so the union-aliasing path is unreachable too; (+ H 0) / (first (enlist H)) produce untagged copies that dl-free rejects — which is exactly the arithmetic-copy escape the PR closes. No ASan double-free or UAF from any variant.
• Pointer disclosure via .attr.drop: real but negligible. (.attr.drop H) does return the raw pointer via the unguarded SORTED branch — but a -RAY_I64 handle already prints as its own pointer value, pre- and post-PR alike, so nothing is disclosed beyond evaluating the handle. ASan-clean (the returned copy is untagged, unre-freeable).
• Leak on last-ref drop without dl-free: real, memory-safe. Confirmed via .mem.ts (LSan can't see it — dl_program_new uses the ray buddy pool, not malloc): dropping a handle without freeing leaks ~2MB (net-bytes: 2097152), vs net-bytes: 0 with dl-free, while a GRAPH handle dropped without .graph.free is net-bytes: 0. So the free-path asymmetry (finalizers for HNSW/GRAPH, none for DLPROG) is genuine.

Recommended follow-up hardening (non-blocking — I'd take them in a separate commit or PR):

• Add DLPROG branches to ray_retain_owned_refs, the copy-detach path, and an rc→0 finalizer, mirroring GRAPH/HNSW. This closes both the latent double-free (should a generic deep-copy-over-atoms builtin ever be added) and the finding-3 leak at once.
• Guard ray_attr_drop_fn against -RAY_I64 handles (add the ray_is_vec check the other SORTED readers have).
• dl-free open-codes the attrs & RAY_ATTR_DLPROG predicate instead of routing through dl_unwrap_program (datalog.c:4530) — the one free-calling path drifting from the accessor is how the next gate-strengthening gets missed; peer frees reuse their unwrap.
• Update the bit-allocation table in heap.h (~55) to record 0x20 / -RAY_I64 / RAY_ATTR_DLPROG — right now the reuse is documented only in the #define block, so the next handle type could claim the same bit.

Approving and merging — the security fix is sound; the above are all defense-in-depth on top of it.