Add ObjC/Swift bindings for the ImageProcessor (#20051)

[metascroy]

Summary:

This diff adds ObjC/Swift bindings for the image processor. Only the pixelbuffer variants get ObjC/Swift bindings. The process/process_yuv variants are only accessible from C++ for now.

Example:

import ExecuTorch

// Configure once
let config = ImageProcessorConfig(
    targetWidth: 224,
    targetHeight: 224,
    normalization: .imagenet()
)
let processor = ImageProcessor(config: config)

// Process a CVPixelBuffer (BGRA/RGBA/NV12/P010)
let tensor: Tensor<Float> = try processor.process(pixelBuffer)
// Output: [1, 3, 224, 224] Float tensor

// For video (reuse tensor to avoid allocations)
let output = Tensor<Float>([1, 3, 224, 224], Array(repeating: 0, count: 3*224*224))
try processor.process(pixelBuffer, into: output)

Differential Revision: D106898406

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20051

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Cancelled Job

As of commit ce4a08f with merge base 502fdbe ():

CANCELLED JOB - The following job was cancelled. Please retry:

• pull / unittest-editable / macos / macos-job (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-codesync]

@metascroy has exported this pull request. If you are a Meta employee, you can view the originating Diff in D106898406.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[metascroy]

@claude review this code

[claude]

Claude finished @metascroy's task in 1m 47s —— View job

---

Code Review: Add ObjC/Swift bindings for the ImageProcessor

• Read changed files
• Review ObjC header (ExecuTorchImageProcessor.h)
• Review ObjC implementation (ExecuTorchImageProcessor.mm)
• Review Swift extension (ExecuTorch+ImageProcessor.swift)
• Review test file (ImageProcessorTest.swift)
• Review build system changes
• Post review feedback

---

Overall: This is a well-structured, clean PR. The API design is thoughtful—good Swift refinements via NS_REFINED_FOR_SWIFT, appropriate immutability, and correct thread-safety documentation. The binding layer is thin and delegates all image-processing logic to the C++ implementation.

Detailed Findings

ExecuTorchImageProcessor.mm

1. tensorFromResult — move from Result::get() (line 57-58)

auto tensorPtr = std::move(result.get());
return [[ExecuTorchTensor alloc] initWithNativeInstance:&tensorPtr];

This is correct: initWithNativeInstance: performs a std::move from the pointed-to TensorPtr before the stack frame unwinds. However, passing a pointer to a local that will be in a moved-from state immediately after is a pattern that could trip up future readers or refactors. A one-line comment like // initWithNativeInstance moves out of tensorPtr would be defensive—but optional per the project's "minimal comments" style.

2. processPixelBuffer:intoTensor:error: — reinterpret_cast from nativeInstance (line 184)

auto* tensorPtr = reinterpret_cast<TensorPtr*>(tensor.nativeInstance);
auto err = process_pixelbuffer_into(*_processor, pixelBuffer, Orientation::UP, **tensorPtr);

This is sound—nativeInstance returns &_tensor (a TensorPtr* behind void*), and **tensorPtr dereferences to the underlying exec_aten::Tensor&. Correct usage of the existing pattern.

3. Singleton thread-safety (lines 75-90)

dispatch_once singletons for zeroToOne and imagenet are correct. Since Normalization is an immutable value type (just float arrays), sharing a single instance is safe.

4. Config defensive copy (lines 150-158)

Good practice copying the config in initWithConfig:. The normalization property is shared by reference, but since ExecuTorchImageNormalization is immutable and only singleton instances exist, this is fine.

ExecuTorch+ImageProcessor.swift

5. process(_:) force-wraps the tensor type (line 54)

return Tensor<Float>(anyTensor)

Looking at the Tensor<T> initializer, this appears to be a non-failable init (the underlying C++ always produces Float tensors). Correct.

6. Convenience init defaults (lines 25-29)

The defaults (.stretch, .center, padValue: 0, .zeroToOne()) are well-chosen for the common case. The gpuMinInputPixels default correctly delegates to the C++ config's kDefaultGpuMinInputPixels.

ExecuTorchImageProcessor.h

7. _Nullable CVPixelBufferRef parameter

Good defensive API design—CVPixelBufferRef is a pointer type, and explicitly accepting + handling nil prevents crashes from accidental nil forwarding in ObjC.

8. Minor alignment nit (lines 55-61)

The colon alignment on the initWithTargetWidth:... selector parameters is slightly inconsistent (resizeMode: / padValue: appear off by one space vs. the others). Cosmetic only.

ImageProcessorTest.swift

9. Test coverage is appropriate for a binding layer:

• Config field round-trip
• Nil pixel buffer error path
• Process-into vs. allocating equivalence
• Letterbox padding bridge
• Default value forwarding

Correctly defers image-processing correctness to the C++ test suite.

Build system

10. All three build paths updated:

• BUCK: adds image_processorApple / image_processorAppleMac deps ✓
• apple_common.cmake: enables EXECUTORCH_BUILD_EXTENSION_IMAGE ✓
• build_apple_frameworks.sh: includes libextension_image.a in the framework ✓

Summary

No blocking issues. The code is correct, well-tested at the binding boundary, and follows existing patterns. The API surface is clean for both ObjC and Swift consumers. Ship it.

---

[metascroy]

@claude review this code

[claude]

Claude finished @metascroy's task in 1m 46s —— View job

---

Code Review: Add ObjC/Swift bindings for the ImageProcessor

• Read changed files
• Review ObjC header (ExecuTorchImageProcessor.h)
• Review ObjC implementation (ExecuTorchImageProcessor.mm)
• Review Swift extension (ExecuTorch+ImageProcessor.swift)
• Review test file (ImageProcessorTest.swift)
• Review build system changes
• Post review feedback

---

Overall: Clean, well-structured PR. Ship it. The binding layer is thin, delegates correctness to C++, and follows existing patterns (nativeInstance, NS_REFINED_FOR_SWIFT, dispatch_once singletons). The API is well-designed for both ObjC and Swift consumers.

Detailed Findings

ExecuTorchImageProcessor.h — No issues

• Good use of NS_REFINED_FOR_SWIFT to hide the raw ObjC init and expose a Swift-friendly convenience init with defaults.
• __attribute__((objc_subclassing_restricted)) on all three classes is appropriate — these are leaf types.
• _Nullable CVPixelBufferRef is defensive and correct.
• Thread-safety documentation is clear and accurate.

ExecuTorchImageProcessor.mm

1. Compile-time enum parity checks (lines 30-33) — Great. static_assert ensures ObjC enums stay in sync with C++ without a runtime cost.

2. tensorFromResult (lines 45-57) — Correct. initWithNativeInstance: moves out of the local TensorPtr before the frame unwinds (verified by checking ExecuTorchTensor.mm:123). The comment on line 55 clarifies this nicely.

3. Custom normalization alpha-slot handling (lines 103-104):

norm.mean[ImageProcessorConfig::kOutputChannels] = 0.0f;
norm.std_dev[ImageProcessorConfig::kOutputChannels] = 1.0f;

Correctly initializes the reserved 4th slot to identity ((x - 0) / 1 = x), matching the Normalization struct's documented invariant.

4. Defensive copy in initWithConfig: (lines 166-173) — Correct defensive copy. Since ExecuTorchImageProcessorConfig is immutable (all readonly properties, no init besides the designated one), this is belt-and-suspenders. Acceptable.

5. Potential improvement (non-blocking): In processPixelBuffer:intoTensor:error: (line 194), the !tensor nil check is somewhat redundant given the parameter is nonnull (via NS_ASSUME_NONNULL_BEGIN). The compiler already warns callers passing nil to a nonnull param. However, it's defensive and low-cost — fine to keep.

ExecuTorch+ImageProcessor.swift

6. Tensor<Float>(anyTensor) (line 72) — Safe. The init(_ tensor: AnyTensor) has a precondition checking tensor.dataType == T.dataType, and the C++ always outputs Float. If the C++ ever changed output type, this would trap at runtime — acceptable since that would be a breaking C++ change requiring binding updates anyway.

7. Convenience init defaults (lines 40-57) — Well-chosen defaults: .stretch, .center, padValue: 0, .zeroToOne(). The gpuMinInputPixels default correctly delegates to the C++ constant via the class property.

8. alwaysGPU/alwaysCPU sentinels (lines 31-32) — Clean. Int.max for alwaysCPU matches kGpuNever = INT64_MAX since Swift Int is 64-bit on all Apple platforms.

ImageProcessorTest.swift

9. Test coverage is appropriate for a binding layer:

• Config field round-trip (testConfigPropertyRoundTrip)
• Default value forwarding (testDefaultInitializerUsesDefaultThreshold)
• Nil pixel buffer error path (testProcessNilPixelBufferReturnsError)
• Wrong-shape tensor error (testProcessIntoWrongShapeReturnsError)
• Process-into vs allocating equivalence (testProcessIntoMatchesAllocatingPath)
• Letterbox padding bridge (testComputeLetterboxPadding)
• Custom normalization flows through (testCustomNormalizationApplied)

Good decision to explicitly not re-test image-processing correctness (resize math, color conversion, format support) — that's owned by the C++ suite.

Build system

10. All three build paths updated correctly:

• BUCK: adds image_processorApple / image_processorAppleMac deps ✓
• apple_common.cmake: enables EXECUTORCH_BUILD_EXTENSION_IMAGE ✓
• build_apple_frameworks.sh: includes libextension_image.a in the framework ✓

Summary

No blocking issues found. The code is correct, well-tested at the binding boundary, and follows existing patterns. The API surface is clean for both ObjC and Swift consumers. LGTM.

---

View job