Skip to content

audio: microwakeword: add mww component - #11135

Draft
singalsu wants to merge 15 commits into
thesofproject:mainfrom
singalsu:mww_mfcc_kpb_development
Draft

audio: microwakeword: add mww component#11135
singalsu wants to merge 15 commits into
thesofproject:mainfrom
singalsu:mww_mfcc_kpb_development

Conversation

@singalsu

Copy link
Copy Markdown
Collaborator

No description provided.

@singalsu
singalsu force-pushed the mww_mfcc_kpb_development branch from 2151ccb to 28a11a0 Compare August 27, 2026 16:51
Comment thread app/boards/intel_adsp_ace15_mtpm.conf
@singalsu
singalsu force-pushed the mww_mfcc_kpb_development branch from 28a11a0 to 2674e49 Compare August 28, 2026 16:37
Comment thread scripts/llext_link_helper.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are correctness issues that will break integration (notably the KPB UUID mismatch with the UUID registry) and should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR introduces a new microWakeWord (MWW) keyword-spotting component and integrates it into SOF’s IPC4/Zephyr-based build, including new topology2 pipelines (KPB-based Wake-on-Voice branches), rimage manifests, and supporting LLEXT/AMS/DP-scheduler plumbing needed to run TFLite Micro–based C++ extensions.

Changes:

  • Add the microwakeword audio component (C/C++ + Kconfig/CMake + rimage TOML) and offline training/tuning scripts.
  • Add topology2 widget/pipeline templates and platform overlays to enable MWW+KPB Wake-on-Voice capture branches on HDA and SDW.
  • Extend DP-scheduler integration (optional DP→DP binding), AMS payload handling, and LLEXT build/export behavior for C++ multi-TU libraries.
File summaries
File Description
zephyr/Kconfig Broadens IPC4 pipeline2.0 + DP scheduler defaults to ACE/CAVS; adds DP→DP bind Kconfig.
zephyr/CMakeLists.txt Adds -Wl,-Bsymbolic-functions for LLEXT shared builds to avoid unresolved local cross-TU calls.
uuid-registry.txt Registers UUID for new mww module.
tools/topology/topology2/sof-hda-generic.conf Adds optional HDA mic MWW/KPB capture include and required class includes.
tools/topology/topology2/platform/intel/sdw-jack-mww-kpb.conf New SDW jack MWW/KPB WoV capture branch overlay topology.
tools/topology/topology2/platform/intel/sdw-dmic-mww-kpb.conf New SDW DMIC MWW/KPB WoV capture branch overlay topology.
tools/topology/topology2/platform/intel/hda-mic-mww-kpb.conf New HDA analog MWW/KPB WoV capture branch overlay topology.
tools/topology/topology2/platform/intel/dmic1-mfcc.conf Removes redundant MFCC include (comment-only change).
tools/topology/topology2/include/pipelines/cavs/src-kpb-be.conf New reusable SRC→KPB backend pipeline class for WoV branching.
tools/topology/topology2/include/pipelines/cavs/host-gateway-src-mfcc-mww-capture.conf Adds a detection pipeline template combining SRC+MFCC+MWW.
tools/topology/topology2/include/pipelines/cavs/host-gateway-micsel-mfcc-mww-capture.conf Adds a stereo-compatible detection pipeline template using micsel+MFCC+MWW.
tools/topology/topology2/include/components/mww.conf Adds topology2 widget class definition for mww.
tools/topology/topology2/include/components/mfcc/mel80.conf Updates exported MFCC config date header.
tools/topology/topology2/include/components/mfcc/mel80_compress.conf Updates exported MFCC config date header.
tools/topology/topology2/include/components/mfcc/mel80_compress_dtx.conf Updates exported MFCC config date header.
tools/topology/topology2/include/components/mfcc/mel40.conf Adds new 40-bin MFCC config blob.
tools/topology/topology2/include/components/mfcc/mel40_compress.conf Adds new compressed-output 40-bin MFCC config blob.
tools/topology/topology2/include/components/mfcc/mel40_10ms.conf Adds new 40-bin, 10ms-hop MFCC config blob for MWW.
tools/topology/topology2/include/components/mfcc/mel40_10ms_compress.conf Adds compressed-output 40-bin, 10ms-hop MFCC config blob.
tools/topology/topology2/include/components/mfcc/default.conf Updates exported MFCC config date header.
tools/topology/topology2/include/components/mfcc/ceps13_compress_dtx.conf Updates exported MFCC config date header.
tools/topology/topology2/include/components/kpb.conf Updates KPB UUID (currently inconsistent with uuid registry; see comment).
tools/topology/topology2/include/common/common_definitions.conf Adds feature flags for enabling MWW/KPB overlays.
tools/topology/topology2/include/common/abi.conf Adds ABI manifest blob definition.
tools/topology/topology2/include/bench/mfccmel40_10ms_s32.conf Adds MFCC mel40_10ms benchmark include (S32).
tools/topology/topology2/include/bench/mfccmel40_10ms_s24.conf Adds MFCC mel40_10ms benchmark include (S24).
tools/topology/topology2/include/bench/mfccmel40_10ms_s16.conf Adds MFCC mel40_10ms benchmark include (S16).
tools/topology/topology2/include/bench/mfcc_controls_playback.conf Adds bench parameter key for mel40_10ms MFCC blob.
tools/topology/topology2/include/bench/mfcc_controls_capture.conf Adds bench parameter key for mel40_10ms MFCC blob.
tools/topology/topology2/development/tplg-targets.cmake Adds development topology targets enabling MWW/KPB branches for HDA + SDW.
tools/topology/topology2/development/tplg-targets-bench.cmake Adds bench target/params for mel40_10ms MFCC benchmark.
tools/topology/topology2/cavs-sdw.conf Adds optional includes and keys for SDW MWW/KPB overlays + required class includes.
tools/topology/topology2/cavs-benchmark-hda.conf Adds benchmark configs for mfccmel40_10ms variants.
tools/rimage/config/wcl.toml.h Includes MWW module TOML when enabled.
tools/rimage/config/tgl.toml.h Includes MWW module TOML when enabled.
tools/rimage/config/tgl-h.toml.h Includes MWW module TOML when enabled.
tools/rimage/config/ptl.toml.h Includes MWW module TOML when enabled.
tools/rimage/config/mtl.toml.h Includes MWW module TOML when enabled.
tools/rimage/config/lnl.toml.h Includes MWW module TOML when enabled.
src/platform/intel/cavs/platform.c Initializes DP scheduler when enabled.
src/library_manager/llext_manager_dram.c Ensures VMA cleanup on restore failure.
src/lib/cpp_new_export.cpp Exports C++ allocation/runtime symbols for LLEXT modules.
src/lib/CMakeLists.txt Builds the new C++ symbol export shim when CONFIG_CPP is enabled.
src/lib/ams.c Adds inline payload data copy into AMS slots; exports ams_send().
src/ipc/ipc4/helper.c Adds optional DP→DP binding support and DP ring buffer attachment changes.
src/ipc/ipc4/ams_helpers.c Exports AMS helper functions for external users/modules.
src/include/sof/lib_manager.h Extends lib manager module struct with export segment + VMA tracking.
src/include/sof/audio/mfcc/mfcc_vad.h Tunes MFCC VAD constants (noise rise alpha + threshold).
src/audio/module_adapter/module_adapter.c Propagates DP domain from extended init; minor sync constant fix.
src/audio/microwakeword/tune/sof_mww_verify.py Adds streaming verification script for quantized MWW models.
src/audio/microwakeword/tune/sof_mww_train_pipeline.sh Adds end-to-end dataset→features→train→verify pipeline runner.
src/audio/microwakeword/tune/sof_mww_prepare_silence_unknown.sh Adds script to prepare silence/unknown classes from Speech Commands v2.
src/audio/microwakeword/tune/sof_mww_plot_mtrace.py Adds mtrace visualization tool for MWW diagnostics.
src/audio/microwakeword/tune/sof_mww_generate_keyword_dataset.sh Adds multi-speaker synthetic keyword dataset generator (has duplicated helper; see comment).
src/audio/microwakeword/tune/sof_mww_generate_keyword_dataset_from_dir.sh Adds real-speech ingestion + augmentation dataset builder.
src/audio/microwakeword/tune/sof_mww_dataset.py Adds feature loader + augmentation utilities for training.
src/audio/microwakeword/tune/sof_mfcc_extract_features.sh Adds batch feature extraction via sof-testbench4 + topology2 bench tplg.
src/audio/microwakeword/tune/README.md Documents offline MWW training/tuning toolchain and workflows.
src/audio/microwakeword/README.md Documents MWW component architecture, dataflow, and deployment notes.
src/audio/microwakeword/mww.toml Adds rimage module manifest entry for MWW.
src/audio/microwakeword/mww_model.h Adds C API surface for the MWW TFLM inference wrapper.
src/audio/microwakeword/mww_model.cc Implements TFLM interpreter setup + streaming inference wrapper.
src/audio/microwakeword/mww_model_data.h Adds generated model-data header declaration.
src/audio/microwakeword/llext/llext.toml.h Adds LLEXT-specific TOML wrapper for MWW module.
src/audio/microwakeword/llext/CMakeLists.txt Adds LLEXT build for the MWW module and its private TFLM library.
src/audio/microwakeword/llext-wrap.c Adds LLEXT portability stubs and PIC-safe math overrides.
src/audio/microwakeword/Kconfig Adds Kconfig options for MWW component and debug/model-loading modes.
src/audio/microwakeword/CMakeLists.txt Adds static + LLEXT build logic for MWW and its private deps.
src/audio/mfcc/tune/setup_mfcc.m Adds MFCC export profiles for mel40 (20ms) and mel40_10ms (+ compress variants).
src/audio/mfcc/mfcc_common.c Clarifies VAD input comment.
src/audio/Kconfig Includes microwakeword Kconfig in audio menu.
src/audio/CMakeLists.txt Adds microwakeword subdir to audio build when enabled.
src/audio/buffers/ring_buffer.c Adds vregion refcount release for DP→DP bind case.
src/audio/buffers/audio_buffer.c Adds DP→DP dual-secondary-buffer sync support and updates DP→DP commentary.
scripts/xtensa-build-zephyr.py Installs symlinks by UUID name for library binaries.
scripts/tensorflow-clone.sh Refactors dependency clone script to fetch + checkout pinned commits under workspace parent dir.
scripts/llext_offset_calc.py Handles ELF with no allocated sections by returning size 0.
app/llext_relocatable.conf Enables export-by-SLID for LLEXT relocatable builds.
app/boards/intel_adsp/Kconfig.defconfig Enables DP→DP bind by default.
app/boards/intel_adsp_cavs25.conf Enables MWW (static), increases heap sizes, and adds related settings for cavs2.5.
app/boards/intel_adsp_ace30_ptl.conf Enables MWW as LLEXT, adjusts library base address, heap sizing, and LLEXT heap.
Review details
  • Files reviewed: 86/88 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/audio/microwakeword/tune/sof_mww_prepare_silence_unknown.sh
Comment thread tools/topology/topology2/include/components/kpb.conf
Comment thread src/lib/ams.c
Comment thread src/audio/microwakeword/tune/sof_mww_generate_keyword_dataset.sh Outdated
When CONFIG_LLEXT_TYPE_ELF_RELOCATABLE is active, bypass appending
static address flags (-Ttext, --section-start, -Tdata) in the linker
helper script. This keeps section base addresses at 0.
Also adjust the offset calculator to avoid integer parsing errors
when all section addresses are set to 0.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
…table modules

Implement page-level virtual memory mapping using Zephyr's sys_bitarray
utility over the library region. Compile section layout at load-time to
allocate virtual addresses and rewrite section sh_addr headers in-place.
This enables Zephyr LLEXT to naturally relocate references.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Without this flag, calls between GLOBAL-visibility functions defined
in different translation units of the SAME llext module are emitted
as PLT calls. Zephyr's llext_link_plt() only resolves PLT symbols
against the base image's export table, this module's own
.exported_sym table, or other already-loaded extensions -- never
against symbols merely defined locally in this module's own .dynsym.

Multi-TU C++ libraries (e.g. TensorFlow Lite Micro) call plenty of
non-exported internal helpers across .cc files, so without this flag
those calls fail to link at load time.
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
LLEXT modules linking C++ code (e.g. TensorFlow Lite Micro) can end up
with undefined references to global operator new/delete and
__cxa_pure_virtual even when built with -fno-exceptions -- some
support code (e.g. TFLM's arena allocators) still emits calls to the
sized deallocation form in generated destructors, and any TU
referencing an abstract class's vtable needs __cxa_pure_virtual
resolvable for its pure-virtual slots.

zephyr/lib/cpp/minimal/cpp_new.cpp and cpp_virtual.c already define
these, but neither is referenced anywhere in the base image build, so
their definitions are never pulled into the link or exported. Add
thin wrappers in src/lib/cpp_new_export.cpp and export them under the
mangled names so LLEXT modules can resolve against the base image.
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The review found concrete correctness issues in new/changed runtime code paths (heap free mismatch in ring buffer cleanup, unsafe AMS memcpy_s handling, and incorrect syscall stub signatures/errno semantics) that should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 85/87 changed files
  • Comments generated: 4
  • Review effort level: Lite

Comment on lines +110 to +113
if (alloc && alloc->vreg) {
if (!vregion_put(alloc->vreg))
rfree(alloc);
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit is do-not-merge [DNM]. The actual DP-DP bind commit is #10562 .

Comment thread src/audio/microwakeword/llext-wrap.c Outdated
Comment thread src/lib/ams.c
Comment thread src/audio/microwakeword/tune/sof_mww_plot_mtrace.py Outdated
lrgirdwo and others added 6 commits September 4, 2026 10:27
Export ams_send(), ams_helper_register_producer(),
ams_helper_unregister_producer(), and ams_helper_prepare_payload() so
an LLEXT module can act as an AMS message producer (e.g. a
keyword-spotting component signaling KPB directly) without needing
these calls statically linked into the base image.
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Consume the lib_uuids dict already populated earlier in the script:
when a library's final .bin file did not yet exist at UUID-collection
time, its UUIDs were deferred into lib_uuids instead of being
symlinked immediately. install_lib() now walks lib_uuids[key] and
creates the deferred <uuid>.bin symlink/copy once the library is
actually installed.
Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Previously binding two DP (Data Processing) scheduled components was
rejected with IPC4_INVALID_REQUEST. Add support for DP-to-DP binding
by creating a dual ring buffer configuration where each DP module
gets its own ring buffer on either side of the intermediate
comp_buffer.

Cascade data through input ring_buffer -> comp_buffer -> output
ring_buffer with rate-limiting on the output side, and manage
DP vregion reference counting and memory contexts.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Call scheduler_dp_init() in platform_init() on cAVS platforms when
CONFIG_ZEPHYR_DP_SCHEDULER is enabled so that DP tasks (such as MFCC
and MWW in the Data Processing domain) can bind to the DP scheduler
without failing with -ENODEV (-19).

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…tend

Add 40-bin mel filterbank configurations (mel40, mel40_compress,
mel40_10ms, and mel40_10ms_compress) to setup_mfcc.m and generate
the corresponding topology blobs for microWakeWord streaming frontends.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Add microWakeWord (MWW) processing module for low-power keyword
spotting:
- Implement module adapter in mww.c with soft mel-log AGC, VAD gating,
  and KPB wake-on-voice notification.
- Implement TFLM bridge in mww_model.cc with MixConv operator resolver,
support for int8 dequantization, and streaming resource variable
resets.
- Add robust circular ring buffer boundary unwrap handling for MFCC
hops.
- Add 2-step debounce verification before triggering KPB drain.
- Add LLEXT wrapper and build system integration for both static and
  dynamic module targets.
- Include initial placeholder model interface in mww_model_data.h/cc.

Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

It introduces an unsafe free path in ring_buffer_free() that can corrupt module allocation lifetime (use-after-free/double-free risk).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

src/audio/buffers/ring_buffer.c:113

  • ring_buffer_free() must not free the module allocation context. alloc is owned by the module (allocated with sof_heap_alloc() in module_adapter_mem_alloc() and freed in module_adapter_mem_free()). Freeing it here via rfree() can cause use-after-free/double-free when the module later tears down, and rfree() is also not the matching deallocator for sof_heap_alloc().
	if (alloc && alloc->vreg) {
		if (!vregion_put(alloc->vreg))
			rfree(alloc);
	}
  • Files reviewed: 85/87 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread src/audio/microwakeword/tune/sof_mww_prepare_silence_unknown.sh Outdated
Update the embedded strawberry wake-word model data array with the
retrained causal streaming MixConv model.

Streaming verification report (threshold 0.65):
=================================================================
microWakeWord Streaming Verification Report (Threshold: 0.65)
=================================================================
  Class              Role         Files   Detected     Rate  Mean Peak
  -------------------------------------------------------------
  silence            Negative      2000          0     0.0%     0.000
  unknown            Negative     25000         21     0.1%     0.002
  strawberry         Positive      3000       2999   100.0%     0.996
  -------------------------------------------------------------
  Overall Wake Word Recall (True Positive Rate) :  99.97% (2999/3000)
  Overall False Alarm Rate (False Positive Rate):   0.08% (21/27000)
  Precision                                     :  99.30%
=================================================================

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…hain

Add an end-to-end offline training, quantization, and verification
toolchain for microWakeWord streaming models using SOF host testbench
MFCC features:
- sof_mfcc_extract_features.sh: Batch-extract real SOF 40-bin mel
  spectrogram features from WAV datasets via sof-testbench4.
- sof_mww_generate_keyword_dataset_piper_tts.sh: Synthesize keyword
  utterances across multiple Piper ONNX neural voices.
- sof_mww_prepare_silence_unknown.sh: Prepare ambient background and
  non-target speech datasets.
- sof_mww_dataset.py: Dataset loader with temporal jitter, silence pool
  background mixing, and hard negative fragment synthesis.
- sof_mww_train.py: Causal streaming MixConv model training with
  quantization-aware calibration and C-array / topology export.
- sof_mww_verify.py: Streaming temporal verification with configurable
  threshold and consecutive-detection debounce.
- sof_mww_train_pipeline.sh: One-shot automation pipeline for training,
  export, and streaming verification.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
…d pipelines

Add Topology2 configuration and pipeline graphs for microWakeWord (MWW)
Wake-on-Voice with KPB:
- Define MWW module component in mww.conf with default control definitions.
- Add HDA Mic capture pipeline (host-gateway-src-mfcc-mww-capture.conf)
  and host-gateway-micsel-mfcc-mww-capture.conf.
- Add SoundWire jack and DMIC MWW branches and DMIC MFCC profiles.
- Add HDA generic and SoundWire MTL/ARL topology targets with KPB and MWW.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Include audio/microwakeword/mww.toml when CONFIG_COMP_MWW is enabled
across platform rimage manifest headers (tgl, tgl-h, mtl, lnl, ptl, wcl)
so the MWW module UUID and entry are registered in base firmware images
and loadable on target devices.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>
Enable CONFIG_COMP_MWW_MODEL_FROM_CONTROL in the cAVS 2.5 board
configuration so that the microWakeWord model is loaded from runtime
bytes control instead of relying on the built-in static C array.

Signed-off-by: Seppo Ingalsuo <seppo.ingalsuo@linux.intel.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The DP-to-DP ring buffer teardown path currently risks incorrect/freeing of shared allocation context and AMS slot message pointer rebinding relies on an invalid cross-core pointer value.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

src/audio/buffers/ring_buffer.c:113

  • ring_buffer_free() drops the vregion refcount and then frees the shared mod_alloc_ctx with rfree(). This ctx is allocated with sof_heap_alloc() (see module_adapter_mem_alloc()) and is also freed via sof_heap_free() in other vregion_put() call sites (e.g. comp_buffer_free()), so using rfree() (and freeing it here at all) risks allocator mismatch or double-free when DP-to-DP bindings are torn down.
	if (alloc && alloc->vreg) {
		if (!vregion_put(alloc->vreg))
			rfree(alloc);
	}
  • Files reviewed: 86/87 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread src/lib/ams.c
Comment on lines 516 to +529
msg = shared_c->slots[slot].u.msg;
if (msg.message && msg.message_length > 0) {
if (msg.message_length <= sizeof(msg_buf)) {
if (memcpy_s(msg_buf, sizeof(msg_buf),
(__sparse_force void *)(shared_c->slots[slot].u.msg_raw + sizeof(msg)),
msg.message_length) != 0) {
ams_release(shared_c);
return -EINVAL;
}
msg.message = msg_buf;
} else {
msg.message = (__sparse_force uint8_t *)(shared_c->slots[slot].u.msg_raw + sizeof(msg));
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants