feat(voice-support): Add voice recording and playback functionality - #692
Merged
haiphucnguyen merged 9 commits intoSep 3, 2026
Merged
Conversation
- Added `AudioPlayer.kt` and `AudioRecorder.kt` for handling audio playback and recording. - Integrated voice recording functionality into `ChatInputField.kt`. - Updated `MessageComponents.kt` to include voice message components. - Added voice settings section in `SettingsView.kt` and `VoiceSettingsSection.kt`. - Updated `AppConfig.kt` to include voice configuration settings. - Added voice-related translations to multiple i18n properties files. - Modified `Main.kt` to include voice settings in the main application view. Refs: askimo-ai#123
There was a problem hiding this comment.
🟡 Changes recommended
There are several confirmed functional issues (e.g., inability to clear the stored voice API key, thread-unsafe Compose state mutation from audio callbacks, and likely OpenAI TTS MP3 playback runtime failures) that should be addressed before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds end-to-end “Voice” support as an optional convenience layer on top of the existing text chat pipeline, including voice configuration, microphone capture + STT, and message TTS playback.
Changes:
- Introduces a
VoiceConfigblock inAppConfigwith secure OpenAI key storage and local endpoint options. - Adds Voice settings UI (new Settings sidebar section + configuration card) and i18n strings for multiple locales.
- Implements core desktop voice services: audio recording (WAV), STT/TTS service registry + implementations, and per-message voice playback UI + shortcut wiring.
File summaries
| File | Description |
|---|---|
| shared/src/main/kotlin/io/askimo/core/config/AppConfig.kt | Adds VoiceConfig/VoiceProvider, YAML sample, resolved vs raw config accessors, and update handling for voice fields |
| desktop/src/main/kotlin/io/askimo/desktop/settings/VoiceSettingsSection.kt | New Voice settings screen for enabling voice, selecting providers/models/endpoints, and managing the OpenAI voice key |
| desktop/src/main/kotlin/io/askimo/desktop/settings/SettingsView.kt | Adds VOICE section to Settings sidebar and renames SKILLS section to AGENTS |
| desktop/src/main/kotlin/io/askimo/desktop/Main.kt | Updates navigation to map prior skills-settings navigation to the AGENTS settings section |
| desktop-shared/src/main/resources/i18n/messages.properties | Adds Voice settings/chat/message voice strings and new shortcut label |
| desktop-shared/src/main/resources/i18n/messages_zh_TW.properties | Adds translations for voice strings |
| desktop-shared/src/main/resources/i18n/messages_zh_CN.properties | Adds translations for voice strings |
| desktop-shared/src/main/resources/i18n/messages_vi_VN.properties | Adds translations for voice strings |
| desktop-shared/src/main/resources/i18n/messages_pt_BR.properties | Adds translations for voice strings |
| desktop-shared/src/main/resources/i18n/messages_ko_KR.properties | Adds translations for voice strings |
| desktop-shared/src/main/resources/i18n/messages_ja_JP.properties | Adds translations for voice strings |
| desktop-shared/src/main/resources/i18n/messages_fr.properties | Adds translations for voice strings |
| desktop-shared/src/main/resources/i18n/messages_es.properties | Adds translations for voice strings |
| desktop-shared/src/main/resources/i18n/messages_de.properties | Adds translations for voice strings |
| desktop-shared/src/main/kotlin/io/askimo/ui/voice/VoiceServices.kt | Adds voice service interfaces, audio format enum, and a registry for provider-based service resolution |
| desktop-shared/src/main/kotlin/io/askimo/ui/voice/impl/TextToSpeechServices.kt | Adds OpenAI and Piper TTS implementations + factories |
| desktop-shared/src/main/kotlin/io/askimo/ui/voice/impl/SpeechToTextServices.kt | Adds OpenAI and local whisper.cpp STT implementations + factories |
| desktop-shared/src/main/kotlin/io/askimo/ui/voice/AudioRecorder.kt | Adds microphone capture + WAV encoding via javax.sound.sampled |
| desktop-shared/src/main/kotlin/io/askimo/ui/voice/AudioPlayer.kt | Adds basic audio playback with pause/resume/stop and single-player coordination helpers |
| desktop-shared/src/main/kotlin/io/askimo/ui/common/keymap/KeyMapManager.kt | Adds Cmd/Ctrl+Shift+M shortcut for toggling voice recording |
| desktop-shared/src/main/kotlin/io/askimo/ui/chat/MessageComponents.kt | Adds per-AI-message 🔊 TTS playback UI and a shared controller for single-playback behavior |
| desktop-shared/src/main/kotlin/io/askimo/ui/chat/ChatInputField.kt | Adds 🎤 recording/transcribe button + keyboard shortcut handler and inserts transcript into the input field |
Review details
- Files reviewed: 22/22 changed files
- Comments generated: 8
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+489
to
+511
| scope.launch { | ||
| val wavBytes = withContext(Dispatchers.IO) { audioRecorder.stop() } | ||
| try { | ||
| val transcript = withContext(Dispatchers.IO) { | ||
| VoiceServiceRegistry.speechToText(AppConfig.voice) | ||
| .transcribe(wavBytes, VoiceAudioFormat.WAV) | ||
| } | ||
| if (transcript.isNotBlank()) { | ||
| val newText = if (inputText.text.isBlank()) { | ||
| transcript | ||
| } else { | ||
| "${inputText.text} $transcript" | ||
| } | ||
| onInputTextChange( | ||
| TextFieldValue(text = newText, selection = TextRange(newText.length)), | ||
| ) | ||
| } | ||
| } catch (e: VoiceServiceException) { | ||
| EventBus.post(AppErrorEvent(title = voiceErrorTitle, message = e.message ?: "Voice transcription failed")) | ||
| } finally { | ||
| voiceRecordingState = VoiceRecordingState.IDLE | ||
| } | ||
| } |
Comment on lines
+176
to
+184
| val ttsService = withContext(Dispatchers.IO) { VoiceServiceRegistry.textToSpeech(AppConfig.voice) } | ||
| val audioBytes = withContext(Dispatchers.IO) { ttsService.synthesize(text) } | ||
| // A newer toggle may have superseded this request while we were synthesizing. | ||
| if (loadingMessageId != messageId) return@launch | ||
| loadingMessageId = null | ||
| playingMessageId = messageId | ||
| player.play(audioBytes, ttsService.outputFormat) { | ||
| if (playingMessageId == messageId) playingMessageId = null | ||
| } |
Comment on lines
+48
to
+69
| val rawStream = AudioSystem.getAudioInputStream(ByteArrayInputStream(audio)) | ||
| val decodedFormat = AudioFormat( | ||
| AudioFormat.Encoding.PCM_SIGNED, | ||
| rawStream.format.sampleRate, | ||
| 16, | ||
| rawStream.format.channels, | ||
| rawStream.format.channels * 2, | ||
| rawStream.format.sampleRate, | ||
| false, | ||
| ) | ||
| val decodedStream = AudioSystem.getAudioInputStream(decodedFormat, rawStream) | ||
|
|
||
| val newClip = AudioSystem.getClip() | ||
| newClip.open(decodedStream) | ||
| newClip.addLineListener { event -> | ||
| if (event.type == LineEvent.Type.STOP && newClip.framePosition >= newClip.frameLength) { | ||
| onFinished?.invoke() | ||
| } | ||
| } | ||
| onFinished = onComplete | ||
| clip = newClip | ||
| newClip.start() |
| * Safe to call even if [start] was never called (returns an empty WAV). | ||
| */ | ||
| fun stop(): ByteArray { | ||
| val targetLine = line ?: return ByteArray(0) |
Comment on lines
+26
to
+35
| class OpenAiTextToSpeechService(private val config: VoiceConfig) : TextToSpeechService { | ||
| private val log = logger<OpenAiTextToSpeechService>() | ||
|
|
||
| override val outputFormat: VoiceAudioFormat = VoiceAudioFormat.MP3 | ||
|
|
||
| override suspend fun synthesize(text: String, speed: Double): ByteArray = withContext(Dispatchers.IO) { | ||
| val apiKey = config.openAiApiKey | ||
| if (apiKey.isBlank()) { | ||
| throw VoiceServiceException("OpenAI API key for voice is not configured. Set it in Settings > Voice.") | ||
| } |
Comment on lines
+1324
to
+1342
| "openAiApiKey" -> { | ||
| val key = value as String | ||
| if (VoiceConfig.isActualKey(key)) { | ||
| val result = VoiceConfig.setSecureOpenAiKey(key) | ||
| when (result.method) { | ||
| StorageMethod.KEYCHAIN -> | ||
| log.debug("Voice OpenAI API key stored securely in keychain") | ||
|
|
||
| StorageMethod.ENCRYPTED -> | ||
| log.warn("Voice OpenAI API key stored with encryption ({})", result.warningMessage) | ||
|
|
||
| StorageMethod.INSECURE_FALLBACK -> | ||
| log.warn("⚠️ Voice OpenAI API key storage: {}", result.warningMessage) | ||
| } | ||
| config.copy(openAiApiKey = VoiceConfig.getKeyPlaceholder()) | ||
| } else { | ||
| config.copy(openAiApiKey = key) | ||
| } | ||
| } |
| ) | ||
| Icon( | ||
| Icons.Default.Edit, | ||
| contentDescription = "Change provider", |
| * [io.askimo.core.providers.ProviderInstance] key — a user may chat exclusively with | ||
| * another provider (Anthropic, Gemini, Ollama...) yet still want OpenAI Whisper/TTS for voice. | ||
| * [useProviderKeyForVoice] offers a convenience toggle so the Settings UI can optionally | ||
| * reuse an existing OpENAI provider instance's key instead of requiring the user to paste it again. |
minhnguyen-ai
force-pushed
the
feature/voice-support
branch
from
September 3, 2026 17:33
ea8ed78 to
bad1ec6
Compare
There was a problem hiding this comment.
🟡 Changes recommended
It introduces several user-impacting correctness/runtime risks (keychain key deletion race on settings open, UI-thread keychain access, cross-thread Compose state mutation, and likely missing MP3 decoding support) that should be addressed before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
desktop/src/main/kotlin/io/askimo/desktop/settings/VoiceSettingsSection.kt:326
findExistingOpenAiProviderKey()performs keychain access viaSecureKeyManager.retrieveSecretKey(...). Calling it directly inside the buttononClickruns on the UI thread and can cause noticeable UI stalls (and on some platforms can block on OS dialogs).
onClick = {
val existingKey = findExistingOpenAiProviderKey()
if (existingKey != null) {
openAiApiKey = existingKey
reuseKeyStatus = "success"
desktop-shared/src/main/kotlin/io/askimo/ui/voice/impl/TextToSpeechServices.kt:30
- This implementation hard-codes OpenAI TTS output as MP3 (
outputFormat = MP3), but the desktop app does not currently declare an MP3javax.sound.sampledSPI decoder dependency (e.g., mp3spi/jlayer). As a result,AudioPlayerwill likely throwUnsupportedAudioFileExceptionat runtime and voice playback won’t work out of the box; either request WAV/PCM output from the TTS backend or bundle an MP3 decoder.
class OpenAiTextToSpeechService(private val config: VoiceConfig) : TextToSpeechService {
private val log = logger<OpenAiTextToSpeechService>()
override val outputFormat: VoiceAudioFormat = VoiceAudioFormat.MP3
- Files reviewed: 22/22 changed files
- Comments generated: 3
- Review effort level: Lite
Comment on lines
+125
to
+142
| // API key loaded async from keychain — starts blank, same pattern as web search / proxy. | ||
| var openAiApiKey by remember { mutableStateOf("") } | ||
| var sttProviderDropdownExpanded by remember { mutableStateOf(false) } | ||
| var ttsProviderDropdownExpanded by remember { mutableStateOf(false) } | ||
| var reuseKeyStatus by remember { mutableStateOf<String?>(null) } | ||
|
|
||
| val showApiKeyField = sttProvider == VoiceProvider.OPENAI || ttsProvider == VoiceProvider.OPENAI | ||
|
|
||
| LaunchedEffect(Unit) { | ||
| val resolved = withContext(Dispatchers.IO) { AppConfig.voice } | ||
| openAiApiKey = if (VoiceConfig.isActualKey(resolved.openAiApiKey)) resolved.openAiApiKey else "" | ||
| } | ||
|
|
||
| // ── Debounced saves for typed fields (keychain I/O for the API key — must NOT block the UI) ── | ||
| LaunchedEffect(openAiApiKey) { | ||
| delay(500.milliseconds) | ||
| withContext(Dispatchers.IO) { AppConfig.updateField("voice.openAiApiKey", openAiApiKey) } | ||
| } |
Comment on lines
+1268
to
1271
| key.isEmpty() -> { | ||
| WebSearchConfig.setSecureBraveKey("") | ||
| config.copy(braveApiKey = "") | ||
| } |
Comment on lines
+526
to
+529
| voiceWaveformSamples.add(level) | ||
| if (voiceWaveformSamples.size > maxWaveformSamples) { | ||
| voiceWaveformSamples.removeAt(0) | ||
| } |
There was a problem hiding this comment.
🟡 Changes recommended
There are confirmed state/concurrency bugs in the new voice UI flows (key persistence and cross-message playback state) that can cause user-visible breakage or data loss.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
desktop/src/main/kotlin/io/askimo/desktop/settings/VoiceSettingsSection.kt:140
- The debounced
LaunchedEffect(openAiApiKey)runs once immediately with the initial blank value and callsAppConfig.updateField("voice.openAiApiKey", ""), which can clear an existing keychain-stored voice OpenAI key before the async keychain load finishes (data loss just from opening the settings screen). Gate the save effect until the initial key load completes.
LaunchedEffect(openAiApiKey) {
delay(500.milliseconds)
withContext(Dispatchers.IO) { AppConfig.updateField("voice.openAiApiKey", openAiApiKey) }
}
desktop-shared/src/main/kotlin/io/askimo/ui/chat/ChatInputField.kt:574
audioRecorder.start { ... }invokes the level callback on the capture thread, but it directly mutatesvoiceWaveformSamples(Compose snapshot state). Writing snapshot state off the UI thread can throw snapshot concurrency exceptions and cause flaky UI updates. Wrap the state mutation inSnapshot.withMutableSnapshot(or dispatch to the UI coroutine context).
audioRecorder.start { level ->
// Invoked from the capture thread — mutating Compose state directly is
// safe/expected here (see ChatViewModel.editMessage for the same pattern),
// and this is a high-frequency, low-cost UI signal, not audio data.
voiceWaveformSamples.add(level)
- Files reviewed: 30/30 changed files
- Comments generated: 1
- Review effort level: Lite
Comment on lines
+190
to
+196
| } catch (e: VoiceServiceException) { | ||
| loadingMessageId = null | ||
| onError(e.message ?: "Voice playback failed") | ||
| } catch (e: AudioPlaybackException) { | ||
| loadingMessageId = null | ||
| onError(e.message ?: "Voice playback failed") | ||
| } |
There was a problem hiding this comment.
🔵 Needs a closer look
Two user-facing correctness/resource issues were identified in the new voice UI/playback paths (stored as PR comments with concrete fixes).
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
desktop-shared/src/main/kotlin/io/askimo/ui/chat/ChatInputField.kt:545
- The transcript is appended using the
inputTextvalue captured when the coroutine was launched; if the user edits the input while transcription is in flight, those newer edits can be overwritten whenonInputTextChangeruns. UserememberUpdatedState(inputText)(or similar) to read the latest text at the moment you apply the transcript.
desktop-shared/src/main/kotlin/io/askimo/ui/voice/AudioPlayer.kt:81 - On natural completion you invoke
onFinished, but the completedClipremains open andclip/onFinishedstay set until the next explicitstop()/play(). This can unnecessarily hold the audio line and leak native resources after playback finishes. Consider closing/clearing the clip on completion (reusingstop()after capturing the callback).
- Files reviewed: 30/30 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
There are concurrency/accessibility issues in newly introduced UI code (cross-thread Compose state mutation in voice waveform updates and a hard-coded contentDescription) that should be corrected before approval.
Review details
Suppressed comments (2)
desktop-shared/src/main/kotlin/io/askimo/ui/chat/ChatInputField.kt:582
voiceWaveformSamples(a Compose SnapshotStateList) is mutated from the audio capture thread. Other background-thread state updates in this codebase are wrapped inSnapshot.withMutableSnapshotto ensure snapshot consistency/atomic updates; doing raw writes here can lead to snapshot contention or runtime snapshot errors under concurrency.
audioRecorder.start { level ->
// Invoked from the capture thread — mutating Compose state directly is
// safe/expected here (see ChatViewModel.editMessage for the same pattern),
// and this is a high-frequency, low-cost UI signal, not audio data.
voiceWaveformSamples.add(level)
desktop/src/main/kotlin/io/askimo/desktop/settings/VoiceSettingsSection.kt:415
- The provider selector's edit icon uses a hard-coded English
contentDescription(not localized) even though it appears purely decorative next to the already-labeled provider name. This is inconsistent with the rest of the UI’s stringResource usage and can produce noisy screen-reader output.
Icon(
Icons.Default.Edit,
contentDescription = "Change provider",
tint = AppTextStyles.primaryContent,
)
- Files reviewed: 30/30 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
There are UI-thread lifecycle/performance issues in the new voice recording and playback integrations that can cause blocking/jank or leave microphone resources open when the composable is disposed.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (3)
Previously missed (2) — in code that hasn't changed since the last review.
desktop-shared/src/main/kotlin/io/askimo/ui/chat/MessageComponents.kt:189
- player.play() performs decoding and opens a javax.sound.sampled.Clip; running it on the UI coroutine dispatcher can cause noticeable jank. Run the playback start on Dispatchers.IO (similar to synthesis) so UI stays responsive.
desktop/src/main/kotlin/io/askimo/desktop/Main.kt:1233 - The callback name
onNavigateToSkillsSettingsnow navigates toSettingsSection.AGENTS, which makes the call site misleading and harder to maintain. Consider renaming the callback (and related View/section naming) to match the newAGENTSterminology.
desktop/src/main/kotlin/io/askimo/desktop/settings/VoiceSettingsSection.kt:415
- The provider selector's edit icon uses a hardcoded English contentDescription (read by screen readers). If the icon is purely decorative, set contentDescription = null; otherwise route it through stringResource so it can be localized.
Icon(
Icons.Default.Edit,
contentDescription = "Change provider",
tint = AppTextStyles.primaryContent,
)
- Files reviewed: 30/30 changed files
- Comments generated: 1
- Review effort level: Lite
Comment on lines
+421
to
+427
| DisposableEffect(Unit) { | ||
| onDispose { | ||
| if (voiceRecordingState == VoiceRecordingState.RECORDING) { | ||
| audioRecorder.cancel() | ||
| Snapshot.withMutableSnapshot { voiceWaveformSamples.clear() } | ||
| } | ||
| } |
haiphucnguyen
approved these changes
Sep 3, 2026
haiphucnguyen
left a comment
haiphucnguyen
left a comment
Collaborator
There was a problem hiding this comment.
Thank you @minhnguyen-ai for this complex issue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AudioPlayer.ktandAudioRecorder.ktfor handling audio playback and recording.ChatInputField.kt.MessageComponents.ktto include voice message components.SettingsView.ktandVoiceSettingsSection.kt.AppConfig.ktto include voice configuration settings.Main.ktto include voice settings in the main application view.Refs: #123