Two Heads Are Better Than One: Async Knowledge Injection for Speech AI with Tandem Architecture
Two Heads Are Better Than One: Async Knowledge Injection for Speech AI with Tandem Architecture
So Kuroki
Yotaro Kubo
Takuya Akiba
Yujin Tang
Manato Yaguchi (intern)
The University of Tokyo
APRIL 2026
We’re excited to introduce KAME: Tandem Architecture for Enhancing Knowledge in Real-Time Speech-to-Speech Conversational AI.
KAME means turtle in Japanese.
- Paper (arxiv): https://arxiv.org/abs/2510.02327 (Accepted at ICASSP 2026)
- Inference code: https://github.com/SakanaAI/kame
- Finetune code: https://github.com/SakanaAI/kame_finetune
- Model: https://huggingface.co/SakanaAI/kame
Overview
A conversation with AI can typically be realized in two ways: Speech-to-Speech (S2S) [1, 2, 3] systems and cascaded systems that connect Speech-to-Text (STT), LLM, Text-to-Speech (TTS) modules [4]. S2S systems are responsive but tend to be less knowledgeable, while cascaded systems can suffer from some lag.
KAME connects an S2S model with a backend LLM to combine their strengths. The S2S model handles the fast response loop, while the backend LLM runs asynchronously on a slower cycle and injects “oracle” signals as they become available. More concretely, as the user’s speech grows, the backend LLM is repeatedly called on the expanding partial transcript to provide progressively refined response signals.
KAME enables the S2S model to start responding immediately, while the LLM runs in parallel to continuously reason about the response. This shifts the paradigm from “think then speak” to “speak while thinking.”
Comparison Between KAME and Moshi
KAME is compared with Moshi [1] to demonstrate its responsiveness and knowledge. Moshi itself is a remarkable system: building a full-duplex S2S model with near human-level response latency is a long-standing challenge, requiring the model to listen, think, and speak simultaneously under tight temporal constraints, and Moshi is a model whose architecture makes this possible.
Yet low latency creates an unavoidable trade-off. The faster a model must respond, the less time it has to think, which limits the knowledge it can deliver. KAME is designed to break this trade-off by letting the system “speak while thinking”: it allows a backend LLM to continue reasoning in parallel without sacrificing responsiveness, so the system can respond quickly and knowledgeably.
In these examples, KAME uses the Moshi-based S2S front-end and gpt-4.1 as the backend LLM. The question audio is generated by converting MT-Bench [5] text prompts into speech using TTS. Scores are obtained by transcribing each spoken answer and evaluating the transcription in text space with an LLM judge, following MT-Bench evaluation.
Reasoning
Question
David has three sisters. Each of them has one brother. How many brothers does David have?
KAME Response
David has three sisters. Each of his brothers has that brother. So all three sisters have three brothers, and each sister has one brother. That brother is David himself. So David is the only brother.
Moshi Response
Since each sister has one brother and David has three sisters, he has three brothers.
STEM
Question
In the field of quantum physics, what is superposition, and how does it relate to the phenomenon of quantum entanglement?
KAME Response
Superposition in quantum physics means that a particle, like an electron or a photon, can exist in more than one state at the same time.
Moshi Response
Superposition is a concept in quantum physics that mentions the act of quantum entanglement as a background process.
Humanities
Question
What are some business etiquette norms when doing business in Japan?
KAME Response
When doing business in Japan, there are several important etiquette norms to keep in mind. First, bow is a common way to greet and show respect. A handshake may also be used, but bowing is more traditional.
Moshi Response
One common etiquette norm in Japan is considering the other person’s status when making offers or expressions of gratitude.
Backend LLM Swapping
KAME allows easy swapping between backend LLMs such as gpt-4.1, claude-opus-4-1, and gemini-2.5-flash. This makes it possible to choose the most suitable model for each use case. In our examples, claude-opus-4-1 tends to score higher on reasoning tasks, while gpt-4.1 tends to score higher on humanities tasks.
Reasoning
Question
David has three sisters. Each of them has one brother. How many brothers does David have?
Claude Response
Let’s start by finding the total. David has a total of three sisters, and each of his sisters have one brother.
Gemini Response
Okay, David has zero brothers. He is the one brother to all three of his sisters.
Humanities
Question
Describe five key principles in evaluating an argument in analytical writing.
KAME Response
Certainly, here are five main principles to keep in mind when evaluating an argument in analytic writing.
Claude Response
I need to determine when an analytic considered the American analytical writing, all Prissions write.
Gemini Response
There are five key principles to consider. First, the rarity of the argument is clear.
Conclusion
KAME combines the responsiveness of a speech-to-speech system with the stronger knowledge access of a backend LLM. Across the examples above, it maintains low latency while delivering more informed responses than a standalone S2S baseline.
The tandem design also makes the backend flexible. By swapping the backend model based on the task, KAME can benefit from the strengths of different LLMs without changing the overall interaction pattern.
References
- Moshi: a speech-text foundation model for realtime dialogue Defossez, A., Mazare, L., Orsini, M., Royer, A., Perez, P., Jegou, H., Grave, E. and Zeghidour, N., 2024. KyutAI.
- LLaMA-Omni: seamless speech interaction with large language models Fang, Q., Guo, S., Zhou, Y., Ma, Z., Zhang, S. and Feng, Y., 2025. International Conference on Learning Representations (ICLR).
- SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y. and Qiu, X., 2023. Findings of the Association for Computational Linguistics: EMNLP, pp. 15757—15773.
- Unmute.sh by KyutAI link KyutAI, ., 2025. KyutAI.
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D. and Xing, E., 2023. Advances in Neural Information Processing Systems (NeurIPS), Vol 36, pp. 46595—46623.