Give each conversation turn an owner
Start with the user-visible contract. Can a new message interrupt speech? Does closing a panel stop the request or only hide its presentation? Should a scene change preserve the conversation? These decisions belong to the product, not to whichever SDK happens to be integrated first.
I use a turn as a useful unit of ownership: it associates the request, accepted text, pending speech and presentation with one identity. The session can own durable conversation history while the turn owns temporary work. A late callback from an old turn must not update the current character or resume audio after the user has moved on.
A generation identifier is one possible safeguard. Check it before accepting an asynchronous result, alongside the owner's lifetime. This complements cancellation; it does not replace stopping work and releasing resources.
Treat text and audio as different streams
Text arriving in small increments does not imply that each increment is ready to speak. A chunk can end halfway through a word, sentence or presentation marker. Buffering for speech needs its own policy: a usable phrase boundary, bounded waiting time and a maximum pending amount.
Keep the displayed transcript separate from the speech queue. A transcript can already contain text whose audio has not played. Likewise, a provider can finish generating while playback still has buffered samples. Distinguish request completion, audio availability and audible completion instead of letting one boolean represent all three.
Define what happens when generation outruns playback: bounded buffering, backpressure where supported, or an explicit truncation policy. An unlimited queue turns a slow speaker into a memory and responsiveness problem.
Design cancellation before the happy path
For a generic museum-guide character, imagine the visitor asks another question while the current answer is playing. Stop accepting old results, cancel outstanding work where supported, stop playback, discard queued audio and release temporary presentation state. The exact sequence must follow the ownership contract; it should not depend on the order in which unrelated components receive callbacks.
Retries need the same care. Retrying a transport failure must not replay an already-spoken sentence or append the same reply twice. Record what was accepted separately from what was merely requested. Decide whether the UI offers retry, continues without voice or reports that the interaction failed.
Keep platform details behind meaningful boundaries
Microphone permission, browser audio activation, native speech libraries and headset suspension are different capabilities. Hide these differences behind focused interfaces for capture, transcription and playback rather than scattering platform conditions throughout conversation logic.
Unity's Web platform documentation is a starting point for browser constraints. Verify the exact Unity version and browsers you support; do not assume that an Editor implementation proves the Web build.
Provider credentials should remain on a trusted backend, not inside a downloadable Unity client. Model-generated text or action requests also remain untrusted input: validate supported actions and arguments before allowing them to affect gameplay. Keep that validation independent from prompt wording.
Test the interruptions users actually cause
Begin with controllable fake providers so failures are repeatable. Then validate the real providers and device builds. Useful acceptance cases include:
- A new turn begins while an old reply is still generating or speaking.
- The scene unloads while transcription or synthesis is pending.
- Microphone permission is denied or capture becomes unavailable.
- The network stalls after some text has already been accepted.
- An old callback arrives after cancellation or after a session restart.
- Voice fails but the text interaction remains usable.
- Closing and reopening the UI leaves no duplicate listeners or audio.
Measure time to first useful text and first audible speech separately, and trace state transitions without logging sensitive conversation content by default. The engineering goal is an interaction that remains understandable when services fail, not just a convincing demo on a clean connection.