Public-mesh push-to-talk: signed live voice bursts in the mesh channel (#1406)

* Live push-to-talk voice for DMs: stream while you talk, voice note as fallback

Holding the mic in a DM now streams AAC frames live over the Noise session
(walkie-talkie style, ~0.5s mouth-to-ear at one hop) while recording the same
audio as a normal voice note. On release the note ships through the existing
fileTransfer pipeline; receivers that heard the live stream absorb it silently
into the same bubble (matched by the burst ID embedded in the file name), so
reliability comes for free and nobody sees duplicates.

Protocol:
- NoisePayloadType.voiceFrame = 0x08 carrying VoiceBurstPacket
  (burstID + seq + START/data/END/CANCELED, length-prefixed AAC frames)
- 210-byte burst-content budget keeps each Noise packet inside the 256-byte
  padding bucket: one BLE frame, never the fragment scheduler
- fire-and-forget: frames are dropped (never queued) without an established
  session; live is only offered when the peer is mesh-reachable

Receive:
- ChatLiveVoiceCoordinator assembles bursts (jitter-ordered, 0.5s gap skip,
  3s idle end, flood/size caps), persists progressively as ADTS .aac so even
  a partial burst is a replayable bubble
- live autoplay only when the conversation is on screen, app active, and the
  new app-info "live voice messages" toggle is on (also gates live sending)
- one-playback-at-a-time via a shared ExclusivePlayback slot

Capture:
- PTTCaptureEngine taps AVAudioEngine, dual-encodes: live AAC frames + the
  finalized .m4a (same 16kHz/mono/16kbps settings as VoiceRecorder)
- VoiceRecordingViewModel now drives a pluggable VoiceCaptureSession; the
  composer HUD shows a pulsing LIVE treatment when streaming

Includes the push-to-talk design doc, 6 new localization keys across all 29
locales, and unit tests for framing, packetizer budget, ADTS output, codec
round-trip, and the assembly/absorb lifecycle. Public-mesh PTT (MessageType
0x29) lands separately on top of this.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Public-mesh push-to-talk: signed live voice bursts in the mesh channel

Extends live PTT from DMs to the public mesh timeline. Holding the mic in
the mesh channel now broadcasts the burst live as signed voiceFrame packets
(MessageType 0x29) while the finalized voice note still ships on release —
new clients hear you as you speak and absorb the note silently into the live
bubble; old clients (and late joiners) keep receiving the note exactly as
before, so mixed-version meshes lose nothing.

Wire/relay:
- MessageType.voiceFrame = 0x29: ephemeral signed broadcast, never
  gossip-synced (SyncTypeFlags maps it to no bit), never padded (padding to
  the 512 block would push every ~490-byte signed packet into fragmentation)
- RelayController treats voiceFrame like media fragments: dense-graph TTL
  clamp contains the sustained ~15 pkt/s per-talker stream, tight 8-25 ms
  jitter keeps multi-hop latency inside the receiver's 350 ms jitter buffer
- inbound gate mirrors public messages: broadcast-only, 30 s freshness cap,
  packet signature verified against the claimed sender's announce before any
  audio reaches the UI

App:
- ChatLiveVoiceCoordinator gains burst scopes: public bubbles land in the
  mesh timeline, autoplay only while that timeline is on screen, and the
  finalized-note absorb is scope-bound (a public note can't replace a DM
  burst or vice versa)
- floor courtesy: while someone talks live in the public channel the
  composer mic tints red and pulses, with an accessibility value naming the
  talker ("%@ is speaking", localized in all 29 locales); holding still
  works — a decentralized mesh has no floor arbiter, the tint just
  discourages talk-over

Tests: relay policy (sparse cap + dense clamp), public bubble + talker
indicator lifecycle, note absorption into the mesh store, and scope-binding
rejection; full suite green (1382 tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* PTT follow-ups from review + field test: peer-ID normalization, toggle gates inbound, drop-path diagnostics

Codex review fixes (#1403):
- makeVoiceCaptureSession normalizes the selected peer with toShort() before
  the reachability/session checks and binds the send target to that same
  routing ID — a conversation selected under the stable 64-hex Noise key no
  longer silently falls back to a classic note while the short-ID session is
  established
- the live-voice toggle now gates inbound bursts too: off means
  classic-notes-only in both directions (no live bubble, partial file, or
  early notification; the finalized note still arrives), with a test

Field-test diagnostics (first device run: DM frames decrypted but no bubble
appeared, with no log evidence of which guard dropped them):
- coordinator logs undecodable frames (size + hex prefix) and blocked drops
- makeAssembly logs directory/file-handle failures instead of returning nil
  silently
- PTTLiveVoiceSession logs capture start and finish (packet/frame/duration
  counts); PTTCaptureEngine logs engine start success/failure with the input
  format; BLEService.sendVoiceFrame logs no-session drops

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix iPhone live-capture failure: dead input unit (AURemoteIO -10851, 0 Hz)

Field testing showed the phone's live capture failing at mic enable with
AURemoteIO -10851 and an input format of 0 Hz / 2 ch — an input unit bound
to an earlier (playback-only or settling) audio session. The Mac, which has
no session lifecycle, captured fine, which is why public bursts from the Mac
worked while phone-side sends degraded from working (first hold) to sporadic
to dead across holds.

Three layers of defense:
- PTTCaptureEngine recreates its AVAudioEngine on every start(), after the
  session is configured, so the input unit binds to the session that is
  active now; a dead input (0 Hz or 0 channels) is now a distinct, logged
  error instead of a silent setup failure
- PTTLiveVoiceSession retries the capture start once after a 150 ms
  route-settle pause
- VoiceRecordingViewModel falls back to the classic VoiceRecorder within the
  same hold if the live engine still cannot start — a route glitch now costs
  the live stream, never the voice note

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Blue mic when the hold will stream live

The mic button now shows readiness at a glance — and doubles as a build
marker for device testing:
- blue: holding will stream live (DM peer reachable with an established
  Noise session, or the public mesh channel)
- accent (orange in DMs): holding records a classic voice note (no session
  yet, peer unreachable, or live voice toggled off)
- red states unchanged (recording, floor busy)

Refactors capture-backend selection into a single liveVoiceTarget() so the
indicator and makeVoiceCaptureSession can never disagree.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Revert the blue live-ready mic to the normal accent color

The build-verification marker did its job; idle mic color goes back to the
accent. The LIVE recording HUD remains the signal for whether a hold is
streaming. Keeps the liveVoiceTarget() refactor so backend selection stays
in one place.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Leave a trace on every mic press and every inbound-frame drop

Field testing read "tap does nothing" as breakage: the mic start is async
(permission check + engine spin-up), so releasing before recording begins
has always been a silent cancel — for classic voice notes too. Every press
now logs which backend it chose and, for quick presses, that it released
before recording started.

Also logs the two remaining silent drops: inbound voice frames rejected by
the live-voice toggle (the one unlogged guard left in the receive path) and
the classic-note fallback now includes the toggle state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix mic hold dying instantly in DMs: sheet swipe gesture starved the composer

Field logs showed every DM mic hold ending 3-10 ms after it began, on both
platforms, while public-channel holds worked — the private sheet wraps its
entire content (composer included) in a high-priority swipe-right-to-leave
DragGesture, and a high-priority ancestor drag cancels the mic button's
press-and-hold within milliseconds. Same starvation mechanism as the DM
image-reveal bug (#1402), hitting a drag instead of a tap.

The swipe-to-leave gesture now lives on the message list only, so the
composer's gestures (mic hold, text field, buttons) are out of its reach and
the swipe still works where users actually swipe.

Also stops touching the capture engine when a hold cancels before the engine
ever started: probing inputNode on a never-started engine instantiates its
input unit against whatever session is active and spams benign-but-alarming
AURemoteIO -10851 errors into field logs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Reorder app info sheet: usage first, then settings, then reference

New section order: HOW TO USE, then the adjustable bits (appearance, voice,
network), then the reference material (features, privacy, symbols legend).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* App info: flow HOW TO USE into one paragraph; "list" and "person" wording

The six how-to-use bullets now read as a single comma-separated paragraph
(same instruction strings, legacy bullet prefix stripped at render). Two
wording updates across all 29 locales: the people icon opens the "list"
(not "sidebar"), and you tap a "person's" name (not a "peer's") to start
a DM.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: jack <jackjackbits@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jack
2026-07-08 09:23:16 +02:00
committed by GitHub
co-authored by jack Claude Fable 5
parent eacd8f0750
commit 78a291ab77
21 changed files with 762 additions and 141 deletions
+131 -24
View File
@@ -11,19 +11,58 @@ import Foundation
protocol ChatLiveVoiceContext: AnyObject {
var nickname: String { get }
var selectedPrivateChatPeer: PeerID? { get }
/// Whether the public mesh timeline is what's on screen (autoplay gate
/// for public bursts).
var isViewingPublicMeshTimeline: Bool { get }
func isPeerBlocked(_ peerID: PeerID) -> Bool
func resolveNickname(for peerID: PeerID) -> String
/// Routes an inbound private message through the full pipeline
/// (store append, unread state, notification, read receipt).
func handlePrivateMessage(_ message: BitchatMessage)
/// Appends directly to the public mesh timeline, bypassing the batched
/// public pipeline: a live bubble must be removable when its burst is
/// canceled or empty, which a pipeline-buffered entry is not (it would
/// re-commit at the next flush).
func appendPublicMeshMessage(_ message: BitchatMessage)
/// Replace-or-append by message ID via the single-writer store intent.
func upsertPrivateMessage(_ message: BitchatMessage, in peerID: PeerID)
/// Replace-or-append by message ID in the public mesh timeline.
func upsertPublicMeshMessage(_ message: BitchatMessage)
@discardableResult
func removePrivateMessage(withID messageID: String) -> BitchatMessage?
/// Removes a message from whichever conversation holds it.
func removeMessage(withID messageID: String, cleanupFile: Bool)
/// Publishes who is currently talking live in the public mesh channel
/// (floor-courtesy indicator on the composer mic), nil when nobody is.
func setActivePublicVoiceTalker(_ nickname: String?)
func notifyUIChanged()
}
extension ChatViewModel: ChatLiveVoiceContext {}
extension ChatViewModel: ChatLiveVoiceContext {
var isViewingPublicMeshTimeline: Bool {
selectedPrivateChatPeer == nil && activeChannel == .mesh
}
func appendPublicMeshMessage(_ message: BitchatMessage) {
_ = appendPublicMessage(message, to: ConversationID(channelID: .mesh))
}
func upsertPublicMeshMessage(_ message: BitchatMessage) {
conversations.upsertByID(message, in: ConversationID(channelID: .mesh))
}
func setActivePublicVoiceTalker(_ nickname: String?) {
if activePublicVoiceTalker != nickname {
activePublicVoiceTalker = nickname
}
}
}
/// Where a live voice burst lives: a Noise DM or the public mesh timeline.
enum VoiceBurstScope: Equatable {
case directMessage
case publicMesh
}
/// Assembles inbound live push-to-talk bursts (`NoisePayloadType.voiceFrame`):
/// orders packets behind a jitter window, persists frames progressively as an
@@ -36,6 +75,8 @@ final class ChatLiveVoiceCoordinator {
private final class Assembly {
let burstID: Data
let peerID: PeerID
let scope: VoiceBurstScope
let nickname: String
let message: BitchatMessage
var messageID: String { message.id }
var messageTimestamp: Date { message.timestamp }
@@ -56,9 +97,11 @@ final class ChatLiveVoiceCoordinator {
var idleTimeout: Task<Void, Never>?
var gapRedrain: Task<Void, Never>?
init(burstID: Data, peerID: PeerID, message: BitchatMessage, fileURL: URL, fileHandle: FileHandle) {
init(burstID: Data, peerID: PeerID, scope: VoiceBurstScope, nickname: String, message: BitchatMessage, fileURL: URL, fileHandle: FileHandle) {
self.burstID = burstID
self.peerID = peerID
self.scope = scope
self.nickname = nickname
self.message = message
self.fileURL = fileURL
self.fileHandle = fileHandle
@@ -69,6 +112,7 @@ final class ChatLiveVoiceCoordinator {
private struct FinishedBurst {
let messageID: String
let peerID: PeerID
let scope: VoiceBurstScope
let fileURL: URL
let messageTimestamp: Date
let expiresAt: Date
@@ -88,11 +132,25 @@ final class ChatLiveVoiceCoordinator {
// MARK: - Inbound frames
/// Inbound DM burst packet (`NoisePayloadType.voiceFrame`).
func handleVoiceFramePayload(from peerID: PeerID, payload: Data, timestamp: Date) {
handle(payload, from: peerID, scope: .directMessage, nickname: context.resolveNickname(for: peerID), timestamp: timestamp)
}
/// Inbound public burst packet (`MessageType.voiceFrame`), already
/// signature-verified by the transport, which resolved the nickname.
func handlePublicVoiceFramePayload(from peerID: PeerID, nickname: String, payload: Data, timestamp: Date) {
handle(payload, from: peerID, scope: .publicMesh, nickname: nickname, timestamp: timestamp)
}
private func handle(_ payload: Data, from peerID: PeerID, scope: VoiceBurstScope, nickname: String, timestamp: Date) {
// Live voice off means classic-notes-only in both directions: no live
// bubble, no partial file, no early notification the finalized
// voice note still arrives through the normal pipeline.
guard PTTSettings.liveVoiceEnabled else { return }
guard PTTSettings.liveVoiceEnabled else {
SecureLogger.debug("PTT: dropping inbound voice frame — live voice is toggled off", category: .session)
return
}
guard let packet = VoiceBurstPacket.decode(payload) else {
SecureLogger.warning("PTT: undecodable voice frame from \(peerID.id.prefix(8))… (\(payload.count) bytes: \(payload.prefix(16).hexEncodedString())…)", category: .session)
return
@@ -103,23 +161,25 @@ final class ChatLiveVoiceCoordinator {
}
if let assembly = assemblies[packet.burstID] {
// The sender is Noise-authenticated; a different peer reusing the
// same burst ID is a collision or a replay drop it.
guard assembly.peerID == peerID else { return }
// The sender is authenticated (Noise session or packet
// signature); a different peer or scope reusing the same burst
// ID is a collision or a replay drop it.
guard assembly.peerID == peerID, assembly.scope == scope else { return }
apply(packet, to: assembly)
return
}
switch packet.kind {
case .start, .frames:
// A data packet with no prior START (lost or mid-burst state)
// A data packet with no prior START (lost or mid-burst join)
// still opens the assembly with the default codec.
guard assemblies.count < TransportConfig.pttMaxConcurrentAssemblies else {
SecureLogger.debug("PTT: dropping burst from \(peerID.id.prefix(8))… — assembly cap reached", category: .session)
return
}
guard let assembly = makeAssembly(burstID: packet.burstID, peerID: peerID, timestamp: timestamp) else { return }
guard let assembly = makeAssembly(burstID: packet.burstID, peerID: peerID, scope: scope, nickname: nickname, timestamp: timestamp) else { return }
assemblies[packet.burstID] = assembly
updatePublicTalkerIndicator()
apply(packet, to: assembly)
case .end, .canceled:
// Control packet for a burst we never saw nothing to do.
@@ -151,8 +211,9 @@ final class ChatLiveVoiceCoordinator {
pruneFinishedBursts()
guard let finished = finishedBursts[burstID] else { return false }
// Bind the note to the burst's authenticated sender.
// Bind the note to the burst's authenticated sender and scope.
guard message.senderPeerID == nil || message.senderPeerID == finished.peerID else { return false }
guard message.isPrivate == (finished.scope == .directMessage) else { return false }
let replacement = BitchatMessage(
id: finished.messageID,
@@ -161,13 +222,18 @@ final class ChatLiveVoiceCoordinator {
timestamp: finished.messageTimestamp,
isRelay: false,
originalSender: nil,
isPrivate: true,
recipientNickname: context.nickname,
isPrivate: finished.scope == .directMessage,
recipientNickname: finished.scope == .directMessage ? context.nickname : nil,
senderPeerID: finished.peerID,
mentions: nil,
deliveryStatus: message.deliveryStatus
)
context.upsertPrivateMessage(replacement, in: finished.peerID)
switch finished.scope {
case .directMessage:
context.upsertPrivateMessage(replacement, in: finished.peerID)
case .publicMesh:
context.upsertPublicMeshMessage(replacement)
}
// The complete .m4a replaces the partial live capture.
WaveformCache.shared.purge(url: finished.fileURL)
@@ -181,7 +247,7 @@ final class ChatLiveVoiceCoordinator {
// MARK: - Assembly lifecycle
private func makeAssembly(burstID: Data, peerID: PeerID, timestamp: Date) -> Assembly? {
private func makeAssembly(burstID: Data, peerID: PeerID, scope: VoiceBurstScope, nickname: String, timestamp: Date) -> Assembly? {
guard let fileURL = Self.makeIncomingURL(burstID: burstID) else {
SecureLogger.error("PTT: cannot resolve incoming media directory for burst \(burstID.hexEncodedString())", category: .session)
return nil
@@ -193,33 +259,46 @@ final class ChatLiveVoiceCoordinator {
return nil
}
let isPrivate = scope == .directMessage
let message = BitchatMessage(
sender: context.resolveNickname(for: peerID),
sender: nickname,
content: "\(MimeType.Category.audio.messagePrefix)\(fileURL.lastPathComponent)",
timestamp: timestamp,
isRelay: false,
originalSender: nil,
isPrivate: true,
recipientNickname: context.nickname,
isPrivate: isPrivate,
recipientNickname: isPrivate ? context.nickname : nil,
senderPeerID: peerID
)
let assembly = Assembly(
burstID: burstID,
peerID: peerID,
scope: scope,
nickname: nickname,
message: message,
fileURL: fileURL,
fileHandle: handle
)
// Full inbound pipeline: store append, unread, notification.
context.handlePrivateMessage(message)
// DM bubbles ride the full inbound pipeline (store append, unread,
// notification). Public bubbles append directly to the store: the
// batched public pipeline can't purge a buffered entry if the burst
// is canceled before the flush.
switch scope {
case .directMessage:
context.handlePrivateMessage(message)
case .publicMesh:
context.appendPublicMeshMessage(message)
}
// Live playback only when the user is looking at this conversation
// with the app frontmost and live voice enabled.
if PTTSettings.liveVoiceEnabled,
PTTSettings.isAppActive,
context.selectedPrivateChatPeer == peerID {
let isViewing = switch scope {
case .directMessage: context.selectedPrivateChatPeer == peerID
case .publicMesh: context.isViewingPublicMeshTimeline
}
if PTTSettings.liveVoiceEnabled, PTTSettings.isAppActive, isViewing {
assembly.player = PTTBurstPlayer()
}
@@ -227,6 +306,13 @@ final class ChatLiveVoiceCoordinator {
return assembly
}
/// Keeps the composer's floor-courtesy indicator pointing at whoever is
/// currently talking live in the public mesh channel.
private func updatePublicTalkerIndicator() {
let talker = assemblies.values.first { $0.scope == .publicMesh }?.nickname
context.setActivePublicVoiceTalker(talker)
}
private func apply(_ packet: VoiceBurstPacket, to assembly: Assembly) {
assembly.receivedBytes += packet.encode().count
let elapsed = Date().timeIntervalSince(assembly.firstPacketAt)
@@ -328,10 +414,11 @@ final class ChatLiveVoiceCoordinator {
try? assembly.fileHandle?.close()
assembly.fileHandle = nil
assemblies.removeValue(forKey: assembly.burstID)
updatePublicTalkerIndicator()
guard assembly.deliveredFrames > 0 else {
// Nothing audible ever arrived drop the empty bubble.
context.removePrivateMessage(withID: assembly.messageID)
removeBubble(of: assembly)
try? FileManager.default.removeItem(at: assembly.fileURL)
context.notifyUIChanged()
return
@@ -342,12 +429,13 @@ final class ChatLiveVoiceCoordinator {
WaveformCache.shared.purge(url: assembly.fileURL)
// Republish so the row re-renders without its LIVE treatment even if
// no finalized note ever arrives to swap in.
context.upsertPrivateMessage(assembly.message, in: assembly.peerID)
republishBubble(of: assembly)
pruneFinishedBursts()
finishedBursts[assembly.burstID] = FinishedBurst(
messageID: assembly.messageID,
peerID: assembly.peerID,
scope: assembly.scope,
fileURL: assembly.fileURL,
messageTimestamp: assembly.messageTimestamp,
expiresAt: Date().addingTimeInterval(TransportConfig.pttFinishedBurstRegistrySeconds)
@@ -356,6 +444,24 @@ final class ChatLiveVoiceCoordinator {
SecureLogger.debug("PTT: burst \(assembly.burstID.hexEncodedString()) finalized (\(assembly.deliveredFrames) frames)", category: .session)
}
private func removeBubble(of assembly: Assembly) {
switch assembly.scope {
case .directMessage:
context.removePrivateMessage(withID: assembly.messageID)
case .publicMesh:
context.removeMessage(withID: assembly.messageID, cleanupFile: false)
}
}
private func republishBubble(of assembly: Assembly) {
switch assembly.scope {
case .directMessage:
context.upsertPrivateMessage(assembly.message, in: assembly.peerID)
case .publicMesh:
context.upsertPublicMeshMessage(assembly.message)
}
}
private func cancelAssembly(_ assembly: Assembly) {
assembly.idleTimeout?.cancel()
assembly.gapRedrain?.cancel()
@@ -363,7 +469,8 @@ final class ChatLiveVoiceCoordinator {
try? assembly.fileHandle?.close()
assembly.fileHandle = nil
assemblies.removeValue(forKey: assembly.burstID)
context.removePrivateMessage(withID: assembly.messageID)
updatePublicTalkerIndicator()
removeBubble(of: assembly)
WaveformCache.shared.purge(url: assembly.fileURL)
try? FileManager.default.removeItem(at: assembly.fileURL)
context.notifyUIChanged()