Files
bitchat/bitchat/Services/Gateway/BridgeDropDedupStore.swift
T
f8ab0a7dc0 Fix restore-path main↔bleQueue deadlock and courier drop amplification (#1425)
* Fix restore-path main↔bleQueue deadlock and courier drop amplification

A device froze permanently in a two-phone test. Debugger stacks showed an
ABBA deadlock: the main actor was in bleQueue.sync (delivery-ack send →
broadcastPacket → readLinkState) while bleQueue was in main.sync
(captureBluetoothStatus reading backgroundTimeRemaining). The load that
lined the two edges up came from a courier-drop amplification storm:
drop dedup was in-memory only while the outbox driving 120s re-deposits
is persisted, so every relaunch republished the same undelivered DM as a
fresh 24h relay drop and every gateway relaunch re-fetched the whole
backlog — ~20 copies of one DM delivered in 40ms, each triggering
decrypt + delivery + ack + handshake work.

Fixes, in rank order:
- Edge B (P0): captureBluetoothStatus no longer main.syncs from bleQueue;
  backgroundTimeRemaining is sampled on main and cached behind a lock.
  Invariant documented: bleQueue must NEVER sync-dispatch to main.
- Edge A (P0, defense in depth): sendDeliveryAck / sendReadReceipt /
  sendPrivateMessage / sendNoisePayload / triggerHandshake hop to
  messageQueue like sendMessage, so no main-actor call path reaches
  readLinkState's bleQueue.sync.
- Drop dedup (P1): publishedDropKeys and seenDropEventIDs persist across
  relaunches (new BridgeDropDedupStore, entries expire with the 24h
  NIP-40 drop window; wiped on panic) — one drop per message ID per 24h
  regardless of relaunch count.
- Receiver dedup (P1): openCourierEnvelope dedups on the inner private
  message ID before delivery, so a duplicate copy costs one decrypt and
  never re-delivers, re-acks, or re-triggers a handshake.
- Handshake gating (P2): queued acks initiate a Noise handshake only for
  reachable peers; mail from absent/rotated identities no longer turns
  each copy into a mesh-wide handshake flood (the ack stays queued and
  flushes when a session eventually establishes).
- Outbox (P3): re-enqueueing a queued message ID carries over its
  depositedCourierKeys so resends stop re-burning the same courier slots.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Review fixes: offline-drop durability, gateway handoff retry, restore-log freshness, coalesced persist

Adversarial review of the storm/deadlock PR surfaced four issues:

- Offline blackhole (must-fix): a deposit made while relays were down
  persisted its dedup key even though the drop only sat in the in-memory
  pending queue — app killed before reconnect meant the relaunch lost the
  drop but the persisted key blocked every re-deposit for 24h. The
  persisted snapshot now excludes keys still pending; they become durable
  only when flushPendingDrops actually publishes them.
- Gateway handoff: seen-event IDs were consumed before the deliverToPeer
  handoff; a failed handoff (peer walked away) permanently dropped the
  event for a single-gateway island. deliverToPeer now reports whether
  the handoff was attempted, and a failure releases the seen slot so a
  relaunch or backlog redelivery retries.
- Restore-path logs: central/peripheral-restore captures logged the init
  sentinel bgRemaining=∞. The cache is now seeded in init's main-thread
  branch and restore captures route through the sampler, which refreshes
  the cached budget before logging.
- Persist cost: the dedup record was a full JSON encode + atomic write on
  the main actor per mutation (once per event during a backlog re-fetch).
  Writes now coalesce behind a 1s window, flushed immediately on
  background/terminate; panic wipe stays immediate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Remove BoundedIDSet.remove, orphaned by the ExpiringIDSet migration

The drop-dedup sets that needed slot release moved to ExpiringIDSet;
remaining BoundedIDSet users only insert and check. Periphery caught it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: jack <jackjackbits@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 18:23:46 +02:00

138 lines
5.2 KiB
Swift

//
// BridgeDropDedupStore.swift
// bitchat
//
// This is free and unencumbered software released into the public domain.
// For more information, see <https://unlicense.org>
//
import BitLogger
import Foundation
/// ID set with per-entry timestamps: entries expire after `lifetime` and the
/// oldest are evicted past `capacity`. The bridge's dedup caches use this
/// instead of `BoundedIDSet` because their contents persist across relaunches
/// and must age out with the 24h drop window they guard.
struct ExpiringIDSet {
private(set) var entries: [String: Date]
let capacity: Int
let lifetime: TimeInterval
init(capacity: Int, lifetime: TimeInterval, entries: [String: Date] = [:], now: Date = Date()) {
self.capacity = capacity
self.lifetime = lifetime
self.entries = entries
prune(now: now)
}
func contains(_ id: String, now: Date) -> Bool {
guard let recorded = entries[id] else { return false }
return now.timeIntervalSince(recorded) <= lifetime
}
/// Returns false when the ID was already present (and unexpired).
@discardableResult
mutating func insert(_ id: String, now: Date) -> Bool {
guard !contains(id, now: now) else { return false }
entries[id] = now
prune(now: now)
return true
}
/// Releases a previously inserted ID so it can be re-added later (e.g. a
/// queued drop evicted before it ever published must become retryable).
mutating func remove(_ id: String) {
entries.removeValue(forKey: id)
}
private mutating func prune(now: Date) {
if entries.contains(where: { now.timeIntervalSince($0.value) > lifetime }) {
entries = entries.filter { now.timeIntervalSince($0.value) <= lifetime }
}
let overflow = entries.count - capacity
guard overflow > 0 else { return }
for (id, _) in entries.sorted(by: { $0.value < $1.value }).prefix(overflow) {
entries.removeValue(forKey: id)
}
}
}
/// Disk persistence for the bridge courier's drop-dedup record. Relays hold
/// drops for the full 24h NIP-40 window and redeliver them on every launch,
/// and the 120s outbox sweep re-deposits anything undelivered — so with
/// in-memory-only dedup every relaunch republished the same message as a
/// fresh drop (fresh throwaway seal, undeduplicatable downstream) and every
/// gateway relaunch re-delivered the whole backlog. Field-verified: ~20
/// copies of one DM delivered in 40ms fed the storm behind a permanent
/// device freeze. Persisting both sides caps this at one drop per message
/// ID per 24h regardless of relaunch count.
///
/// Contents are opaque IDs (message UUIDs, relay event IDs) — no plaintext,
/// no peer identities — so until-first-unlock protection matches
/// `NostrProcessedEventStore`, and the file must load during a
/// locked-background restoration relaunch. Wiped on panic with the rest of
/// the courier state.
final class BridgeDropDedupStore {
struct Snapshot: Codable {
var publishedDropKeys: [String: Date]
var seenDropEventIDs: [String: Date]
}
private let fileURL: URL?
/// - Parameter fileURL: Overrides the on-disk location (tests). Ignored
/// when `persistsToDisk` is false.
init(persistsToDisk: Bool = true, fileURL: URL? = nil) {
self.fileURL = persistsToDisk ? (fileURL ?? Self.defaultFileURL()) : nil
}
func load() -> Snapshot {
guard let fileURL,
let data = try? Data(contentsOf: fileURL),
let snapshot = try? JSONDecoder().decode(Snapshot.self, from: data) else {
return Snapshot(publishedDropKeys: [:], seenDropEventIDs: [:])
}
return snapshot
}
func save(_ snapshot: Snapshot) {
guard let fileURL else { return }
guard !(snapshot.publishedDropKeys.isEmpty && snapshot.seenDropEventIDs.isEmpty) else {
try? FileManager.default.removeItem(at: fileURL)
return
}
do {
try FileManager.default.createDirectory(
at: fileURL.deletingLastPathComponent(),
withIntermediateDirectories: true
)
let data = try JSONEncoder().encode(snapshot)
var options: Data.WritingOptions = [.atomic]
#if os(iOS)
options.insert(.completeFileProtectionUntilFirstUserAuthentication)
#endif
try data.write(to: fileURL, options: options)
} catch {
SecureLogger.error("Failed to persist bridge drop dedup record: \(error)", category: .session)
}
}
/// Panic wipe: forget which drops we published or handled.
func wipe() {
guard let fileURL else { return }
try? FileManager.default.removeItem(at: fileURL)
}
private static func defaultFileURL() -> URL? {
guard let base = try? FileManager.default.url(
for: .applicationSupportDirectory,
in: .userDomainMask,
appropriateFor: nil,
create: true
) else { return nil }
return base
.appendingPathComponent("courier", isDirectory: true)
.appendingPathComponent("bridge-drop-dedup.json")
}
}