Fix restore-path main↔bleQueue deadlock and courier drop amplification (#1425)

* Fix restore-path main↔bleQueue deadlock and courier drop amplification A device froze permanently in a two-phone test. Debugger stacks showed an ABBA deadlock: the main actor was in bleQueue.sync (delivery-ack send → broadcastPacket → readLinkState) while bleQueue was in main.sync (captureBluetoothStatus reading backgroundTimeRemaining). The load that lined the two edges up came from a courier-drop amplification storm: drop dedup was in-memory only while the outbox driving 120s re-deposits is persisted, so every relaunch republished the same undelivered DM as a fresh 24h relay drop and every gateway relaunch re-fetched the whole backlog — ~20 copies of one DM delivered in 40ms, each triggering decrypt + delivery + ack + handshake work. Fixes, in rank order: - Edge B (P0): captureBluetoothStatus no longer main.syncs from bleQueue; backgroundTimeRemaining is sampled on main and cached behind a lock. Invariant documented: bleQueue must NEVER sync-dispatch to main. - Edge A (P0, defense in depth): sendDeliveryAck / sendReadReceipt / sendPrivateMessage / sendNoisePayload / triggerHandshake hop to messageQueue like sendMessage, so no main-actor call path reaches readLinkState's bleQueue.sync. - Drop dedup (P1): publishedDropKeys and seenDropEventIDs persist across relaunches (new BridgeDropDedupStore, entries expire with the 24h NIP-40 drop window; wiped on panic) — one drop per message ID per 24h regardless of relaunch count. - Receiver dedup (P1): openCourierEnvelope dedups on the inner private message ID before delivery, so a duplicate copy costs one decrypt and never re-delivers, re-acks, or re-triggers a handshake. - Handshake gating (P2): queued acks initiate a Noise handshake only for reachable peers; mail from absent/rotated identities no longer turns each copy into a mesh-wide handshake flood (the ack stays queued and flushes when a session eventually establishes). - Outbox (P3): re-enqueueing a queued message ID carries over its depositedCourierKeys so resends stop re-burning the same courier slots. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Review fixes: offline-drop durability, gateway handoff retry, restore-log freshness, coalesced persist Adversarial review of the storm/deadlock PR surfaced four issues: - Offline blackhole (must-fix): a deposit made while relays were down persisted its dedup key even though the drop only sat in the in-memory pending queue — app killed before reconnect meant the relaunch lost the drop but the persisted key blocked every re-deposit for 24h. The persisted snapshot now excludes keys still pending; they become durable only when flushPendingDrops actually publishes them. - Gateway handoff: seen-event IDs were consumed before the deliverToPeer handoff; a failed handoff (peer walked away) permanently dropped the event for a single-gateway island. deliverToPeer now reports whether the handoff was attempted, and a failure releases the seen slot so a relaunch or backlog redelivery retries. - Restore-path logs: central/peripheral-restore captures logged the init sentinel bgRemaining=∞. The cache is now seeded in init's main-thread branch and restore captures route through the sampler, which refreshes the cached budget before logging. - Persist cost: the dedup record was a full JSON encode + atomic write on the main actor per mutation (once per event during a backlog re-fetch). Writes now coalesce behind a 1s window, flushed immediately on background/terminate; panic wipe stays immediate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Remove BoundedIDSet.remove, orphaned by the ExpiringIDSet migration The drop-dedup sets that needed slot release moved to ExpiringIDSet; remaining BoundedIDSet users only insert and check. Periphery caught it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: jack <jackjackbits@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 17:45:21 +00:00 · 2026-07-09 18:23:46 +02:00
parent 304460ee83
commit f8ab0a7dc0
13 changed files with 754 additions and 55 deletions
@@ -0,0 +1,137 @@
+//
+// BridgeDropDedupStore.swift
+// bitchat
+//
+// This is free and unencumbered software released into the public domain.
+// For more information, see <https://unlicense.org>
+//
+
+import BitLogger
+import Foundation
+
+/// ID set with per-entry timestamps: entries expire after `lifetime` and the
+/// oldest are evicted past `capacity`. The bridge's dedup caches use this
+/// instead of `BoundedIDSet` because their contents persist across relaunches
+/// and must age out with the 24h drop window they guard.
+struct ExpiringIDSet {
+    private(set) var entries: [String: Date]
+    let capacity: Int
+    let lifetime: TimeInterval
+
+    init(capacity: Int, lifetime: TimeInterval, entries: [String: Date] = [:], now: Date = Date()) {
+        self.capacity = capacity
+        self.lifetime = lifetime
+        self.entries = entries
+        prune(now: now)
+    }
+
+    func contains(_ id: String, now: Date) -> Bool {
+        guard let recorded = entries[id] else { return false }
+        return now.timeIntervalSince(recorded) <= lifetime
+    }
+
+    /// Returns false when the ID was already present (and unexpired).
+    @discardableResult
+    mutating func insert(_ id: String, now: Date) -> Bool {
+        guard !contains(id, now: now) else { return false }
+        entries[id] = now
+        prune(now: now)
+        return true
+    }
+
+    /// Releases a previously inserted ID so it can be re-added later (e.g. a
+    /// queued drop evicted before it ever published must become retryable).
+    mutating func remove(_ id: String) {
+        entries.removeValue(forKey: id)
+    }
+
+    private mutating func prune(now: Date) {
+        if entries.contains(where: { now.timeIntervalSince($0.value) > lifetime }) {
+            entries = entries.filter { now.timeIntervalSince($0.value) <= lifetime }
+        }
+        let overflow = entries.count - capacity
+        guard overflow > 0 else { return }
+        for (id, _) in entries.sorted(by: { $0.value < $1.value }).prefix(overflow) {
+            entries.removeValue(forKey: id)
+        }
+    }
+}
+
+/// Disk persistence for the bridge courier's drop-dedup record. Relays hold
+/// drops for the full 24h NIP-40 window and redeliver them on every launch,
+/// and the 120s outbox sweep re-deposits anything undelivered — so with
+/// in-memory-only dedup every relaunch republished the same message as a
+/// fresh drop (fresh throwaway seal, undeduplicatable downstream) and every
+/// gateway relaunch re-delivered the whole backlog. Field-verified: ~20
+/// copies of one DM delivered in 40ms fed the storm behind a permanent
+/// device freeze. Persisting both sides caps this at one drop per message
+/// ID per 24h regardless of relaunch count.
+///
+/// Contents are opaque IDs (message UUIDs, relay event IDs) — no plaintext,
+/// no peer identities — so until-first-unlock protection matches
+/// `NostrProcessedEventStore`, and the file must load during a
+/// locked-background restoration relaunch. Wiped on panic with the rest of
+/// the courier state.
+final class BridgeDropDedupStore {
+    struct Snapshot: Codable {
+        var publishedDropKeys: [String: Date]
+        var seenDropEventIDs: [String: Date]
+    }
+
+    private let fileURL: URL?
+
+    /// - Parameter fileURL: Overrides the on-disk location (tests). Ignored
+    ///   when `persistsToDisk` is false.
+    init(persistsToDisk: Bool = true, fileURL: URL? = nil) {
+        self.fileURL = persistsToDisk ? (fileURL ?? Self.defaultFileURL()) : nil
+    }
+
+    func load() -> Snapshot {
+        guard let fileURL,
+              let data = try? Data(contentsOf: fileURL),
+              let snapshot = try? JSONDecoder().decode(Snapshot.self, from: data) else {
+            return Snapshot(publishedDropKeys: [:], seenDropEventIDs: [:])
+        }
+        return snapshot
+    }
+
+    func save(_ snapshot: Snapshot) {
+        guard let fileURL else { return }
+        guard !(snapshot.publishedDropKeys.isEmpty && snapshot.seenDropEventIDs.isEmpty) else {
+            try? FileManager.default.removeItem(at: fileURL)
+            return
+        }
+        do {
+            try FileManager.default.createDirectory(
+                at: fileURL.deletingLastPathComponent(),
+                withIntermediateDirectories: true
+            )
+            let data = try JSONEncoder().encode(snapshot)
+            var options: Data.WritingOptions = [.atomic]
+            #if os(iOS)
+            options.insert(.completeFileProtectionUntilFirstUserAuthentication)
+            #endif
+            try data.write(to: fileURL, options: options)
+        } catch {
+            SecureLogger.error("Failed to persist bridge drop dedup record: \(error)", category: .session)
+        }
+    }
+
+    /// Panic wipe: forget which drops we published or handled.
+    func wipe() {
+        guard let fileURL else { return }
+        try? FileManager.default.removeItem(at: fileURL)
+    }
+
+    private static func defaultFileURL() -> URL? {
+        guard let base = try? FileManager.default.url(
+            for: .applicationSupportDirectory,
+            in: .userDomainMask,
+            appropriateFor: nil,
+            create: true
+        ) else { return nil }
+        return base
+            .appendingPathComponent("courier", isDirectory: true)
+            .appendingPathComponent("bridge-drop-dedup.json")
+    }
+}