Fix restore-path main↔bleQueue deadlock and courier drop amplification (#1425)

* Fix restore-path main↔bleQueue deadlock and courier drop amplification A device froze permanently in a two-phone test. Debugger stacks showed an ABBA deadlock: the main actor was in bleQueue.sync (delivery-ack send → broadcastPacket → readLinkState) while bleQueue was in main.sync (captureBluetoothStatus reading backgroundTimeRemaining). The load that lined the two edges up came from a courier-drop amplification storm: drop dedup was in-memory only while the outbox driving 120s re-deposits is persisted, so every relaunch republished the same undelivered DM as a fresh 24h relay drop and every gateway relaunch re-fetched the whole backlog — ~20 copies of one DM delivered in 40ms, each triggering decrypt + delivery + ack + handshake work. Fixes, in rank order: - Edge B (P0): captureBluetoothStatus no longer main.syncs from bleQueue; backgroundTimeRemaining is sampled on main and cached behind a lock. Invariant documented: bleQueue must NEVER sync-dispatch to main. - Edge A (P0, defense in depth): sendDeliveryAck / sendReadReceipt / sendPrivateMessage / sendNoisePayload / triggerHandshake hop to messageQueue like sendMessage, so no main-actor call path reaches readLinkState's bleQueue.sync. - Drop dedup (P1): publishedDropKeys and seenDropEventIDs persist across relaunches (new BridgeDropDedupStore, entries expire with the 24h NIP-40 drop window; wiped on panic) — one drop per message ID per 24h regardless of relaunch count. - Receiver dedup (P1): openCourierEnvelope dedups on the inner private message ID before delivery, so a duplicate copy costs one decrypt and never re-delivers, re-acks, or re-triggers a handshake. - Handshake gating (P2): queued acks initiate a Noise handshake only for reachable peers; mail from absent/rotated identities no longer turns each copy into a mesh-wide handshake flood (the ack stays queued and flushes when a session eventually establishes). - Outbox (P3): re-enqueueing a queued message ID carries over its depositedCourierKeys so resends stop re-burning the same courier slots. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Review fixes: offline-drop durability, gateway handoff retry, restore-log freshness, coalesced persist Adversarial review of the storm/deadlock PR surfaced four issues: - Offline blackhole (must-fix): a deposit made while relays were down persisted its dedup key even though the drop only sat in the in-memory pending queue — app killed before reconnect meant the relaunch lost the drop but the persisted key blocked every re-deposit for 24h. The persisted snapshot now excludes keys still pending; they become durable only when flushPendingDrops actually publishes them. - Gateway handoff: seen-event IDs were consumed before the deliverToPeer handoff; a failed handoff (peer walked away) permanently dropped the event for a single-gateway island. deliverToPeer now reports whether the handoff was attempted, and a failure releases the seen slot so a relaunch or backlog redelivery retries. - Restore-path logs: central/peripheral-restore captures logged the init sentinel bgRemaining=∞. The cache is now seeded in init's main-thread branch and restore captures route through the sampler, which refreshes the cached budget before logging. - Persist cost: the dedup record was a full JSON encode + atomic write on the main actor per mutation (once per event during a backlog re-fetch). Writes now coalesce behind a 1s window, flushed immediately on background/terminate; panic wipe stays immediate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Remove BoundedIDSet.remove, orphaned by the ExpiringIDSet migration The drop-dedup sets that needed slot release moved to ExpiringIDSet; remaining BoundedIDSet users only insert and check. Periphery caught it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: jack <jackjackbits@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 21:05:20 +00:00 · 2026-07-09 18:23:46 +02:00
parent 304460ee83
commit f8ab0a7dc0
13 changed files with 754 additions and 55 deletions
@@ -24,6 +24,7 @@ struct BridgeCourierServiceTests {
        var localPeers: [(peerID: PeerID, noiseKey: Data)] = []
        var held: [CourierEnvelope] = []
        var sealResult: CourierEnvelope?
+        var deliverResult = true

        private(set) var publishedEvents: [NostrEvent] = []
        private(set) var openedSubscriptions: [[String]] = []
@@ -35,8 +36,8 @@ struct BridgeCourierServiceTests {

        let service: BridgeCourierService

-        init() {
-            service = BridgeCourierService()
+        init(dedupStore: BridgeDropDedupStore? = nil) {
+            service = BridgeCourierService(dedupStore: dedupStore)
            service.bridgeEnabled = { [weak self] in self?.bridgeOn ?? false }
            service.relaysConnected = { [weak self] in self?.relaysConnected ?? false }
            service.publishEvent = { [weak self] event in self?.publishedEvents.append(event) }
@@ -49,7 +50,10 @@ struct BridgeCourierServiceTests {
                return self?.sealResult
            }
            service.openEnvelope = { [weak self] envelope in self?.openedEnvelopes.append(envelope) }
-            service.deliverToPeer = { [weak self] envelope, peer in self?.delivered.append((envelope, peer)) }
+            service.deliverToPeer = { [weak self] envelope, peer in
+                self?.delivered.append((envelope, peer))
+                return self?.deliverResult ?? false
+            }
            service.heldEnvelopes = { [weak self] cooldown in
                self?.heldCooldowns.append(cooldown)
                return self?.held ?? []
@@ -175,6 +179,134 @@ struct BridgeCourierServiceTests {
        #expect(fixture.sealRequests.count == 1)
    }

+    @Test func publishedDropDedupSurvivesRelaunch() throws {
+        // Regression (field-verified amplification storm): the outbox that
+        // drives re-deposits is persisted, but the sender-side drop dedup was
+        // in-memory only — every relaunch republished the same undelivered
+        // message as a fresh drop, and relays hold each for 24h.
+        let fileURL = FileManager.default.temporaryDirectory
+            .appendingPathComponent("bridge-dedup-\(UUID().uuidString).json")
+        defer { try? FileManager.default.removeItem(at: fileURL) }
+        let recipientKey = Fixture.randomKey()
+        let messageID = UUID().uuidString
+
+        let fixture = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
+        fixture.sealResult = makeEnvelope(recipientKey: recipientKey)
+        #expect(fixture.service.depositDrop(content: "hello", messageID: messageID, recipientNoiseKey: recipientKey))
+        #expect(fixture.publishedEvents.count == 1)
+        // Persistence is coalesced; a real launch flushes within a second or
+        // on backgrounding — tests flush explicitly.
+        fixture.service.flushDedupSnapshot()
+
+        // "Relaunch": a fresh service over the same store must refuse to
+        // publish the same message ID again (before even re-sealing it).
+        let relaunched = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
+        relaunched.sealResult = makeEnvelope(recipientKey: recipientKey)
+        #expect(!relaunched.service.depositDrop(content: "hello", messageID: messageID, recipientNoiseKey: recipientKey))
+        #expect(relaunched.publishedEvents.isEmpty)
+        #expect(relaunched.sealRequests.isEmpty)
+    }
+
+    @Test func seenDropEventDedupSurvivesRelaunch() throws {
+        // Same storm, gateway side: relays redeliver the whole 24h drop
+        // backlog on every launch; a relaunch must not re-open (and re-ack)
+        // events it already handled.
+        let fileURL = FileManager.default.temporaryDirectory
+            .appendingPathComponent("bridge-dedup-\(UUID().uuidString).json")
+        defer { try? FileManager.default.removeItem(at: fileURL) }
+
+        let fixture = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
+        let myKey = try #require(fixture.myKey)
+        fixture.service.refresh()
+        let event = try makeDropEvent(for: makeEnvelope(recipientKey: myKey))
+        fixture.service.handleDropEvent(event)
+        #expect(fixture.openedEnvelopes.count == 1)
+        fixture.service.flushDedupSnapshot()
+
+        let relaunched = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
+        relaunched.myKey = myKey
+        relaunched.service.refresh()
+        relaunched.service.handleDropEvent(event)
+        #expect(relaunched.openedEnvelopes.isEmpty)
+    }
+
+    @Test func offlineQueuedDropStaysRedepositableAfterRelaunch() throws {
+        // A deposit made while relays are down only joins the in-memory
+        // pending queue. Its dedup key must NOT be durable yet: if the app is
+        // killed before relays connect, the relaunch loses the queued drop —
+        // a persisted key would then block every 120s re-deposit for 24h and
+        // the message would silently never reach a relay.
+        let fileURL = FileManager.default.temporaryDirectory
+            .appendingPathComponent("bridge-dedup-\(UUID().uuidString).json")
+        defer { try? FileManager.default.removeItem(at: fileURL) }
+        let recipientKey = Fixture.randomKey()
+        let messageID = UUID().uuidString
+
+        let fixture = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
+        fixture.relaysConnected = false
+        fixture.sealResult = makeEnvelope(recipientKey: recipientKey)
+        #expect(fixture.service.depositDrop(content: "later", messageID: messageID, recipientNoiseKey: recipientKey))
+        #expect(fixture.publishedEvents.isEmpty)
+        // Even a flush while the drop is still pending must exclude its key.
+        fixture.service.flushDedupSnapshot()
+
+        // "App killed before relays connected": pendingDrops were memory-only.
+        let relaunched = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
+        relaunched.sealResult = makeEnvelope(recipientKey: recipientKey)
+        #expect(relaunched.service.depositDrop(content: "later", messageID: messageID, recipientNoiseKey: recipientKey))
+        #expect(relaunched.publishedEvents.count == 1)
+    }
+
+    @Test func publishedPendingDropBecomesDurableAfterFlush() throws {
+        // Counterpart: once the queued drop actually publishes on reconnect,
+        // its key becomes durable and a relaunch must not republish.
+        let fileURL = FileManager.default.temporaryDirectory
+            .appendingPathComponent("bridge-dedup-\(UUID().uuidString).json")
+        defer { try? FileManager.default.removeItem(at: fileURL) }
+        let recipientKey = Fixture.randomKey()
+        let messageID = UUID().uuidString
+
+        let fixture = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
+        fixture.relaysConnected = false
+        fixture.sealResult = makeEnvelope(recipientKey: recipientKey)
+        #expect(fixture.service.depositDrop(content: "later", messageID: messageID, recipientNoiseKey: recipientKey))
+        fixture.relaysConnected = true
+        fixture.service.flushPendingDrops()
+        #expect(fixture.publishedEvents.count == 1)
+        fixture.service.flushDedupSnapshot()
+
+        let relaunched = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
+        relaunched.sealResult = makeEnvelope(recipientKey: recipientKey)
+        #expect(!relaunched.service.depositDrop(content: "later", messageID: messageID, recipientNoiseKey: recipientKey))
+        #expect(relaunched.publishedEvents.isEmpty)
+    }
+
+    @Test func failedGatewayHandoffReleasesSeenSlot() throws {
+        // A gateway's deliverToPeer handoff is best-effort: when it fails
+        // (the peer walked away between relay fetch and mesh send), the drop
+        // event must stay retryable — for a single-gateway mesh island this
+        // gateway is the recipient's only carrier.
+        let fixture = Fixture()
+        let peerKey = Fixture.randomKey()
+        let peer = PeerID(str: "aabbccdd00112233")
+        fixture.localPeers = [(peer, peerKey)]
+        fixture.service.refresh()
+        let event = try makeDropEvent(for: makeEnvelope(recipientKey: peerKey))
+
+        fixture.deliverResult = false
+        fixture.service.handleDropEvent(event)
+        #expect(fixture.delivered.count == 1)
+
+        // Redelivery (relaunch/backlog re-fetch) retries the handoff …
+        fixture.deliverResult = true
+        fixture.service.handleDropEvent(event)
+        #expect(fixture.delivered.count == 2)
+
+        // … and a successful handoff consumes the event for good.
+        fixture.service.handleDropEvent(event)
+        #expect(fixture.delivered.count == 2)
+    }
+
    @Test func distinctDropsUseDistinctThrowawayKeys() {
        let fixture = Fixture()
        let keyA = Fixture.randomKey()
@@ -0,0 +1,118 @@
+//
+// BridgeDropDedupStoreTests.swift
+// bitchat
+//
+// This is free and unencumbered software released into the public domain.
+// For more information, see <https://unlicense.org>
+//
+
+import Foundation
+import Testing
+@testable import bitchat
+
+@Suite("Bridge drop dedup persistence")
+struct BridgeDropDedupStoreTests {
+
+    // MARK: - ExpiringIDSet
+
+    @Test func entriesExpireAfterLifetime() {
+        let start = Date(timeIntervalSince1970: 1_700_000_000)
+        var set = ExpiringIDSet(capacity: 8, lifetime: 60)
+
+        let inserted = set.insert("a", now: start)
+        #expect(inserted)
+        #expect(set.contains("a", now: start))
+        let duplicate = set.insert("a", now: start.addingTimeInterval(30))
+        #expect(!duplicate)
+
+        // Past the lifetime the slot is free again.
+        let later = start.addingTimeInterval(61)
+        #expect(!set.contains("a", now: later))
+        let reinserted = set.insert("a", now: later)
+        #expect(reinserted)
+    }
+
+    @Test func capacityEvictsOldestFirst() {
+        let start = Date(timeIntervalSince1970: 1_700_000_000)
+        var set = ExpiringIDSet(capacity: 2, lifetime: 3600)
+
+        set.insert("oldest", now: start)
+        set.insert("middle", now: start.addingTimeInterval(1))
+        set.insert("newest", now: start.addingTimeInterval(2))
+
+        let check = start.addingTimeInterval(3)
+        #expect(!set.contains("oldest", now: check))
+        #expect(set.contains("middle", now: check))
+        #expect(set.contains("newest", now: check))
+    }
+
+    @Test func removeReleasesSlot() {
+        let now = Date()
+        var set = ExpiringIDSet(capacity: 8, lifetime: 3600)
+        set.insert("a", now: now)
+        set.remove("a")
+        #expect(!set.contains("a", now: now))
+        let reinserted = set.insert("a", now: now)
+        #expect(reinserted)
+    }
+
+    @Test func initPrunesExpiredPersistedEntries() {
+        let now = Date()
+        let set = ExpiringIDSet(
+            capacity: 8,
+            lifetime: 3600,
+            entries: [
+                "stale": now.addingTimeInterval(-7200),
+                "fresh": now.addingTimeInterval(-60),
+            ],
+            now: now
+        )
+        #expect(!set.contains("stale", now: now))
+        #expect(set.contains("fresh", now: now))
+    }
+
+    // MARK: - Store round trip
+
+    @Test func snapshotRoundTripsThroughDisk() {
+        let fileURL = FileManager.default.temporaryDirectory
+            .appendingPathComponent("bridge-dedup-store-\(UUID().uuidString).json")
+        defer { try? FileManager.default.removeItem(at: fileURL) }
+        let recorded = Date(timeIntervalSince1970: 1_700_000_000)
+
+        let store = BridgeDropDedupStore(fileURL: fileURL)
+        store.save(BridgeDropDedupStore.Snapshot(
+            publishedDropKeys: ["msg-1": recorded],
+            seenDropEventIDs: ["event-1": recorded]
+        ))
+
+        let reloaded = BridgeDropDedupStore(fileURL: fileURL).load()
+        #expect(reloaded.publishedDropKeys["msg-1"] == recorded)
+        #expect(reloaded.seenDropEventIDs["event-1"] == recorded)
+    }
+
+    @Test func wipeRemovesTheRecord() {
+        let fileURL = FileManager.default.temporaryDirectory
+            .appendingPathComponent("bridge-dedup-store-\(UUID().uuidString).json")
+        defer { try? FileManager.default.removeItem(at: fileURL) }
+
+        let store = BridgeDropDedupStore(fileURL: fileURL)
+        store.save(BridgeDropDedupStore.Snapshot(
+            publishedDropKeys: ["msg-1": Date()],
+            seenDropEventIDs: [:]
+        ))
+        store.wipe()
+
+        let reloaded = BridgeDropDedupStore(fileURL: fileURL).load()
+        #expect(reloaded.publishedDropKeys.isEmpty)
+        #expect(reloaded.seenDropEventIDs.isEmpty)
+    }
+
+    @Test func nonPersistingStoreStaysEmpty() {
+        let store = BridgeDropDedupStore(persistsToDisk: false)
+        store.save(BridgeDropDedupStore.Snapshot(
+            publishedDropKeys: ["msg-1": Date()],
+            seenDropEventIDs: [:]
+        ))
+        #expect(store.load().publishedDropKeys.isEmpty)
+    }
+}
@@ -446,6 +446,34 @@ struct MessageRouterTests {
        #expect(transport.sentCourierMessages.count == 1)
    }

+    @Test @MainActor
+    func enqueueReplacementCarriesOverDepositedCourierKeys() async {
+        // Re-sending a queued message ID replaces the outbox entry; the
+        // replacement must inherit which couriers already carry the message,
+        // or the deposit retry re-burns the same courier slots (duplicate
+        // sealed copies to the same peer).
+        let recipient = PeerID(str: "00000000000000aa")
+        let recipientKey = Data(repeating: 0xBB, count: 32)
+        let courier = PeerID(str: "00000000000000cc")
+        let courierKey = Data(repeating: 0xCC, count: 32)
+
+        let transport = MockTransport()
+        transport.connectedPeers.insert(courier)
+        transport.updatePeerSnapshots([Self.snapshot(courier, key: courierKey, verified: true)])
+
+        let router = MessageRouter(
+            transports: [transport],
+            courierDirectory: Self.directory(recipient: recipient, recipientKey: recipientKey)
+        )
+        router.sendPrivate("Hello", to: recipient, recipientNickname: "Peer", messageID: "ck1")
+        #expect(transport.sentCourierMessages.count == 1)
+
+        // Same message ID re-sent (e.g. a resend while still queued): the
+        // courier already carrying it must not receive a second copy.
+        router.sendPrivate("Hello", to: recipient, recipientNickname: "Peer", messageID: "ck1")
+        #expect(transport.sentCourierMessages.count == 1)
+    }
+
    @Test @MainActor
    func courierBecameAvailable_ignoresTheRecipientThemselves() async {
        let recipient = PeerID(str: "00000000000000aa")