permissionlesstech/bitchat
1
0
Fork 0
mirror of https://github.com/permissionlesstech/bitchat.git synced 2026-07-26 04:05:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

Fix restore-path main↔bleQueue deadlock and courier drop amplification (#1425)

Browse Source 
* Fix restore-path main↔bleQueue deadlock and courier drop amplification

A device froze permanently in a two-phone test. Debugger stacks showed an
ABBA deadlock: the main actor was in bleQueue.sync (delivery-ack send →
broadcastPacket → readLinkState) while bleQueue was in main.sync
(captureBluetoothStatus reading backgroundTimeRemaining). The load that
lined the two edges up came from a courier-drop amplification storm:
drop dedup was in-memory only while the outbox driving 120s re-deposits
is persisted, so every relaunch republished the same undelivered DM as a
fresh 24h relay drop and every gateway relaunch re-fetched the whole
backlog — ~20 copies of one DM delivered in 40ms, each triggering
decrypt + delivery + ack + handshake work.

Fixes, in rank order:
- Edge B (P0): captureBluetoothStatus no longer main.syncs from bleQueue;
  backgroundTimeRemaining is sampled on main and cached behind a lock.
  Invariant documented: bleQueue must NEVER sync-dispatch to main.
- Edge A (P0, defense in depth): sendDeliveryAck / sendReadReceipt /
  sendPrivateMessage / sendNoisePayload / triggerHandshake hop to
  messageQueue like sendMessage, so no main-actor call path reaches
  readLinkState's bleQueue.sync.
- Drop dedup (P1): publishedDropKeys and seenDropEventIDs persist across
  relaunches (new BridgeDropDedupStore, entries expire with the 24h
  NIP-40 drop window; wiped on panic) — one drop per message ID per 24h
  regardless of relaunch count.
- Receiver dedup (P1): openCourierEnvelope dedups on the inner private
  message ID before delivery, so a duplicate copy costs one decrypt and
  never re-delivers, re-acks, or re-triggers a handshake.
- Handshake gating (P2): queued acks initiate a Noise handshake only for
  reachable peers; mail from absent/rotated identities no longer turns
  each copy into a mesh-wide handshake flood (the ack stays queued and
  flushes when a session eventually establishes).
- Outbox (P3): re-enqueueing a queued message ID carries over its
  depositedCourierKeys so resends stop re-burning the same courier slots.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Review fixes: offline-drop durability, gateway handoff retry, restore-log freshness, coalesced persist

Adversarial review of the storm/deadlock PR surfaced four issues:

- Offline blackhole (must-fix): a deposit made while relays were down
  persisted its dedup key even though the drop only sat in the in-memory
  pending queue — app killed before reconnect meant the relaunch lost the
  drop but the persisted key blocked every re-deposit for 24h. The
  persisted snapshot now excludes keys still pending; they become durable
  only when flushPendingDrops actually publishes them.
- Gateway handoff: seen-event IDs were consumed before the deliverToPeer
  handoff; a failed handoff (peer walked away) permanently dropped the
  event for a single-gateway island. deliverToPeer now reports whether
  the handoff was attempted, and a failure releases the seen slot so a
  relaunch or backlog redelivery retries.
- Restore-path logs: central/peripheral-restore captures logged the init
  sentinel bgRemaining=∞. The cache is now seeded in init's main-thread
  branch and restore captures route through the sampler, which refreshes
  the cached budget before logging.
- Persist cost: the dedup record was a full JSON encode + atomic write on
  the main actor per mutation (once per event during a backlog re-fetch).
  Writes now coalesce behind a 1s window, flushed immediately on
  background/terminate; panic wipe stays immediate.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Remove BoundedIDSet.remove, orphaned by the ExpiringIDSet migration

The drop-dedup sets that needed slot release moved to ExpiringIDSet;
remaining BoundedIDSet users only insert and check. Periphery caught it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: jack <jackjackbits@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jack
2026-07-09 18:23:46 +02:00
committed by GitHub
co-authored by jack Claude Fable 5
parent 304460ee83
commit f8ab0a7dc0
13 changed files with 754 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bitchatTests/Services/BridgeCourierServiceTests.swift
+135 -3
View File
@@ -24,6 +24,7 @@ struct BridgeCourierServiceTests {
var localPeers: [(peerID: PeerID, noiseKey: Data)] = []
var held: [CourierEnvelope] = []
var sealResult: CourierEnvelope?
var deliverResult = true
private(set) var publishedEvents: [NostrEvent] = []
private(set) var openedSubscriptions: [[String]] = []
@@ -35,8 +36,8 @@ struct BridgeCourierServiceTests {
let service: BridgeCourierService
init() {
service = BridgeCourierService()
init(dedupStore: BridgeDropDedupStore? = nil) {
service = BridgeCourierService(dedupStore: dedupStore)
service.bridgeEnabled = { [weak self] in self?.bridgeOn ?? false }
service.relaysConnected = { [weak self] in self?.relaysConnected ?? false }
service.publishEvent = { [weak self] event in self?.publishedEvents.append(event) }
@@ -49,7 +50,10 @@ struct BridgeCourierServiceTests {
return self?.sealResult
}
service.openEnvelope = { [weak self] envelope in self?.openedEnvelopes.append(envelope) }
service.deliverToPeer = { [weak self] envelope, peer in self?.delivered.append((envelope, peer)) }
service.deliverToPeer = { [weak self] envelope, peer in
self?.delivered.append((envelope, peer))
return self?.deliverResult ?? false
}
service.heldEnvelopes = { [weak self] cooldown in
self?.heldCooldowns.append(cooldown)
return self?.held ?? []
@@ -175,6 +179,134 @@ struct BridgeCourierServiceTests {
#expect(fixture.sealRequests.count == 1)
}
@Test func publishedDropDedupSurvivesRelaunch() throws {
// Regression (field-verified amplification storm): the outbox that
// drives re-deposits is persisted, but the sender-side drop dedup was
// in-memory only — every relaunch republished the same undelivered
// message as a fresh drop, and relays hold each for 24h.
let fileURL = FileManager.default.temporaryDirectory
.appendingPathComponent("bridge-dedup-\(UUID().uuidString).json")
defer { try? FileManager.default.removeItem(at: fileURL) }
let recipientKey = Fixture.randomKey()
let messageID = UUID().uuidString
let fixture = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
fixture.sealResult = makeEnvelope(recipientKey: recipientKey)
#expect(fixture.service.depositDrop(content: "hello", messageID: messageID, recipientNoiseKey: recipientKey))
#expect(fixture.publishedEvents.count == 1)
// Persistence is coalesced; a real launch flushes within a second or
// on backgrounding — tests flush explicitly.
fixture.service.flushDedupSnapshot()
// "Relaunch": a fresh service over the same store must refuse to
// publish the same message ID again (before even re-sealing it).
let relaunched = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
relaunched.sealResult = makeEnvelope(recipientKey: recipientKey)
#expect(!relaunched.service.depositDrop(content: "hello", messageID: messageID, recipientNoiseKey: recipientKey))
#expect(relaunched.publishedEvents.isEmpty)
#expect(relaunched.sealRequests.isEmpty)
}
@Test func seenDropEventDedupSurvivesRelaunch() throws {
// Same storm, gateway side: relays redeliver the whole 24h drop
// backlog on every launch; a relaunch must not re-open (and re-ack)
// events it already handled.
let fileURL = FileManager.default.temporaryDirectory
.appendingPathComponent("bridge-dedup-\(UUID().uuidString).json")
defer { try? FileManager.default.removeItem(at: fileURL) }
let fixture = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
let myKey = try #require(fixture.myKey)
fixture.service.refresh()
let event = try makeDropEvent(for: makeEnvelope(recipientKey: myKey))
fixture.service.handleDropEvent(event)
#expect(fixture.openedEnvelopes.count == 1)
fixture.service.flushDedupSnapshot()
let relaunched = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
relaunched.myKey = myKey
relaunched.service.refresh()
relaunched.service.handleDropEvent(event)
#expect(relaunched.openedEnvelopes.isEmpty)
}
@Test func offlineQueuedDropStaysRedepositableAfterRelaunch() throws {
// A deposit made while relays are down only joins the in-memory
// pending queue. Its dedup key must NOT be durable yet: if the app is
// killed before relays connect, the relaunch loses the queued drop —
// a persisted key would then block every 120s re-deposit for 24h and
// the message would silently never reach a relay.
let fileURL = FileManager.default.temporaryDirectory
.appendingPathComponent("bridge-dedup-\(UUID().uuidString).json")
defer { try? FileManager.default.removeItem(at: fileURL) }
let recipientKey = Fixture.randomKey()
let messageID = UUID().uuidString
let fixture = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
fixture.relaysConnected = false
fixture.sealResult = makeEnvelope(recipientKey: recipientKey)
#expect(fixture.service.depositDrop(content: "later", messageID: messageID, recipientNoiseKey: recipientKey))
#expect(fixture.publishedEvents.isEmpty)
// Even a flush while the drop is still pending must exclude its key.
fixture.service.flushDedupSnapshot()
// "App killed before relays connected": pendingDrops were memory-only.
let relaunched = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
relaunched.sealResult = makeEnvelope(recipientKey: recipientKey)
#expect(relaunched.service.depositDrop(content: "later", messageID: messageID, recipientNoiseKey: recipientKey))
#expect(relaunched.publishedEvents.count == 1)
}
@Test func publishedPendingDropBecomesDurableAfterFlush() throws {
// Counterpart: once the queued drop actually publishes on reconnect,
// its key becomes durable and a relaunch must not republish.
let fileURL = FileManager.default.temporaryDirectory
.appendingPathComponent("bridge-dedup-\(UUID().uuidString).json")
defer { try? FileManager.default.removeItem(at: fileURL) }
let recipientKey = Fixture.randomKey()
let messageID = UUID().uuidString
let fixture = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
fixture.relaysConnected = false
fixture.sealResult = makeEnvelope(recipientKey: recipientKey)
#expect(fixture.service.depositDrop(content: "later", messageID: messageID, recipientNoiseKey: recipientKey))
fixture.relaysConnected = true
fixture.service.flushPendingDrops()
#expect(fixture.publishedEvents.count == 1)
fixture.service.flushDedupSnapshot()
let relaunched = Fixture(dedupStore: BridgeDropDedupStore(fileURL: fileURL))
relaunched.sealResult = makeEnvelope(recipientKey: recipientKey)
#expect(!relaunched.service.depositDrop(content: "later", messageID: messageID, recipientNoiseKey: recipientKey))
#expect(relaunched.publishedEvents.isEmpty)
}
@Test func failedGatewayHandoffReleasesSeenSlot() throws {
// A gateway's deliverToPeer handoff is best-effort: when it fails
// (the peer walked away between relay fetch and mesh send), the drop
// event must stay retryable — for a single-gateway mesh island this
// gateway is the recipient's only carrier.
let fixture = Fixture()
let peerKey = Fixture.randomKey()
let peer = PeerID(str: "aabbccdd00112233")
fixture.localPeers = [(peer, peerKey)]
fixture.service.refresh()
let event = try makeDropEvent(for: makeEnvelope(recipientKey: peerKey))
fixture.deliverResult = false
fixture.service.handleDropEvent(event)
#expect(fixture.delivered.count == 1)
// Redelivery (relaunch/backlog re-fetch) retries the handoff …
fixture.deliverResult = true
fixture.service.handleDropEvent(event)
#expect(fixture.delivered.count == 2)
// … and a successful handoff consumes the event for good.
fixture.service.handleDropEvent(event)
#expect(fixture.delivered.count == 2)
}
@Test func distinctDropsUseDistinctThrowawayKeys() {
let fixture = Fixture()
let keyA = Fixture.randomKey()