electrum backend + mwebsync + rescan when ImportAccount + ImportMwebScanKey + filter listunspent/listchaintxns by account#18
Open
electrum backend + mwebsync + rescan when ImportAccount + ImportMwebScanKey + filter listunspent/listchaintxns by account#18
Conversation
Refactor `handleNewHeader` to sequentially process blocks, ensuring complete block notification and watched address tracking. Key changes: - Process blocks from last known height to current - Fetch and validate each block header - Separate reorg detection from block processing - Improve logging and error handling - Ensure consistent block caching and notifications
This change adds REST API capabilities to the Electrum backend, enabling: - Fetching full block information - Retrieving transaction details - Finding transaction indices within blocks - Validating channel-related transactions Key changes include: - Added RESTClient to electrum package - Updated ElectrumNotifier and ChainClient to support REST URL - Modified configuration to require REST URL in Electrum mode - Implemented methods for block and transaction retrieval
This change adds support for fetching the actual pkScript for Taproot outputs when the script cannot be derived directly from the witness. It retrieves the full funding transaction to extract the correct output script for historical spend detection.
Ensure proper synchronization and order of operations when updating transaction notifier during block connected events. The height is now updated first, and confirmation/spend checks run concurrently to prevent potential race conditions.
Implements two critical optimizations for Electrum backend wallet sync: 1. Parallel address scanning using 10 concurrent workers with semaphore control, matching ElectrumX server INITIAL_CONCURRENT limit. Applied to FilterBlocks, Rescan, and NotifyReceived operations. 2. Session-level address history cache to eliminate redundant GetHistory queries across multiple block batches. Cache is invalidated on reorg detection to ensure data freshness. Performance improvements for 400k block sync (500k→900k) with 5000 addresses: - Per-batch time: 13 min → 2 min (6.5x faster) - Multi-batch sync: 6.7 hours → 13 min (30x faster) - Network queries: 1,000,000 → 5,000 (200x reduction) Changes: - Add addressScanConcurrency constant (10 workers) - Add historyCache map with RWMutex for thread-safe caching - Implement getHistoryWithCache() for cache-aware history fetching - Implement clearHistoryCache() for cache invalidation on reorg - Implement sendNotification() helper for non-blocking notifications - Refactor FilterBlocks() to use worker pool pattern - Refactor Rescan() to parallelize address scanning - Refactor NotifyReceived() to parallelize background scanning - Increase notification channel buffer: 100 → 2100 - Modify filterAddressInBlocks() to use cached history - Add cache clearing on reorg detection in handleNewHeader() The cache optimization is critical: without it, syncing 200 batches queries each of 5000 addresses 200 times (1M queries). With caching, each address is queried once and reused across all batches (5K queries).
…overy Rewrite FilterBlocks to iterate blocks one at a time and return on the first match, matching the contract expected by the wallet recovery loop (same as neutrino's FilterBlocks). The previous implementation returned ALL transactions from the entire block range associated with a single block, causing the wallet to re-discover the same txids at different block heights on subsequent scans — producing 607 DB entries for only 151 unique txids. The new approach queries all address histories upfront (cached), builds a height index, then iterates req.Blocks sequentially with O(1) lookups. On the first matching block, it fetches txs, runs BlockFilterer, and returns immediately. This preserves RPC performance while fixing the root cause. Also removes the notifiedTxs dedup mechanism and merkle proof verification added in a previous attempt that did not address the underlying issue and added unnecessary complexity and latency.
Pull reviewers statsStats of the last 30 days for lnd:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.