Skip to content

[server] Optimize RemoteLogFetcher with async prefetch for recovery#3132

Open
Kaixuan-Duan wants to merge 2 commits intoapache:mainfrom
Kaixuan-Duan:remote-log-fetcher-prefetch
Open

[server] Optimize RemoteLogFetcher with async prefetch for recovery#3132
Kaixuan-Duan wants to merge 2 commits intoapache:mainfrom
Kaixuan-Duan:remote-log-fetcher-prefetch

Conversation

@Kaixuan-Duan
Copy link
Copy Markdown

Purpose

Linked issue: close #3091
This PR improves KV recovery performance by reducing wait time between remote log segments in RemoteLogFetcher.

Brief change log

  • Add a dedicated single-thread download executor for async prefetch.
  • Prefetch the next fetchable remote segment while consuming the current one.
  • Reuse prefetched file when available; fallback to synchronous download on prefetch failure.
  • Ensure cleanup on close: cancel in-flight prefetch, close active iterator/resources, shutdown executor.
  • Add a regression test for repeated fetch() to ensure previous iterator cleanup.

Tests

  • ./mvnw -pl fluss-server -Dtest=RemoteLogFetcherTest -DfailIfNoTests=false -Dspotless.check.skip=true test

API and Format
No API change. No storage/log format change.

Documentation
No user-facing feature. No documentation update required.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[server] RemoteLogFetcher optimize to async downloading

1 participant