Add change feed pull model support#3649
Closed
Pilchie wants to merge 11 commits intoAzure:release/azure_data_cosmos-previewsfrom
Closed
Add change feed pull model support#3649Pilchie wants to merge 11 commits intoAzure:release/azure_data_cosmos-previewsfrom
Pilchie wants to merge 11 commits intoAzure:release/azure_data_cosmos-previewsfrom
Conversation
Pilchie
force-pushed
the
feature/change-feed-pull-model-3458
branch
from
f4362d6 to
e92439e
Compare
Pilchie
changed the base branch from
main
to
release/azure_data_cosmos-previews
Pilchie
force-pushed
the
feature/change-feed-pull-model-3458
branch
2 times, most recently
from
139f991 to
3e9bd2a
Compare
We need to do a release ASAP for a customer need, and we're depending on unreleased code in azure_core. This PR reverts that temporarily and moves our dependency from path-based to depending on the released versions of core libraries (0.32). I'll open a follow-up PR after this is released to go back to referencing the live code, so that we can release again after the next Core release wave.
Implement change feed pull model for azure_data_cosmos crate with V2 support for partition splits and merges. New public APIs: - ContainerClient::query_items_change_feed<T>() -> FeedPager<T> - ContainerClient::read_feed_ranges() -> Vec<FeedRange> New types: - change_feed::FeedRange - Serializable feed range for persistence - change_feed::ChangeFeedStartFrom - Beginning, Now, or PointInTime - change_feed::ChangeFeedMode - LatestVersion or AllVersionsAndDeletes - QueryChangeFeedOptions - Options for change feed queries - ReadFeedRangesOptions - Options for reading feed ranges Internal infrastructure: - Continuation token serialization/deserialization - Composite continuation tokens with split/merge handling - Change feed state machine for tracking position Fixes Azure#3458
Remove leftover conflict markers in the cosmos Cargo.toml after rebasing onto release/azure_data_cosmos-previews, and refresh Cargo.lock to match the dependency set (azure_core workspace + base64). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The method on ChangeFeedStartFromInternal was never called, causing a dead_code warning treated as error in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pin azure_core to version 0.32.0 with default-features = false instead of inheriting the workspace version (0.33.0), which was incorrectly merged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Pilchie
force-pushed
the
feature/change-feed-pull-model-3458
branch
from
3e9bd2a to
bf4d1e1
Compare
Pilchie
force-pushed
the
feature/change-feed-pull-model-3458
branch
from
bf4d1e1 to
089c9b7
Compare
Update cosmos_change_feed.rs to use correct public APIs: - Use azure_core::Uuid instead of uuid crate - Use ContainerProperties::new() instead of struct literal - Use QueryChangeFeedOptions builder methods instead of private field access - Add .await on async container_client() calls - Pass None for run_with_shared_db options parameter - Use FeedItemIterator as Stream<Item=Result<T>> directly - Handle Result from FeedRange::to_string_representation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The change feed request was passing None for PipelineSendOptions, causing 304 (Not Modified) responses to be treated as errors. Now passes SUCCESS_CODES to accept 304 and returns Ok(None) to cleanly end the change feed stream when no more changes exist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rewrite query_items_change_feed to use ChangeFeedState for multi-partition round-robin instead of hardcoded pk_range_id 0. Read the etag response header for change feed continuation instead of x-ms-continuation. - Make query_items_change_feed async to resolve partition key ranges upfront via the partition key range cache - Expose continuation tokens on every FeedPage so callers can save and resume via with_continuation_token() - Wire up with_feed_range() for scoped/parallel processing - Add FeedRange::for_full_range() factory - Validate AllVersionsAndDeletes requires ChangeFeedStartFrom::Now - Add ETAG constant for change feed response headers - Remove old single-partition send_change_feed_request from pipeline - Delete unused scaffolding (test-only constructors, unused fields) Add 9 new emulator tests covering event counting, continuation token resume, fromNow, fromPointInTime, small page size, full range, parallel feed ranges, and mode restriction validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
|
This is going to be different in the driver world, so I'm going to close it for now, we can keep the code around in your fork branch! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement change feed pull model for azure_data_cosmos crate with V2 support for partition splits and merges.
New public APIs:
New types:
Internal infrastructure:
Fixes #3458