Fix preferred_locations stale ordering after hub region change

[tvaron3]

[Internal]

Depends on this pr #3861 that should be merged and reviewed in first.

Problem

LocationCache::update() permanently bakes in the region order from the first account-property fetch. On each call it clones the accumulated preferred_locations, extends it with any new read regions via a HashSet dedup check, and stores the result back. Because existing regions are already in the set, their order never changes — even when the gateway returns a new region order after a hub region change.

Impact: After a hub region change, the SDK ignores the updated region ordering from the gateway. Traffic continues routing based on the stale order from the initial account fetch rather than reflecting the new hub.

Fix

Add an immutable initial_preferred_locations field to LocationCache that preserves the user-specified preferred regions from construction. On each update() call, the effective preferred list is rebuilt from this immutable copy rather than from the accumulated state:

• Customer provided preferred regions: Always honored as the priority prefix; gateway-returned regions that are not in the customer list are appended in the gateway's current order (which updates on each refresh).
• No customer preference: The effective list is derived entirely from the gateway's response order, correctly reflecting hub changes.

Changes

• location_cache.rs: Added initial_preferred_locations field to LocationCache, stored in new(), used in update() to rebuild effective preferred list from the original user intent on every account refresh.
• Added two new tests:
  • preferred_locations_not_permanently_mutated_no_user_preference — verifies gateway order updates take effect when no user preference is set
  • preferred_locations_not_permanently_mutated_with_user_preference — verifies user prefix is preserved while gateway tail updates

[Copilot]

Pull request overview

Fixes stale preferred_locations ordering in the Cosmos routing LocationCache after hub region changes, ensuring the SDK re-respects the gateway-provided region order on each account refresh (while still honoring any user-provided preferred-region prefix).

Changes:

• Rebuild locations_info.preferred_locations on each LocationCache::update() from an immutable initial_preferred_locations baseline to avoid permanently baking in the first gateway ordering.
• Add unit tests covering both “no user preference” and “user preference prefix preserved” scenarios across consecutive updates with reordered gateway regions.
• Update retry handling for 403/3 (WriteForbidden) to gate PK-range endpoint marking behind PPAF/PPCB eligibility checks; add a changelog entry for the location ordering fix.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
sdk/cosmos/azure_data_cosmos/src/routing/location_cache.rs	Adds initial_preferred_locations, changes update() to rebuild effective ordering each refresh, and adds regression tests.
sdk/cosmos/azure_data_cosmos/src/retry_policies/client_retry_policy.rs	Adjusts 403/3 retry path to use per-partition failover / circuit breaker eligibility checks before marking endpoints unavailable.
sdk/cosmos/azure_data_cosmos/CHANGELOG.md	Adds an unreleased “Bugs Fixed” entry documenting the hub-region ordering fix.

Comments suppressed due to low confidence (1)

sdk/cosmos/azure_data_cosmos/src/routing/location_cache.rs:216

• existing is built once from effective_preferred_locations and never updated when you push new regions. That means duplicates in read_locations (or duplicates already present in initial_preferred_locations) can still produce duplicate entries in preferred_locations, despite the dedup intent. Consider making the set mutable and inserting each region as it’s appended (and optionally normalizing/deduping the initial list as well) so the preferred list is guaranteed unique.

        // Use HashSet for O(1) lookups instead of O(n) linear search
        let existing: HashSet<RegionName> = effective_preferred_locations.iter().cloned().collect();

        // Extend with read locations not already in preferred locations - O(n)
        for location in &read_locations {
            if !existing.contains(&location.name) {
                effective_preferred_locations.push(location.name.clone());
            }

[FabianMeiswinkel]

LGTM - Thx!

[tvaron3]

Application region is now mandatory so this change is not relevant currently.