You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: Spanner Change Stream reading can become stuck due to watermark of partition moving backwards #36470
This shows up in Dataflow as the errors in harness logs: generic::invalid_argument: SDK reported a smaller watermarkhold: 2025-09-11T02:45:27.292+00:00 than the computed lower bound: 2025-09-11T02:51:11.775+00:00
Partitions are added to the metadata table with their start timestamp. The detectnewchangestreams fn outputs new restrictions for these partitions with the minimum watermark of all partitions and then during processing of a partition the timestamp is strictly increasing.
This was root-caused to possible reordering of updating watermarks in the metadata table. It is exacerbated by watermark caching and filtering that was added to change streams in 2.67.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Activity
Update CHANGES.md to show issue apache#36470 is fixed in 2.69.0 release.
tianz101 commentedon Oct 10, 2025
LGTM, thanks!
Update CHANGES.md to show issue apache#36470 is fixed in 2.69.0 release.
Update CHANGES.md to show issue apache#36470 is fixed in 2.69.0 release.
Update CHANGES.md to show issue #36470 is fixed in 2.69.0 release. (#…
liferoad commentedon Oct 30, 2025
Close this since 2.69 is released.