You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: Performance regression and stuck jobs due to gcs-connector v3 upgrade #39548
It has been reported that some pipelines are running slower after upgrading Beam SDK to 2.74+. Particularly, users observed increased CPU, memory usage, and wall time for those jobs.
Root Cause
We suspect the culprit is gcs-connector, since we bumped its version from 2.x to 3.x in release 2.74.
Specifically, in gcs-connector v3, GoogleCloudStorageReadOptions.DEFAULT changed its default fadvise behavior from SEQUENTIAL to AUTO. This change disrupts expected sequential read throughput and caching behavior, which leads to the observed performance degradation or even stuck jobs.
References
A separate investigation documented in the upstream Hadoop connectors repository has confirmed this behavior: GoogleCloudDataproc/hadoop-connectors#1762
Internal bugs: 535192796, 512366613, 535194786.
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Fixed by #39445 (commit eb9e7a3), which overrides the default fadvise back to SEQUENTIAL for GoogleCloudStorageReadOptions and supports serialization for remote workers.
Activity
damccorm commentedon Nov 9, 2026
Fixed by #39445 (commit eb9e7a3), which overrides the default fadvise back to
SEQUENTIALforGoogleCloudStorageReadOptionsand supports serialization for remote workers.