You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Failing Test]: Many BigQuery direct read Python PostCommit failing possibly due to fastavro regression #28811
If you are using our released docker image it should not be hit because fastavro dependency is included. If you are building custom environment, explicitly install fastavro<1.8.4 (pip install "fastavro<1.8.4")
Also, we don't want to pin a tighter upper bounds on the dependencies since users will miss out on using latest minor releases until the next beam release comes out
We pin dependencies and we know that transitive dependencies cannot break users.
We do semver loose dependencies and know that transitive dependencies can break users.
In scenario 2 we can't really treat transitive dependency changes as a panic situation or release blocker, because the behavior is actually by design.
To be very clear: I like scenario 2 better and I think dependency pinning should wait until the final deployment of an application.
For test coverage: Even though we claim we are compatible with a semver range, we have not tested that whole range. In reality it just means the same thing as being pinned to the top of the range. We have no idea if Beam 2.51.0 actually works with lower versions of fastavro. It was never tested.
Currently we release the docker image per release which contains fixed version of dependency, which partly mitigate the potential issue of choice 2. In this case, the same test does not break on Dataflow, which builds docker image via :sdks:python:container:py3x:docker task. Other portable runners in DOCKER mode would not be impacted either (most production use cases). It fails test because the direct runner and flink runner (LOOPBACK mode) determines beam's dependency from setup.py during the test.
Activity
yupbank commentedon Oct 4, 2023
This one is hitting us in production, any work arounds ? e.g. any health version ? currently using
apache_beam[gcp]==2.50.0Abacn commentedon Oct 4, 2023
If you are using our released docker image it should not be hit because fastavro dependency is included. If you are building custom environment, explicitly install fastavro<1.8.4 (
pip install "fastavro<1.8.4")tvalentyn commentedon Oct 6, 2023
@yupbank see recommendations about building reproducible environments on https://beam.apache.org/documentation/sdks/python-pipeline-dependencies/ to prevent an issue like this affecting your production.
riteshghorse commentedon Oct 6, 2023
Should we limit pin fastavro to <1.8.4?
beam/sdks/python/setup.py
Line 245 in af9da31
tvalentyn commentedon Oct 6, 2023
we need a bug on fastavro side; to unblock tests, we could temporarily add
!=1.8.4to the condition.tvalentyn commentedon Oct 6, 2023
they might be working on patch release to remediate.
riteshghorse commentedon Oct 6, 2023
Created an issue for fastavro fastavro/fastavro#720
kennknowles commentedon Oct 9, 2023
But the root danger is still here: if fastavro releases a bad version in our allowed range, all users break. Is our dependency pinning strategy wrong?
AnandInguva commentedon Oct 9, 2023
We tend to follow
SemVerapproach for pinning dependencies for most of the packages. https://semver.org/#why-use-semantic-versioning. Usually, we set the versions of a dependency to be less than the next major upgrade. In this case, its 2.0. It is up to the fastavro library not to make backward compatible changes until there is an major upgrade.https://devhints.io/semver
Also, we don't want to pin a tighter upper bounds on the dependencies since users will miss out on using latest minor releases until the next beam release comes out
kennknowles commentedon Oct 10, 2023
We have to choose. Either:
In scenario 2 we can't really treat transitive dependency changes as a panic situation or release blocker, because the behavior is actually by design.
To be very clear: I like scenario 2 better and I think dependency pinning should wait until the final deployment of an application.
For test coverage: Even though we claim we are compatible with a semver range, we have not tested that whole range. In reality it just means the same thing as being pinned to the top of the range. We have no idea if Beam 2.51.0 actually works with lower versions of fastavro. It was never tested.
Abacn commentedon Oct 10, 2023
Currently we release the docker image per release which contains fixed version of dependency, which partly mitigate the potential issue of choice 2. In this case, the same test does not break on Dataflow, which builds docker image via
:sdks:python:container:py3x:dockertask. Other portable runners in DOCKER mode would not be impacted either (most production use cases). It fails test because the direct runner and flink runner (LOOPBACK mode) determines beam's dependency from setup.py during the test.