You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: BigQueryIO Storage API write autoUpdateSchema may cause data corruption #26789
It is found that the schema returned in response may have different ordering for fields. It may cause either persistent write errors or data written into wrong BigQuery field.
This will be triggered under the following circumstances:
The STORAGE_API_AT_LEAST_ONCE write method is used
The table schema is explicitly set by the user (as opposed to leaving it empty for the sink to infer)
The table schema set by the user has fields listed in a different order than in the actual table.
There are several possible mitigations:
- Ensure that the schema specified has fields listed in the exact same order as in the BigQuery table.
- Stop specifying the schema, which will cause the sink to infer the schema from the destination table (note: this may not perform well when writing to large numbers of unique tables).
- Use a different insert method.
This was revealed from another bug that makes schemaUpdate always true (fixed in #26752)
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Run another STORAGE_API_AT_LEAST_ONCE pipeline, where
set .withSchema with a tableSchema that field order are shuffled
or Assemble TableRow order adding fields shuffled
Both pipeline succeeded and wrote the expected results (checked the table data). Code snippet in #26796
Nevertheless this means a more common use case of (writing to an existing BigQuery Table with STORAGE_API_AT_LEAST_ONCE method, explicitly set .withSchema(tableSchema) where tableSchema has arbitrary field order) does not trigger this bug.
The bug would only be triggered when schema update actually happened, as tested in #26794
Update: @reuvenlax entered #26794 with the fix. I verified that running test on modified master it fails with
rg.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Unknown fields set in append! 1: 6
2: "name6"
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:374)
...
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unknown fields set in append! 1: 6
2: "name6"
at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.lambda$process$10(StorageApiWritesShardedRecords.java:567)
at org.apache.beam.sdk.io.gcp.bigquery.RetryManager$Operation.run(RetryManager.java:132)
at org.apache.beam.sdk.io.gcp.bigquery.RetryManager.run(RetryManager.java:248)
at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.process(StorageApiWritesShardedRecords.java:740)
Caused by: java.lang.RuntimeException: Unknown fields set in append! 1: 6
2: "name6"
at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService$1.appendRows(FakeDatasetService.java:573)
at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.lambda$process$10(StorageApiWritesShardedRecords.java:565)
at org.apache.beam.sdk.io.gcp.bigquery.RetryManager$Operation.run(RetryManager.java:132)
at org.apache.beam.sdk.io.gcp.bigquery.RetryManager.run(RetryManager.java:248)
Tested with both ./gradlew :sdks:java:io:google-cloud-platform:test --tests org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testUpdateTableSchemaUseSet[3]
and ./gradlew :sdks:java:io:google-cloud-platform:test --tests org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testUpdateTableSchemaUseSet[4]
(STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE)
if do not set threshold and num of stream to 1 for the test, error message is
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: req
at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:374)
......
Caused by: java.lang.RuntimeException: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: req
at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.lambda$process$10(StorageApiWritesShardedRecords.java:567)
at org.apache.beam.sdk.io.gcp.bigquery.RetryManager$Operation.run(RetryManager.java:132)
at org.apache.beam.sdk.io.gcp.bigquery.RetryManager.run(RetryManager.java:248)
at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.process(StorageApiWritesShardedRecords.java:740)
Caused by: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: req
at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:79)
at com.google.protobuf.DynamicMessage$Builder.buildParsed(DynamicMessage.java:394)
Activity
Abacn commentedon May 19, 2023
I tried the steps in the description by
.withSchemawith a tableSchema that field order are shuffledBoth pipeline succeeded and wrote the expected results (checked the table data). Code snippet in #26796
Nevertheless this means a more common use case of (writing to an existing BigQuery Table with STORAGE_API_AT_LEAST_ONCE method, explicitly set
.withSchema(tableSchema)where tableSchema has arbitrary field order) does not trigger this bug.The bug would only be triggered when schema update actually happened, as tested in #26794
Abacn commentedon May 19, 2023
Update: @reuvenlax entered #26794 with the fix. I verified that running test on modified master it fails with
Tested with both
./gradlew :sdks:java:io:google-cloud-platform:test --tests org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testUpdateTableSchemaUseSet[3]and
./gradlew :sdks:java:io:google-cloud-platform:test --tests org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testUpdateTableSchemaUseSet[4](STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE)
Merge pull request #26794: #26789 Fix auto schema update when schema …
Merge pull request apache#26794: apache#26789 Fix auto schema update …
liferoad commentedon May 22, 2023
#26794 was merged. @Abacn can you verify whether it completely addresses the current issue?
Abacn commentedon May 22, 2023
Yes (see above comment). #26794 fixes on master; and #26810 cherry pick into release-2.48.0 will close this once the cherry-pick merged.
Merge pull request #26794: #26789 Fix auto schema update when schema …
Merge pull request apache#26794: apache#26789 Fix auto schema update …