Skip to content

[Bug]: BigQueryIO Storage API write autoUpdateSchema may cause data corruption #26789

Closed

Description

@Abacn

What happened?

It is found that the schema returned in response may have different ordering for fields. It may cause either persistent write errors or data written into wrong BigQuery field.

This will be triggered under the following circumstances:

  • The STORAGE_API_AT_LEAST_ONCE write method is used
  • The table schema is explicitly set by the user (as opposed to leaving it empty for the sink to infer)
  • The table schema set by the user has fields listed in a different order than in the actual table.

There are several possible mitigations:
- Ensure that the schema specified has fields listed in the exact same order as in the BigQuery table.
- Stop specifying the schema, which will cause the sink to infer the schema from the destination table (note: this may not perform well when writing to large numbers of unique tables).
- Use a different insert method.

This was revealed from another bug that makes schemaUpdate always true (fixed in #26752)

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

  • Component: Python SDK
  • Component: Java SDK
  • Component: Go SDK
  • Component: Typescript SDK
  • Component: IO connector
  • Component: Beam examples
  • Component: Beam playground
  • Component: Beam katas
  • Component: Website
  • Component: Spark Runner
  • Component: Flink Runner
  • Component: Samza Runner
  • Component: Twister2 Runner
  • Component: Hazelcast Jet Runner
  • Component: Google Cloud Dataflow Runner

Activity

  1. self-assigned this
    on May 19, 2023
  2. Abacn commented on May 19, 2023

    @Abacn
    ContributorAuthor

    I tried the steps in the description by

    1. Run a pipeline that write some data to a table
    2. Run another STORAGE_API_AT_LEAST_ONCE pipeline, where
    • set .withSchema with a tableSchema that field order are shuffled
    • or Assemble TableRow order adding fields shuffled

    Both pipeline succeeded and wrote the expected results (checked the table data). Code snippet in #26796

    Nevertheless this means a more common use case of (writing to an existing BigQuery Table with STORAGE_API_AT_LEAST_ONCE method, explicitly set .withSchema(tableSchema) where tableSchema has arbitrary field order) does not trigger this bug.

    The bug would only be triggered when schema update actually happened, as tested in #26794

  3. Abacn commented on May 19, 2023

    @Abacn
    ContributorAuthor

    Update: @reuvenlax entered #26794 with the fix. I verified that running test on modified master it fails with

    rg.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: Unknown fields set in append! 1: 6
    2: "name6"
    
    	at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:374)
            ...
    Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unknown fields set in append! 1: 6
    2: "name6"
    
    	at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.lambda$process$10(StorageApiWritesShardedRecords.java:567)
    	at org.apache.beam.sdk.io.gcp.bigquery.RetryManager$Operation.run(RetryManager.java:132)
    	at org.apache.beam.sdk.io.gcp.bigquery.RetryManager.run(RetryManager.java:248)
    	at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.process(StorageApiWritesShardedRecords.java:740)
    Caused by: java.lang.RuntimeException: Unknown fields set in append! 1: 6
    2: "name6"
    
    	at org.apache.beam.sdk.io.gcp.testing.FakeDatasetService$1.appendRows(FakeDatasetService.java:573)
    	at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.lambda$process$10(StorageApiWritesShardedRecords.java:565)
    	at org.apache.beam.sdk.io.gcp.bigquery.RetryManager$Operation.run(RetryManager.java:132)
    	at org.apache.beam.sdk.io.gcp.bigquery.RetryManager.run(RetryManager.java:248)
    
    

    Tested with both ./gradlew :sdks:java:io:google-cloud-platform:test --tests org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testUpdateTableSchemaUseSet[3]
    and
    ./gradlew :sdks:java:io:google-cloud-platform:test --tests org.apache.beam.sdk.io.gcp.bigquery.BigQueryIOWriteTest.testUpdateTableSchemaUseSet[4]
    (STORAGE_WRITE_API and STORAGE_API_AT_LEAST_ONCE)

    • if do not set threshold and num of stream to 1 for the test, error message is
    org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.RuntimeException: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: req
    	at org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish(DirectRunner.java:374)
            ......
    Caused by: java.lang.RuntimeException: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: req
    	at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.lambda$process$10(StorageApiWritesShardedRecords.java:567)
    	at org.apache.beam.sdk.io.gcp.bigquery.RetryManager$Operation.run(RetryManager.java:132)
    	at org.apache.beam.sdk.io.gcp.bigquery.RetryManager.run(RetryManager.java:248)
    	at org.apache.beam.sdk.io.gcp.bigquery.StorageApiWritesShardedRecords$WriteRecordsDoFn.process(StorageApiWritesShardedRecords.java:740)
    Caused by: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: req
    	at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:79)
    	at com.google.protobuf.DynamicMessage$Builder.buildParsed(DynamicMessage.java:394)
    
    
  4. added a commit that references this issue on May 21, 2023
  5. added a commit that references this issue on May 21, 2023
  6. liferoad commented on May 22, 2023

    @liferoad
    Contributor

    #26794 was merged. @Abacn can you verify whether it completely addresses the current issue?

  7. added this to the 2.48.0 Release milestone on May 22, 2023
  8. Abacn commented on May 22, 2023

    @Abacn
    ContributorAuthor

    Yes (see above comment). #26794 fixes on master; and #26810 cherry pick into release-2.48.0 will close this once the cherry-pick merged.

  9. added a commit that references this issue on May 22, 2023
  10. added a commit that references this issue on May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions