You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: Managed IO for Iceberg cannot query by partition #33497
Lets say, I have field1 (string type) as my partition field, and there are many other fields, field2, field3, field4....
Querying the data with " select * from iceberg_table where field1 = 'field1_example_value'; " returns no data
but when querying with " select * from iceberg_table where field2 = 'field2_example_value'; " returns the data as expected.
I have ingested the data through Managed IO of Iceberg,
pipeline.apply(Managed.write(Managed.ICEBERG).withConfig(config_map);
Please have a look into this issue.
Hi @ahmedabu98 ,
I am using hive catalog, for which hive metastore running at port 9083,
Yes, I have created the iceberg table beforehand with the query,
CREATE TABLE catalog_name.schema_name.table_name( field1 varchar, field2 varchar... )
WITH (format = 'PARQUET', format_version = 2, location = 's3a://bucket_name/storage_path', partitioning = ARRAY['field1'])
I have created this by using Trino Iceberg Connector, referring to https://trino.io/docs/current/connector/iceberg.html
But while investigating, I found a bigger problem. Most records contain wrong/duplicate values for the partitioned column. For example, the above query gives me this result:
When actually I would only expect one result: {long=811, str=value_811}
I checked the warehouse and the str=value_811 partition has only one datafile. Directly reading this one datafile gives me the expected record {long=811, str=value_811}:
The key difference being the additional idToConstants passed to the reader. I noticed this was added to our Iceberg source recently in #33332, and AFAICT it replaces the record's column value with the stored partition value.
So the datafile is indeed part of the table but the partition value takes precedence. Seems that we are probably setting the wrong partition value.
Ahh the problem is that we're using a single PartitionKey object to resolve partitions, and using the same object to create writers.
What ends up happening is a race condition on which partition we resolve to by the end of the bundle. That partition value gets set for all writers in that bundle.
Instead, we should be making one copy of partitionKey for each new writer.
Activity
damccorm commentedon Jan 6, 2025
@ahmedabu98 any ideas here?
ahmedabu98 commentedon Jan 6, 2025
Hey @saathwik-tk, can you provide a little more context?
Also a reproducible example would be helpful to find an RCA
saathwik-tk commentedon Jan 8, 2025
Hi @ahmedabu98 ,
I am using hive catalog, for which hive metastore running at port 9083,
Yes, I have created the iceberg table beforehand with the query,
CREATE TABLE catalog_name.schema_name.table_name( field1 varchar, field2 varchar... )
WITH (format = 'PARQUET', format_version = 2, location = 's3a://bucket_name/storage_path', partitioning = ARRAY['field1'])
I have created this by using Trino Iceberg Connector, referring to https://trino.io/docs/current/connector/iceberg.html
and my configuration map looks as below,
Map<String,Object> configProperties = ImmutableMap.<String,Object>builder()
.put("io-impl","org.apache.iceberg.aws.s3.S3FileIO")
.put("catalog-impl","org.apache.iceberg.hive.HiveCatalog")
// few s3 configurations as required
.build();
Map<String, Object> catalogConfig = ImmutableMap.<String, Object>builder()
.put("uri","my-thrift-uri:my-port-no")
.put("warehouse","my-s3-warehouse-path")
.put("type", "hive")
.build();
ImmutableMap<String, Object> config_map = ImmutableMap.<String, Object>builder()
.put("table", "iceberg.iceberg_table")
.put("catalog_name", "iceberg_catalog")
.put("catalog_properties", catalogConfig)
.put("config_properties",configProperties)
.build();
ahmedabu98 commentedon Jan 9, 2025
I've been trying to reproduce it but haven't gotten something yet.
Are you running this on a specific Beam version? or against HEAD?
ahmedabu98 commentedon Jan 9, 2025
Nevermind I got a reproduction:
Code snippet
I get no results when executing this query in BigQuery:
This works however:
ahmedabu98 commentedon Jan 9, 2025
But while investigating, I found a bigger problem. Most records contain wrong/duplicate values for the partitioned column. For example, the above query gives me this result:

When actually I would only expect one result:
{long=811, str=value_811}I checked the warehouse and the
str=value_811partition has only one datafile. Directly reading this one datafile gives me the expected record{long=811, str=value_811}:Code snippet
Another case:
The following query gives me no results:
But when I manually read the datafile under partition
str=value_812, I find the correct record:{long=812, str=value_812}ahmedabu98 commentedon Jan 9, 2025
So it looks like we're writing the correct records to the correct partitions, but we get the wrong results when reading it back...
I can see the issue even when reading it with the Beam Iceberg connector:
Code snippet
kennknowles commentedon Jan 9, 2025
I may have missed it, but what is the result of
SELECT * FROMapache-beam-testing.ahmedabualsaud.my_table_with_spec` ?I'm trying to distinguish which of these is true, and maybe I missed it but I don't see the answer here:
ahmedabu98 commentedon Jan 9, 2025
It's the latter of those
kennknowles commentedon Jan 9, 2025
OK that's probably the simpler case. It sounds like that data file isn't "part of the table".
ahmedabu98 commentedon Jan 9, 2025
Okay when I read datafiles like this, I do see the bad data:
Code snippet
The key difference being the additional
idToConstantspassed to the reader. I noticed this was added to our Iceberg source recently in #33332, and AFAICT it replaces the record's column value with the stored partition value.So the datafile is indeed part of the table but the partition value takes precedence. Seems that we are probably setting the wrong partition value.
ahmedabu98 commentedon Jan 9, 2025
Ahh the problem is that we're using a single PartitionKey object to resolve partitions, and using the same object to create writers.
What ends up happening is a race condition on which partition we resolve to by the end of the bundle. That partition value gets set for all writers in that bundle.
Instead, we should be making one copy of partitionKey for each new writer.
ahmedabu98 commentedon Jan 9, 2025
@saathwik-tk I opened a PR to fix this in #33549. I ran another set of questions and from what I can tell it also fixes your original issue.
Cherrypick #33549 to Release 2.62.0 branch (Iceberg partition value r…