Dropping a previous Iceberg sort order field causes org.apache.iceberg.exceptions.ValidationException #8614
Closed as not planned
Description
Apache Iceberg version
1.3.1 (latest release)
Query engine
Spark
Please describe the bug 🐞
If you have ever used a column in write sort order (as it's called in the Spark Iceberg DDL), attempting to drop the column will result in org.apache.iceberg.exceptions.ValidationException.
Repro:
DROP TABLE IF EXISTS local.udevelopers_sorted;
CREATE TABLE IF NOT EXISTS local.udevelopers_sorted (
username string,
firstname string,
lastname string)
USING ICEBERG;
ALTER TABLE local.udevelopers_sorted WRITE ORDERED BY lastname;
ALTER TABLE local.udevelopers_sorted RENAME COLUMN lastname TO deprecated_lastname;
SELECT * FROM local.udevelopers_sorted;
ALTER TABLE local.udevelopers_sorted WRITE ORDERED BY username;
ALTER TABLE local.udevelopers_sorted DROP COLUMN lastname;
SELECT * FROM local.udevelopers_sorted;
I think the solution is to add similar handling to how Iceberg handles partition columns & identifier columns (e.g. special drop function etc.)
As a temporary work around for folks you can make it a partition column then drop the partition column, but this feels ugly to me.
DROP TABLE IF EXISTS local.udevelopers_sorted;
CREATE TABLE IF NOT EXISTS local.udevelopers_sorted (
username string,
firstname string,
lastname string)
USING ICEBERG;
ALTER TABLE local.udevelopers_sorted WRITE ORDERED BY lastname;
SELECT * FROM local.udevelopers_sorted;
ALTER TABLE local.udevelopers_sorted WRITE ORDERED BY username;
-- Hack, add it to identifier fields so we can do a "partial" drop where it stays in the schema and we don't
-- corrupt the metadata.
ALTER TABLE local.udevelopers_sorted ADD PARTITION FIELD lastname;
ALTER TABLE local.udevelopers_sorted DROP PARTITION FIELD lastname;
SELECT * FROM local.udevelopers_sorted;