Skip to content

Spark: Add view support to SparkSessionCatalog #11388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
danielcweeks merged 2 commits into from
Dec 3, 2024
Merged

Spark: Add view support to SparkSessionCatalog #11388

danielcweeks merged 2 commits into from
Dec 3, 2024

Conversation

@nastra
Copy link
Contributor

@nastra nastra commented Oct 24, 2024 •
edited
Loading

No description provided.

@github-actions github-actions bot added the spark label Oct 24, 2024
@nastra nastra force-pushed the spark-session-catalog-with-views branch 4 times, most recently from 13f39eb to 47ac7f9 Compare October 24, 2024 09:32
@nastra nastra mentioned this pull request Oct 24, 2024
Closed
@nastra nastra force-pushed the spark-session-catalog-with-views branch from 47ac7f9 to c89d454 Compare October 24, 2024 10:40
@nastra nastra force-pushed the spark-session-catalog-with-views branch 4 times, most recently from e0eb166 to 1401a5f Compare October 24, 2024 10:51
@nastra nastra force-pushed the spark-session-catalog-with-views branch from 1401a5f to 2d8e5de Compare November 22, 2024 07:29
@github-actions github-actions bot removed the OPENAPI label Nov 22, 2024
@nastra nastra force-pushed the spark-session-catalog-with-views branch from 2d8e5de to 37f6c7b Compare November 22, 2024 07:34
@nastra nastra requested a review from danielcweeks November 22, 2024 07:34
danielcweeks
danielcweeks reviewed Dec 3, 2024
View reviewed changes
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java Outdated
properties);
} else if (isViewCatalog()) {
try {
getSessionCatalog().dropView(ident);
Copy link
Contributor

@danielcweeks danielcweeks Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if we want to allow this. If a catalog implementation doesn't support replace view, this could technically be a lossy operation. For example, if someone has DROP but not CREATE permissions, then we would drop the view and not be able to create leaving this in a bad state. I feel like it's safer to just fail and require that the user perform the drop.

Copy link
Contributor Author

@nastra nastra Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was mostly following Spark's example in https://github.com/apache/spark/blob/6cd13344bffd411289ec20170ae2da2035bb2859/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ViewCatalog.java#L124-L152 but I agree that we maybe don't want to do this non-atomically. If needed, we can always add that back later

@nastra nastra force-pushed the spark-session-catalog-with-views branch from 730cf39 to 252de8b Compare December 3, 2024 16:37
@nastra nastra requested a review from danielcweeks December 3, 2024 17:27
@danielcweeks danielcweeks merged commit 6501d29 into apache:main Dec 3, 2024
31 checks passed
@nastra nastra deleted the spark-session-catalog-with-views branch December 3, 2024 18:43
@nastra nastra mentioned this pull request Dec 4, 2024
Merged
@nastra nastra self-assigned this Dec 5, 2024
@nastra nastra mentioned this pull request Dec 16, 2024
Merged
zachdisc pushed a commit to zachdisc/iceberg that referenced this pull request Dec 23, 2024
@nastra @zachdisc
Spark: Add view support to SparkSessionCatalog (apache#11388)
d406604 
* Spark: Add view support to SparkSessionCatalog

* Don't replace view non-atomically
@nastra nastra mentioned this pull request Feb 28, 2025
Closed
@lintingbin
Copy link
Contributor

lintingbin commented Nov 11, 2025

spark.sql.catalog.spark_catalog = org.apache.iceberg.spark.SparkSessionCatalog spark.sql.catalog.spark_catalog.type = hive
We use the above configuration to work with Iceberg. Views created before this PR were in Hive format by default, but now views are created in Iceberg format, which can cause compatibility issues. Moreover, there is currently no way to create views in Hive format. I wonder if there is any method now to create Hive views, as most of our views are just column transformations and do not need to save any files.

@nastra
Copy link
Contributor Author

nastra commented Nov 11, 2025

@lintingbin can you please open a new issue for this?

@lintingbin lintingbin mentioned this pull request Nov 11, 2025
Open
3 tasks
@lintingbin
Copy link
Contributor

lintingbin commented Nov 11, 2025

@lintingbin can you please open a new issue for this?

Okay, a new issue has been created.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@danielcweeks danielcweeks danielcweeks approved these changes

Assignees

@nastra nastra

Labels

build spark

Projects

View support
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nastra @lintingbin @danielcweeks