[SPARK] Fix flakey test

[Fokko]

Feature Request / Improvement

I noticed this one the other day:

TestDataFrameWrites > testFaultToleranceOnWrite() > format = orc FAILED
    java.io.IOException: Failed to delete temp directory /tmp/junit10947448527164022571. The following paths could not be deleted (see suppressed exceptions for details): <root>
        at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.createIOExceptionWithAttachedFailures(TempDirectory.java:431)
        at org.junit.jupiter.engine.extension.TempDirectory$CloseablePath.close(TempDirectory.java:312)

I don't think we want to fail if a temp path cannot be deleted.

Query engine

None

[manuzhang]

Duplicate of #10480?

[nk1506]

@Fokko / @manuzhang , With TempDirectory , CleanupMode is ALWAYS as default. Means it always try to delete the temp directory after the test case execution.
What will be harm if we disable the clean up? Means something like @TempDir(cleanup = CleanupMode.NEVER) this we can do.
Since with every test and format the directory location is unique. It will never overlap with each other. Only immediate side effects I am seeing here is occupying this space for sometime.

The cause of this issue is

if (this.cleanupMode != CleanupMode.NEVER && (this.cleanupMode != CleanupMode.ON_SUCCESS || !TempDirectory.selfOrChildFailed(this.extensionContext))) {
                    FileOperations fileOperations = (FileOperations)this.extensionContext.getStore(TempDirectory.NAMESPACE).getOrDefault("file.operations", FileOperations.class, TempDirectory.FileOperations.DEFAULT);
                    SortedMap<Path, IOException> failures = this.deleteAllFilesAndDirectories(fileOperations);
                    if (failures.isEmpty()) {
                        return;
                    }

                    throw this.createIOExceptionWithAttachedFailures(failures);
                }

WDYT ?

[ismailsimsek]

Might be related to unclosed resources, similar reports here and here