FIX Make fetch_openml atomically cache the download

[siavrez]

Adding a tempfile name generator
change _retry_with_clean_cache function

Reference Issues/PRs

#21798

What does this implement/fix? Explain your changes.

Any other comments?

[ogrisel]

I would rather download the file into a temporary folder using the tempfile module from the standard library:

with tempfile.TemporaryDirectory(dir=data_home) as tmpdir:
    # call the private function that downloads the file before moving it to its final location

# tmpdir is automatically deleted when exiting the context manager

EDIT: let's pass dir=data_home to make sure we are on the same filesystem so that the move operation is guaranteed to be atomic.

[ogrisel]

This looks good to me. Just a quick suggestion below.

Could you please document this enhancement in the doc/whats_new/v1.1.rst file?

[ogrisel]

LGTM. Thanks!

[thomasjpfan]

Thank you for working on this @siavrez !

[thomasjpfan]

_open_openml_url is only used in fetch_openml:

Suggested change

	- |Fix| The fetch_* functions are now thread safe. Data is first downloaded
	- |Fix| :func:`datasets.fetch_openml` is now thread safe. Data is first downloaded

[thomasjpfan]

I think this is mostly safe as long as the temporary directory and the destination are in the same filesystem. According to shutil.move docs, shutil.move is the same as os.rename on the same filesystem.

If there is actually a copy, I can see issues such as:

• If two threads try to copy to the same location at the same time, I think the file can end up corrupted.
• If one thread is in the process of copying over the file, another thread will see that file exist and attempt to read it, which would fail since the file is not fully copied yet.

I do not know a great way forward. Using os.rename directly may break use cases where the temporary directory is not on the same filesystem as the cache. And copying itself may lead to errors as stated above.

[siavrez]

My original idea was to use a naming scheme based on process id or thread Id and create a soft symlik instead of renaming. If same process wants to clean cache remove the original and symlink If another process tries clean cache create a new file based on its id and for reading it can use the original symlink. What do you think?
@thomasjpfan

[ogrisel]

I think the tempfile is guaranteed to be in the same filesystem and therefore the renaming to be atomic because we create the tempfolder as a subfolder.

So I think the current code is fine. Maybe @siavrez you can add an inline comment to explain this.

[thomasjpfan]

I think the tempfile is guaranteed to be in the same filesystem and therefore the renaming to be atomic because we create the tempfolder as a subfolder.

Ah I see it now. Thank you for the explanation.

[thomasjpfan]

LGTM