Optimize plugin fails with "No such file or directory" in .cache directory

[chipsenkbeil]

Context

I recently tried adding the optimize plugin to work with a variety of PNGs I use on https://distant.dev/. It appears that I am hitting the issue described in #5098, so taking a stab at reporting it.

I'm using commit fa5bd3f45024328d973977db80cccd5b1df89228 of Insiders.

Bug description

Running mkdocs build or mkdocs serve with the optimize plugin configured within mkdocs.yml results in a FileNotFoundError.

❯ mkdocs build
INFO     -  [macros] - Macros arguments: {'module_name': 'macros', 'modules': [], 'render_by_default': True, 'include_dir': '',
            'include_yaml': [], 'j2_block_start_string': '', 'j2_block_end_string': '', 'j2_variable_start_string': '', 'j2_variable_end_string': '',
            'on_undefined': 'strict', 'on_error_fail': True, 'verbose': False}
INFO     -  [macros] - Found local Python module 'macros' in: /Users/senkwich/projects/distant.dev
INFO     -  [macros] - Found external Python module 'macros' in: /Users/senkwich/projects/distant.dev
INFO     -  [macros] - Extra filters (module): ['pretty']
INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /Users/senkwich/projects/distant.dev/site

  Optimizations:
Traceback (most recent call last):
  File "/Users/senkwich/Library/Python/3.9/bin/mkdocs", line 8, in <module>
    sys.exit(cli())
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/mkdocs/__main__.py", line 250, in build_command
    build.build(cfg, dirty=not clean)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/mkdocs/commands/build.py", line 332, in build
    config.plugins.run_event('post_build', config=config)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/mkdocs/plugins.py", line 522, in run_event
    result = method(**kwargs)
  File "/Users/senkwich/Library/Python/3.9/lib/python/site-packages/material/plugins/optimize/plugin.py", line 113, in on_post_build
    size_opt += os.path.getsize(target)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '.cache/plugin/optimize/assets/favicon.png'

Related links

• Optimize Plugin

Reproduction

example.zip

I've removed everything but a blank index.md page as it appears unrelated to which images are actual referenced. Additionally, I cleared out the majority of images and other assets alongside all css and javascript.

mkdocs.yml

nav:
  - 'index.md'

theme:
  name: material
  favicon: assets/favicon.png
  logo: assets/logo.svg

plugins:
  - optimize

Steps to reproduce

1. Create a couple of PNGs in the assets directory
   • For me, a minimal layout is

     assets/
         favicon.png
         logo.svg
         images/
             architecture/
                 picture.png
     

2. Attempt to run mkdocs build or mkdocs serve
3. Observe error about a file not existing in the .cache directory
   • In my case, there is a path to .cache/plugin/optimize/assets/images/architecture without anything inside, nor anything else in the parent directories.

Browser

No response

Before submitting

• I have read and followed the bug reporting guidelines.
• I have attached links to the documentation, and possibly related issues and discussions.
• I assure that I have removed all customizations before submitting this bug report.
• I have attached a .zip file with a minimal reproduction.

[squidfunk]

Thanks for reporting! Unfortunately, I'm not able to see the error you reported (multiple runs, same result):

INFO     -  Cleaning site directory
INFO     -  Building documentation to directory: /Users/squidfunk/Downloads/example 6/site
INFO     -  Optimized media file: assets/images/favicon.png 1.9 kB
INFO     -  Optimized media file: assets/favicon.png 1.7 kB
INFO     -  Optimized media file: assets/images/architecture/authenticate-with-server.png 93.5 kB ↓ 242.8 kB [72.2%]

  Optimizations:
    *.png 97.0 kB ↓ 242.8 kB [71.4%]

INFO     -  Documentation built in 1.24 seconds

My gut tells me it might be a race condition. Does the error occur when you set concurrency to 1?

plugins:
  - optimize:
      concurrency: 1

[squidfunk]

Additionally, please check if #5098 (comment) fixes it.

[squidfunk]

Okay, so 647614b27 is an attempt to mitigate the issue. After diving into concurrent futures, my understanding was that ThreadPoolExecutor.shutdown would automatically wait for all futures to be resolved or cancelled. However, I'm not sure whether this is implementation dependent. Thus, I've opted for making awaiting all futures more explicit by using wait. Could you please check the tip of master and see if this mitigates the issue? Unfortunately, I'm not able to reproduce, but making things more explicit is likely a good idea.

[squidfunk]

/cc @yrashk, @uPagge ⤴️

[chipsenkbeil]

My gut tells me it might be a race condition. Does the error occur when you set concurrency to 1?

plugins:
 - optimize:
     concurrency: 1

Additionally, please check if #5098 (comment) fixes it.

@squidfunk setting concurrency: 1 for optimize did not seem to impact the issue, but print_gain_summary: false enabled it to succeed. I'm not seeing any actual images in .cache though. Is that to be expected?

If it is a race condition, my machine hits it consistently.

---

I also tried https://github.com/squidfunk/mkdocs-material-insiders/commit/647614b2710e544009555fa3a7c358aa712bf5c4 and using concurrency: 1 yielded same error and print_gain_summary: false continued to succeed with nothing in the cache.

[JuLeeAtPlanet]

Just to share: this allowed the pipeline to succeed, but many images missing.

[squidfunk]

Hmm, I'm a little bit lost here, as I'm not able to observe the issue. I also tried to run it under the exact same Python version as reported in the OP + I'm also on macOS, but no success – everything works as expected 😐 If somebody that can observe the issue can assist, I'd be more than happy to collaborate in a PR on the Insiders repository.

[squidfunk]

Wait, I think I know what might be happening: do you have pngquant installed? The plugin needs pngquant to optimize PNG images. The problem is that it will currently silently fail when pngquant is not available. I'll fix that and improve error reporting, so it's apparent what's happening.

Just to share: this allowed the pipeline to succeed, but many images missing.

Are all of those images PNGs?

[squidfunk]

I've improved the resilience of the optimize plugin in 523653774:

• Upon start, the plugin checks for pngquant and notifies the user if it is not installed
• The plugin should keep running in serve mode when it encounters errors, similar to the social plugin

I'm pretty convinced this should fix the reported errors, and/or point the author into the right direction, i.e., if pngquant is not installed. Our documentation explains how to install pngquant.

[JuLeeAtPlanet]

I didn't install your latest, but I did add pngquant via brew to the pipeline and successfully built. The images I dynamically populate with a main.py macro on landing pages aren't showing up, but that could be another issue? At any rate, this ticket is about the gitlab pipeline failing and it no longer fails once I added brew and installed pngquant that way.