Avoid triggering @property methods on plugins when looking for hookimpls during registration

[pirate]

Hey all, thanks for making pluggy! I'm loving using it so far. 🥳

---

Currently pluggy fairly aggressively inspects every plugin's attrs upon registration in order to look for potential @hookimpl-marked methods.

Based on the existing implementation in PluginManager.register() -> ... -> parse_hookimpl_opts(), it's clear that the intention is to only look for Callable methods marked with the @hookimpl decorator (it skips over any non-methods / undecorated-methods it finds in the process).

If a plugin is class-based, the current implementation using inspect.isroutine(getattr(plugin, name)) has the unintended consequence of evaluating every single @property method on the passed plugin object, which can have side effects because property methods can execute arbitrary code upon access!

This PR corrects this by pre-checking if a given plugin attr is a @property (or pydantic field), before attempting to getattr(plugin, name) to pass it to inspect.isroutine(...).

[pirate]

Codecov is complaining about whitespace lines and comment lines not being covered?

[RonnyPfannschmidt]

Looks good to me, wondering if we should take static getattr from pytest for fetching declarations

[pirate]

Ok I've updated the PR:

• Removed pydantic-specific logic
• Abstracted @property check to helper function _attr_is_property(obj, name) -> bool
• Exhaustively tested the following situations:
  • proper handling of: modules, class, objects, Pydantic models, Django models being used as namespaces
  • namespaces containing: @property, @classproperty, @classmethod, @staticmethod, ClassVars, normal variables, and normal methods

For a full demo of the exhaustive tests showing that it works in those cases ^ (including pydantic-specific tests that I did not put in the PR) see here:

https://gist.github.com/pirate/66f12beac594c99c697cd5543a1cb77b

[RonnyPfannschmidt]

this looks good to me, Thanks!

i'd like to get a final ok from @bluetech

[RonnyPfannschmidt]

i recently re-discovered inspect.getattr_static
i wonder if that one may be nice for a followup change

[bluetech]

When I last looked at it, the main downside of getattr_static was that it was very slow. Though may this has improved (I think I saw mention of that).

[bluetech]

Thanks for removing the pydantic bit. I still have a question on the AttributeError case.

[bluetech]

If I understand correctly, this is to handle a proxy object which returns some attribute from dir(obj) but getattr(obj, attr) raises AttributeError.

To me this seems like a malfunctioning proxy implementation, which should be fixed by properly overriding __dir__ or similar.

[pirate]

That's correct @bluetech, but unfortunately some common classes/module namespaces in the wild don't behave stricly like a barebones class or MappingTypes. e.g. pydantic models, benedict dicts, django model classes. Unfortunately you cant "fix" __dir__ on these classes becauase they are managed by the pydantic/django/etc. and it would break the library.

The case is easy to catch though, becauase getting an AttributeError at that point while iterating through dir.keys() is surprising/rare, and the fix is easy, just check the base class:

try:
    method = getattr(plugin, name)
except AttributeError:   # means it's not a normal object
    # oops, we actually want the base class it's wrapping,
    # not the wrapper proxy object.
    method = getattr(type(plugin), name)

With the patch you can define classes with @hookimpl methods on them like normal:

from django.db import models
from pydantic import BaseModel

from .plugin_spec import hookimpl

class FancyGizmoPlugin(BaseModel): 
    # or any other "wrapped" ^ base class from django, pydantic, dataclass, etc.
    # numpy 
    
	name: 'Some Fancy New Gizmo!'
	
	@hookimpl
	def get_all_plugins(self):
	    return {self.name: self}
	    
    @hookimpl
    def finished_loading(self):
		print(f'{self.name} finished loading succesfully!')
	
    @hookimpl
    def do_some_cool_thing(self):
        ...

Real example in production: https://github.com/ArchiveBox/ArchiveBox/blob/9241a45bb8bb1ebce04fdd9031c2524cea06a337/archivebox/abx/archivebox/base_configset.py#L213

[bluetech]

Unfortunately you cant "fix" dir on these classes becauase they are managed by the pydantic/django/etc. and it would break the library.

I think it's a bug in pydantic (returning an attr from __dir__ but not being able to get it), and if you report it they might fix it. But as much as I dislike workarounds, I guess we can live with one for this.

Do you have a use case for the method = getattr(type(plugin), name) fallback? It doesn't make sense to me to grab from the class. Can you live with the fix being to change getattr(plugin, name) to getattr(plugin, name, None)?

[pirate]

The only case it fixes is if an instance proxy sets up some dynamic field at __init__ time that shadowed a hookimpl method defined on the base class. It would prevent pluggy from detecting the hookimpl on the class, but I could see that being intentional/not worth edge-casing, so I can live without it if you'd prefer to remove it.

[bluetech]

Let's start with getattr(plugin, name, None) and consider something more elaborate separately if someone needs it.