Please provide a fallback from `regex` to the standard `re` module

[mgorny]

Context

I am maintaining Python packages for Gentoo Linux. mkdocs-autorefs package supports PyPy3, and it recently gained a dependency on mkdocs-material.

Description

Some of the mkdocs-material plugins currently depend on the regex package. This package relies on CPython implementation details and has significant technical debt, and support for other Python implementations such as PyPy has been rejected upstream.

Related links

• https://pypi.org/project/regex/#pypy
• split offset wrong due to UTF-8 issues on PyPy3 mrabarnett/mrab-regex#374 (comment)

Use Cases

It will enable full functionality of the plugin when using alternate Python implementations such as PyPy.

Visuals

No response

Before submitting

• I have read and followed the change request guidelines.
• I have verified that my idea is a change request and not a bug report.
• I have ensured that, to the best of my knowledge, my idea will benefit the entire community.
• I have included relevant links to the documentation, related issues, and discussions to underline the need for my idea.

[squidfunk]

Thanks for suggesting.

This is a reasonable request, but requires some careful thought. We're currently in feature freeze due to foundational work, so we have no bandwidth to implement this. However, if you want to go ahead, we can consider and collaborate on a PR. We're primarily using the regex crate to add support for Chinese search, which requires functionality that re does not have.

Since we are not planning to follow up here, we're closing this as out of scope, but as said, PRs can be considered.

[mgorny]

Thanks for the response. I've drafted it as #8032.

[facelessuser]

@mgorny I'm not sure about what technical debt is being mentioned here.

The linked issue seemed to state that pypy had a difference with the below example.

>>>> t = 'Fête\n'
>>>> re.match('(?s).*', t)
<_sre.SRE_Match object; span=(0, 8), match='Fête\n'>
>>>> regex.match('(?s).*', t)
<regex.Match object; span=(0, 6), match='Fête\n'>

No doubt it probably did, but now it doesn't seem to be the case.

>>>> t = 'Fête\n'
>>>> re.match('(?s).*', t)
<re.Match object; span=(0, 6), match='Fête\n'>
>>>> import regex
>>>> regex.match('(?s).*', t)
<regex.Match object; span=(0, 6), match='Fête\n'>

[facelessuser]

With that said, there may be some environments where Regex simply can't be used. In those cases I use this package (full disclosure I maintain it): https://github.com/facelessuser/backrefs.

It allows you to use Re with Unicode properties. It basically manipulates the pattern replacing Unicode properties with sequences that are equivalent and work in Re. Granted, it won't be as performant and it doesn't support true grapheme clusters which simply cannot be implemented in a reasonable way as a regex pattern. You need real coded logic to handle that, something that can't be accomplished with a wrapper.

I'm not sure how deep Material delves into Regex features, but if they are mainly tapping into Unicode properties, something like the above package could likely provide sufficient functionality as a fallback. But if it delves deeper than that, then maybe not.

[facelessuser]

One thing to note is that the mentioned package uses script_extension by default for things like IsHan vs Regex which uses script for things like IsHan. Usage of script_extensions (scx) matches perl regex.

>>> import regex
>>> text = '勇𣌀hi你好👋（）【】「」{}[]()'
>>> regex.sub(r'\p{ishan}', '', text)
'hi👋（）【】「」{}[]()'
>>> from backrefs import bre
>>> bre.sub(r'\p{ishan}', '', text)
'hi👋（）{}[]()'
>>> bre.sub(r'\p{script: han}', '', text)
'hi👋（）【】「」{}[]()'
>>> bre.sub(r'\p{scx: han}', '', text)
'hi👋（）{}[]()'

[mgorny]

I'm not sure about what technical debt is being mentioned here.

The _regex.c is over 26k lines of C code, basically organically grown over 25 years. I suspect that's the reason author rejected adding PyPy support — there was even discussion of explicitly preventing it from being installed on PyPy, but I don't think that was ever done. In any case, the responses were pretty hostile, and we gave up on trying to maintain PyPy support.

[facelessuser]

That's fair. I'm not sure if the issues were in pypy or Regex, but I don't see the specifically mentioned ones there anymore, but yeah, the project doesn't explicitly support pypy, so there is always risk there if support is needed.

Anyway, there are potentially other avenues as well as the obvious one of simply turning off any support that requires regex if it is missing. That would be unfortunate, but also an option.

[squidfunk]

Let's continue discussions in #8032.