Problem with french language

[Seraf]

Hello,

In the french language we use these letters :
àâäçèéêëîïôöùûü and as special character : ' @

' can be used in words like : "aujourd'hui" (means today in english)
@ can be used to give an email address to perform some custom action with.

Can you update the regexp to handle these utf8 characters please ?

Thanks a lot for your amazing work !!

[kirsle]

This has been requested a couple times now. I'm starting some work on getting UTF-8 support in RiveScript, beginning with the Perl version (commit: aichaos/rivescript-perl@884708a)

The support is "experimental" for the Perl version, but once I get some people to test it and make sure it's reasonably sound I'll begin bringing the support over to the Python version. In the mean time, if you'd like to add this support yourself you can do so and send me a pull request. :)

[Seraf]

Hello Kirsle,

I don't think I own too much the code to do something like that. I tried a lot but don't have success with.
Anyway, I found by replacing the regexp nasties with : re_nasties = re.compile('[^\p{L}0-9 ]') it handles all letters I want to use. Hope it can help.
When I will have a better level and comprehension of your code, I hope I will be able to do the pull request you want. Thanks anyway for your work !

[Seraf]

Hello Kirsle,

I'm working on a patch.
I changed the nasties regexp by : re_nasties = re.compile(r'^\w\s', re.UNICODE)
Now I have some difficulties to match the regexp :

It works : it returns the right sentence sent. The problem is now located in the function "_getreply" :

                regexp = '^'+regexp+'$'
                print "I pass here the regexp :/"+regexp+'/'
                print "and the message :/"+msg+'/'   
                match = re.match(regexp, msg)

Give :
[RS] Try to match "jarvis il y a quoi au cinema ?" against il y a quoi au cinema ? (jarvis il y a quoi au cinema ?)
I pass here the regexp :/^jarvis il y a quoi au cinema ?$/
and the message :/jarvis il y a quoi au cinema ?/
None
[...]
[RS] Try to match "jarvis il y a quoi au cinema ?" against * (jarvis (.+?))
I pass here the regexp :/^jarvis (.+?)$/
and the message :/jarvis il y a quoi au cinema ?/
<_sre.SRE_Match object at 0x14c5288>
[RS] Found a match!

I'm totally lost ... I have exactly the same result, a "if regexp==msg" return true ... If you have any idea, I'll take these !

the code above is located in :

                #Search their topic for a match to their trigger.
                if not foundMatch:
                    if isAtomic:
                    else:
                        >Here<

Thanks !

[Seraf]

Should be fixed with this commit : https://github.com/Seraf/rivescript-python/commit/0997a6278643b5c7f83f30ae033c20faf32f92c9

[kirsle]

I'm seeing the same issue on my end. I have a branch for implementing UTF-8 support here: https://github.com/kirsle/rivescript-python/tree/unicode

In my case I'm trying to match the atomic trigger "comment ça va", and even though my input exactly matches it (I also have debug prints showing two matching strings for the message and what it tries to match it to), it doesn't match even with an == comparison.

I suspect Python treats one of them (the user-provided one, I bet) as an ASCII string, and the other as a Unicode string, and it doesn't see them as equals for that reason.