Add "--annotate" option to `%%cython` magic. #2225

bfroehle · 2012-07-31T20:37:28Z

Closes #1976 and replaces pull-request #2147 (~~Windows only~~).

In addition, I've removed the ctx parameter from cythonize which is not actually parsed in Cython > 0.13. The previous coded didn't work with Cython of that age either (as Cython.Build didn't exist as a module yet).

With Cython 0.14 and 0.15 there is a bug where the force flag to cythonize isn't getting picked up, so the annotated HTML isn't being generated if the user already ran %%cython for that same bit of code. To trigger, run something like:

%%cython
<CODE>

%%cython -a
<CODE>

In my testing, everything seems to work correctly in 0.15.1 and later.

takluyver · 2012-07-31T21:00:50Z

IPython/extensions/cythonmagic.py

@@ -189,6 +192,10 @@ def f(x):
        module = imp.load_dynamic(module_name, module_path)
        self._import_all(module)

+        if args.annotate:
+            with open(os.path.join(lib_dir, module_name + '.html')) as f:


What will this do with files containing non-ascii characters? I suspect we may need to use io.open and specify utf-8, but it's worth checking what it does.

It works fine for me in Linux. But I suppose I should duplicate a similar methodology as the code above:

pyx_file = os.path.join(lib_dir, module_name + '.pyx') pyx_file = py3compat.cast_bytes_py2(pyx_file, encoding=sys.getfilesystemencoding()) with io.open(pyx_file, 'w', encoding='utf-8') as f: f.write(code)

Any idea why the cast_bytes_py2 line is there? That seems totally crazy to me...

Yes, something like that - although we should check that the HTML Cython generates is in UTF-8. That's the most logical choice, though.

A bit of digging turns up that the need for cast_bytes_py2 is a quirk of distutils - we shouldn't need that for reading the HTML file:

#1770 (comment)

bfroehle · 2012-07-31T23:17:11Z

@takluyver What's the best way to alert the user if the annotated HTML file cannot be opened? Do we have some display class for that? Or do I print to stderr?

bfroehle · 2012-07-31T23:30:26Z

IPython/extensions/cythonmagic.py

+        if args.annotate:
+            with open(os.path.join(lib_dir, module_name + '.html')) as f:
+                annotated_html = f.read()
+            return display.HTML(annotated_html)


Is there any other way to return the HTML? I know @minrk suggested putting it in the notebook directory and providing a link.

That is the only other way really. In the long run we can't really assume that the notebook directory is served by notebook server in library code. In contexts where the kernel runs on a different host than the notebook server, this directory won't be available. Because of this I think that using display.HTML is a better option.

takluyver · 2012-07-31T23:31:00Z

Print to stderr, I think. Or maybe just let the exception raise. Is there any obvious reason it wouldn't be able to open the file?

bfroehle · 2012-07-31T23:33:46Z

Yes, with Cython < 0.15.1, if the user compiles the same bit of Cython code first without and then with --annotate, cythonize doesn't successfully rebuild the file (and hence doesn't produce the annotated html).

I might be able to work around it, haven't looked into it yet.

takluyver · 2012-07-31T23:38:56Z

OK, then I guess a message to stderr is in order.

bfroehle · 2012-08-01T00:24:59Z

@ellisonbg Since you wrote the original %cython magic functions, do you want to look this one over too?

ellisonbg · 2012-08-01T05:38:56Z

I will try to have a look soon, thanks!

On Tue, Jul 31, 2012 at 5:25 PM, Bradley M. Froehle
[email protected]
wrote:

@ellisonbg Since you wrote the original %cython magic functions, do you want to look this one over too?

Reply to this email directly or view it on GitHub:
#2225 (comment)

Brian E. Granger
Cal Poly State University, San Luis Obispo
[email protected] and [email protected]

dhirschfeld · 2012-08-01T17:24:12Z

Without the module name twiddling I did in #2147 the --force flag doesn't work - I get LinkErrors after the source has been compiled the first time which means you can't annotate an already compiled cell.

This issue is present in master, but means that to recompile the source you either need to restart the kernel or alter the source in some way so its hash is different.

Also the link-args are missing from this version, but perhaps it's good practice to keep that a separate issue?

bfroehle · 2012-08-01T17:31:10Z

Can you share a copy of the traceback? I don't have that same problem in Ubuntu 12.04. I assume you are in Windows?

bfroehle · 2012-08-01T17:42:55Z

Also I cherry-picked your link-args option into this branch for your ease.

dhirschfeld · 2012-08-02T10:07:16Z

Yep, I'm using win32 Python 2.7.3 on an x64 box.

I put a copy of the traceback at https://gist.github.com/3235999/.

I think the pertinent information however is what is printed out in the console where I started the notebook (cython example) I used for testing:

c:/dev/bin/mingw32/bin/../lib/gcc/mingw32/4.6.1/../../../../mingw32/bin/ld.exe: cannot open output file C:\Users\dhirschfeld\.ipython\cython\_cython_magic_6eb34c8621696ead6242b637e2311da9.pyd: Permission denied
collect2: ld returned 1 exit status

Using process explorer I can see that python.exe is holding a reference to that pyd file so I think it can't be overwritten when you try to recompile the code with the same filename.

Just to be clear, to recreate the problem I run the cell magic once which works fine, then I run again with the --force flag and I get the permission denied error on the pyd.

bfroehle · 2012-08-02T15:11:41Z

Okay. Well the solution of adding a -force suffix to the filename isn't necessarily correct either, as that will then break on the third time around.

Perhaps we should os.remove the current .so (i.e., .dll ?) file first? Or will that be impossible, too? Or should we just not allow --force anyway?

dhirschfeld · 2012-08-02T16:31:35Z

I'd come to the same conclusion that it would only postpone the link error, but I was sure I had tested it and it worked. It turns out that was only because the args.force wasn't being respected when passed in as part of the context and I was picking up an old .html file from my .ipython/cython directory.

os.remove unfortunately gives the same access denied error:

import os
os.remove(r'C:\Users\dhirschfeld\.ipython\cython\_cython_magic_6eb34c8621696ead6242b637e2311da9.pyd')
---------------------------------------------------------------------------
WindowsError                              Traceback (most recent call last)
<ipython-input-4-5ab4fa3a02de> in <module>()
      1 import os
----> 2 os.remove(r'C:\Users\dhirschfeld\.ipython\cython\_cython_magic_6eb34c8621696ead6242b637e2311da9.pyd')

WindowsError: [Error 5] Access is denied: 'C:\\Users\\dhirschfeld\\.ipython\\cython\\_cython_magic_6eb34c8621696ead6242b637e2311da9.pyd'

A more robust solution which works well for me is:

if force or not os.path.isfile(module_path):
    if os.path.isfile(module_path):
        module_name += datetime.now().strftime('_%Y%m%dT%H%M%S')
        module_path = os.path.join(lib_dir, module_name+so_ext)

...with the appropriate from datetime import datetime at the top of the file.

bfroehle · 2012-08-02T17:00:56Z

Yes, timestamping is robust, but interferes with the desired caching properties. Perhaps only with --force as you suggest.

Let's think about this for a bit so we can settle on a good solution.

Certainly we could separate the cythonize and compile steps, which would remove some interference.

takluyver · 2012-08-02T17:12:28Z

Is there any way to force Python to let go of the pyd file so it can be overwritten? If we drop any references to the module, would it unload it?

bfroehle · 2012-08-02T17:40:53Z

@takluyver There isn't any sane way to unload modules in Python. In this case we could delete the module, hope that we've dropped all references, and then use cytpes to call FreeLibrary to unload the module. Too complicated to make robust.

What I'll suggest is the following:

if args.force:
    {add time.time() or similar to the key we hash to get the module name}

if args.annotate and not exists(html_source):
    {run cythonize with force=True, annotate=True}
elif not exists(compiled_module):
    {run cythonize}

if not exists(compiled_module):
    {run build_ext}

{load module, import all of its names}

if args.annotate:
    {return annotated HTML source}

takluyver · 2012-08-02T17:48:18Z

I had a feeling that might be the case. Never mind, carry on.

bfroehle · 2012-08-02T17:50:29Z

Also, for another issue, our code to determine the extension of the compiled file is not robust.

We should probably use something similar to Cython to figure it out instead.

dhirschfeld · 2012-08-03T15:20:08Z

Your proposal sounds good to me - I don't think there's any good way around having to recompile a forced compile. Given that it's a cell magic the time to (re-)compile shouldn't be to onerous.

bfroehle · 2012-08-03T18:26:14Z

Pinging @dhirschfeld and @ellisonbg. I think this revised version of %%cython is nearly complete and ready to go. Please test it at your convenience.

I've implemented --force by adding time.time() to the hashed key used to determine the module name, and the overall logic is similar to my proposal a few comments ago.

dhirschfeld · 2012-08-05T09:39:03Z

IPython/extensions/cythonmagic.py

+        if args.annotate:
+            html_file = os.path.join(lib_dir, module_name + '.html')
+            if not os.path.isfile(html_file):
+                need_cythonize = True


To get this to work for me I need annotate to imply force otherwise I get the file not found error below.

Simply setting args.force = True in the same block as need_cythonize = True solves the problem for me.

Cython completed successfully but the annotated source could not be read. [Errno 2] No such file or directory: u'C:\\Users\\dhirschfeld\\.ipython\\cython\\_cython_magic_6eb34c8621696ead6242b637e2311da9.html'

Thanks for the catch. Should be fixed by setting force=True below.

… timestamp). Otherwise it breaks in Windows since the existing .pyd file cannot be overwritten, and is otherwise ineffective since the module won't be able to be imported again.

bfroehle · 2012-08-05T20:51:11Z

Yes, of course. Fixed by setting force = True as you suggest.

ellisonbg · 2012-08-09T23:26:12Z

I tested this and looked over the code. My only comment is that the .c files linked to in the annotation HTML has a URL that is unreachable. For it to work it would need to have a URL that begins with files. But maybe that is outside the scope of this PR.

ellisonbg · 2012-08-09T23:26:37Z

Also, what is the status of the other comments on the PR. Is everything else ready?

…yway).

bfroehle · 2012-08-10T00:09:53Z

Yes, I think everything else here is ready. I just pushed a little commit to address the link to the .c file.

ellisonbg · 2012-08-10T00:11:22Z

I would go ahead with the merge.

Add "--annotate" option to `%%cython` magic.

fperez · 2012-08-10T03:22:19Z

Sorry I didn't pitch in during review, and this is something that can easily be done now... I think this is an awesome improvement, but it would be good to update the example notebook we have, showing these features to the users. I hope very soon we'll have things clean enough to start building our example notebooks as part of our documentation, and even in their current form they serve as great mini-tutorials of our various magics and extensions. So it would be great to update the cython one with a brief demo of these new capabilities.

Thanks again for the great improvement, I think the cython magic is now pretty much feature-complete, at least in the key functionality.

Add "--annotate" option to `%%cython` magic.

bfroehle added 4 commits July 31, 2012 12:47

Keep magic_arguments module vs function clear.

ac3a2e7

ctx is ignored by Cython > 0.13.

f61f74d

Don't unpack args.force for one use.

0b26a2b

Produce annotated cython source.

2330766

takluyver reviewed Jul 31, 2012
View reviewed changes

bfroehle reviewed Jul 31, 2012
View reviewed changes

bfroehle added 2 commits July 31, 2012 17:03

Use io.open ensure the file is read as UTF-8.

1eff338

Print message on stderr if annotated html cannot be read.

4fc8285

bfroehle mentioned this pull request Aug 1, 2012

Added --annotate option to %%cython magic. #2147

Closed

Added link-args argument to %%cython.

8d92014

bfroehle added 2 commits August 3, 2012 10:39

Clean up docstrings.

8ca2fb7

Determine the shared object extension more robustly.

a1daa02

Add --cplus option for c++.

3abe888

dhirschfeld reviewed Aug 5, 2012
View reviewed changes

Implement --force by choosing a unique module name (using the current…

af2449a

… timestamp). Otherwise it breaks in Windows since the existing .pyd file cannot be overwritten, and is otherwise ineffective since the module won't be able to be imported again.

Remove link to generated C/C++ file (since it would be a 404 error an…

0059bd8

…yway).

bfroehle added a commit that referenced this pull request Aug 10, 2012

Merge pull request #2225 from bfroehle/cython_annotate

2092555

Add "--annotate" option to `%%cython` magic.

bfroehle merged commit 2092555 into ipython:master Aug 10, 2012

bfroehle mentioned this pull request Aug 10, 2012

Add link to C file in Cython failures and -a for annotated html #1976

Open

bfroehle mentioned this pull request Aug 10, 2012

Update cython_extension.ipynb to include --annotate examples #2286

Closed

mattvonrocketstein pushed a commit to mattvonrocketstein/ipython that referenced this pull request Nov 3, 2014

Merge pull request ipython#2225 from bfroehle/cython_annotate

4bc7944

Add "--annotate" option to `%%cython` magic.

Uh oh!

Add "--annotate" option to %%cython magic. #2225

Add "--annotate" option to %%cython magic. #2225

Uh oh!

Conversation

bfroehle commented Jul 31, 2012

Uh oh!

takluyver Jul 31, 2012

Choose a reason for hiding this comment

Uh oh!

bfroehle Jul 31, 2012

Choose a reason for hiding this comment

Uh oh!

takluyver Jul 31, 2012

Choose a reason for hiding this comment

Uh oh!

bfroehle commented Jul 31, 2012

Uh oh!

bfroehle Jul 31, 2012

Choose a reason for hiding this comment

Uh oh!

ellisonbg Aug 9, 2012

Choose a reason for hiding this comment

Uh oh!

takluyver commented Jul 31, 2012

Uh oh!

bfroehle commented Jul 31, 2012

Uh oh!

takluyver commented Jul 31, 2012

Uh oh!

bfroehle commented Aug 1, 2012

Uh oh!

ellisonbg commented Aug 1, 2012

Uh oh!

dhirschfeld commented Aug 1, 2012

Uh oh!

bfroehle commented Aug 1, 2012

Uh oh!

bfroehle commented Aug 1, 2012

Uh oh!

dhirschfeld commented Aug 2, 2012

Uh oh!

bfroehle commented Aug 2, 2012

Uh oh!

dhirschfeld commented Aug 2, 2012

Uh oh!

bfroehle commented Aug 2, 2012

Uh oh!

takluyver commented Aug 2, 2012

Uh oh!

bfroehle commented Aug 2, 2012

Uh oh!

takluyver commented Aug 2, 2012

Uh oh!

bfroehle commented Aug 2, 2012

Uh oh!

dhirschfeld commented Aug 3, 2012

Uh oh!

bfroehle commented Aug 3, 2012

Uh oh!

dhirschfeld Aug 5, 2012

Choose a reason for hiding this comment

Uh oh!

bfroehle Aug 6, 2012

Choose a reason for hiding this comment

Uh oh!

bfroehle commented Aug 5, 2012

Uh oh!

ellisonbg commented Aug 9, 2012

Uh oh!

ellisonbg commented Aug 9, 2012

Uh oh!

bfroehle commented Aug 10, 2012

Uh oh!

ellisonbg commented Aug 10, 2012

Uh oh!

fperez commented Aug 10, 2012

Uh oh!

Uh oh!

Add "--annotate" option to `%%cython` magic. #2225

Add "--annotate" option to `%%cython` magic. #2225