Graphical code profiling in IPython

Posted on

As you might know, IPython is the new Matlab. The great thing about using IPython notebooks for scientific data analysis is that Python is very extensible. I have been adding many little helper functions to my environment that make developing a lot easier.

Here’s a little helper function that evaluates a given Python statement and turns it into a graphical summary of where the largest portions of CPU runtime are spent. The output looks like this (click for larger view):

from nbhelpers import profviz
statement = 'for i in range( 1000 ): svd( dot( standard_normal( ( 10, 200 ) ), standard_normal( ( 200, 10 ) ) ) )'
profviz( statement )

Inline graphical profiling for IPython
Is it not beautiful? And this sits right in your notebook, as a zoomable SVG – no need to open a separate window.

As a side effect, it puts two files in your working directory, ‘profile.pstats’ and ‘profile.pstats.svg’, the former containing the ‘raw’ profiling data and the latter containing the SVG itself. If you figure out a way to do this without writing temporary files to disk, let me know (although I think it may be useful to save the profiling data for later, which is why I made the filename a parameter).

You need the following programs installed on your machine (should be included in most Linux distributions):

  • graphviz
  • gprof2dot
def profviz( statement, filename = 'profile.pstats', sortkey = 'cumu' ):
    from IPython.display import display, SVG
    from cProfile import run
    from subprocess import call
    from pstats import Stats
    run( statement, filename )
    Stats( filename ).strip_dirs().sort_stats( sortkey ).print_stats()
    call( 'gprof2dot.py -f pstats "{0}" | dot -Tsvg -o "{0}.svg"'.format( filename ), shell = True )
    display( SVG( filename = filename + '.svg' ) )

I put this function into a file called nbhelpers.py, and then usually source all the helpers by

from nbhelpers import *

at the beginning of each notebook. You could also automate this by modifying the IPython configuration file.

Leave a Reply

Your email address will not be published. Required fields are marked *