From netbeans to eclipse for python development

Netbeans has been my favorite IDE for a very long time. It was unbeatable when I was still developing in Java, and when I switched to Python for my main work it also provided support for it. But lately the netbeans community seemed to have abandoned Python - at least, there is no built-in python support for any release of the 7.x series, and the available plugins are hackish and lack some of the functionality that was present still in 6.9.
So it was clear: Python has no future in netbeans. And this meant that netbeans had no future as my IDE ;)
I was inclined to actually believe the hype about eclipse - all eclipse users I know are very enthusiastic about telling you how great their IDE is. I already tried it several times, but never really got into it, always falling back to netbeans 6.9. With the recent release of Eclipse Juno I decided to give it another shot. 
Installation is no problem, you just have to unpack the zipped file and take care that the binary is in your PATH. Netbeans had this nice installer which creates directories for you and stuff, but that's not really an important point. Installation of plugins was a bit rough - Netbeans gives you a much smoother experience in this regard. But hey, it's a developer tool, and a Real Programmer should be able to figure out, right? ;)
What I didn't understand was the concept of the workspace, which eclipse forces you to choose on every startup. What the hell was so important about that workspace to justify asking me all the time which one I want to use? I never figured out until David Higgins explained to me at a recent workshop: The workspace is simply a place where your settings are stored, and has nothing to do with where your projects are located. For example, it allows you for example to use different settings when you're developing in Java and Python, or C++, for that matter. So I checked that "Don't ask anymore" box and was done with workspace selection  ;) .
I had to learn a few new shortcuts and customized other ones to be like Netbeans, which went smoothly once I discovered that hitting Ctrl-Shift-L twice gets me directly into the Keyboard Shortcuts dialog, which sports a nice quick filter widget to quickly identify any command you'd like Eclipse to perform. On the downside, I had to say goodbye to nice things like a new, separate console popping up for every process I start, or the more straightforward integration of SVN in netbeans. But then again, eclipse embraces Python while Netbeans just dumped support for it, so the decision to migrate is a no-brainer. And in addition I switched to Git anyway ;) 

AAAARRRRRRGH!! Matlab!!

Happy Matlab licensing trouble - yay!

We have 5 Matlab licenses in our lab. Originally, we bought them as "Concurrent licenses", that is, Matlab allowed us to have 5 instances of Matlab running in parallel, regardless which computer they are running on. At some point, Mathworks somehow transformed these licenses into "Designated Computer" licenses, that is, each license must be associated with a designated computer and will only run on that machine. Although this was an obviously bad deal, we didn't care too much about that change back then, since we were busy doing more important stuff than caring about licensing issues.

Anyway, Matlab is a dying species in our lab since most of us are using Python for scientific computing, except for a few legacy scripts. But every now and then, I need to run one of those legacy scripts.

I do much development on my laptop, but for numbercrunching I use our compute server. Hence, I need my computing environment on both machines, although not necessarily at the same time. I had one of these designated computer licenses, and thanks to Mathworks' provident care, I was able to deactivate and reactivate them over the web when switching between computers. So I changed the designated computer a few times between those machines. Today I wanted to change again, but Mathworks wouldn't let me:

"No more machine transfers available for this license."

WTF?

OK, you're forcing me to port even my old scripts to python. Pity you. I spent already too much time struggling with licensing issues - time which I would much more like to spend on research. Goodbye Matlab.

Towards fast scientific python

Python seems to come of age in its role as an universal language for scientific computing. It already has a good standing in the computational neuroscience community. The Neural Ensemble project gathers some initiatives that use Python as the primary language for neuronal simulation and data analysis. Large simulator projects like Nest and NEURON adopted Python as their primary command language already a few years ago. The core of those simulators is still written in C/C++, which delivers good performance, but leads to interfacing issues with the command language. Those issues can be addressed by clever software design, but a pure-python implementation of a simulator is much more convenient regarding maintainability and extendibility. The problem is, that pure Python will lag behind the speed of compiled languages like C/C++ by an order of magnitude.

The Brian simulator is designed to be a simulator written entirely in python. To cope with the speed of C/C++-based simulators, Brian can generate compiled code from the python network model. This code can also be compiled for graphics processors (GPUs), which promise high speedups for computational problems that can be parallelized efficiently. The Brian developers describe how to do just that in their article on vectorized algorithms for neuronal simulations, which is one of my current favorite papers.

Today, and that was the initial motivation for this post, I came across the announcement for the new version of Theano, a compiler for evaluation mathematical expressions on CPUs and GPUs. I haven't tried it out yet, but it looks definitely promising. But the really interesting fact is that there is vivid development toward making Python not only an ubiquitous language for scientific computing (a goal which has largely been achieved already), but also an alternative in terms of performance to established software packages.

Without licence fees, and fully open source.

Have your python toolchain in $HOME

In my previous post I explained how to install scipy from source on openSUSE.

What makes it particularly nice is that I can now carry most of my toolchain in my $HOME. I make python include modules from within my $HOME by setting $PYTHONPATH to something like

/home/micha/mypython/lib64/site-packages

. The good thing is that you can install any python package in your $HOME by using the --prefix option to setup.py:

python setup.py install --prefix=$HOME/mypython

For even more python goodness I tell easy_install to put everything there by having a file called .pydistutils.cfg in my $HOME with the contents

[install]
prefix=/home/micha/mypython

So everytime I easy_install a package, it is automatically put into my $HOME directory. That makes it much easier to reinstall or upgrade the system. Since most python-related stuff is now in my $HOME and not in the system, rebuilding my python-toolchain basically consists of installing python and distutils. Isn't that great :)

Update: I learnt from Brandon Rhodes that virtualenv will set up everything for you automatically. Awesome python goodness.

Installing scipy 0.7.0 on openSUSE

I had to rebuild parts of my toolchain because I messed up my OS and needed to reinstall. In the process of searching for nice numpy and scipy packages for openSuSE (which failed), I discovered that now its actually possible to do

easy_install numpy
easy_install scipy

(provided that you have the python-distutils package installed). That's great. But... it doesn't work! At least not on openSUSE. I could convince numpy to install somehow. I don't remember exactly, I think I at least needed to install gfortran, maybe also blas and lapack from the scientificlinux-repository in the build service.

For scipy then it was a little bit more work. It kept complaining that it did not find BLAS and LAPACK, even though I edited numpy's site.cfg file so that it should be aware of the location of the shared libs.

It turned out that to install scipy I had to:

1. Download BLAS sources and unpack them, e.g. to $HOME/Apps/BLAS
   
2. edit make.inc in that directory, changing the FORTRAN line to

   FORTRAN  = gfortran

   
   
3. build BLAS by calling make in the BLAS dir
   
4. DL and unpack LAPACK to $HOME/Apps/lapack-3.2
   
5. edit make.inc.example in that dir, changing the BLASLIB line to

   BLASLIB = $(HOME)/Apps/BLAS/blas$(PLAT).a

   and saving that file as make.inc
   
6. build LAPACK by typing make in the lapack dir.
   
7. Download scipy, unpack it and start the build process:

   python setup.py install

   
   

That takes quite a while. It seems it builds LAPACK and BLAS again, so maybe you don't have to build it first, but I guess you need at least to make the appropriate modifications to the respective make.inc files. Comments on that are welcome.

But most important: it finally worked :)

Update: At least I thought it worked. It didn't :D Problem was that import scipy produced a symbol not found error. Maybe adjusting ldconfig's path could fix this, but I don't have time to look into this. Installed numpy & scipy from the scientificLinux repo (link see above).

Netbeans + python = happy happy

Netbeans is now available with python support as an early access (read: beta) build. Install went smoothly, it just updated my existing netbeans 6.5 install with the python capability. Hassle-free, netbeans-style. Great!

At first glance python support looks great - I can create projects from existing sources, it allows code navigation by function names, and all that other netbeans goodness. During the next few days I'll give it a shot - I'm looking forward to finding out whether it beats Eric4...

Design patterns in python

Found a blog post that summarizes several links on how to implement design patterns in python. Have to check it out once I finished my abstract for Neurocomp '08!

openSUSE 11.0 (and why I'll wait with the upgrade)

I tried Beta 1 and Beta 2 of the newest SUSE Release. Beta 1 was already very streamlined for a Beta, and Beta 2 put something on top.

On the upside, the overall feel is very snappy and responsive. Suspend to disk and RAM is faster than ever on my Thinkpad T60, and the kernel supports more energy saving fanciness. They even polished KDE4 such that it can almost replace trusty KDE3. But, alas, only almost...
On the downside, KDE4 has still some issues, like Drag-n-drop not yet working with files on the desktop. That's a minor issue, I know, but then again what's a full blown desktop environment for if you can't drag'n drop? But since they also ship KDE 3.5.9, this wouldn't keep me from upgrading.

More important, I couldn't get my T60's WiFi to work; it's an Atheros 5418 which needs madwifi drivers from svn, and they don't seem to work with 11.0's NetworkManager. I didn't even dare to mention that on the 11.0 mailing list since I expect the answer would be something like "This Madwifi release is not part of the distribution, so we don't support it" (which has a point, I admit). In addition on the hardware side, the radeonHD driver that is used by default for my Mobility X1300 doesn't like compositing in KDE4.

Finally, openSUSE still has no numpy/scipy in the standard repository, and the "science" repo which provided numpy and related stuff hasn't upgraded to 11.0 yet.
Taken together, as much as I'm looking forward to using KDE 4 and saving battery with the modern kernel, as long as my WiFi's not working and numpy/scipy support is not there I'll have to force myself not to upgrade my system... and I already know that I'll have a hard time doing so, because from what I have seen in the Betas, openSUSE 11.0 will kick ass :)

More Emacs + Python goodness

Ryan McGuire posted ver informative entry on how to improve your emacs + Python experience. And then, he links to a cool screencast demonstrating Emacs's capabilities as an IDE. Nice!

Python IDE summary

I'm still looking for the most comfortable way to edit python source. At the moment I use Emacs & Rope, which works well. But still, it doesn't come close to what Netbeans provides for Java editing. In particular, I miss universal code completion, readily available documentation (think tooltips) and quick code navigation.

Fortunately I'm not the only one searching for a better way to edit python. Jonathan Ellis summarized an "IDE smackdown" by a LUG that has some interesting points. Nice!

Python + IDE smackdown

Python + Emacs, contd.

On my neverending voyage towards the ultimate python IDE I stumbled across another nice tool: ropemacs! Ropemacs is an emacs binding to the rope library for refactoring python code. It also uses Pymacs, which aims towards employing Python as an extension language for emacs.

Installing Pymacs, rope, and ropemacs was fairly easy thanks to the excellent installation instructions - Kudos to the developers of rope and Pymacs! Good instructions are not a common thing for emacs extensions, it seems.

I think ropemacs will accompany my journey for the next few thousand lines of code, and we'll see how well we get along... I'll keep you posted! :)

7 reasons why test-driven development rocks

Did I ever mention that test-driven development rocks? No? Well, it sure does :)
Here are my top 7 reasons:

• Know when you're done. Back when I didn't have tests prior to implementing functions I never was sure when the implementation was actually done. There was always this unspecific feeling of having forgotton something...


• Reward. Everytime a test passes, I get this satifying feeling of having accomplished something :)


• Efficiency. Writing tests forces me to think about solutions instead of thinking about problems.


• Better software design. Writing tests before implementing the actual method forces me to think about what exactly I expect the method to do. Methods that are awkward to test are awkward to use, and require redesign.


• Refactoring. You simply cannot refactor your code without having tests in place. But if you have tests, refactoring is a breeze.


• Confidence. Having tests in place makes me confident that my code does what I want it to do. Well, most of the time... But when I encounter a bug I add another test and regressions won't hurt me.


• Example code. Test functions are concrete examples how to use a given function in my code.

I would have liked to have 10 top reasons, but couldn't figure out the remaining three... If you have one, please comment!

Neural Ensemble

The Neuralensemble project brings together several efforts to make simulations of neurons easier. Currently, I spend most of my time working with the tools they created. Since a few days I have an account on their TRAC server, so I thought this is a good time to blog about it.

The project which got me into neuralensemble is PyNN , which provides a unified front end to several widely used neural simulators, e.g. NEURON, NEST and PCSIM. PyNN greatly facilitates setting up neural networks, because it provides an abstraction from simulator-specific languages. As the name suggests, it's written in Python, and lets you specify the network in Python, too.

For experiment design and analysis of simulated data there is Neurotools. It's written for PyNN (in Python), but can also be used independently from it. One of the most promising design goals is its ability to control large experiment sets, e.g. to systematically investigate the impact of certain parameters on network function. I'm sure every scientist who does computational modeling has written code to deal with this tasks at least once. Having a common API for this is a great thing.

Neuralensemble is a dynamic community which is still very open to all sorts of contributions. So if you're into modeling of neural networks, you should definitely give it a shot!

Python IDE revisited

Today Komodo edit got released under various open source licenses. Since I'm still looking for a decent Python IDE I could not wait to try it.

My very first impression is a very good one! It looks like Eclipse, but I can live with that... especially because it lets me use Emacs keybindings. Killer feature :). Beyond that, code completion seems to do a good job in Komodo Edit, something I never got around to set up in emacs.

Of course, I still need to check out things like code navigation, refactoring and other IDE goodies... soon to be reported here!

Yet another python primer

Another python tutorial, this time specially for life science researchers. Published back in November last year, but it took me until today to run across it. It seems to be focused towards bioinformatics, at least from what I can tell by quickly going through it. Might be useful for students entering the field, not knowing which lagnuage to learn.

Citation: Bassi S (2007) A Primer on Python for Life Science Researchers. PLoS Comput Biol 3(11): e199

Python + Emacs, cont.

I came across this blog post which states some very interesting additional resources for python development in emacs. Very cool!

The quest for a Python IDE - Back to good old emacs

Since I started using Python last year, I've been looking for a nice IDE that supports that language. When I was programming in Java, Netbeans made programming REAL fun. I was really missing such a powerful tool for Python.
Not that I haven't tried a few. First, there is jpydbg for netbeans. Or at least it seems to be, since I never managed to start a project with it. It has been written for jedit, and it does not really integrate into netbeans' look and feel, let alone it's usabilit concept.
Then there was Eric. It made a good impression, but unfortunately, it stopped working on my machine for some obscure error relating to QT4 libs not being found. This ain't Eric's fault, other Qt4 also won't work under my account.... I have no clue which config file causes confusion here. But still, I can't use it.
So I went on to Eclipse. As an e- netbeans user, I was quite reluctant to take that step, somehow it felt to me like switching from Linux to Windows again ;)... Nevertheless, I've been using it for about half a year now, but still haven't got comfortable with it. I'm sure it offers every tool a programmer might probably want at some point in his life, but I just haven't managed finding out how to open the toolbox... I couldn't even manage to update an SVN repo from the IDE! Don't get me wrong, I'm sure there is an easy way how to do this, but I just couldn't figure it out. And I don't want to spend my time digging through docs and forums and mailing lists when I really want to spend it programming. Especially if the solution to my problem is only an "svn update" away... on the command line.
And this was the point when I remembered my good old friend emacs. Long time back, when desktop envirenments under Unix still needed their diapers changed regularly, I used emacs for all programming related tasks. And how I liked it. And of course, today it comes with a python mode, with syntax highlighting and all bells and whistles. Who needs more?
Well, actually I need more... :) I want version control, refactoring tools, auto completion and quick navigation through large source trees. And naturally, emacs seems to offer it all. There's psvn.el for SVN. Navigation has always been the domain of Speedbar (M-x speedbar). For refactoring, there seems to be BicycleRepairMan. Haven't used it yet, but voices all over the web praise it. And finally, for auto-completion Pymacs claims to do the job. This will be the next thing to try for me.
So for now, I'm happy again with emacs!