Adding testing to pip

Python packaging has been in a bit of a state of disarray for as long as I’ve been using it. Pip has come along to make installing python packages easier. It has a lot of features that are useful, but they have been talked about in many other blog posts.

Today I want to talk about adding testing to pip. If you are familiar with the Perl community, then you probably know about CPAN. It is basically Pypi for Perl. They have a command, called cpan, which allows you to install packages in a similar way to pip.

One of the steps that a package goes through before being installed on your system is that the tests are run. This allows you to know if the package that you have installed is actually going to work on your system. It may be broken on your platform, or you may be missing a library that it thought you had. Currently, pip has no way to test packages when they are being installed. I went looking for a way to make that happen.

It should be noted that pip is based on setuptools. Setuptools is what parses and understands most of the logic inside of your setup() function in the setup.py for your project (which you have, right?). Setuptools has an option called test_suite, which allows you to run setup.py test on your package, and have it run the unit tests. This is done by calling whatever python function is defined in test_suite.

I added the ability for pip to run setup.py test on a package that it is installing. It is executed by running pip install --test <package>. The implementation is on a ticket on bitbucket, and in a repository on github.

If you want to check it out, go ahead and clone my repo and check out the test_command branch. Then you can simply run

python pip.py install --test wsgiref

for an example package. If a package doesn’t have a test_suite, then it simply doesn’t run anything.

Note that if the tests fail, it doesn’t impact the installation of the package. The python community’s tests aren’t quite good enough,and almost any Django package you try this on will not have any tests. I wrote about how to add testing to your django package, but the process is long and involved. I’m working to improve the situation for Django and hopefully having the ability to run tests in the package management tool will spur people to add testing ability to their setup scripts!

Nose makes this really easy, by simply adding test_suite = 'nose.collector' to your setup.py, nose will run your tests correctly. This is the level of support that I am hoping to implement for Django.

On a side node, I talked to Ian Bicking about this, and he suggested writing the test command as a separate command, so you would be able to do pip test wsgiref, if it was installed. This has some other problems, which I will talk about after I have implemented this functionality.

I would love to hear feedback, or if anyone has ideas for improving testing in the python and django communities. I have lots of ideas, and I will be writing more of them up over the following weeks.



Hey there! I'm Eric and I work on communities in the world of software documentation. Feel free to email me if you have comments on this post!