License Creative Commons Attribution Non-Commercial Share Alike 3.0-US
Keywords
distribution (3) distutils (1) metadata (3) modules (1) packages (1) prerequisites (1) python (33) setup.py (4) setuptools (4) tips (5) tricks (7)
Permissions
Viewable by Everyone
Editable by All Siafoo Users
Hide
Meet people who work on similar things as you – get help if you need it Join Siafoo Now or Learn More

Use setup.py to Deploy Your Python App with Style Atom Feed 0

Beginning with the simplest possible setup.py file, learn how to deploy your Python application the right way. After you learn the basics, discover how to create virtual packages, set up automatic build tagging, and deal with conflicting distributions, alternative prerequisites, and weird errors.

Question: You've just written an awesome Python app, and you want to share it with the world. What do you do?

Answer: You use setuptools and distutils to deploy your application. [1]

Honestly deploying your Python application might be the worst documented common task in the world. There is plenty of information (some of it outdated), but nothing that seems to describe what you typically need to do. Hopefully this tutorial will fill that void.

1   The Basics

1.1   Definitions

module
The most basic unit of code reusability in Python. What you usually import. Typically a single file.
package
A group of modules and other packages. Typically a folder on the filesystem, distinguised by an __init__.py file inside (empty or otherwise - anything you want to import straight from the package should go in here).
distribution
A package or group of packages meant to be installed at the same time. What you are releasing here. Source distributions are usually represented folder, called a 'root package', with a setup.py file inside.
site-packages directory
The place in your filesystem where all of the installed distributions live.

1.2   File Layout

You probably want to set up your distribution like this one. If you'd like, you can download it and follow along.

  • root_package

    directory name doesn't matter

    • setup.py

    • mypackage

      the folder containing your Python code; should be the name you specify to 'packages' below

      • __init__.py

        tells Python that this directory is indeed a package; can be blank

      • mysource.py

        your actual code; name doesn't matter

If you're actually in Root Folder you can enter the Python console and access code from source.py:

# 's
1from mypackage.mysource import some_function_in_mysource()
2some_function_in_mysource()

Once your package is installed you can do that from anywhere!

1.3   setup.py

A minimalist setup.py file looks like this:

# 's
1from setuptools import setup
2
3setup(name='MySoftware',
4 packages=['mypackage']
5)

This code tells setuptools about your distribution. It is named MySoftware, which will be the file name of any egg or tarball you make, and the code can be found in the package 'mypackage' (a folder with an __init__.py in it relative to this script).

If you have a package like above, this is all you need to install a basic distribution:

# 's
1python setup.py install

Go to a different folder, run python, run the 'from mypackage...' statements above, and wha-la, you'll see you can access your software from anywhare on the system.

1.4   Install vs. Develop

When you're installing a distribution, you can install it in develop mode instead with python setup.py develop. Using this mode, a special link file is created in your site-packages directory. This link points back to the current folder or 'root package'. Any changes you make to the software here will be reflected immediately without having to do an install again.

When you're ready to install for good do a real install.

2   Multiple Packages

You'll notice that our distribution above was pretty simple: we only had one package. In real life, anything you're thinking about installing has several, probably several packages deep.

You need to specify to the setup command all of the packages you want to be included as a part of this distribution. They don't all have to have the same base package (here, "mypackage"), but this is typical.

You can always list all of the packages by hand:

# 's
1packages = ['mypackage', 'mypackage.subpackage', 'mypackage.othersubpackage', 'mypackage.subpackage.subsubpackage']

As you can see you need to list every directory/package in your code. This is a pain and I guarantee you'll mess it up somehow, I did.

A much better option is to use the find_packages function:

# 's
1from setuptools import setup, find_packages
2setup(name='MySoftware',
3 packages=find_packages()
4)

This function searches the current directory, recursively looking for any packages (subdirectories with __init__.py files). Run it yourself if you like. It takes two keyword arguments:

  • exclude, a list of packages to exclude (for example 'mypackage.excludepackage')
  • where, an URL-style path (that means slashes) to scan for packages if not the current directory

Just use it. It is smarter than all of us.

3   Distribution Metadata

This seems like a good time to mention all those boring things like version number, author, etc. You pass these as arguments to setup:

# 's
1setup(name='MyPackage',
2 version='0.1a',
3 description='This is some stuff that I wrote',
4 author='Nobody Special',
5 author_email='dev_null@example.com',
6 url='http://www.example.com/MyPackage',
7 packages=find_packages()
8)

I think these are self-explanatory, except for maybe url, which is the home page for the package. If you're interested a list of all the possible metadata can be found in the distutils documenation.

4   Requiring Prerequisites

To require other distros as prerequisites to yours, you can use the install_requires argument. You provide it a list of strings, each string is a distribution name and optionally, a relational oprerator and version number. For example:

# 's
 1setup(
2 name='MySoftware',
3 packages=find_packages(),
4 install_requires=[
5 'Pylons>=0.97',
6 'SQLAlchemy==0.5',
7 'Genshi<=0.4',
8 'tw.forms!=1.9.2',
9 'Elixir'
10 ]

The name here is the name of a distribution. Specifying no version number like the 'Elixir' example means that any version of the package is okay. If none is found on the current computer the latest will be downloaded. Specifying a comparison operator and a version number means that your software will only work with that version or range of versions. For example, here our software will only work with a Pylons version greater or equal to 0.97.

Let me also note that while some of the distutils documentation seems to suggest that you can require packages with the keyword argument 'require', this doesn't actually seem to do anything.

4.1   How Versions Work

It's obvious which numeric versions are greater or less than to each other. But what happens when packages are tagged non-numerically? setuptools decides that tags alphabetically before the word final (and a few special tags), separated from the numbers by nothing or a period, are considered to come before the plain release. Words that come after 'final', or anything separated by a dash, are considered to come after the plain release.

For example, 1.0.a.dev1 < 1.0.a < 1.0b < 1.0.c < 1.0dev1 < 1.0e < 1.0 < 1.0port < 1.0-mod < 1.0.1. Note that adjoining letter tags need to have a dot between them to be recognized appropriately.

For more information visit the setuptools documentation.

4.2   Downloading Prerequisites

When you install a package with prerequisites, setuptools will automatically search PyPI, the python package repository, to find what you are looking for.

If your software relies on software that is not in PyPI, like other distributions you have created, you can also tell setuptools to check another repository.

You'll need to create a file called setup.cfg in the same directory as setup.py. [2] Add this to your setup.cfg:

[easy_install]
find_links = http://path.to.my.website.com/

You don't need to have this be a special repository; setuptools is smart enough to find download links on a regular FTP or HTTP page, assuming you've labeled them normally (something like 'MyPackage-0.1.tar.bz2').

5   More Advanced

Here are a few more advanced tricks you might want to try...

5.1   Automatic Build Tagging

If you're using subversion, you can have setuptools automatically tag each of your builds. Create or open your setup.cfg file [2] in the same directory as setup.py and add the lines:

tag_build = .dev
tag_svn_revision = true

Now when you install something it will automatically append the tag '.dev' and the subversion revision number. For example if your version number is 0.1, you will get an egg or tarball called 'MySoftware-0.1a.dev-r1234'. Or something like that. This does seem to cause problems on some computers.

5.2   Multiple Distributions, One (Virtual) Package

Sometimes you want to release multiple distributions for the same project. Maybe you have a client and a server distribution, or maybe you have a lot of optional plugins. You could have them each installed with a different base package name, but wouldn't it be cool if they appeared to be in the same package?

To do this you need to specify a namespace_packages argument in each of the distribution's setup.py files. This is a list of the packages in this distribution that will act as 'virtual' packages.

# 's
1setup(name='MySoftware',
2 packages=find_packages(),
3 namespace_packages=['mypackage']
4)

In the __init__.py file for each of the 'virtual' or namespace packages, add the lines:

# 's
1import pkg_resources
2pkg_resources.declare_namespace(__name__)

The __name__ variable is a magic variable set to the name of the current package or module. You're letting Python know that you intend to use this package as a virtual package.

Warning

Only one __init__.py file will be loaded for each virtual package, and which one is not determined until runtime. Therefore it is important to either put nothing but these lines in the file or to have each file be exactly the same.

5.3   Specifying Conflicting Packages

Currently there is no native way to tell a user that your distribution conflicts with another one on the system. How would this happen? Perhaps you have a stripped down version and a full version, and having both installed might cause problems. In any case the best solution I have found is to check for the conflicting distribution and alert the user. You can add this to the top of your setup.py file:

# 's
 1import sys
2from pkg_resources import require, DistributionNotFound, VersionConflict
3
4try:
5 require('ConflictingDistribution')
6 print
7 print 'You have ConflictingDistribution installed.'
8 print 'You need to remove ConflictingDistribution from your site-packages'
9 print 'before installing this software, or conflicts may result.'
10 print
11 sys.exit()
12
13except (DistributionNotFound, VersionConflict):
14 pass
This snippet is licensed under the Public Domain license

In this snippet, we use the pkg-resources require function to attempt to load a certain distribution. You could also load, and error on, only a certain version of the package using the same syntax as before, 'ConflictingPackage<1.0' for example.

5.4   Requiring One Package OR Another

Sometimes your software might work equally well with either of two packages. You can use a similar method as above to require one package OR the other:

# 's
 1from setuptools import setup, find_packages
2from pkg_resources import require, DistributionNotFound
3
4# Load the PreferedDist only unless the LessPreferedDist is already installed
5try:
6 require('LessPreferedDist')
7 required='LessPreferedDist'
8except DistributionNotFound:
9 required = 'PreferedDist'
10
11setup(
12 name='MyPackage',
13 packages=find_packages(),
14 #...
15 install_requires=[
16 required,
17 'SomeOtherPackage'
18 ]
19)
This snippet is licensed under the Public Domain license

This code checks to see if your less preferred distribution is already installed. If it is, it is set as the required distribution, and if not your more preferred package is set as the required distribution.

The result of this is that if either is installed, that package will be set as the required distribution, and installation will proceed. If neither is, the more preferred distribution will be set as the required distribution and be installed. Whew.

6   Errors You May Get

For the longest time, I was constantly getting an error when I tried to do a python setup.py develop or install. This frustrated me for many months, and I've finally figured out the reason (but not the solution): it is subversion's fault.

This is the error:

File "build/bdist.linux-i686/egg/setuptools/command/sdist.py", line 98, in entries_finder
NameError: global name 'log' is not defined

It turns out that if you have setup automatically tagging your builds with an tag_svn_revison (check your setup.cfg), it does some magic with svn that causes it to crash. The 'solution' is just to rename your .svn directory and to move it back after running setup. I've written a little alias in my .bashrc file to do this for me. Or of course you could just get rid of the build tagging.

7   Conclusion

This is just an introduction to the power of setup.py. You can create tarballs, binary distributions, eggs, or even submit distros to PyPI using setup. For more info see the distutils docs

If you think I've missed anything, have any better ways to do anything here, or know of any good references on the internets, let me know. Or, even better, you are welcome to update this article yourself. That is the point of Siafoo after all. : )

8   References

setuptools docs

distutils docs

[1]setuptools actually builds on top of distutils, but for our purposes they are the same thing.
[2](1, 2) The setup.cfg file should be in the same directory as setup.py. It is in INI format, meaning that you have key = value pairs in sections called 'stanzas', topped by a title of the form [title].