License Creative Commons Attribution Non-Commercial Share Alike 3.0-US
Keywords
hacks (1) HTTP errors (1) internal redirect (1) middleware (5) Paste (1) Pylons (10) setup.py (4) setuptools (4)
Permissions
Viewable by Everyone
Editable by All Siafoo Users
Hide
Writing an article is easy - try our reStructured Text demo Join Siafoo Now or Learn More

Pylons Tips, Tricks, and Hacks Atom Feed 1

Learn how to do new things in Pylons. Create one app to be middleware for others, without overwriting the configuration of one of them. Make internal redirects. Configuring multiple apps in one file. Solve odd errors. And add your own tricks.

Pylons is a great way to tie together a bunch of different open-source code. But that is exactly the source of much of its frustration: it is sometimes hard to figure out even what package you need to be looking at to fix a bug or get something done. Documentation as well is fragmented, there are good basic tutorials but for a lot of advanced features you have to dive into the generated docs.

Following are a couple of tricks I've learned in the last year or two of Pylons programming. Feel free to add your own!

1   Quick Tricks

We'll start with some easy things that are hard to figure out how to do. I think that makes sense...

1.1   Internal Redirects

Doing an internal redirect is almost trivial, but it took me over a year of Pylons programming to run across how to do it. You'll need to add some middleware to your middleware.py file. Place it where it says 'CUSTOM MIDDLEWARE HERE', between the Cache middleware and Error Handling:

# 's
1from paste.recursive import RecursiveMiddleware
2app = RecursiveMiddleware(app)

Now all you have to do to do an internal redirect is:

# 's
1from paste.recursive import ForwardRequestException
2raise ForwardRequestException('/url/to/forward/to')

And if for some reason you want to run a request on your own application and get the result:

# 's
1from pylons import request
2result = request.environ['paste.recursive.include']('/path/to/run/request/at')

For more information, see the Paste documentation.

1.2   Multiple Apps, One File

Did you know that you can configure multiple Pylons apps in one .ini file? This could be useful if you are making one app middleware for another or just want to have less configuration files.

Each app just has to have a named stanza like [app:some_name], the default one is 'main' but you can have as many as you want:

# 's
1[app:main]
2# Some configuration
3
4[app:other_app]
5# Some other configuration

To run setup or start any of the apps you just add '#app_name' to the .ini file name in the command. Specifying '--name=app_name' does the same thing. If there is no app name specified, the 'main' app is loaded. For example:

# 's
1paster setup-app config.ini#other_app
2paster serve config.ini --name=app_3
3paster serve config.ini

The first command sets up the app specified in config.ini as [app:other_app], the second serves [app:app_3] and the third serves [app:main] as you've learned to expect.

You can do cool things with this like load initial data for your middleware app from the host app's .ini file. You'd setup your initial data imports in the middleware's websetup.py and then run paster setup-app config.ini#middleware_app.

2   Making One App Middleware For Another

Except for one (rather large) problem, turning a Pylons app into middleware is easy. All you need to initialize the middleware when the host is initialized, and then on each request to check whether the host or the middleware should be handling the request.

And don't worry, I tell you how to solve the problem too.

2.1   Initializing the Middleware App

You use the PasteDeploy function loadapp to load a Pylons application into memory. This is exactly the same way your host application is loaded when you do a paster serve:

# 's
1from paste.deploy import loadapp
2middleware_app = loadapp(loadapp('config:/path/to/config/file', name='middleware_app')

This code loads whatever app is specified by your config file under the [app:middleware_app] stanza. Your best bet is to add this stanza to the same configuration file as your host app:

# 's
1[app:main]
2use = egg:host_app
3# Other configuration options for your host app, this section already should exist
4
5[app:middleware_app]
6use = egg:middlware_app
7# Other configuration options for your middleware app

If you have access to the global configuration data ('config' in the default Pylons middleware.py), you can access this configuration file with config['file'].

2.2   Handling Requests

To handle requests, you wrap the host app in a callable. This callable gets called instead of the host app, and can decide whether to pass control on to the host app or deflect it to the middleware app. We'll use a class with a __call__ method; here we have any URLs that start with base_url handled by the middleware app instead of the host app.

# 's
1def __call__(self, environ, start_response):
2 if environ['PATH_INFO'].startswith(self.base_url):
3 return self.middleware_app(environ, start_response)
4 else:
5 return self.host_app(environ, start_response)

Just doing this, however, is going to give you some extra trouble with routing and URL creation in your middleware app. You can either add the base_url to all of the routing and all of the URLs, or you can do something like this before calling the middleware app:

# 's
1environ['PATH_INFO'] = environ['PATH_INFO'][len(self.base_url):] or '/'
2environ['SCRIPT_NAME'] = self.base_url

PATH_INFO is the actual URL that the application sees. We're chopping off the base_url here so the application treats the base_url as its root, '/'. SCRIPT_NAME is a magic variable that the Routes and Pylons use to construct URLs for you when you do a redirect_to or url_for. It tells them the base path of your app (where it lives); this path is prepended to any generated URLs. (You'll still have to prepend it manually if you write out any URLs yourself.)

2.3   But Wait, There's More!

You'd think it really would be this easy, wouldn't you? However some 'design choices' on the part of Paste and Pylons make this a little more difficult than it could be. And by a little I mean a lot. And by a lot I mean "Are you ready to hack?"

2.3.1   The Problem

The Problem is that Pylons has this variable called pylons.config. It stores everything about your pylons app. And for some reason instead of being bound to your specific app instance in some way, it is proxied by process id. In other words, the same config variable is used everywhere for one process. Yikes.

Oh and it's not a single variable either, it's a stack of configs. When you start up Pylons it pushes a default one onto the stack. When a new Pylons application is initialized, it pushes its config onto the top. And when you start a second app, it pushes its to the top. Unfortunately accessing pylons.config just gives you the top config in this stack, which is that for the most recent app initialized. Effectively the new app overwrites the config of the old app.

It actually appears the the pylons.config acts this way when you're not in a request. In a request, all of the Pylons data seems to be automagically located by the RegistryMiddleware, as long as that middlware has access to the right data in the first place.

When you add your middleware app in middleware.py, you have to initialize it. This initialization occurs in the middle of setting up the middleware for the host app, and effectively overwrites all the values in the pylons.config. The RegistryMiddleware is one of the last middlewares to initialize, and so receives and apparently stores the wrong config. It stores the config of the middleware app FOR the host app.

When you execute a request on the host app, the RegistryMiddleware pops out what it got in – the wrong configuration information. So you have the wrong data, and your host app either doesn't work or crashes.

2.3.2   The Solution

So what is the solution? The only one that works in all situations (setup, running, testing) is also a huge hack. Once your middleware App's initialization is done, it doesn't need its config on the stack anymore. Its RegistryMiddleware already has its config, and you're not planning on running any setup/tests/whatever that access its config directly from the host app (hopefully). So we just pop it off the stack.

Note that this code is written for Pylons 0.9.7 and is released into the Public Domain:

# 's
1import pylons
2
3process_configs = pylons.config._process_configs
4this_conf = [conf for conf in process_configs if conf['pylons.package'] == 'name_of_my_middleware_package'][0]
5process_configs.remove(adjector_dict)

You can also just do process_configs.pop(0), but somehow I don't trust that quite as much. Here I know what I am popping. After that everything works hunky-dory (or something like that). The host application's RegistryMiddleware gets the top config on the stack, which is the one for the host application, and each app's RegistryMiddleware provides the right configurations to each app when requests occur. Setup and tests seem to work fine too.

2.4   Putting It All Together

We want to be able to add the middleware in the middleware.py file; the best place is (not surprisingly) where it says 'CUSTOM MIDDLEWARE', right above the if full_stack stuff:

# 's
1app = MyMiddleware(app, config, '/my_base_url')

This can instantiate a class, which will load and initialize the middleware app. The class wraps and replaces the host app. When the 'app' is later called, we are really calling the middleware class, which can then call either the middleware app or the host app:

# 's
 1import pylons
2from paste.deploy import loadapp
3
4class MyMiddleware(object):
5 def __init__(self, app, config, base_url):
6 self.app = app
7
8 self.middleware_app = loadapp('config:%s' % config['__file__'], name='middleware_app')
9 self.base_url = base_url
10
11 # Remove the config from the config stack; otherwise the host app gets *very* confused
12 # We should be done initializing the middleware app, so this isn't used again anyways.
13 # The RegistryMiddleware takes care of this from now on (during requests).
14
15 process_configs = pylons.config._process_configs
16 this_conf = [conf for conf in process_configs if conf['pylons.package'] == 'name_of_my_middleware_package'][0]
17 process_configs.remove(adjector_dict)
18
19 def __call__(self, environ, start_response):
20 if self.base_url and environ['PATH_INFO'].startswith(self.base_url):
21 environ['PATH_INFO'] = environ['PATH_INFO'][len(self.base_url):] or '/'
22 environ['SCRIPT_NAME'] = self.base_url
23 return self.adjector_app(environ, start_response)
24
25 else:
26 return self.app(environ, start_response)
This snippet is licensed under the BSD license

Make sure you also add a [app:middleware_app] stanza to your host app's

Using this setup, whenever you go to anything that starts with /my_base_url the middleware app will take over. The rest of the time it will leave you alone. Phew.

2.5   Alternative Methods

Instead of adding middleware to middleware.py, you can add it directly from your .ini file. On the plus side, this somehow magically stops the configuration from being overwritten during setup and serving without any dirty hacks; presumably the apps are being loaded serially instead of in parallel. However this method:

  • Completely breaks setup and tests unless you patch PasteScript and PasteDeploy
  • Doesn't solve the config-overwrite problem for tests; this might not be a big problem for you though
  • Works for "no apparent reason", which is a little scary

If you're still interested, you'll need to patch PasteScript and PasteDeploy: #344, #345, #347. You can also just download the patched versions from the iCapsid server (subject to randomly disappear) at http://beta.icapsid.net/src.

You'll need to create a function that instantiates your class:

# 's
1def make_middleware(app, global_conf, **app_conf):
2 return AdjectorMiddleware(app, global_conf)

You'll need to tell paste where to find that class by modifying the 'entry points' section of your middleware app's setup.py. Add the lines:

# 's
1[paste.app_factory]
2main = path.to.this.module:make_app

Finally you should add some code to your host application's .ini. The simplest method is to just add a filter-with directive to your [app:main] stanza:

# 's
1filter-with = egg:middleware_app

I doubt this way is worth it, but I spent so long getting it to work I figured you might as well hear about it : ).

3   Weird Errors (and their Solutions)

3.1   Custom HTTP Error Messages

Ever wish your error messages in Pylons would actually say what you told them to say? In many (but not all?!) setups, you would do something like raise HTTPNotFound('So and so not found') but the specific message would get lost somewhere between the raise and the web browser.

To solve this problem, I refer you to Stou's excellent article Custom HTTP Error messages in Pylons.

You may also want to take a look at a virgin Pylons 0.9.7 setup; error messages seem to work in mine.

3.2   An Odd Setup Error

For a while now, I've gotten this error whenever I try to do a python setup.py develop or install. I don't know if it is a Mac OS X error, a SVN error, or what:

File "build/bdist.linux-i686/egg/setuptools/command/sdist.py", line 98, in entries_finder
NameError: global name 'log' is not defined

It turns out that this is being caused by setup automatically tagging your builds... or trying to at least. This is enabled by default in Pylons. Even though it doesn't externally tag a build when you do a setup, apparently that step is happening somewhere.

To solve it you can either:

  • Temporarily rename your .svn directory while you do a setup. I just wrote a script to do this for me.

  • Comment out/delete this line in your setup.cfg file:

    # 's
    1#tag_svn_revision = true

4   Conclusion

I always feel like I should write something at the end of an article, but it always comes down to the same thing: feel free to comment or contribute. After all, collaboration is why we made the site; otherwise I might as well of put this in my blog and forgot about it.

5   References and Further Reading

The quite excellent Pylons Book (I especially found Chapters 16 and 17 interesting)

Pylons WSGI Support

The paste.recursive Module

Really everything you could ever want to know about SCRIPT_NAME and PATH_INFO