Jkx@home » twisted

Python web developpement: the dilemma / Act 2

admin — Tue, 14 Sep 2004 22:23:11 +0000

After a long trip around the Python web developpement tools, it’s time to fix. In a previous post (please read all comments before ..) we have talk for a long time about various web frameworks. Here is my personnal conclusion:

A web framework has nothing to do with templates. It need to allow to wrap templates easily, but it should’nt deal with them directly. (I have done DTML in a past life.. no thanks:)
A web framework has nothing to deal w/ the object-store. It should let me store objects as i want (without too much pain)

In the previous post we talk about the different approach: Threads, Mod_Python, Twisted, and Fork (in comments)

I have tested quite a bunch of framework right now, all have some nice feature, and drawback. To conclude my tour and explain my choice, some little points:

Webware

I really enjoy Webware, because the API is really easy learn, and work like a charm. But as explain before, I have too much trouble w/ the threads in.. I hope cheetah will cleanup the way it includes Kits.. a long thread on the mailing list.

Mod_Python

I don’t like the mod_python Publisher etc etc, I feal too close-to-apache. So i guess the I will choose MP Servlets. Performance are really good, and doesn’t need to maintain a server process on the host. This is good news because i want to host a couple of little websites. The major drawback is that there is no way to maintain (or even limit) a pool of objects between request. (without apache tweak) Another bad point is that debugging is awfull, need to restart all the time ..
But mod_python offer a lot for deployement..

Twisted

This is the most flamewar subject, I ever seen on the Python community. I used twisted, and yes it’s really fun and powerfull. But I don’t want to mix deferred and threads (for DB .. and others stuffs). If I want to mix differents networks stuffs, twisted is my tool. but as previouly said (comments) I ‘m not really fuent with twisted.web

Skunk-Web

Hum, this is a fork based server.. I haven’t tested it a lot because it’s really template base, and I want to use Cheetah or ZPT not another-templating-o-matic. And fork, may cause issue if you have some hits (but skunk really seems to work well under heavy load so.. )

And now..

So, yes, i haven’t choose one of this. In fact, in the previous post somebody ask me about Quixote, so i decided to test it. And I found an interesting stuff,
Quixote use: scgi (mod_scgi + daemon), old-fashion-cgi or mod_python, medusa or twisted.

So I can:

use mod_python in production area. (nothing more than apache :)
use mod_scgi if i want to run the server as a different user. No need to use su-exec or fight w/ ownerships…
use medusa to develop. And that’s a really good point because i can use pdb to debug,and avoid to restart mod_python again and again.
use old-fashion-cgi on system that only provides this..
test Durus + Dulcinea for the fun.

There is some drawbacks of course:

really small documentation right now (check the Quixote Wiki)
I don’t really like the way it map the URL namespace to the modules
come w/ custom templating system, but cheetah has a handler (infos in the wiki)

As Phil said: This is your new toy, until the next.

He’s right. I dream one day, Python web’s framework will be more unified. Quixote is a good example of what power we can achieve if the framework cover a large amount of different approach in a unified way.

Twisted in EndUserLand

admin — Wed, 14 Jul 2004 12:20:50 +0000

As nobody notice that, the new apt-proxy use python (not perl as the old one). This is really a good news because i love Python and the new apt-proxy has some better performance than the oldest perlish one. The key value here is the Twisted Matrix framework, now apt-proxy can download multiple .deb at the time without a bunch of process or other weakness. This is a great new for debian users but specially for twisted developper because i think it’s the first end-user app that use twisted (not speaking about webserver of other developpers geeky stuffs).

– Nobody for a twisted bittorrent server ? This would save a bunch of CPU cycles too !

Python web developpement: the dilemma

admin — Sun, 20 Jun 2004 12:16:47 +0000

There is a bunch of Python based web framework. So much that it’s a really hard choice to do. Most of them came w/ a special templating language, and a different approach.

Dilemma 1

I really think most of the frameworks, have nothing to do with templating. We should give up with that mixOmatic. A framework is a framework not a templating language. So while looking for a good framework, never looks at its templating system, since you should use the one you want. (not the one that come with the framework).

Dilemma 2

The second thing that run me in trouble is the approach. I find about 3 different way to handle requests, and still wondering what is the best choice

Thread approach

Webware has this kind of approach. It maintain a pool of servlets, and use apache as front-end to dispatch to webware server. This approach is really interesting since you can really tweak performance by maintaining the some cached stuff .. By the other side, you have all the drawbacks of this way to using:

A lot of python module aren’t thread safe. Even the SQLObject which is normally written in this way have trouble with thread. Phil lost part of the night to discover that SQLO + SQLite run give him a bunch of errors..
Really hard to deploy: Since the servlet pool doesn’t support advanced caching, every single servlet load stay loaded all the time. (I want to deploy a bunch of website .. something like 20 .. with low traffic that isn’t the right way)

Mod_Python approach

There is a lot of frameworks that use this way. The one i really enjoy is MP Servlets. The major feature is that it use Apache 2 thread behaviour, so you don’t have to use python threads. This sound really good, and you don’t have to launch a python server on the host. But the major drawback is that there is no way to maintain (or even limit) a pool of objects between request. (This is possible but you need to tweak the apache way to handle request to disable threading for a proceess… and this isn’t a good approach for virtual hosted website).

Twisted approach

Twisted try to fix the thread problem by using a single loop, and async request handling. With that you can:

enjoy the python world without thread nightmare
desging small dedicated servers

But once again this way have a major drawback has you will be unable to use stuff like SQLObject since Twisted require some specials database connection to avoid locking (remember single loop). And i don’t want to use the old fashiong sql way to build website. Another thing to remember about Twisted is that you need to use apache to proxy the request to the host, and i don’t really think that this way is really pretty for performance.

Conclusion ?

I don’t have a good conclusion. I know Zope can fix some of this problem, but Zope is a too hard to developp / maintain for me. Zope3 tend to be more simple for a lot of ways, but still under developpement, and need a lot of work to learn right now.

Update: Nobody seems to have a decent solution :) Really strange no ?

Twisted or not

admin — Tue, 11 Nov 2003 12:10:37 +0000

I spent a couple of hours to look at http://www.twistedmatrix.com/ . After a little of hacking, i think i understand how it works, mainly static.File. But i still need to work on this since i want to use it to share some large files and this need a little work to avoid locking the whole system.

Perhaps using static.File as core of my system will do the trick but as i want to do some bandwith limitation i need to write some custom stuffs. ..