Ouch: Plone 2.0 vs Plone 2.1

I run into a little test of Plone 2.1 today. And the whole stuff, seems to be slower than Plone 2.0.5. I decided to do a little benchmark, and yes Plone 2.1 seems to be really slower..

Plone 2.0.5 (python 2.3.4)

Total transferred:      239130 bytes
HTML transferred:       234620 bytes
Requests per second:    5.52 [#/sec] (mean)
Time per request:       181.261 [ms] (mean)
Time per request:       181.261 [ms] (mean, across all concurrent requests)
Transfer rate:          128.54 [Kbytes/sec] received

Plone 2.1 (python 2.3.5)

Total transferred:      240740 bytes
HTML transferred:       237440 bytes
Requests per second:    3.54 [#/sec] (mean)
Time per request:       282.476 [ms] (mean)
Time per request:       282.476 [ms] (mean, across all concurrent requests)
Transfer rate:          83.19 [Kbytes/sec] received

Grr, here is two point to read:

  • The 2.1 version is really slower look at the transfer rate..
  • But this is astomish slow

The bench was done on:

  • pure zope httpd (no apache or squid frontend)
  • without debug enable
  • with default HTTPCache and RAMCache enable (default install setting)
  • a dual Xeon at 2.6Ghz with 1Go of RAM ! (I know python only use 1 CPU but !!!)

I first think my setup is buggy, but no.. nobody with a good CMS in stock ?



Related Posts

12 thoughts on “Ouch: Plone 2.0 vs Plone 2.1

  1. Compare that to a purpose built Quixote app:

    target: a moderately complex internationalized document object in a CMS:

    lighttpd / scgi:
    HTML transferred:       4167500 bytes
    Requests per second:    134.99 [#/sec] (mean)
    Time per request:       7.41 [ms] (mean)
    Transfer rate:          1150.92 [Kbytes/sec] received
    

    Medusa:
    HTML transferred:       4167500 bytes
    Requests per second:    127.16 [#/sec] (mean)
    Time per request:       7.86 [ms] (mean)
    Transfer rate:          1079.22 [Kbytes/sec] received
    

    Target: Root page, only a few objects resolving and minor security check

    lighttpd / scgi:
    HTML transferred:       1802000 bytes
    Requests per second:    309.21 [#/sec] (mean)
    Time per request:       3.23 [ms] (mean)
    Time per request:       3.23 [ms] (mean, across all concurrent requests)
    Transfer rate:          1173.47 [Kbytes/sec] received
    

    Medusa:
    Requests per second:    268.24 [#/sec] (mean)
    Time per request:       3.73 [ms] (mean)
    

    These tests run on a single CPU development workstation running X / xfce at the same time and a swack of other stuff; 1GB ram, 2.4ghz processor.

    Of course, with Plone/Zope I could inherit a swack of software and perhaps write very little myself; so there’s real time and productivity saved there… at the same time, I’d have to invest a lot of energy in learning the platform before proficiency could be achieved. Plusses and minuses.

    Quixote – or any of the simpler frameworks – certainly have a place.

  2. One Question:

    Why do you run Zope in Debug-Mode for Benchmarking? This may have a significant impact on your Performance.

    And, yes with ZEO your System will run remarkably faster if you have more than 1 CPU…

    Kind Regards
    Maik

  3. Sorry for the debug statement, I read "with debug enabled" :-o

    I Should get another cup of coffee quick ;-)

  4. In fact, Plone doesn’t seem to suffer from concurrency:

    -c1 -n100:
    Requests per second:    2.77 [#/sec] (mean)
    Time per request:       360.872 [ms] (mean)
    Time per request:       360.872 [ms] (mean, across all concurrent requests)
    -c5 -n100:
    Requests per second:    2.82 [#/sec] (mean)
    Time per request:       1771.659 [ms] (mean)
    Time per request:       354.332 [ms] (mean, across all concurrent requests)
    -c10 -n100:
    Requests per second:    2.82 [#/sec] (mean)
    Time per request:       3540.563 [ms] (mean)
    Time per request:       354.056 [ms] (mean, across all concurrent requests)
    -c20 -n200:
    Requests per second:    2.84 [#/sec] (mean)
    Time per request:       7053.295 [ms] (mean)
    Time per request:       352.665 [ms] (mean, across all concurrent requests)
    -c50 -n500:
    Requests per second:    2.78 [#/sec] (mean)
    Time per request:       18011.671 [ms] (mean)
    Time per request:       360.233 [ms] (mean, across all concurrent requests)
  5. Here’s the same, Quixote, medusa. It doesn’t suffer from a big load at it either. Lightweight has its dividends ;-)

    Server Software: Medusa/1.11 Document Length: 6743 bytes

    ab -c 1 -n 500 http://dev.mikewatkins.net:8080/testfolder/xyz/mydoc/
    ab -c 2 -n 500 http://dev.mikewatkins.net:8080/testfolder/xyz/mydoc/
    ab -c 4 -n 500 http://dev.mikewatkins.net:8080/testfolder/xyz/mydoc/
    
    Concurrency Level:      1
    Time taken for tests:   4.335 seconds
    Requests per second:    115.34 [#/sec] (mean)
    Time per request:       8.67 [ms] (mean)
    Time per request:       8.67 [ms] (mean, across all concurrent requests)
    Transfer rate:          795.27 [Kbytes/sec] received
    
    Concurrency Level:      2
    Time taken for tests:   4.273 seconds
    Requests per second:    117.01 [#/sec] (mean)
    Time per request:       17.09 [ms] (mean)
    Time per request:       8.55 [ms] (mean, across all concurrent requests)
    Transfer rate:          806.81 [Kbytes/sec] received
    
    Concurrency Level:      4
    Time taken for tests:   4.294 seconds
    Requests per second:    116.44 [#/sec] (mean)
    Time per request:       34.35 [ms] (mean)
    Time per request:       8.59 [ms] (mean, across all concurrent requests)
    Transfer rate:          804.47 [Kbytes/sec] received
    

    And then to extremes. The system is very responsive to a real user while this runs:

    Concurrency Level:      20
    Time taken for tests:   20.550 seconds
    Complete requests:      1500
    Failed requests:        0
    Broken pipe errors:     0
    Total transferred:      10349395 bytes
    HTML transferred:       10121243 bytes
    Requests per second:    72.99 [#/sec] (mean)
    Time per request:       274.00 [ms] (mean)
    Time per request:       13.70 [ms] (mean, across all concurrent requests)
    Transfer rate:          503.62 [Kbytes/sec] received
  6. running plone 2.0.5, zope 2.7.5 with 512mb ram was fine.
    running plone 2.1, zope 2.8.0 on the same machine gives a noticeable lag and is pretty much IMHO "unuseable".

  7. I don’t know enough about your Quixote app to say this, but you may be comparing apples to oranges, here. For your Quixote app (and the Plone app) that you tested, was it a ‘portalish’ application, which would need to do the necessary security/permission checks, did it need to aggregate different information based upon the personalization rules of the authenticated user’s roles, etc. as would be the case with a test of an out-of-the-box Plone site?

    Also, did you have debug mode enabled for the Quixote app? I don’t think that would cause the variance in requests/second that you’re reporting here, but enabling debug mode for any performance tests (of most any CMS or framework) would pretty much be a useless comparison to what you would expect to see when debug is disabled.

    I think you may want to compare Quixote to other frameworks such as Django, rather than to full-fledged CMS systems, for a good comparison of like tools.

    On the other hand, if you were to compare Plone’s out-of-the-box performance with that of say Documentum, Interwoven, Vignette, Joomla/Mambo, Bricolage or other CMS tools (with none in debug mode), I think that may be more meaningful as Plone is more functional than a mere framework (where you build the functionality), and with that additional functionality comes a requirement to know what one is doing when configuring for high-volume application scenarios. It can surely be done, is documented, and the community seems very helpful whenever I’ve needed to ask for help in this area.

    I think as a rule of thumb, just about any framework can beat out just about any CMS in terms of performance. You hit it on the head when stating that you have to weigh the time to build features versus the time to configure the CMS for optimal performance.

    Cheers!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>