Ouch: Plone 2.0 vs Plone 2.1

I run into a little test of Plone 2.1 today. And the whole stuff, seems to be slower than Plone 2.0.5. I decided to do a little benchmark, and yes Plone 2.1 seems to be really slower..

Plone 2.0.5 (python 2.3.4)

Total transferred:      239130 bytes
HTML transferred:       234620 bytes
Requests per second:    5.52 [#/sec] (mean)
Time per request:       181.261 [ms] (mean)
Time per request:       181.261 [ms] (mean, across all concurrent requests)
Transfer rate:          128.54 [Kbytes/sec] received

Plone 2.1 (python 2.3.5)

Total transferred:      240740 bytes
HTML transferred:       237440 bytes
Requests per second:    3.54 [#/sec] (mean)
Time per request:       282.476 [ms] (mean)
Time per request:       282.476 [ms] (mean, across all concurrent requests)
Transfer rate:          83.19 [Kbytes/sec] received

Grr, here is two point to read:

  • The 2.1 version is really slower look at the transfer rate..
  • But this is astomish slow

The bench was done on:

  • pure zope httpd (no apache or squid frontend)
  • without debug enable
  • with default HTTPCache and RAMCache enable (default install setting)
  • a dual Xeon at 2.6Ghz with 1Go of RAM ! (I know python only use 1 CPU but !!!)

I first think my setup is buggy, but no.. nobody with a good CMS in stock ?

Tags: , ,

Related posts :

admin September 27th, 2005


12 Responses to “Ouch: Plone 2.0 vs Plone 2.1”

  1. # Mike Watkinson 28 Sep 2005 at 11:52 pm

    Compare that to a purpose built Quixote app:

    target: a moderately complex internationalized document object in a CMS:

    lighttpd / scgi:
HTML transferred:       4167500 bytes
Requests per second:    134.99 [#/sec] (mean)
Time per request:       7.41 [ms] (mean)
Transfer rate:          1150.92 [Kbytes/sec] received


    Medusa:
HTML transferred:       4167500 bytes
Requests per second:    127.16 [#/sec] (mean)
Time per request:       7.86 [ms] (mean)
Transfer rate:          1079.22 [Kbytes/sec] received

    Target: Root page, only a few objects resolving and minor security check

    lighttpd / scgi:
HTML transferred:       1802000 bytes
Requests per second:    309.21 [#/sec] (mean)
Time per request:       3.23 [ms] (mean)
Time per request:       3.23 [ms] (mean, across all concurrent requests)
Transfer rate:          1173.47 [Kbytes/sec] received


    Medusa:
Requests per second:    268.24 [#/sec] (mean)
Time per request:       3.73 [ms] (mean)

    These tests run on a single CPU development workstation running X / xfce at the same time and a swack of other stuff; 1GB ram, 2.4ghz processor.

    Of course, with Plone/Zope I could inherit a swack of software and perhaps write very little myself; so there’s real time and productivity saved there… at the same time, I’d have to invest a lot of energy in learning the platform before proficiency could be achieved. Plusses and minuses.

    Quixote - or any of the simpler frameworks - certainly have a place.

  2. # ogriselon 29 Sep 2005 at 7:35 pm

    You might be interested in having a look at funkload to do automated benchmark of your favorite CMS.

    http://public.dev.nuxeo.com/~ben/funkload/

    Patches, bug reports, feature requests should go to :

    http://svn.nuxeo.org/trac/pub/newticket

    under the "funkload" component.

  3. # Maikon 30 Sep 2005 at 8:54 am

    One Question:

    Why do you run Zope in Debug-Mode for Benchmarking? This may have a significant impact on your Performance.

    And, yes with ZEO your System will run remarkably faster if you have more than 1 CPU…

    Kind Regards
    Maik

  4. # Maikon 30 Sep 2005 at 8:55 am

    Sorry for the debug statement, I read "with debug enabled" :-o

    I Should get another cup of coffee quick ;-)

  5. # freeformzon 30 Sep 2005 at 9:26 am

    What program did you use and what arguments did you pass to said program? Just curious…

  6. # Jkxon 30 Sep 2005 at 10:05 am

    Ab2 -n 10 (I don’t test concurrent request, it will be worst..)

  7. # bcroqon 30 Sep 2005 at 12:38 pm

    In fact, Plone doesn’t seem to suffer from concurrency:

    -c1 -n100:
Requests per second:    2.77 [#/sec] (mean)
Time per request:       360.872 [ms] (mean)
Time per request:       360.872 [ms] (mean, across all concurrent requests)
    -c5 -n100:
Requests per second:    2.82 [#/sec] (mean)
Time per request:       1771.659 [ms] (mean)
Time per request:       354.332 [ms] (mean, across all concurrent requests)
    -c10 -n100:
Requests per second:    2.82 [#/sec] (mean)
Time per request:       3540.563 [ms] (mean)
Time per request:       354.056 [ms] (mean, across all concurrent requests)
    -c20 -n200:
Requests per second:    2.84 [#/sec] (mean)
Time per request:       7053.295 [ms] (mean)
Time per request:       352.665 [ms] (mean, across all concurrent requests)
    -c50 -n500:
Requests per second:    2.78 [#/sec] (mean)
Time per request:       18011.671 [ms] (mean)
Time per request:       360.233 [ms] (mean, across all concurrent requests)
  8. # Michael Watkinson 30 Sep 2005 at 9:43 pm

    Here’s the same, Quixote, medusa. It doesn’t suffer from a big load at it either. Lightweight has its dividends ;-)

    Server Software: Medusa/1.11 Document Length: 6743 bytes

    ab -c 1 -n 500 http://dev.mikewatkins.net:8080/testfolder/xyz/mydoc/
ab -c 2 -n 500 http://dev.mikewatkins.net:8080/testfolder/xyz/mydoc/
ab -c 4 -n 500 http://dev.mikewatkins.net:8080/testfolder/xyz/mydoc/

    Concurrency Level:      1
Time taken for tests:   4.335 seconds
Requests per second:    115.34 [#/sec] (mean)
Time per request:       8.67 [ms] (mean)
Time per request:       8.67 [ms] (mean, across all concurrent requests)
Transfer rate:          795.27 [Kbytes/sec] received

Concurrency Level:      2
Time taken for tests:   4.273 seconds
Requests per second:    117.01 [#/sec] (mean)
Time per request:       17.09 [ms] (mean)
Time per request:       8.55 [ms] (mean, across all concurrent requests)
Transfer rate:          806.81 [Kbytes/sec] received

Concurrency Level:      4
Time taken for tests:   4.294 seconds
Requests per second:    116.44 [#/sec] (mean)
Time per request:       34.35 [ms] (mean)
Time per request:       8.59 [ms] (mean, across all concurrent requests)
Transfer rate:          804.47 [Kbytes/sec] received

    And then to extremes. The system is very responsive to a real user while this runs:

    Concurrency Level:      20
Time taken for tests:   20.550 seconds
Complete requests:      1500
Failed requests:        0
Broken pipe errors:     0
Total transferred:      10349395 bytes
HTML transferred:       10121243 bytes
Requests per second:    72.99 [#/sec] (mean)
Time per request:       274.00 [ms] (mean)
Time per request:       13.70 [ms] (mean, across all concurrent requests)
Transfer rate:          503.62 [Kbytes/sec] received
  9. # Michael Watkinson 30 Sep 2005 at 9:43 pm

    And if your blog "preview" worked I might be able to format that! :D

    Cheers…

  10. # Jkxon 30 Sep 2005 at 10:25 pm

    The preview works .. except that firefox or moz block it ..

  11. # julianon 07 Oct 2005 at 2:30 am

    running plone 2.0.5, zope 2.7.5 with 512mb ram was fine.
    running plone 2.1, zope 2.8.0 on the same machine gives a noticeable lag and is pretty much IMHO "unuseable".

  12. # sprkenon 21 Nov 2005 at 5:33 am

    I don’t know enough about your Quixote app to say this, but you may be comparing apples to oranges, here. For your Quixote app (and the Plone app) that you tested, was it a ‘portalish’ application, which would need to do the necessary security/permission checks, did it need to aggregate different information based upon the personalization rules of the authenticated user’s roles, etc. as would be the case with a test of an out-of-the-box Plone site?

    Also, did you have debug mode enabled for the Quixote app? I don’t think that would cause the variance in requests/second that you’re reporting here, but enabling debug mode for any performance tests (of most any CMS or framework) would pretty much be a useless comparison to what you would expect to see when debug is disabled.

    I think you may want to compare Quixote to other frameworks such as Django, rather than to full-fledged CMS systems, for a good comparison of like tools.

    On the other hand, if you were to compare Plone’s out-of-the-box performance with that of say Documentum, Interwoven, Vignette, Joomla/Mambo, Bricolage or other CMS tools (with none in debug mode), I think that may be more meaningful as Plone is more functional than a mere framework (where you build the functionality), and with that additional functionality comes a requirement to know what one is doing when configuring for high-volume application scenarios. It can surely be done, is documented, and the community seems very helpful whenever I’ve needed to ask for help in this area.

    I think as a rule of thumb, just about any framework can beat out just about any CMS in terms of performance. You hit it on the head when stating that you have to weigh the time to build features versus the time to configure the CMS for optimal performance.

    Cheers!

Comments RSS

Leave a Reply