I spent part of the day to install a virtual mail server, and i discover that there is a really simple way to secure courier-imap. By defaut when you use this kind of server the auth is made against the standard pam module. This is really simple, but what happend if someone is able to read your imap password. Yes he get the account password.
Courier imap can do the auth against a lot of stuff. Simply change the authmodulelist=authpam to authmodulelist=authuserdb. This will use the file /etc/courier/userdb.
userdb "username" set home=/home/username mail=/home/username/Maildir uid=UID gid=GID userdbpw | userdb "username" set imappw makeuserdb
So now you can use a different password for your account and your imap, and you can choose who can access to imap server.