IEEE Single Precision Floating Point

Written by kjfletch on May 20, 2012 Categories: Code, Computer Science, Maths, Scheme Tags: , , ,

I find that if I have to keep revisiting a subject over and over I haven’t fully understood it. One such subject I have revisited a lot since my university days is the IEEE floating point representations.

By converting numbers to the IEEE float format in example form I hope to put the subject to bed and keep the knowledge lodged in my brain once and for all.

Dusting off (Clements, 2000)[1] Section 4.8.3 IEEE floating point format gives us the representation of a 32-bit floating point number.

1 bit 8 bits 23 bits
S Biased exponent Fractional mantissa

Giving the value as the formula

n = -1^S * 2^(E-B) * 1.F

Where S is the sign, E is the biased exponent, B is the bias to be removed from the exponent and F is the fractional mantissa.

The bias is used so that the exponent can represent negative numbers. With a bias as 127 the exponent can be in the range -126 to +127 (An additional two values are set aside to represent zero and -infinity/+infinity/NaN.

The fractional mantissa is the normalised fraction of the number (the most significant bit and radix point removed). A little explanation;

If we take a decimal value -2345.125 we can normalise this number into scientific notation as such: -2.345125e3. To achieve this algorithmically we can use

(define (log10 n)
  (/ (log n) (log 10)))
 
(define (normalise10 d)
  (if (= d 0)
      (list 0 0) ; Treat zero as a special case!
      (let ((ex (truncate (log10 (abs d)))))
	(list (/ d (expt 10 ex)) 
	      (inexact->exact ex)))))
 
(normalise10 -2345.125)
;; => (-2.345125 3)

Normalising for base 2 (binary) numbers can use the same method, calling log2 in place of log10

(define (log2 n)
  (/ (log n) (log 2)))
 
(define (normalise2 d)
  (if (= d 0)
      (list 0 0) ; Treat zero as a special case!
      (let ((ex (truncate (log2 (abs d)))))
	(list (/ d (expt 2 ex))
	      (inexact->exact ex)))))
 
(normalise2 -2345.125) 
;; => (-1.14508056640625 11)
;; Although we normalise for base 2, the result is
;; displayed in decimal. As a curiosity we can check
;; that the number has been normalised to binary by
(string-append (number->string -1.14508056640625 2) "e" 
	       (number->string 11 2))
;; => "-1.00100101001001e1011"

We now have all the required components to make an IEEE float: a sign, a fraction and an exponent. We could write a crude function to package up a float like this

(define (ieee-single d)
  (let ((v (make-bitvector 32 #f)))
    (if (not (= d 0))
	(begin
	 (let ((bias 127)
	       (normal (normalise2 d)))
 
	   ;; Write the sign bit into the bitvector.
	   (bitvector-set! v 0 (< d 0))
 
	   ;; Write the biased exponent into the bitvector.
	   (let loop ((n (+ bias (cadr normal))) (i 8))
	     (if (not (= 0 n))
		 (let ((r (remainder n 2))
		       (q (quotient n 2)))
		   (bitvector-set! v i (= r 1))
		   (loop q (- i 1)))))
 
	   ;; Write the normalised fractional mantissa into the
	   ;; bitvector. We don't need to bother with the "1.", the
	   ;; ieee745 assumes this bit.
	   (let loop ((fraction (- (abs (car normal)) 1)) (i 9))
	     (if (not (= 0.0 fraction))
		 (let* ((f (* 2 fraction))
			(bit (truncate f)))
		   (bitvector-set! v i (= bit 1))
		   (loop (- f bit) (+ i 1))))))))
    v))
 
(ieee-single -2345.125)
;; => #*11000101000100101001001000000000
;;      S|--E---||----------M----------|

If a number can’t be exactly represented in an IEEE single precision floating point number the above function will fail. In another post I will discuss this problem and how values are approximated.

[1] The Principles of Computer Hardware, Third Edition. Alan Clements. Oxford University Press.

No Comments

emacs as an MDA

Written by kjfletch on May 7, 2012 Categories: Code, Emacs Tags: , , , , , , , ,

Ah, emails; yet again. Previously I had discussed elid as a way of fetching mails from an IMAP server from within emacs. I have been using this to deliver my mails from multiple servers/mailboxes and pass them to an external MDA configuration (maildrop).

Introducing elmda

To remove the external dependency I have created elmda; some basic MDA functionality for elisp. Example configuration:

(defun my-mda (msg)
  "MSG is the content of a single email"
  (with-temp-buffer
    (insert msg)
    (cond ((elmda-is-recipient "home@example.com")
           (elmda-deliver-maildir "~/mailboxes/home-inbox"))
          ((elmda-is-recipient "work@example.com")
           (elmda-deliver-maildir "~/mailboxes/work-inbox"))
          ;; Fallback: just stick mails in a general inbox
          ((elmda-deliver-maildir "~/mailboxes/general-inbox")))))

Passing this function an email message will filter it into the appropriate maildir mailbox depending on the recipient.

The power comes from bolting elmda to elid:

(require 'elid)
(require 'elmda)
 
(setq elid-remotes
      '((:name "mail"
         :user "fred"
         :host "mail.example.com"
         :port 993
         :mailboxes ("^INBOX$")
         :authstream tls
         :authtype nil
         :mda my-mda)))
 
(defun my-mda (props msg)
  "MSG is the content of a single email"
  (with-temp-buffer
    (insert msg)
    (cond ((elmda-is-recipient "home@example.com")
           (elmda-deliver-maildir "~/mailboxes/home-inbox"))
          ((elmda-is-recipient "work@example.com")
           (elmda-deliver-maildir "~/mailboxes/work-inbox"))
          ;; Fallback: just stick mails in a general inbox
          ((elmda-deliver-maildir "~/mailboxes/general-inbox")))))
 
(elid-deliver "mail")

Where to Find elmda

You can find the repository for elmda bzr branch http://bzr.kjfletch.co.uk/emacs-mda/trunk or get the latest from here.

Maturity

Both elmda and elid are at a very immature stage. I have written them to scratch a personal itch and they wont suit everyone. Feel free to modify (and share) more functionality, additional mailbox backends, etc.

Also, both packages have a tested status: “Hmm, seems to work.”

No Comments

Sieve of Eratosthenes

Written by kjfletch on April 24, 2012 Categories: Code, Maths, Scheme Tags: , , , ,

The Sieve of Eratosthenes is a method of producing a list of prime numbers by eliminating multiples.

;; Implementation of the Sieve of Eratosthenes in scheme
(define (sieve-of-eratosthenes n)
  (let ((v (make-vector (- n 1) #t)))
    (let loop ((i 0) (primes '()))
      (cond ((> i (- n 2)) primes)
	    ((vector-ref v i)
	     (let loop ((j (+ i i 2)))
	       (if (<= j (- n 2))
		   (begin (vector-set! v j #f)
			  (loop (+ j i 2)))))
	     (loop (+ i 1) (cons (+ i 2) primes)))
	    (else (loop (+ i 1) primes))))))
 
(sieve-of-eratosthenes 10)
  => (7 5 3 2)

A nice explanation of the algorithm is given on Programming Praxis.

Optimisations

The Sieve can be subjected to some optimisations; First, we can eliminate all even numbers outright (as they are sifted by the initial sweep of 2). Second, When sifting for a number we can start at the square of this number (due to all lower numbers having already been eliminated by previous sweeps). Third, We can stop sifting when the current number is larger than the square root of N.

An example of the Sieve of Eratosthenes with these optimisations is provided on Programming Praxis, implemented by Jos Koot

(define (primes n)
  (let* ((max-index (quotient (- n 3) 2))
         (v (make-vector (+ 1 max-index) #t)))
    (let loop ((i 0) (ps '(2)))
      (let ((p (+ i i 3)) (startj (+ (* 2 i i) (* 6 i) 3)))
        (cond ((>= (* p p) n)
               (let loop ((j i) (ps ps))
                  (cond ((> j max-index) (reverse ps))
                        ((vector-ref v j)
                          (loop (+ j 1) (cons (+ j j 3) ps)))
                        (else (loop (+ j 1) ps)))))
              ((vector-ref v i)
                (let loop ((j startj))
                  (if (<= j max-index)
                      (begin (vector-set! v j #f)
                             (loop (+ j p)))))
                      (loop (+ 1 i) (cons p ps)))
              (else (loop (+ 1 i) ps)))))))
 
;; There are one million primes less than or equal to 15485863:
(length (primes 15485863))
  => 1000000

It is not immediately obvious how the calculations in this function were derived so I will dissect the function. The implementation starts with a vector large enough to keep all odd numbers between 3 and n (inclusive): (+ (quotient (- n 3) 2) 1) each element is initially set to #t

Index :  0  1  2  3  4  5  6  7  8  9 10 11 12 ... N
Vector: #t #t #t #t #t #t #t #t #t #t #t #t #t ...
Value :  3  5  7  9 11 13 15 17 19 21 23 25 27 ...

The mapping between an index and a value is then

Index to value

Given in the primes function as (+ i i 3) or (+ j j 3). The inverse function (which takes a value and returns the index) is

Value to index

We can combine these two functions to calculate the index of the square of any given index:

Index of square of value of index

This is used in the primes function to give the index to start sifting for any given number: (let ((startj (+ (* 2 i i) (* 6 i) 3))))

No Comments

Emacs Speaks Email (introducing elid)

Written by kjfletch on April 22, 2012 Categories: Code, Emacs, SysAdmin Tags: , , , , , , , , ,

In this post I will be continuing down the email rabbit hole. But first, a recap.

In Remote IMAP Syncing with MDA Delivery I discussed the requirement to access IMAP servers and deliver new emails locally through an MDA. I prototyped a system using OfflineIMAP and wrote my own MDA cache in a bash script.

In Sync Multiple IMAPs and Deliver to an MDA I improved the system by allowing it to deliver from multiple IMAP servers.

Although the prototype system works it has some downsides:

  • - Emails are duplicated making my mailbox storage twice what is required. OfflineIMAP keeps all my mails in a cache so that it can perform sync.
  • - Authentication credentials are separate from my outgoing mails (SMTP). Although, It is possible to write some python in the OfflineIMAP configuration to access the same authentication credentials, this is just another complexity to the system.
  • - Relies on external dependencies (python, OfflineIMAP, etc) to the rest of my mail system.

As my aim is to handle all aspects of my email within emacs I’ve been investigating an emacs solution and came up with elid (tentative name).

What elid Does

It connects to IMAP servers (using imap.el and auth-source.el), downloads all messages it hasn’t already seen (using a simple UID cache) and delivers them to an arbitrary elisp function. Simple!

How to Use elid

A simple configuration is given in the comment header block of elid.el:

(require 'elid)
(setq elid-remotes
      '((:name "mail"
         :user "fred"
         :host "mail.example.com"
         :port 993
         :mailboxes ("^INBOX$")
         :authstream tls
         :authtype nil
         :mda my-mda)))
 
(defun my-mda (props msg)
  (with-temp-buffer
    (insert msg)
    (eq 0 (call-process-region (point-min)
                            (point-max)
                            "maildrop" nil nil nil
                            "/home/fred/maildroprc"))))
(elid-deliver "mail")

This simple configuration, in English:

  • - Connnect to a single remote IMAP server (we’ll call it mail).
  • - My username is fred, we’ll be connecting to mail.example.com at port 993
  • - The mailboxes we’re interested in (regexps); a single mailbox called exactly INBOX
  • - Authenticate over TLS with the default/best authenticate type
  • - Send messages to the function my-mda

my-mda is the function which accepts a single message and a list of properties (giving the origin of the message; host, inbox, username, etc).

In this example it passes messages to an external MDA (maildrop). my-mda should return t if the message was successfully delivered.

Where to Find elid

You can find the repository for elid bzr branch http://bzr.kjfletch.co.uk/emacs-email-delivery/trunk. Or you can download the latest version of elid.el from here

No Comments

Organising Media Files

Written by kjfletch on April 11, 2012 Categories: Code, SysAdmin Tags: , , , , ,

For years I have stored my photos and home movies in an “each event in a subdirectory” structure. It has never served me well. As time has gone by and the data has grown, the number of source devices has increased (cameras, video cameras, phone camera, my wife’s phone camera, etc) as well as the number of file formats (and file name formats); this structure has become completely unworkable and a nightmare to sync!

I have spent the last few days tracking down all my media with the goal of creating a flat directory with each file prefixed with a time/date stamp. I’ll rely on a software package to organise them into categories, events, tags, locations, etc virtually.

This would be nice and easy apart from a little niggle; somewhere along the way a lot of my files lost their modification timestamps! Luckily this file-set was mainly JPEGs so I could rectify it by taking the DateTimeOriginal property out of the JPEG header.

I came up with a script that I can reuse to pull new media off of my devices and store them in the required format. The script runs in 2 modes (depending on command line argument):

Directory Mode

$ media-archive.sh -d DIRECTORY -o OUTPUTDIR [-t] [-v]
Where -t enters test mode (we’ll come to that), and -v is verbose.

mkdir -p ${OUTPUTDIR}
 
FINDARGS=" \"${DIRECTORY}\" -type f "
FINDARGS+=' "(" -not -iwholename "*.trash-*" ")"'
FINDARGS+=' "("'
FINDARGS+='     "(" -iname "*.jpg" -not -iname "sample*.jpg" -not -iwholename "*.thumbnails*" ")"'
FINDARGS+=' -or "(" -iname "*.cr2" ")"'
FINDARGS+=' -or "(" -iname "*.3gp" ")"'
FINDARGS+=' -or "(" -iname "*.mp4" ")"'
FINDARGS+=' -or "(" -iname "*.AVI" ")"'
FINDARGS+=' ")"'
 
if [ -z "${TESTMODE}" ]; then
    FINDARGS+=" -exec \"${SCRIPT}\" ${VERBOSE} -f \"{}\" -o \"${OUTPUTDIR}\" \";\""
else
    find "${DIRECTORY}" -type f > .findall
    eval find $FINDARGS > .findmatch
    diff .findall .findmatch
    exit
fi
 
print_verbose "find $FINDARGS"
eval find $FINDARGS

The find expressions represent the set of all possible file types from my current devices (including some files to ignore). This will change when new devices are added to my arsenal.

The -exec on the find call simply recalls the script in file mode -f for each found file.

Note that passing in the -t (test) flag runs find with no -exec argument; pipes the output to a file and compares it (diff) with the output of running find with no filename constraints. This gives you a list of all files NOT matched by the expressions. It is basically a sanity check that your filters aren’t missing anything vital, what good would an incomplete backup be?

File Mode

$ media-archive.sh -f FILE -o OUTPUTDIR [-v]
Where -v is verbose mode.

FILETYPE=`file "${FILE}"`
USEMFALLBACK=0
 
print_verbose ""
print_verbose ${FILE}
 
if [ ! -z "$(echo ${FILETYPE} | grep "JPEG image data")" ]; then
    print_verbose "IMAGE TYPE: JPEG"
    TAKEN=`identify -verbose "${FILE}" | grep exif:DateTimeOriginal | sed "s/    exif:DateTimeOriginal: //" | sed "s/[ :]/_/g"`
    if [ -z "${TAKEN}" ]; then
        USEMFALLBACK=1
        echo "WARNING: Using modification time fallback for ${FILE}"
    fi
elif [ ! -z "$(echo ${FILETYPE} | grep "Canon CR2 raw image data")" ]; then
    USEMFALLBACK=1
elif [ ! -z "$(echo ${FILETYPE} | grep " AVI,")" ]; then
    print_verbose "VIDEO TYPE: AVI"
    USEMFALLBACK=1
elif [ ! -z "$(echo ${FILETYPE} | grep " MPEG ")" ]; then
    print_verbose "VIDE TYPE: MPEG"
    USEMFALLBACK=1
else
    echo "Error: Unhandled file format ${FILETYPE}"
    exit
fi
 
if [ "${USEMFALLBACK}" -eq 1 ]; then
    TAKEN=$(stat -c %y "${FILE}")
    TAKEN=${TAKEN:0:19}
    TAKEN=$(echo "${TAKEN}" | sed "s/[ :\-]/_/g")
fi
 
print_verbose "Time Taken: ${TAKEN}"
NEWNAME=${TAKEN}_`basename "${FILE}"`
print_verbose "New Name: ${NEWNAME}"
OUTPUTFILE=${OUTPUTDIR}/${NEWNAME}
print_verbose "Output File: ${OUTPUTFILE}"
 
if [ ! -e ${OUTPUTFILE} ]; then
    cp -p "${FILE}" "${OUTPUTFILE}"
else
    print_verbose "Skipping file, output file already exists!"
fi

File mode works out what file type it is working with (by using file FILENAME). For JPEGs it extracts the DateTimeOriginal field out of the header using ImageMagick‘s identify program. If the field cannot be found it falls back on using the modification date of the file.

All other file types use the modification date of the file, Hopefully a timestamp is stashed in these files somewhere (I haven’t looked into it yet). Also, identify is slow. I would like to discover an alternative method of extracting the timestamps from JPEGs (comments welcome).

The rest of the script contains the definition of print_verbose, the handling and checking of the arguments and the main if branch of which operating mode the script is in (directory or file).

print_verbose(){
    if [ ! -z "${VERBOSE}" ]; then
	echo $@
    fi
}
 
SCRIPT=${0}
 
while [ ! -z "${1}" ]; do
    if [ "-d" = "${1}" ]; then
	shift
	DIRECTORY=${1}
    elif [ "-f" = "${1}" ]; then
	shift
	FILE=${1}
    elif [ "-o" = "${1}" ]; then
	shift
	OUTPUTDIR=${1}
    elif [ "-t" = "${1}" ]; then
	TESTMODE=${1}
    elif [ "-v" = "${1}" ]; then
	VERBOSE=${1}
    else
	echo "Error: Unknown argument ${1}"
	exit
    fi
    shift
done
 
# Check that correct arguments have been passed in
if [ -z "${DIRECTORY}" -a -z "${FILE}" ]; then
    echo "Error: Not specified a file or directory to archive! Try -d <directory> or -f <file> arguments."
    exit
fi
 
if [ ! -z "${DIRECTORY}" -a ! -z "${FILE}" ]; then
    echo "Error: Specified BOTH file AND directory mode, exiting!"
    exit
fi
 
if [ -z "${OUTPUTDIR}" ]; then
    echo "Error: No output directory specified, use the -o <dir> argument."
    exit
fi
 
# Run in directory mode
if [ ! -z "${DIRECTORY}" ]; then
  ...
fi
# Run in file mode
if [ ! -z "${FILE}" ]; then
  ...
fi
No Comments

Sync Multiple IMAPs and Deliver to an MDA

Written by kjfletch on March 20, 2012 Categories: Code, SysAdmin Tags: , , , , , , ,

Continuing the overhaul of my email system I have been working toward syncing multiple IMAP servers and piping them to the same MDA configuration.

All mail activities are contained in ~/Email which can be tarballed into a backup very easily (including gnus flags, marks, etc… but we’ll get to that in a later post).

Currently I have a single remote IMAP synced (with OfflineIMAP) to a local Maildir ~/Email/offlineimap.mailboxes/. I’ve changed the OfflineIMAP configuration to create multiple accounts which drop their mailboxes in ~/Email/offlineimap.mailboxes/${ACCOUNT_NAME} and changed my delivery script accordingly.

The delivery script now expects to be passed an ${ACCOUNT_NAME} as the first argument; it will deliver mail for this account assuming mailboxes are under ~/Email/offlineimap.mailboxes/${ACCOUNT_NAME}

The delivery cache is now derived from the ${ACCOUNT_NAME} such that it exists in ~/Email/offlineimap.mailboxes/${ACCOUNT_NAME}.delivered

The MDA (maildrop) is now passed 2 custom parameters: ${ACCOUNT_NAME} and ${MAILBOX} stating which account and mailbox the email originated from (this information can be used in the filtering rules to decide where to route the mail).

No mailboxes are created in the delivery script, this is now handled by maildrop.

#!/bin/bash
 
ACCOUNT=$1
MAIL_DELIVERED=0
 
if [ -z ${ACCOUNT} ]; then
  echo "Need account name as first argument!"
  exit -1
fi
 
INPUT_MAILBOXES=offlineimap.mailboxes/${ACCOUNT}
CACHE=offlineimap.mailboxes/${ACCOUNT}.delivered
 
if [ ! -e ${INPUT_MAILBOXES} ]; then
  echo "Mailboxes don't exist for account: ${INPUT_MAILBOXES}"
  exit -1
fi
 
# Make sure the cache directory exists
[ ! -e ${CACHE} ] && mkdir ${CACHE}
 
for i in ${INPUT_MAILBOXES}/* ; do
  THIS_MAILBOX=`basename ${i}`
 
  for STATEBOX in "cur" "new" ; do
    for EMAIL in ${i}/${STATEBOX}/* ; do
      # Create a cache name from the email file name, remove any
      # flags so the same mail doesn't get redelivered if the
      # server flags change.
      CACHENAME=`basename ${EMAIL}`
      CACHENAME=`echo ${CACHENAME} | sed -e "s/:2,.\\+/:2,/"`
      CACHENAME="${CACHE}/${CACHENAME}"
 
      if [ -e ${EMAIL} ] && [ ! -e ${CACHENAME} ] ; then
        echo "Delivering mail: ${EMAIL}"
        maildrop maildroprc ${ACCOUNT} ${THIS_MAILBOX} < ${EMAIL}
        if [ "$?" -eq "0" ]; then
          # mark the mail as delivered if maildrop delivered it
          touch ${CACHENAME}
          # TODO: we could mark the mail as READ in the
          # INPUT_MAILBOX by moving it from "new" to "cur"
          # and setting the appropriate flag.
          MAIL_DELIVERED=`expr ${MAIL_DELIVERED} + 1`
        fi
      fi
    done
  done
done
 
if [ ${MAIL_DELIVERED} -gt 0 ] ; then
  notify-send -i stock_mail-unread "New Email: ${ACCOUNT}" "${MAIL_DELIVERED} new email(s)"
fi

A little helper script create-mailboxes.sh:

[ -z $1 ] && echo "Require a base directory as first parameter!" && exit -1
 
# make the root #{1}
[ ! -e $1 ] && mkdir -p $1
 
FIRST=1
 
# make all mailboxes from ${2} to ${n} under ${1}
for i in "$@"; do
  if [ ${FIRST} -eq 0 ]; then
    MAILBOX=${1}/${i}
    [ ! -e ${MAILBOX} ] && maildirmake ${MAILBOX}
  fi
  FIRST=0
done

This allows maildrop to create destination mailboxes (if they don’t already exist) as part of the maildroprc config:

MAILBOXES=$HOME/Email/mailboxes
`./create-mailboxes.sh "$MAILBOXES" INBOX BLOG DEVELOPMENT`

As the origin account and mailbox get passed through to maildrop I can use them in filter rules:

ACCOUNT=$1
SRC_MAILBOX=$2
 
if ($ACCOUNT == "account1" && $SRC_MAILBOX == "INBOX")
{
  ...
}

My main script is then a simple matter of calling OfflineIMAP and my delivery script with account names:

#!/bin/bash
 
cd ~/Email
[ ! -e offlineimap.mailboxes ] && mkdir offlineimap.mailboxes
 
for ACCOUNT in "account1" "account2"; do
  offlineimap -c ./offlineimaprc -a "${ACCOUNT}" && ./deliver-mail.sh ${ACCOUNT}
done
No Comments

Remote IMAP Syncing with MDA Delivery

Written by kjfletch on March 18, 2012 Categories: Code, SysAdmin Tags: , , , , , , , , ,

Recently I have been sorting out my email system. My goal was to use my remote IMAP server as a constant data store for all my emails; have a local synchronised version of it (where only new emails get downloaded); then have new emails passed to a Mail Delivery Agent (MDA) to sort it into my mailboxes.

After playing around with various Mail Retrieval Agents (MRAs) such as fetchmail and getmail I realised that they weren’t going to fit into the solution. Generally MRAs can be put into two fetch modes; fetch all mail and fetch unread mail. Fetch all mail is to be used when you wish to flush the remote server of emails after each fetch and store all your mail locally (or to transfer it somewhere else). Fetching unread mail would be used for a one-to-one relationship between your server and your client (Mail User Agent – MUA).

I asked at superuser.com if there was already a solution that I could use and got the response “Kinda: OfflineIMAP”.

OfflineIMAP allows you to keep a local and remote server in sync with each other, only transferring new/modified mail. Unfortunately, it does not allow you to dispatch mail to an MDA directly.

The solution was to get OfflineIMAP to dump my mail into a local Maildir. Then write a script which works out which mails are new and send them to an MDA! Easy.

#!/bin/bash
 
CACHE=delivered.cache                 # The cache of delivered mails.
OUTPUT_MAILBOXES=mailboxes            # Where mail will be delivered to.
INPUT_MAILBOXES=offlineimap.mailboxes # The OfflineIMAP sync of my remote mailboxes.
MAIL_DELIVERED=0                      # The number of mails delivered this pass.
 
# Make sure the mailbox directory
[ ! -e ${OUTPUT_MAILBOXES} ] && mkdir ${OUTPUT_MAILBOXES}
 
# Make custom mailboxes
for i in "INBOX" "DEVELOPMENT" "BLOG" ; do
    [ ! -e ${OUTPUT_MAILBOXES}/${i} ] && maildirmake ${OUTPUT_MAILBOXES}/${i}
done
 
# Make sure the cache directory exists
[ ! -e ${CACHE} ] && mkdir ${CACHE}
 
for i in ${INPUT_MAILBOXES}/* ; do
    # Make sure the mailbox exists
    THIS_MAILBOX=`basename ${i}`
    DST_MAILBOX=${OUTPUT_MAILBOXES}/${THIS_MAILBOX}
    [ ! -e ${DST_MAILBOX} ] && maildirmake ${DST_MAILBOX}
 
    for STATEBOX in "cur" "new" ; do
	for EMAIL in ${i}/${STATEBOX}/* ; do
	    # Create a cache name from the email file name, remove any
	    # flags so the same mail doesn't get redelivered if the
	    # server flags change.
	    CACHENAME=`basename ${EMAIL}`
	    CACHENAME=`echo ${CACHENAME} | sed -e "s/:2,.\\+/:2,/"`
	    CACHENAME="${CACHE}/${CACHENAME}"
 
	    if [ -e ${EMAIL} ] && [ ! -e ${CACHENAME} ] ; then
		echo "Delivering mail: ${EMAIL}"
		MAIL_DELIVERED=`expr ${MAIL_DELIVERED} + 1`
		maildrop maildroprc ${THIS_MAILBOX} < ${EMAIL}
		# mark the mail as delivered if maildrop delivered it
		if [ "$?" -eq "0" ]; then
		    touch ${CACHENAME}
                    # TODO: we could mark the mail as READ in the
		    # INPUT_MAILBOX by moving it from "new" to "cur"
		    # and setting the appropriate flag.
		fi
	    fi
	done
    done
done
 
if [ ${MAIL_DELIVERED} -gt 0 ] ; then
    notify-send -i stock_mail-unread "New Email" "${MAIL_DELIVERED} new email(s)"
fi

Then a little script which syncs my OfflineIMAP mailboxes and delivers new mail. This is used as a cron job.

cd ~/Mail
offlineimap -c ./offlineimaprc && ./mailboxes-to-mda.sh

I wont bore you with the details of my OfflineIMAP or maildrop configs. I will point out though that I pass the remote mailbox name through to maildrop; if the mailbox is “INBOX” it does filter rules to decide which local mailbox it should deliver to, else it just delivers the mail to a local mailbox with the same name as the remote mailbox.

No Comments

Emacs Config (Mark 2)

Written by kjfletch on November 10, 2011 Categories: Code, Emacs Tags: ,

I’ve previously written about my emacs configuration, discussing the skeleton of my init.el. Recently I have been tinkering; this is a showcase of the results.

I’ll remind you that a primary goal of my config is to work out of the box so that I can can go to a new computer, branch my ~/.emacs.d, start emacs and then start working.

For this to work the config shouldn’t grumble if packages are missing and core functionality should continue unhindered.

At the basic level packages are passed to the initialisation system with a list of hooks to run upon successful loading. If a package fails to load a message will notify the user. The user has a chance to add custom/local packages using the same system.

(defvar init-package-load-list nil
  "A list of symbols representing packages to be loaded.")
(defvar init-package-load-hooks nil
  "A hook list to be called when a package is loaded. Functions
  should follow the form `(lambda (x) ...)' where the `x' will be
  symbol representing the package loaded.")
(defvar init-font-search-list nil
  "A list of fonts (as strings) to search for. The first one to
be discovered will be used.")
 
(defmacro add-init-package-load-hook (pkg fun)
  "Adds a callback function to the package load hook
list. The callback will fire when the given package is loaded."
  (let ((p (eval pkg))
	(f (eval fun)))
    `(add-hook 'init-package-load-hooks
	       (lambda (x)
		 (cond
		  ((equal x ',p)
		   (funcall ',f)))))))
 
(defun add-init-package (package &amp;rest function-list)
  "Adds a package to be loaded and a list of
functions to be called when it is loaded."
  (cond 
   ((and (listp function-list) (not (null function-list)))
    (let ((function (car function-list)))
      (add-init-package-load-hook package function))
    (apply 'add-init-package package (cdr function-list)))
   ((add-to-list 'init-package-load-list package))))
 
(defun load-packages (pkg-list)
  "Call `require' for all packages in the package
list and run package load hooks."
  (when (not (null pkg-list))
    (let ((pkg (car pkg-list)))
      (if (require pkg nil t)
	  (run-hook-with-args 'init-package-load-hooks pkg)
	(message "Warning: config failed to load feature '%s'!" pkg)))
    (load-packages (cdr pkg-list))))

We load a local config file if it exists. Giving the user chance to add packages to the list.

(setq emacs-user-file "~/.emacs-user.el")
 
(if (file-exists-p emacs-user-file)
    (load-file emacs-user-file))

Packages are loaded using the above system.

(add-init-package 'ediff 'my-ediff-config)
(add-init-package 'multi-eshell 'my-multi-eshell-config)
(add-init-package 'multi-shell)
(add-init-package 'whitespace 'my-whitespace-config)
(add-init-package 'linum 'my-linum-config)
(add-init-package 'erc 'my-erc-config)
(add-init-package 'color-theme 'my-color-theme-config)
(add-init-package 'jabber)
(add-init-package 'geiser-install)
(add-init-package 'paredit 'my-paredit-config)
(add-init-package 'python 'my-python-config)
(add-init-package 'dired+)
(add-init-package 'ruby-mode 'my-ruby-config)

We then run the load

(load-packages init-package-load-list)

An example of loading an additional (user) package from the ~/.emacs-user.el script

(add-init-package 'emms-setup (lambda ()
				(emms-standard)
				(emms-default-players)))
No Comments

Implementation of Lisp Primitives in Scheme

Written by kjfletch on November 4, 2011 Categories: Code, Scheme Tags: , ,

While learning the Scheme language I have found it hard to break the imperative habits I’ve built up over my whole programming life. I am able to write Scheme to solve problems but with a voice in my head telling me I’m doing it wrong. Rather than muddle through I have decided to swallow my pride and restart my scheming at the basics.

Here I will be implementing very simple functions that provide the primitive behaviour of Lisp style languages. This is merely an exercise to better understand some basics, Scheme already provides functions to perform these behaviours.

You will notice a heavy use of recursion. To collect cons cells to create a list.

Mapping

Perform an operation on each element of a list and produce a list of equal size containing the collected results of the operations.

For example to multiply each element of a list by n we could implement a simple mapping function

(define (multall lst n)
  (cond
   ((pair? lst) (cons (* (car lst) n) (multall (cdr lst) n)))
   (else lst)))
 
(multall '(1 2 3 4 5) 5)
;; => (5 10 15 20 25)

We can make this more general by creating a map function which allows the caller to apply any operation to the list elements.

(define (mapcar fun lst)
  (cond
   ((pair? lst) (cons (fun (car lst)) (mapcar fun (cdr lst))))
   (else lst)))
 
(mapcar (lambda (x) (* x 5)) '(1 2 3 4 5))
;; => (5 10 15 20 25)

Reduce/Fold

To apply an operation to a list and produce a single output.

For example we could create a function which takes a list of numbers and reduces the list to a sum of those numbers

(define (sum lst)
  (cond
   ((pair? lst) (+ (car lst) (sum (cdr lst))))
   ((null? lst) 0)))
 
(sum '(1 2 3 4))
;; => 10

Again, it would be more useful to provide a function which allows the caller to apply any operation to reduce a list.

(define (reduce fun lst)
  (cond
   ((pair? lst)
    (fun (car lst) (reduce fun (cdr lst))))
   (else (fun))))
 
(reduce + '(1 2 3))
;; => 6
 
(reduce + '(1))
;; => 1
 
(reduce + '())
;; => 0

Make List

A function that creates a list of a given number of elements.

Here we implement a simple function which accepts the size of the list and two optional parameters: 1) the init value of a new elements and 2) an operation to increment the initial value between elements.

(define (make-list count . rest)
  (define (inc->incfun inc)
    (cond
     ((procedure? inc) inc)
     ((number? inc) (lambda (x) (+ x inc)))
     (else #nil)))
 
  (let ((init (if (pair? rest) (car rest) #nil))
	(inc (if (and (pair? rest) (pair? (cdr rest))) 
		 (inc->incfun (cadr rest)) #nil)))
    (if (<= count 0) '()
	(cons init 
	      (make-list (- count 1)
			 (if inc (inc init) init) inc)))))
 
;; (make-list 3)
;; => (#nil #nil #nil)
;; Note: Initialise to #nil as initial value not specified.
 
(make-list 3 0)
;; => (0 0 0)
;; Note: All initialised to 0 as increment operation not specified.
 
(make-list 0)
;; => ()
 
(make-list 3 0 (lambda (x) (+ x 10)))
;; => (0 10 20)
;; Note: Increment operation defined to plus 10 for each element.
 
(make-list 3 0 10)
;; => (0 10 20)

Range

Here we build upon make-list to produce a list of numbers in a given rage.

The first version of range specifies the start of the range, the number of elements to return and, optionally, a step value (which will be 1 if not specified).

(define (range start count . step)
  (make-list count start (if (pair? step) (car step) 1)))
 
(range 0 3)
;; => (0 1 2)
 
(range 0 3 2)
;; => (0 2 4)
 
(range 2 3 (lambda (x) (* x x)))
;; => (2 4 16)

The second version of range accepts a start value, an end value and a step.

(define (range* start end step)
  (range start (+ (round (/ (- end start) step)) 1) step))
 
(range* 0 5 2)
;; => (0 2 4)
 
(range* -5 -10 -2)
;; => (-5 -7 -9)
No Comments

Running Debian Services When Synology NAS Boots

Written by kjfletch on October 20, 2011 Categories: Debian Tags: , , , , , , , ,

Previously we looked at installing a Debian system on a Synology NAS (DS110j). Now I’m going to show you how to start Debian services when the NAS boots.

In this example I will show how to run a mumble server (murmur) including dependencies (dbus). For this I installed the mumble server and configured it (to set the SuperUser password).

chroot-ds# apt-get install mumble-server
chroot-ds# dpkg-reconfigure mumble-server

On the host system we write a script which will be executed on boot. We’ll drop the script in /usr/syno/etc/rc.d/ with the name Z01debian-services.sh it’s contents:

mount -o bind /proc /volume1/debian/squeezechroot/proc
chroot /volume1/debian/squeezechroot /etc/init.d/dbus start
chroot /volume1/debian/squeezechroot /etc/init.d/mumble-server start

The script should then be given the correct permissions to allow it to run

ds# chmod 755 /usr/syno/etc/rc.d/Z01debian-services.sh

The next time the NAS is booted your Debian services will be started.

No Comments

Switch to our mobile site