Th.Oughts : Hacking Gnus expiry

When mail grows in size, one thing that hurts the most for Gnus users is mail expiration. The default rule, as far as I understand it, is to run the process for a group when you exit it. Some admins/users set gnus-group-expire-all-groups to run when you exit gnus. So, an easy out is

(remove-hook 'gnus-summary-prepare-exit-hook
             'gnus-summary-expiry-articles)

(remove-hook 'gnus-Exit-group-hook
             'gnus-group-expire-all-groups)

However, the problem now is that you need to remember to manually run expiry once in a while. Someone like me can easily forget that!

Another option is to fetch mail locally before serving to Gnus. I have used offlineimap with pretty good results. But note that the simplest backend to use - nnfolder is also terribly slow when handling large amounts of mail. So, you would have to end up running a local dovecot instance too. On resource starved hardware, this could be a problem.

Another way, I got around this is by timing by expiration process. Mail expiry is meant to be fire-and-forget, but you really don't need to run it more than say, once a day.

  1. On the first run, when exiting a group, create an assoc list of the following format

    ((group1 . time1) (group2 . time2) ...
    
  2. Once we have a list with some entries is when the fun starts. We probably don't need to run the expiry process on a group that was just processed. We could check that with this example code snippet

    (let ((entry gnus-newsgroup-name))
      (if (check-entry-in-ignore-list entry)
          ;;use the default behavior
          (gnus-summary-expire-articles)
        (progn
          (let ((last-expiry-time (cdr (assoc entry gnus-expiry-table))))
            (if (and last-expiry-time
                     ( > (- (float-time) (string-to-number last-expiry-time))
                         gnus-group-expiry-time-limit))
                (progn
                  (let ((time (number-to-string(float-time))))
                    (setq gnus-expiry-table
                          (acons entry time gnus-expiry-table)))
                  (gnus-summary-expire-articles)))))))
    
  3. And finally, when exiting a group, we check times for all groups. But we are not done yet! We need to store the times we last ran expiration to a file

    (defun write-string-to-file (string file)
      "Simple Function to write a string of the form - \"group value\""
       (if string
           (with-temp-buffer
             (insert string)
             (when (file-writable-p file)
               (write-region (point-min)
                             (point-max)
                             file))))
    
  4. Things are different the next time you fire up Gnus. We need to load up the last known entries from the file we just wrote (I will call it .gnus-expire)

    (let ((file gnus-expire-log-file))
      (if (file-exists-p file)
          (let ((entries (with-current-buffer
                             (find-file-noselect file)
                           (split-string
                            (save-restriction
                              (widen)
                              (buffer-substring-no-properties
                                                       (point-min)
                               (point-max)))
                            "\n" t))))
            (mapcar (lambda (tuple)
                      (let ((groupname (car (split-string tuple)))
                            (value (cadr (split-string tuple))))
                        ;(message "%s %s" groupname value)
                        (setq gnus-expiry-table
                              (acons groupname value gnus-expiry-table))))
                    entries)))))
    

And now, we have a populated associative list to perform step 2 above.

Here's the complete source

;;; expiry-hack.el - Don't expire often!
;; Author: Bandan Das <bsd@makefile.in>
;;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; This assoc list will store the expiry times
;; associated with groups. If there's a .gnus-expiry-file,
;; this gets filled up with the last time expiry was
;; performed
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
(defvar gnus-expiry-table nil
  "This is the run-time assoc list")
(defvar gnus-expire-log-file "~/.gnus-expire"
  "The file where the expire times are written")
(defvar gnus-group-ignore-list nil
  "Don't apply our expire logic for anything here")
(defvar gnus-group-expiry-time-limit 86400
  "When to run expiry")
;; remove some hooks first

(defun gnus-read-last-expiry-times()
  "Read the expiry times from the file, populates gnus-expiry-table"
  (let ((file gnus-expire-log-file))
    (if (file-exists-p file)
        (let ((entries (with-current-buffer
                           (find-file-noselect file)
                         (split-string
                          (save-restriction
                            (widen)
                            (buffer-substring-no-properties
                             (point-min)
                             (point-max)))
                          "\n" t))))
          (mapcar (lambda (tuple)
                    (let ((groupname (car (split-string tuple)))
                          (value (cadr (split-string tuple))))
                      ;(message "%s %s" groupname value)
                      (setq gnus-expiry-table
                            (acons groupname value gnus-expiry-table))))
                  entries)))))

(defun write-string-to-file (string file)
  "Simple Function to write a string of the form - \"group value\""
   (if string
       (with-temp-buffer
         (insert string)
         (when (file-writable-p file)
           (write-region (point-min)
                         (point-max)
                         file)))))

(defun check-entry-in-ignore-list (entry)
  "Just check an entry against the ignore list"
  (setq match nil)
  (setq count 0)
  (setq found nil)
  (while
      (and (not found) (< count (length gnus-group-ignore-list)))
    (when (string-match (elt gnus-group-ignore-list count) entry)
      (setq found t))
    (setq count (+ count  1)))
  found)

(defun gnus-group-custom-expire-single-group()
  "This is called when you exit a group, functionally
   similar to the function below to expire all groups"
  (let ((entry gnus-newsgroup-name))
    (if (check-entry-in-ignore-list entry)
        ;;use the default behavior
        (gnus-summary-expire-articles)
      (progn
        (let ((last-expiry-time (cdr (assoc entry gnus-expiry-table))))
          (if (and last-expiry-time
                   ( > (- (float-time) (string-to-number last-expiry-time))
                       gnus-group-expiry-time-limit))
              (progn
                (let ((time (number-to-string(float-time))))
                  (setq gnus-expiry-table
                        (acons entry time gnus-expiry-table)))
                (gnus-summary-expire-articles))))))))

;gnus-group-marked
(defun gnus-group-custom-expire-all-groups ()
  "This function gets the list of all groups,
  and creates a new list out of it based on -
   - if there is an ignore list
   - If we have matching entry in gnus-expiry-table
  and we need to process expiry"
  (interactive)
  (setq to-write nil)
  (save-excursion
    (gnus-message 5 "Running custom expire...")
    (let ((gnus-group-tmp-list (mapcar (lambda (info)
                                       (gnus-info-group info))
                                     (cdr gnus-newsrc-alist)))
          (to-write nil))
      (dolist (entry gnus-group-tmp-list)
        (if (check-entry-in-ignore-list entry)
            (add-to-list 'gnus-group-marked entry) ;;default behavior - add to expiry list
          (progn
            (let ((last-expiry-time (cdr (assoc entry gnus-expiry-table))))
              (if (and last-expiry-time
                       ( > (- (float-time) (string-to-number last-expiry-time))
                         gnus-group-expiry-time-limit))
                  (add-to-list 'gnus-group-marked entry))
              ;but write it so it gets accounted for the next expiry!
              (setq to-write (concat to-write
                                     (concat
                                      entry " "
                                      (number-to-string (float-time))
                                      "\n")))))))
      (write-string-to-file to-write gnus-expire-log-file))
    (gnus-group-expire-articles nil))
  (gnus-group-position-point)
  (gnus-message 5 "Expiring...done"))


(provide 'expiry-hack)

There are comments.

All posts

  1. kvmclock notes
  2. Pyblosxom + git blogging
  3. Linked list design in the Linux kernel
  4. Make Controller on Linux
  5. ThinkBlink
  6. Home dir versioning
  7. Coolest Patch ever
  8. Versioning your home dir
  9. Multiple route support (equalize)