Using VimWiki as a distributed, encrypted lab notebook for programming

So what’s the most important coding tool for a software engineer in the long-term? There’s a pretty decent argument that it’s a lab notebook. I’ve kept one for years and it’s one of the more useful tools I have, but I found that my notebook, apart from not quite measuring up to how a lab notebook should be kept, wasn’t quite as useful as it could be because when you have five or six notebooks covering a bit over a decade’s worth of notes and no search function, then how do you use them daily?

I’ve been looking around for a while for a software version of a paper notebook without much success (this isn’t ludditism – we just don’t have very much that can match what paper and pen can do for this kind of task). I’ve used a simple text editor on a cron job for a few years as a way to track what I do during the day – every 30 minutes, up pops a vim window with a ready-inserted line listing the date and time, I scribble in what I’ve done since the last time the window popped up and I save and quit – it takes about 15 seconds and it gives me a file listing everything I’ve done at the end of the day/week/month/year which is handy for things like preparing reports, doing job reviews, that sort of thing. It’s simplicity itself to set up, just call a simple script using a simpler cronjob entry:

[cc escaped=”true” lang=”bash”]

#!/bin/bash

# Cron script for 30-min activity journal
#——————————————-

export DISPLAY=:0
echo -n -e “\n[” `date` “] :\n” >> ~/.journal
/usr/bin/gvim -U ~/.journal.gvimrc -geometry 100×40+512+400 + ~/.journal

[/cc]

[cc escaped=”true” lang=”bash”]

# m h dom mon dow command
0,30 8-20 * * mon-fri ~/.journal.sh

[/cc]

Simple and effective… but not a lab notebook. It does help test a few criteria for what I need that notebook program to do to fit in with my workflow though:

  • It has to be *fast* and familiar to use. ’nuff said, really. 
  • It has to be distributed. I work on lots of machines; I need it to be available on them, or else I’d have to have it on something like a netbook that I’d carry everywhere and that’s just not practical for me (besides, what if I forgot it or it got stolen?)
  • It has to be encrypted. Everything I work on is NDA’d at one level or another; and design/debugging notes would definitely be too sensitive to leave lying around in plaintext format. With a physical notebook, I keep it locked away; but this is supposed to be better than that option…

Colleagues have used various products for this over the years – mindmapping software; emacs in one mode or another; and various other software. But none of those really appealed. Mediawiki did seem to be as good a fit as I could find, but something that depends on an entire LAMP stack to run is hardly lightweight; and while I could host it somewhere public, that’s not really very secure (I’d spend more time making sure the full LAMP stack was up to date and mediawiki too than I want to). Besides, I’d rather this be console-accessible if possible (yes, some of us are still happier that way 🙂 ).

I’ve been using vim since around 1993 or so; at this point it’s wired into my fingers. So when I saw vimwiki, it seemed ideal. For those who’ve not encountered vimwiki before, it creates a directory, and then every file in that directory becomes part of a rudimentary text-based wiki (which it can turn into a set of HTML pages so it can handle images and so forth, but you can also navigate it from within vim). It also has a diary function which works in a sub-directory of the wiki directory.

It doesn’t have any support for encryption or distribution. But that’s quite solvable.

The encryption is easy enough – you could use the blowfish encryption in (post-v7.3) vim but that proved a bit awkward as you had to reenter the password every time you navigated down a link (and I don’t always have post-7.3 vim available). This password entering every minute or so broke up my workflow, so no thanks. My netbook and work laptops all have whole-disk-encryption, so I just left the vimwiki directory as normal on those laptops, and on the machines where I don’t have whole-disk-encryption, I use eCryptFS to create an encrypted directory and put the wiki under that. Very simple indeed, but quite effective. Now even theft of the physical hard drive isn’t a major concern.

The distribution was equally simple; you could use any DVCS, but I’m fond of mercurial, so I decided to use that. You have to tweak the vimwiki script ( .vim/ftplugin/vimwiki.vim ) to call it:

[cc escaped=”true” lang=”vim”]

augroup vimwiki
au! BufRead /home/mdennehy/vimwiki/index.wiki  !hg pull;hg update
au! BufWritePost /home/mdennehy/vimwiki/*  !hg add ;hg commit -m ” “;hg push
augroup END

[/cc]

But that’s a simple tweak at best. And you want to have ssh setup with keys for the easiest workflow, but you have that already, right? 😀

Then just modify the crontab script:

[cc escaped=”true” lang=”bash”]

#!/bin/bash
# Cron script for 30-min activity journal
#——————————————-

export DISPLAY=:0
/usr/bin/gvim -U ~/.journal.gvimrc -geometry 100×40+512+400 -c “call vimwiki#diary#make_note(v:count1)” + -c “r !date +’\%n= \%H\%Mh =\%n'”[/cc]

And add an Awesome keybinding and menu entry:

[cc escaped=”true” lang=”lua”]

vimwiki_cmd = “/usr/bin/gvim -U /home/mdennehy/vimwiki/.gvimrc -c ‘call vimwiki#base#goto_index(v:count1)'”

mymainmenu = awful.menu({ items = { { “awesome”, myawesomemenu, beautiful.awesome_icon },


{ “VimWiki”,vimwiki_cmd }
}
})

awful.key({ modkey, }, “w”, function () awful.util.spawn(vimwiki_cmd) end),

[/cc]

And now whenever I hit -w from within Awesome, it pops up a gVim window open at the root of the wiki; every 30 minutes it pops up a gVim window in today’s diary page with the time inserted automatically for a log entry; and whenever I hit save or switch buffers, it syncs the files up to a central server’s encrypted area.

Distributed, encrypted, fast and useful. I’ve been using it in the job for the last few months now and it does almost everything I need. I do still keep around the paper notebook though – no matter how good the program, we still don’t have anything that can do everything paper can do (doodle, take cornell format notes, sketch diagrams easily for later capture, that sort of thing), but vimwiki’s search function alone is making it the day-to-day workhorse and it’s making my life a lot easier. Notes on development, patent ideas, job review reports, sysadmin notes, notes on papers I’m writing, and a daily log, all in one easy-to-use package. Damn useful tool.

Tags: , , , , ,

22 comments

  1. Did you ever try Evernote? Seems like it does mostly everything you need.

  2. I use evernote at the moment on my S2, but it doesn’t even come close to doing everything I need, it wants to store all my data (which rather thoroughly breaks every NDA I’ve ever signed), and it’s not fast enough, and I’ve not yet found a console app for it.

  3. check out [freeplane](http://freeplane.sourceforge.net/wiki/index.php/Main_Page) – its usefull to organize knowledge in a tree-like maner – very slick with keyboard shortcuts (no mouse)

  4. Doesn’t make the cut – too heavyweight, won’t run on a console, not encrypted, not distributed.

  5. Try aes.io – it’s not console, but encrypted, and has several ways to organize notes (folders/flat tags/hierarchical tags)

  6. No, because that would mean giving my data to a third party beta service, which would both break NDAs *and* tie a fairly critical part of my working toolset to some company’s beta test programme!

  7. I really like this, but unfortunately OS X doesn’t come with gvim. Here’s a similar version to your crontab that I created to get it to work. (The following requires macvim https://code.google.com/p/macvim/)

    report.sh:

    #!/bin/bash
    export DISPLAY=:0
    echo -n -e “\n[” `date` “] :\n\n” >> ~/.journal
    /Applications/MacVim.app/Contents/MacOS/Vim -c “normal G” -c ‘startinsert’ ~/.journal -g

    This will also place your cursor at the end of file and start vim in Insert Mode.

  8. What about Simplenote, have you tried it all?

    It’s OS/device agnostic, as it takes any format (that is plain text).

  9. Nope; won’t be trying it either because:

    • it’s a commercial service run by someone else; if they fold, I’d be stuffed.
    • I need closer control of the data than they offer – if I can’t be sure exactly where the files are and what their security is, I could be in breach of NDAs without knowing it.
    • It’s too heavyweight compared to vim.
  10. Thanks for the useful post! Bemused by the other commenters suggesting solutions that obviously don’t fit your requirements…

    I had something similar set up using gnote and lipsync[0]; console access wasn’t as important to me, I didn’t need (or rather didn’t think I needed) full versioning, and wanted to sync other (binary) stuff too. I eventually dropped it because lipsync was too flaky, but it’ll probably be a worthy alternative to a DVCS in a solution like yours some day.

    [0] https://github.com/philcryer/lipsync

  11. I did try vimwiki before, but was not happy when it stopped suddenly working after upgrade. Since then I’m more careful and happy with gf way of linking or UTL plugin.
    Project site looks ok, but after noticing few bugs like “ENTER doesn’t seem to follow link in OS X terminal”, “new 2.0.1 version incompatibilities” I’ll wait…

  12. How about dropbox for the distribution? Too heavyweight?

  13. No, dropbox would work (and a lot of vimwiki users use it for this); but like I’ve been saying on HN, dropbox is cloud storage, and like I’ve been saying, cloud storage and EU/US law and NDAs just make a big hairy mess that I don’t want to play with: http://www.mcgarrsolicitors.ie/2009/09/11/cloud-computing-european-data-protection-dangers/

  14. I took a stab at this with macvim and OSX LaunchAgents too. Neat Idea.

    https://gist.github.com/4197022

  15. Very cool to see you are using GVIM for Powershell scripting. Syntax hightighling using Peter Norvost’s plugins works great. I use the vibrantink color scheme myself. Your scheme is pretty cool too.Curious though .do you have omnifunc (auto-completion) working? If so, how? I think it would be very cool to have powertab like auto-completion for powershell scripts. Using the VIM supertab plugin, I have it working for everything but powershell scripts. When I try to use it, it says something to the effect of omnifunc is not set (but as I mentioned, I only see this for .PS1 files).CK

  16. I ve been using vimwiki recently and is great.

    As for the doodles and drawings thing, you could use another software for that and just insert the resulting image via local file or url into the vimwiki.

    Regarding the cornell thing… you can create auto formatted tables using vimwiki-tables

    Distribution … you could use btsync, a decentralized way of syncing files and folders with encrypted communications included.

  17. The thing with the doodles, the diagrams and so on – it’s speed. There just aren’t any software packages that are as fast or as flexible as paper is for that kind of thing, not yet.

  18. Do you synchronise the decrypted files with DVCS? Or is you encrypted directory within your ~/vimwiki folder?

    Im confused that the mercurial commands seem to be executed within the vimwiki’s directory

  19. I sync the decrypted files using DVCS (Mercurial to be exact) but over SSH links and the files are being encrypted at a layer below the DVCS level (either that portion of the filesystem or the entire hard drive is encrypted below the normal open()/read()/write()/close() API level so it’s transparent to the DVCS). End result is that Mercurial never has to worry about encryption but the files remain secure and encrypted at all times anyway, even when content is in flight between machines).

  20. It was only after I posted the question I noticed you are using eCryptFs and not EncFs as I initially miss-read.

    It is a great setup, if you own the hardware on all involved machines. I am doing something similar with Encfs, since I don’t own all the machines, just accounts on some of them.

  21. Have you tried Zim wiki with Truecrypt for encryption? Zim wiki is already a pretty cool knowledge management tool designed for nerds. Couple it with truecrypt and you got a beast.

    Here: http://dotpad.blogspot.in/2012/12/zim-with-truecrypt-and-dropbox-my-final.html

  22. What a brillant setup and helpful article. Thank you.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.