James Richardson
Different Way of Thinking About Systems

As you know, I've been migrating my sytems to guix, which at the moment consists of packaging things that aren't yet present. I'm starting to wonder if the current standard of putting systems together is wrong. I know, we've been doing this way since the beginning, and saying this is wrong is perhaps too strong a statement. But maybe it is not.

What is a system?

Before I make a statement such as "The current standard of putting systems together is wrong.", I should probably define what is meant by system. I am using the word system to refer to a thing composed of a kernel and an operating system composed of packages. I'm thinking of a Linux kernel, with a GNU operating system, but the arguments can be abstracted to UNIX-like systems, such as any of the proprietary ones left (e.g. AIX, Solaris, or HP/UX), the *BSDs, and possibly systems around the GNU Hurd. For sake of concreteness, I'll assume a Linux kernel and a GNU operating system, typically any mainline GNU/Linux distribution available. The vast majority of my GNU/Linux work has been with Debian, so much of this may be colored by such experience.

A system is then a Linux kernel and a GNU operating system, which is composed of packages. The Linux kernel is also usually packaged along with the operating system and there is typically a bootstrap process which facilitates installing the operating system and kernel onto a machine, either physical or virtual.

So by saying the current standard of putting systems together is wrong, I'm really getting to the point that the way we package things is wrong.

Why are mainstream package managers doing things the wrong way?

In short, they depend on /usr and its children to maintain state. What's wrong with that? /usr is mutable and its mutability is not strictly managed by the package manager. Although package managers allow mutation in a controlled matter, they largely ignore users. If I as a user am developing a python application that depends on a particular version of a python library and the system administrator upgrades the python library to a newer version that has a different api, my application just broke.1 If I read the FHS, it says /usr/local is reserved for local software, or by de facto standards software that is not part of the operating system. It doesn't say how to handle cases where software in /usr/local depends on packages in the operating system or specific version of packages in the operating system. There typically doesn't seem to be a way of marking operating system packages as required by locally installed software or handling the case of having multiple versions of a given package installed.

Is there a better way?

Probably. Typical package managers are imperative, maintain meta-state (e.g. which packages are installed, at which versions, and dependencies) database and keep state in /usr.

Instead of an imperative approach (e.g. how to install package X), can we use a declarative approach (e.g. install package X)? Can we also remove state from the system and have state contained within the package definition? The answer to both questions is yes. I'm not going to attempt to argue the reasons in this blog post, because I wouldn't be able to so without spending much more time than I would like. Instead I will refer you to Eelco Dolstra's PhD thesis, The Purely Functional Software Deployment Model. For a much lighter read, see NixOS: A Purely Functional Linux, which speaks to building an entire stateless operating system.


  1. I know about language specific package managers (.e.g. pip, gems, cpan), I don't think they help outside of the single user case. ↩

James Richardson
Why Don't I Document Things

Einstein defines insanity as doing the same thing repeatably and expecting different results. By this definition, I'm probably insane1. I add services to my infrastructure, say for instance, MediaGoblin. I didn't document what I had to do to install it, get it to run or anything. I can remember. Well I didn't remember last time I did something. That's ok, I'll remember this time as I was paying better attention. Oh, I understand now, I'm insane, expecting I'll remember why I did something next time, unlike every previous time. As another example, I'm sure there is a reason I run the house with a 10/25 netmask. Maybe to keep game consoles off of my main network, I don't know. I think sane people would use a couple of 10/8 networks or even something like 10.0.0/24 and 10.0.1/24 and squash these into a 10.0/16 at the edge if needed. But being insane, I split the network at 10.0.0.0/25. I neglected to document why I did this.

But documentation is hard

Writing documentation is hard. Writing good documentation is even harder. Harder still is writing good documentation that is actually useful. On top of that documentation is hard, it's also not fun.

Now that we know documentation is hard and not fun, and the we are insane thinking we can rely on our memories in lieu of documentation, how do we resolve this dilemma?

Removing (some) insanity

Well, the obvious thing would be to document everything. I know, such is hard and not fun. I can't do much about the hard part. Perhaps practice. I know from this blogging thing I'm doing lately, blogging is becoming easier, perhaps the same works for documentation.

The easiest way for me to write text is with Emacs. I like org-mode and use it for most everything else. Why not use org-mode for system documentation? Well, I've done so in the past, even to the point of publishing a web site with said documentation. Well, it wasn't really workable and rather cumbersome. My new idea is instead to use markdown and publish documentation to an ikiwiki instance, the same software and workflow that powers this bliki. I get to use the same workflow as I use for publishing this site; I have nothing new to learn, there is no impedance mismatch. I use emacs to create markdown files, commit them into git, push to the remote, which builds the website.

I have created a site for my own use on my intranet which seems to be working out quite well. As I'm going through thinking about these things, I am realizing we perhaps need a new approach to thinking about system construction.


  1. I'm probably insane by other definitions, also. ↩

James Richardson
NSA Looses Control Over Hacking Tools

The New York Times has a fairly complete write up of NSA's loss of control over cyberweapons.

Remember this next time the NSA or any Government/State agency says "trust us with a backdoor into your smartphone, or other encryption". How long do you think it will be before the shadow brokers or some other organization releases their "master key"?

James Richardson
Restructured DNS Zones

Well, it is time for my DNS infrastructure to evolve, again. I run services in the jamestechnotes.com domain behind my cable modem (shhhh! don't tell my provider). Initially I used the DNS services provided by my registrar and only published public names. I quickly discovered running a mail server behind a cable modem is, well, nigh impossible. The ip is listed in a dynamic pool which most (if not all) mail servers consider a spam ip from which mail will not be accepted.

I purchased a small linode, moved my email, followed soon by many other services I run. This worked out quite well, as the linode has a higher uptime than my servers ;). DNS became rather interesting. I still wanted to keep all the hosts behind my cable modem in DNS. I needed to keep hosts and various (SRV, MX) records in DNS for public facing services. I never really liked split-horizon or split-brain configuration. Always seemed like a small error could either break the system or expose internal names to the internet.

I decided to run a public dns managed by linode's dns servers and a private dns server only accessible from nodes behind the cable modem. This worked quite well at first. The issue came when I added a new public service. I had to update the DNS at linode and I had to update the DNS locally. I would usually forget one or the other. There had to be a better way.

I added another subdomain into the mix. Everything behind the cable modem went into the lab01.jamestechnotes.com domain, my linode became lab02.jamestechnotes.com. The dns server for lab01.jamestechnotes.com is behind my cable modem, only accessible to my private network. I am now able to publish my private network to its own dns server without leakage or interference with my public zones. I still create A records in the jamestechnotes.com space to advertise public services. I could do cnames into lab02.jamestechnotes.com, but I'm not a big fan of cnames. I think cnames should be reserved for redirecting a name to a name into another zone under different control. I control jamestechnotes.com and lab02.jamestechnotes.com, so I don't really see the point of a cname, as it does put extra load on resolvers.

I've been running this way for a while now. It seems to work much better for me than trying to keep everything in a flat name space. I did have to add lab01.jamestechnotes.com to linode. I still a service that goes through my cable modem, that I need to be able to access from outside. All that is left now is ?update-dns-with-cable-modem-ip.

James Richardson
Ikiwiki and org-mode

I like emacs. I practically live in it at work. I practically live in it outside of work, also. If it learned how to browse the web reasonably well, it would probably be the only application I use. One of the best features of emacs is org-mode. Org is essentially a markup language similar to markdown or wiki-text. org-mode uses this to achieve TODO list, GTD type scheduling, and many other things. At $WORK, I'm now required to Jira to manage projects, tasks, etc. There is even a project org-jira that attempts to integrate org-mode and jira. It's not working all that great for me, creating a very bad impedance mismatch, but that is for a different post.

I've looked a blogging with org-mode, but never found a reasonable platform that supports both org-mode and the features I want. Ikiwiki has many features I like that I've not found elsewhere. I like the tag cloud, the simple way links work, and the way blogs and wikis are together. I know Chris Gray wrote a plugin so ikiwiki supports org-mode natively, sort of. It requires a running eamcs to convert the org to html for ikiwiki to consume. I didn't really want to have an emacs running on my webhost and it seemed to not support all the ikiwiki features I liked. In the end, I just use markdown for ikiwiki, mostly.

There are a few things I still maintain in org, mostly technical papers, talks, and the résumé. When I publish these to my wiki, I just export the document to markdown. I am then able to enclose markdown specific bits in #+BEGIN/#+END markdown blocks, so I get all features of publishing with org-mode including publishing to my wiki.

The biggest downside I have at the moment is I now have the org document and the generated markdown file in git.

James Richardson
Résumé in Org-mode

Résumés and LinkedIn profiles are 2 things I hate to maintain. As an IT professional in the 21st century, they are a kind of must have. Even if one is happy with their position, most of us have no way of knowing if their company will decide to offshore the work force. It seems best to have a professional presence in the web to easily be found by recruiters. LinkedIn, résumés, and the major job sites seem to be the best way to accomplish this.

As a developer, I firmly adhere to the DRY method, thus would like a single source for my résumé that can be used everywhere. I've not quite achieved that as LinkedIn doesn't provide proper API's (at least that I could find) to manage profiles. I have to cut and paste things to keep LinkedIn updated. sigh

Way back in the early 2000's, possibly before I was maintaining the résumé in LaTeX and converting it to pdf with pdflatex which worked out quite well. People, mostly recruiters, still insisted on a Word formatted résumé. I don't think recruiters in general read résumés, they just scan them into some sort of database or hack off headers and personal identifying information to send to their clients. I suspect it is easier to do so with Word than with a pdf. Pandoc, at least at the time, didn't really do a satisfactorily job and conversion to Word (or odt).

I discovered org-mode, I don't know when, but a long time ago. Org-mode is an emacs mode for managing org files. Org is, at its core, a markup language, but emacs uses such mark up to maintain TODO lists, note taking, outlining, GTD, and many other things. Org also has exporters which can export org to (amongst others) LaTex, markdown, and odt.

Now I have my résumé in org. It is still plain text so git can track changes. I can export to LaTeX and then pdf, so I get a copy I can send to people or upload to job boards. I can also export to plain text so I can cut-n-paste into my LinkedIn profile. As an added bonus, I can export to markdown, so I have a copy online in my wiki.

Still I have few things to automate. I'm not sure the best place to add the hook at the moment. Should I add the hook to emacs, so when it saves the resume.org file, it causes the pdf to be generated and uploaded to the web, then exports it to markdown for inclusion into the wiki or should this be in git commit hook?

James Richardson
Shutting Down Personal Blog

I've shut down my personal blog. The domain is being redirected to this post as way of explanation.

My blogging efforts are solely directed at my technical blog.

Why did I shutdown the personal blog?

I have trouble maintaining the technical one here. I can add personal bits as I see fit, it is after all my site ;).

I don't particularly want a bunch of random personal bits floating around the interwebs for people to mine. One of the reasons I set the thing up was to encourage the children to write. That was a total failure.

I shall have ngnix redirect the site here. My technical site is here.

Posted
License: cc by-nd 4.0
James Richardson
Interruptions Are Like Exceptions

Interruptions and "in the zone"

I hate to get a phone call or an instant messaging popup on my desktop. It usually (perhaps always) breaks me out of the "zone". What is the zone? Being "in the zone" means I have all the bits of the problem I'm working in my head have tuned the rest of the world out and can actually be productive. We all know knowledge workers work best by being in the zone or getting into the flow where they are fully concentrated on their work and have fully tuned out their environment. Writers, software developers, engineers, scientist, and even basketball players will tell you about being in the zone.

Getting into the zone is not easy. Various places on the web claim from 15 to 30 minutes to get into the zone. Obviously, the time is dependent on the complexity of the problem and the mental state of the person. If I'm tired, it's harder for me to get in the zone. If I know I'm going to have to stop and attend a typically agendaless meeting in an hour or so, will it's sometimes hard to get the motivation to even start.

Maker Schedule vs Manager Schedule

This leads into concept that Paul Graham called the maker's schedule (as opposed to the manager's schedule, where task switches are the norm). Personally, I clearly work on the maker's schedule. Having an interruption is like throwing an exception. It doesn't merely cause me to switch tasks, it changes the entire mode in which I work. My brain doesn't save stack frames properly (or at all). When I return from the exception, there is no where to return, so I have to drop core and start over. That takes effort. If I know ahead of time that I will have to stop for a meeting, or for lunch, or to leave to go to the house, it sometimes seems like wasted effort to even attempt to get back in the zone. I don't have any concrete measurements, but if feels like I get more done when I work from home and tune out the workplace than when I go to the cubical farm and have the noise from other cubicals, people walking by (not stopping, just walking down the aisle), constantly in my head. Wish my brain had a feature where it would make all of that into white noise.

How do I deal with this?

Apparently, not very well. I keep the ringer silenced on my phone quite often. Work gets bent out of shape at times over this. The spousal unit goes ballistic if I don't answer the phone or a text message immediately. I don't have have voice mail setup on my cell. My preferred way of being contacted is email, preferably encrypted. I realize this creates an impedance mismatch between myself and people who want to contact me primarily over non email channels. Perhaps that's because I'm an intp and have my own way of looking at things.

I don't like phone calls, unless they are prearranged with a purpose, otherwise it is just like and angendaless meeting.

Summary

Given that you don't know what I'm doing, try not to interrupt me. I understand that you thing you are more important than me (you aren't) or that what you need to tell me is so important (it isn't) I have to stop what I'm doing to do what you want. Emails are nice (except at work, where outlook insists on interrupting me, and I have to use outlook at work because they make me. I prefer to not use proprietary software anywhere, but that's a different topic) and can even be encrypted so prism can't read them. Voice mails, text messages, and many forms of instant messaging, not so much.

Emails (outside of work) will not interrupt me. Neither will IRC messages. My Emac's status line keeps be abreast of those things. It doesn't beep, doesn't pop up things. It's just there in my subconscious, along with anything else on the status bar. Emacs is nice that way. It doesn't cause an exception.

So don't IM me, telling me hello, or asking if I'm there. If what you got is so important it just has to be over IM, just tell me what it is. Skip the niceties or the time waste or whatever the hello, or are you there questions are supposed to achieve. You've already interrupted me, just tell me what you want, I might can help. I will not (usually) respond to a hello or an are you there? question on IM.

Please don't leave me a voice mail (you can't on my personal phone) asking me to call you back without telling me what you want. I won't return the call.

Also don't send me an email asking me if it's ok to call. It isn't. If the email specifies what you with to talk about, fine. I'll send an email asking for clarification or more information or whatever. If there's sufficient interest, then we can setup a properly agendad (what is the verb form of agenda?) meeting over the phone.

I usually have more tolerance for family, peers, coworkers, and friends (and occasionally recruiters). Family, peers, coworkers, and friends also are used to my idiosyncrasies.

The initial impetus for this post was to rant about phone calls, but it evolved into a ranty post about interruptions. I still don't have a really good way to deal with interruptions. There is room for improvement on my usual workflow...

Posted
License: cc by-nd 4.0
James Richardson
Why I'm Migrating to Guix

Most of you know that I've been using Debian rather happily for a very long time, since hamm was stable, possibly before. I like Debian for many reasons, the social contract, its commitment to free software, although it seems to favor open source over free.1 and things are done in public. They also make it fairly trivial to use non-free software, which may be a good thing or a bad thing.

Back in the late 90's or early 2000's, I inherited the responsibilities of managing the subscriber email system for the phone company I was working for. I started looking to rip out sendmail for something better. Not that such is really relevant, other than qmail was a thing back then, which led me to the discovery of daemontools. I happily used daemontools until I discovered its replacement, runit, which I still use today. Until Jessie it was quite easy to not use the default SysV init system and use runit, even as PID 1. With Jessie it became harder, I didn't need the hassle so I started looking for other distributions.

Distributions using the Linux kernel and not using systemd are, well, scarce. Distributions using the Linux kernel, with systemd as the default init, but with the choice of using something else are even scarcer2. One of the advantages of free software is choice. If I don't like something, I can change it. Systemd has largely taken that choice away from GNU/Linux users. I think by design.

I'm not quite sure when I discovered Guix. I had seen Nix earlier. It seemed a strange way to build an operating system. If NixOS is strange, and GuixSD shares many of the ideas, why is the former strange and the latter not? I never stated the GuixSD isn't strange, it does indeed have very different ideas and semantics of how a GNU/Linux distribution should be put together. I happen to like it. GuixSD is committed to free software and is very hackable, as it composes packages as guile modules, even system services are composed as guile modules. This means that GuixSD doesn't use systemd, but not for systemd hater reasons.

I am now actively working to migrate all of my boxen and VPSes to GuixSD. The biggest blocks for me right now are lack of lvm support (I really don't want to give up lvm support for my file server), lack of a decent ikiwiki package, although I have it mostly working, with the modules I use, and lack of git-annex.

More about my work on guix is here.


  1. At one time a believe free and open were the same. I now understand differences. ↩

  2. One may even say nonexistent. I've not looked at every distro on distrowatch and probably none that aren't, so that's just a guess. ;) ↩

James Richardson
Removed Everything From Cloud Hosting

Github recently changed their ToS. At my read it seems to be rather anti-GPL, or really anti-copyleft. As I'm not a lawyer, I'm not going to do an analysis here, but see https://www.mirbsd.org/permalinks/wlog-10_e20170301-tg.htm and FSF Response.

I removed all my repos from github, gitlab (which I don't really think was bad) and bitbucket (which may have been worse). I did such the wrong way. I actually delete my account. What I should have done reinitialize the git index and have left just a readme file pointing to the new location. I had to recreate my account to I can contribute to projects there, as the projects I'm interested insist on using github's pull request model.

I now self host repos and use mr to sync with repo.or.cz for backup purposes.

I have started a code wiki where I talk about projects where I have interest or involvement.

Posted
License: cc by-nd 4.0
James Richardson
I Hate Writing Résumés

I hate writing résumés. I hate the entire application process. There has got to be a better way.

I know there is LinkedIn, but I don't like their new ToS. I still have an account, but will delete it soon, probably.

I also don't want to actively hunt for other work. I already have a job. The work isn't bad. I would prefer to be able to use more free/libre tools. I like the people on my team, and many others in the company. I don't particularly like senior management at the moment, but that will probably change at some point. Hunting for work is a drag; I have better things to do.

I also don't want to stay ignorant of what the market looks like. I found the job I have now the usual way with résumés, recruiters, and LinkedIn. I found the job before that the same way, and the job before that (maybe that one was before LinkedIn), and before that, etc. Albert Einstein defined insanity as doing something over and over again expecting different results. Perhaps I'll try something different.

Maybe I'll try to build my personal brand. I can become known as an expert in things I'm interested, rather than things my employer is interested. Typically they are coincident. I shall build a thing called a résumé into this wiki and see what happens. The trick will be to structure the data in such a way that the same source can be used to feed the wiki and an actual résumé that recruiters will want.

James Richardson
I'm bothered by phrases in many Terms of Services (ToS)

Several months ago, Github updated their ToS, which read to me more like a license grant rather than a ToS. I'm not sure GPL'ed code is compatible with the new ToS, apparently others do not share my opinion, which is fine. I'm not going to rehash the arguments here as they are not hard to find and I'm not a lawyer. I did take the action of deleting my Github account. I also deleted my Gitlab and Bitbucket accounts, as I was getting little to no value add from these, and now I have 2 less ToS to monitor.

I logged into LinedIn, they also have a new ToS. I didn't see nothing in it particular bad, per se. I don't like some of the phrases saying they can use my data as they see fit. I'm considering deleting my account there. I still occasionally receive an email via LinkedIn. My current employer seems to be offshoring jobs, so I may need LinkedIn at some point in the near future. I'm trying to build my own personal brand with this wiki and becoming more active in the Free Software and Open Source communities. Don't know how successful I shall be, as this is rather foreign to me.

James Richardson
Perl and AIX

I have been working with AIX since I started my present job back in February, 2011. I wasn't really familiar with AIX at the time. I've learned a bit since then, mostly that it is an odd system. Seems IBM, looked at BSD and System V, took the best parts out, and merged what was left and called it AIX, but I digress.

As there are not really any HLL languages available for AIX, I wrote a lot of stuff in perl. I'm not a particularly big fan of perl, but it does seem to be a necessary evil for sysadm work. I'd much rather write Python or most any lisp. ;) The really annoying thing about perl on aix is it is a horribly ancient version, 5.8 to be exact. I wrote a program that takes source tarballs with a gnu build system or similar and transforms such into an installp package.

Apparently, at some point, I upgraded a couple of core libraries, such as File::Copy. I needed to use the cp variant so that files could be copied including permission bits, much like the the cp command found on most POSIXish systems.

I did not take proper notes. When the box migrated to AIX 7.1, perl was upgraded to a merely ancient version of perl, 5.10. The upgraded core libraries were no longer upgrade, my program had hard to find bugs deep in its innards. I found this entire thing rather frustrating, especially as most of it was actually self inflicted.

I think I shall sort out better ways to do document what I have done, perhaps looking at readme driven development and/or test-driven development. I'm also looking at Guix to sort out even better ways to manage things.

James Richardson
Debian releases Stretch

Debian released Debian 9 (Stretch) . I've been running Debian as my primary distribution since Hamm, which was a really long time ago. I think the first distribution I used was Yggdrasil, which is no longer. I may have first used Slackware but I disremember.

When Jessie was released, I decided to look for alternatives as I do not like systemd. Is that enough to make me abandon Debian after all these years? In and of itself, no, of course not, there are plenty of other packages within Debian that I do not use. The issue I have is that I have no choice. Not that I'm a big fan of SysVInit either. I've been using Daemontools and then runit as my init system since I discovered them1. Starting with Jessie, it became harder to use a different init system, even with Sid using a non default init system is becoming tedious. I have better things to do with my time, so I chose to leave Debian. I have to choose to either learn systemd2 or use a distribution that hasn't drank the systemd kool-aid. I choose to do the latter.

As I have chosen not to drink the systemd kool-aid, I am now faced with choosing either a distribution that doesn't require systemd or use a kernel other than Linux. I looked briefly at the *BSD's, but found them to be too much anti-GPL for my tastes. I then looked at GNU Hurd, but found it lacking too much to use as my primary OS. I then discovered Nix and NixOS, which i think, in general, are moving in the right direction. Soon after I discovered Guix and GuixSD. Guix seemed to be the right fit for me. I am on a journey to migrate all of my boxen to GuixSD and run Guix on top of everything else I can.

I had wanted to move to GuixSD before Stretch was released. That didn't happen. My primary laptop has been running GuixSD for a while. The spousal unit's laptop has been running GuixSD for a few weeks.


  1. Probably late 1990's when looking to move from sendmail at the ISP I was working at. ↩

  2. I have to learn systemd anyway for $WORK. ↩

James Richardson
SouthEast LinuxFest 2017

I attended SouthEast LinuxFest June 9-11, 2017. I've been attending every years for the past 4 years or so. This one, as usual, had a very good participant turnout. Some of the bigger vendors such as RedHat and Rackspace were not present. Otherwise, I thought it was good event and I learned many things and met some interesting people. Even picked up few more signatures on my gpg key.

This year I did a crazy thing for an introvert. I presented a ?talk on functional package management with Guix. This was the first time I ever gave a talk in public. Overall, the experience was fun and educational. I ended up talking too fast and finished early, but that actually seemed a good thing as there were many interesting questions.

I hope to give a talk next year.

New contact info

I finally got around to updating my contact info. I now have a new email address which I publish. Hopefully, I have a good spam solution.

I no longer use xmpp/jabber. Guess I got tired of talking to myself. If you use such and wish to contact me via that channel, let me know, I may set the server back up.

My preferred method of contact is encrypted email.