My 2018 in review

I know it’s filler for news orgs, but I like year in review #content, so here’s mine. This is the public portion of a basic internal self-audit I did this year and plan to do again next year.


I successfully executed two large projects as a self-employed IT guy back in my hometown. I set up much of the networking and inventory for a convenience store in collaboration with a gas station technician, the clerks, and the owners. I also set up a two-building enterprise-grade hotel WiFi network in collaboration with management. I left a few other things unfinished when I changed jobs, including one big project for a neighbor, so I had to write refund checks and send some sincere apologies. Still, I’m mostly satisfied with the work I did.

That said, I’m pleased I’m now doing something else. I like my job at Earlham. I work with a group of talented students administering high-performance computing clusters and other campus-scale server systems – a phenomenal way to boost my experience and grow as a member of a community. I’m competent enough now in my technology skills to explore my prospects in software development or system administration after Earlham.

I’m starting out, so I’m experiencing many of the challenges of starting out – but it’s a good start.


I moved!

When I graduated college in December 2016, I wasn’t sure what I wanted to do, and moving back home was my safety net (I’m fortunate that I had it). After over a year, that had to change. When a job opened at Earlham, I applied and was selected. I moved a few weeks later. Lifestyle, more than any factors in being self-employed, drove me to Earlham.

There’s plenty of criticism of the idea that changing your environment improves happiness over time. I take that point, but for me it has been an important step.

That said, I still don’t get out much socially, and that’s the next major change I need to make in my life.


Like a lot of us, consuming culture takes up a lot of my leisure time. For details see my previous post!

Creativity probably goes in this category as well, and for that see my series on making things and why I think it’s important. I intend to share more creative work in the future.


After a year of stagnation (on net) in 2017, the year 2018 was a transitional year for me. It contained some important moments and provided a lot of clarity, though it was fundamentally about setup, not action.

I like that I took in a lot more culture, which I interpret as one way of participating in our shared experience as a society. I’m also satisfied having learned to run a business, moved, changed jobs, and succeeded at each of those steps. I have a sense of substantial progress over December of last year.

Thanks to the work done in 2018, it’s possible I’ll have a very good year in 2019. Shortly I’ll publish a post looking forward, discussing how I intend to make the most of next year.

My 2018 in culture and media

This was part of my upcoming year-in-review post. It got too long so I spun it off. It’s possible to consume a lot of media in a year.

(Assume links are not safe for work – I haven’t vetted all of them recently.)


The big one was Robert Caro’s The Power Broker, his stunning doorstopper about Bob Moses, which I spent months reading on and off in 2017 and 2018 before finishing. I love long books and this is a good example of a book that absolutely had to be long in order to capture the story and its central character. In addition, I read and enjoyed two Vonnegut novels (God Bless You, Mr. Rosewater and Cat’s Cradle) and Maureen Johnson’s young adult mystery Truly Devious.

I also read for professional growth. I reread Cal Newport’s Deep Work, which I consider the definitive book about how to do creative or technical work. I also read Peak: Secrets from the New Science of Expertise by Anders Ericsson and Robert Pool, which covers similar themes.

I spent way more time reading articles than reading books, which is typical for me. Rather than share all of them, here are some favorite articles on politics, pop culture, and tech in the past year:

In the new year I’d like to shift my balance of reading in favor of books and away from articles, but that’s not my top priority so it may not happen.


If the book list was too short, this one is probably too long. I swap out podcasts relatively often based on mood and interest, but here are some highlights:

Movies and TV watched

I started a classic film kick around a year ago. I’ve started with big-name classics. I was joined midway through the year by the excellent Unspooled podcast, got my heart broken by the loss of Filmstruck (I’m likely to subscribe to the Criterion Channel soon!), and came to love film and the art of film. In order to not forget, I’ve built a spreadsheet of everything I’ve seen. I’m tempted to write some movie reviews and whatnot.

Here are some movies I watched for the first time this year and especially liked – spanning genres, tones, etc. Not all fall into that “classic film” category, but as I don’t see many movies in theaters most are from before 2018. The list is roughly in the order I watched them, not a ranking:

  • Citizen Kane
  • Love, Simon
  • Treasure of the Sierra Madre
  • Titanic
  • O.J.: Made in America
  • Bride of Frankenstein
  • Little Shop of Horrors
  • Quiz Show
  • The Iron Giant
  • Duck Soup
  • Bohemian Rhapsody

(Note: “here’s a list of movies I like” is a good opportunity to not tell me why I should hate them instead. 🙂 )

For film YouTube I like:

Each of those links goes to one of my favorite of their videos.

For television, my highlights were Breaking Bad (another great opportunity: don’t tell me I should have seen this much sooner!), The Flash, and Queer Eye.


I played through all of the Kingdom Hearts series this year, my first sustained gaming experience since the Banjo series I loved as a kid. And KH is great! Looking forward with everyone else to the release of Kingdom Hearts 3 in a few weeks. For a completely different experience but one that makes use of a greater range of the capabilities of the PS4, check out Horizon Zero Dawn.

Coda: More stuff

This doesn’t really belong anywhere but it’s all stuff I liked.

POST-SCRIPT: While compiling this post I noticed a need to diversify my sources next year. Fortunately, a nice thing about the Internet is I know I can find a lot of great content by creators of color, women, etc.

The sanitation layer of the Internet

Facebook is clean, while the rest of the Internet is not.

Along with network effects, birthday reminders, news- and information-gathering, and other surface-level benefits, the fundamental clean user experience is what (for now) gives Facebook an edge over everything else online.

Facebook is, in no particular order…

  • safe to log into.
  • trustworthy with regard to basic tools like adding friends, joining groups, following people, and sharing links.
  • devoid of NSFW imagery and similar content.
  • pleasant to look at, with soothing blues and grays.
  • familiar.
  • easy to explain.
  • a just-fine networking tool.
  • well-understood by the general public.

The rest of the Internet is comparably bad. To quote Tim Wu:

Today, you wander off the safe paths of the Internet and it’s like a trap. You know, you click on the wrong thing, suddenly fifty pop-ups come up, something says, hey, you’ve been infected with a virus, click here to fix it, which of course, if you do click on it, it does infect you with a virus, it’s teeming with weird listicles and crazy things like, reason number four and how you can increase your sperm count or something, and you have to kind of constantly control yourself. You have to be on guard, it’s worse than, it’s a mixture of being in a bad neighborhood and a used car sales place and a casino and a infectious disease ward, all combined into one, and that is not relaxing. Yeah, let’s just put it that way.

We could add: those obnoxious full-page ad spreads, malicious JavaScript, phishing links, link rot, post-GDPR cookie warnings on every other website, and programs that install extensions to your browser because you were in a hurry and didn’t un-check that one box in that one view.

Compared to Facebook, much of the Internet is…

  • dangerous.
  • untrustworthy.
  • rife with NSFW content.
  • ugly, gaudy, tacky.
  • weird and full of tricks.
  • devoid of obvious context.
  • hard to explain.

In other words, it’s everything that the Facebook timeline has been designed and engineered to avoid.

If social media companies and their services are as bad as a lot of people think, we should understand the advantages they do provide as part of imagining and creating alternatives. This is one of those advantages.

Misadventures in source control

Or, what Present!Me ever do to Past!Me?

I observed a while ago on Twitter that learning git (for all its headaches) was valuable for me:

This was on my mind because I recently set about curating (and selecting for either skill review or public presentation) all the personal software projects I worked on as a student. It was a vivid reminder of how much I learned then and in the few years since.

Every day since then I have observed more version control errors of mine, and at some point I thought it worth gathering my observations into one post. Here is a non-comprehensive list of the mistakes I observed in my workflows from years past:

  • a bunch of directories called archive, sometimes nested two or three deep
  • inconsistent naming scheme so that archive and old in multiple capitalization flavors were together
  • combinations of the first two: I kid you not, cs350/old-string-compare/archive/archive/old is a path to some files in my (actual, high-level, left-as-it-was-on-final-exam-day) archive
  • multiple versions OF THE SAME REPO with differing levels of completion, features, etc. (sure, branching is tricky but… really?)
  • no apparent rhyme or reason in the sorting at all – a program to find the area under a curve by dividing it up into trapezoids and summing the trapezoid area was next to a program to return a list of all primes less than X, and next to both of those was a project entirely about running software through CUDA, which is a platform not a problem
  • timestamps long since lost because I copied files through various servers without preserving metadata when I was initially archiving
  • inconsistent use of README‘s that would inform of me of, say, how to compile a program with mpicc rather than gcc or how to submit a job to qsub
  • files stored on different servers with no real reason for any of them to be in any particular place
  • binaries in some directories but not others
  • Makefiles in some directories but not others

(You may have noticed that parallelism is a recurring theme here, and that’s because it was a parallel and distributed computing course where I realized that my workflows weren’t right. I didn’t learn how to fix that problem in time to go from a B to an A in the course, but after that class I did start improving my efficiency and consistency.)

To be fair to myself and to anyone who might find this eerily familiar: I never learned programming before college, so much of my college years were spent catching up on the basics that a lot of people already knew when they got there. Earlham is a place that values experiment, learn-by-doing, jumping into the pool rather than (or above and beyond) reading a book about swimming, etc. Which is good! I learned vastly more that way than I might have otherwise.

What’s more, I understand that git isn’t easy to pick up quickly and poses problems for accessibility to newcomers. Still I can’t help but look at my own work and consider it vastly superior to trying to make this up as you go. It’s well worth the time to learn.

Git and related software carpentry were not something I learned until quite a while into my education. And that’s a bit of a shame, to me: if you’re trying to figure out (as I clearly was) how to manage a workflow, do appropriate file naming, etc. concurrently with learning to code, you end up in a thicket of barely-sorted, unhelpfully-named, badly-organized code.

And then neither becomes especially fun, frankly.

I’ve enjoyed the coding I’ve done since about my junior year in college much more than before that, because I finally learned to get out of my own way.

What counts as making?

As I pursue becoming a maker of things, the two things I want to make are software and writing. If I can spend time making those two things, I’ll be happy.

Obviously you can make more than those things, and for anyone reading this I want to be generous with my definition of “making”. Here’s a non-exhaustive list of what I think we can include:

  • writing, which I mean in the almost limitless fashion that includes “the Great American Novel”, screenplays, and 10,000-word blog posts but also fan fiction, appliance manuals, etc.
  • coding
  • running a home media server
  • starting a business
  • film or video
  • music, including performances of pieces you didn’t yourself write
  • a podcast
  • handcrafts
  • woodworking
  • metalworking
  • social media posts are borderline, but if you’re doing it in a systematic way and not just for marketing, chit-chat, or blowing off steam, it counts
  • photography (it’s not all phone snaps)
  • illustration
  • learning to cook new meals or improve your skills in what you already know how to cook
  • science experiments
  • home decoration and design

I would also consider the following as making, or at least making-adjacent. Their inclusion may be controversial as they arguably don’t produce some external artifact for others’ enjoyment, but for many people they offer the same types of satisfaction that making does: purpose, fulfillment, legacy, impact, sense of progress, agency, a valuable way to spend finite time on this earth.

  • athletics, or at least fitness with emphasis on improvement, technique, and maybe participation in some group activity
  • teaching

This should go without saying in the age of open-source software and ubiquitous blogging/microblogging, but whether or not you make money from something does not factor into whether it’s “making.”

A plan for making

In my last post I discussed why I’ve concluded that I need to make more. This post looks forward, and it will explain how I plan to start and continue creating.

In that pile of articles I’ve saved up, I can find a lot of advice about habits, systems, workflows, etc. But I’m mostly drawing from the work of Cal Newport, whose Deep Work and So Good They Can’t Ignore You both changed my outlook on work within the last year or two.

I’d note that all, some, or none of this may be broadly applicable:

  • If you’re already hold a job where making is what you do from 9-5 (the sort of job I’d like to have!), this probably won’t be especially useful.
  • Since this is a post about getting started in a creative habit, it’s also focused on projects conducted by an individual, not a team. The patterns in those two kinds of work are different.
  • I’m posting this mostly as a public expression of a private commitment to myself.
  • This is by no means binding. I’ll be revisiting this list in the future and adjusting my methods based on how well it actually works for me.

The short version, as I currently imagine it, is this:

  • Clear distractions.
  • Make the time and space.
  • Do the work.
  • Share it, with 0 or more people.

That’s it!

Details follow on how to go from that design to an implementation that can last.

0. Clear distractions.

This is item 0 because it’s not actually part of the creative process. That’s very important to say: clearing distractions is about getting out of your own way, not a method of creating. By itself it does not add value (economic, social, personal, spiritual, etc.) to the world. It’s about making space for what you actually want.

Action steps:

  • Freedom: Run the Freedom app. I’ve experimented with it for a year now, and the way that works best for me is to run a fairly strict blocker on a repeating schedule – but not to disable quit during sessions. This lets me look things up if, say, I’ve blocked Reddit but there might be good information on /r/learnprogramming about something I’m learning. I recommend blocking social media, news, and all sites you mindlessly click to when you want to waste time while you’re working.
  • Pick simple tools or tools you know well. Choosing the “perfect” tools can easily chew up a lot of your time. (Of course, if there is a tool that is THE tool for what you’re doing, learn that.)
  • Have your environment ready. The time to pick your instrumental Spotify playlist for audio while you’re working is before the work session, not during it. Ditto lighting, seating, temperature, colors in your terminal, choice of word processor, etc.

1. Before making anything else, make time and space.

Put creating on your calendar. If necessary, consider it an appointment. A creative work session is a time during which you are unavailable except in emergencies. No email, no phone, etc.

Actions to take:

  • Add it to the calendar. I’m going to start with 30 to 60 minutes each day, longer when I need more depth. I’d like to go for 90 on, say, weekends. If you can do this in such a way that it accomplishes tasks for your job, that’s excellent and you could probably do it during work hours. If not, make sure to reserve the time outside work, at least a few days a week. Either way, add it to your calendar and stick to that appointment.
  • Be prepared to fill that time. Don’t arrive unprepared to make. “Write”, for example, might be on the calendar, but if you don’t know what to write, the path of least resistance is to hem and haw and not do anything. Prevent that by knowing what you’re going to work on in advance.

2. Do the work.

This is the only stage that counts. It’s the only non-subjective item on this list.

None of the rest of this matters, for purposes of this project, if it doesn’t ultimately help make this happen.

At the chosen time, in the chosen environment, and for the duration of the chosen time, create. Words and drawings onto paper and screen. Code into editor. Sound waves into air.

Action steps:

  • Make!
  • If you have 60 to 90 minutes, use the time to do deep work: focusing on a single cognitively-demanding task for an extended period of time to improve your own skillset. This may require a little more advance planning in the previous step.

3. Share your work, with 0 or more people.

For me, this is the hardest stage. Part of my motivation for publishing this list is to let me isolate it, in my mind, from the other stages.

All those questions I listed in my previous post about managing feedback, responses, comments, traffic, etc.? They all crop up now, when you “put it out there” for others to see.

This is also the most difficult stage to turn into action steps, because it varies so much by your medium. But here’s my attempt:

  • Create a site or blog for each part of your work. I’ve got my personal digital home right here, a developer blog hosted at GitHub, both private and public code repos on different platforms, and a blog for my writing.
  • Automate sharing tools, if you’d like. I’d rather pick and choose as I go, but the option to share across social networking accounts exists on most web tools you’d want to use.

Some caveats

There are a ton of lifestyle factors that matter to making as well: manage time well, stay healthy, exercise, get outside away from screens, etc. But for the specific work of making, this is the path that makes most sense to me: clear, schedule, work, share.

I also believe readers ought to willfully disregard any of this that won’t work for them because of personality, lifestyle, schedule, etc. But if you’re wondering how to start putting in the work, this is one possible framework that might at least provide some useful tactics.

On making

For a long time I’ve been operating in one mode of learning and personal growth, and I’m in the process of pivoting to another.

Since sometime in college, I have followed what we might call a consume-curate-archive model of learning. I read online frequently, so I have hundreds of articles saved to Instapaper about a range of topics. I read books. I guzzle podcasts, and I use Tweetdeck to follow a lot of carefully-refined Twitter lists about politics, tech, culture, and more.

This is a process of finding inputs, highlighting or annotating them, and stashing them away. It’s a kind of orderly digital hoarding, and I’ve been doing it for a long time.

At the same time, my creative output slowed to a crawl. I’m not sure if there was a direct 1-to-1 tradeoff of those two (any writer will tell you that reading is essential to better writing), but certainly I was gathering a lot of inputs and not using them to produce rare and valuable outputs.

There are a lot of reasons for that. Basic lifestyle is one factor, especially after college. Unsure what I wanted to do, I felt directionless and frustrated. After being across the country for college, I moved back home and worked for myself for about a year. (A contemporary indicator of the simmering desire to make was the joy I felt designing and creating my solopreneur business.) After a few months on my current job, at Earlham, I feel better consistently. Maybe I should have created in spite of my mood, but I didn’t, and now I feel like I can again.

I also experience typical human anxieties about reactions. Some of my work can be private, but some should be public, and how do I sort which is which? We know deliberate practice, “learn by doing” guided by consistent feedback, is the only path to mastering something, but how do I implement that? Say I decide writing is what I want to do. Do I have to spend all my time keeping track of website comments and worrying about traffic? Can I make money from it? Should I? I feel like I have a lot to say, but am I then violating my own privacy just to get attention or make a buck? If I succeed, do I have to be a “public figure” (I don’t want to be)? There’s a halting problem to be modeled here, and my conjecture is that this kind of thinking never terminates on its own.

All of that was enough to smother my creativity for a long time.

That’s no way to live, to work, or to grow professionally. That’s why, in the last few months, I’ve been gradually switching into a creation-first model of learning – a model in which producing some artifact (at a high level it doesn’t much matter what kind) is the guiding principle, rather than accumulating a lot of neatly-arranged inputs.

This is the first in a series of posts I’ll be writing about making. In my next post, I’ll share how I’m planning to orient myself, on a regular basis, into a pattern of creativity.

Generate an idea list

It’s been a while since I had a side project going, so I’ve been itching do to something creative. But I’ve been in the developer’s equivalent of writer’s block for a while, hung up on a lack of ideas or at least a lack of good ideas.

I think I’ve now replenished the idea supply, thanks to this exercise.

I spent a few hours tonight generating 101 ideas for software projects. Half or more of them will never become anything. A few are absolute garbage. A bunch of them might be good afternoon projects, possibly worth sharing, but aren’t especially inspired.

A few seem actually good, and they’re what the exercise is all about.

If you’re in a creative rut, especially if (like me) you’re fairly young or early in your career, I highly recommend this.

Here’s how I did it:

  • Pick a number, let’s say N. I picked N=101 because it’s (in this context) a big number that would be a stretch. I also like that it’s JUST over 100, so your 100th isn’t a complete fluke.
  • Open a notes app, spreadsheet, word processor, text file with line counters, dead-tree notepad, etc.
  • Start writing your list of N items.
  • Fill the list in one sitting exactly, if at all possible. (You can always revisit this!)
  • Put it down afterward. I just finished mine and don’t intend to look it over until at least tomorrow. I want to see it with fresh eyes to choose the best prospects.

I suggest these criteria for whether to add an idea to the list:

  • Be specific…: An item should be a short but complete description of a project. One sentence or phrase is enough if it tells you everything you need to know.
  • … but vague enough: Stay away from describing programming languages, environments, web servers, etc. – those questions can come if you decide a project is worth actually implementing.
  • Make the project scope appropriate for team size: I was thinking up individual side projects, so I wanted ideas I could implement on my own using mostly the Internet for external information. A group/team/company project would be different.
  • Otherwise reserve judgment: If you must, add a parenthetical statement afterward on the problem (e.g., a few of my ideas raised concerns about copyright, but I added them anyway because as a technical matter they would be doable). You can also bold an idea you feel especially good about in the moment. If possible don’t get any more judgy than that.
  • It can be based on something you’ve seen, but modify it and be mindful of law and ethics: Inspiration is inevitable, especially at the idea stage. Be sure to make it your own, though, or it’s not much of an “idea”. Worry about copyright, terms of use, etc., if you ever consider publishing or distributing what you make. For now, add the idea if you think it would work for you given your context and purpose.

Fundamentally, this is about quantity not quality.

This is my first time doing such an exercise in the context of software, so by all means change the rules to whatever works for you. These worked for me, though, so I suspect they may work for others.

A final cautionary note: I worked alone, and I suspect I would have created a totally different list working in a group (i.e. brainstorming). Your mileage may vary there.

Looking for bugs in all the wrong places

When I took Earlham’s Networks and Networking class, we implemented Dijkstra’s algorithm.

Dijkstra’s algorithm is an algorithm for finding the shortest paths between nodes in a graph, which may represent, for example, road networks. It was conceived by computer scientist Edsger W. Dijkstra in 1956 and published three years later.

The algorithm exists in many variants; Dijkstra’s original variant found the shortest path between two nodes, but a more common variant fixes a single node as the “source” node and finds shortest paths from the source to all other nodes in the graph, producing a shortest-path tree.


I got my implementation (in Python) close, but not quite right, by the time the deadline hit for submission.

And more deeply than I have for any coding project up till now, I always felt bad about falling short on this one. I trained much of my perfectionism out of me to become a CS major and decent programmer, but this particular hangup hit hard. In hours of work, I couldn’t find where my implementation was going wrong, or why it was going wrong so consistently.

I submitted my code for grading unhappily, then put it down to focus on other things. I felt like I’d reached my upper limit as a programmer (though I knew in my mind that this was probably not the case). The source code lay quietly in a directory for a couple of years.

Today I’m happy to report that – judged exclusively my own irrational metric, success in implementing Dijkstra’s algorithm – I underestimated myself.

This semester I’m helping teach the same networks class. Since we may assign Dijkstra’s algorithm at some point, I decided to review my old code and maybe try to make it work.

I spent about two hours today, Sunday, reading that rusty old code, tweaking it, running the new version, and parsing its output. I added debug statement after debug statement. I ran it on different input files.

Then I noticed a mistake in the output. Somehow, an edge of weight 1 was being read as an edge of weight 100000000 (the value I used to approximate infinite cost, i.e. the cost of moving directly between two nodes that do not share an edge). In effect, that edge would never be part of a shortest-path between any combination of source and destination. This was bad, because in fact that edge was part of many such shortest-paths in this network.

I went back to some of the most basic pieces of the code and found a possible problem. It was small, easy to fix but hard to detect. I edited a single line of code and ran the program.

It worked.

As it turns out, I’d gotten the implementation right. The core of the assignment, Dijkstra’s algorithm itself, had worked on the input it received.

Visually, here’s the network I had:

And here’s the network the program thought I had:

So what did I get wrong?

Believe it or not: counting.

You see, I had set a variable for the number of nodes N in the network graph. I also had a two-dimensional list describing the network, where each item in the list was an edge in the graph, itself represented by a list containing two nodes and the weight to go between them. Crucially, there are at most N^2 edges in such a graph.

My fatal flaw: rather than saying “for each possible edge in the network, read a line from the file”, I said, for each node in the network read a line in the file. In other words, for my graph with up to N^2 edges, I would only be loading the data about N of them. In this case, the program read only 4 lines, and the edge of weight 1 was described on the 5th line.

(This might have been obvious had I tested the code more thoroughly on one of the larger network files we had. Alternatively, the combination of edges being missed might have obscured the result a lot. A copy of the same input file, but with the lines reversed, would have been the most useful second test case.)

After switching the variable that the index would be checked against, everything worked as I expected.

The code still has problems. I intend to clean it up and streamline it. But the implementation now consistently returns correct output.

The concrete lessons of this experience for me are:

  • Don’t just write debug statements. Write clear and meaningful debug statements. Be specific.
  • Check your I/O, indices, and other such basic features of the code. You can have the greatest algorithm of all time (though I did not!), but if the program isn’t handling exactly what you expect it to, you won’t get the results you want.
  • Vary the input. Vary the input. Vary the input.
  • Don’t let one project, however important or complex or valuable, determine your feelings about your personal skillset.

Finally, while I emphasized the specific and silly programming error here, failure to count correctly wasn’t a root cause of my mistake. The root causes were factors removed from coding altogether: rushing to completion and getting too tangled in the weeds to think holistically about the problem. I don’t think it’s a coincidence that I solved this problem after spending a lot of time in my life disciplining those tendencies.

The perks of being a VM

Several of the CS department’s servers are virtual machines. While running VM’s adds complexity, it also lets you do things like octuple* system RAM in five minutes from your laptop.

For context, Earlham CS runs a Jupyterhub server for the first- and second-semester CS students. We want to provide a programming environment (in this case Python, a terminal, and a few other languages) so students can focus on programming instead of administration, environment, etc. Jupyter is handy for that purpose.

The issue: Each notebook takes a relatively large amount of RAM. There are 60 or so intro CS students here. The Xen virtual machine hosting Jupyter was simply not equipped for that load. So at the request of my colleagues teaching the course, I visited a lab today. After observing the problem, we took five minutes to shut the server down, destroy the volume, change a single number in a single config file, and bring it all back to life with a boosted configuration. We’ve had no additional problems – so far. 🙂

Running a VM is frequently more complex than running on bare hardware. But the alternative is this:

I wish I had some of the “upcoming maintenance” email notifications we sent out in my ECCS sysadmin days for comparison. They were basically “no email, no websites for several days while we rebuild this server from parts, mmmkay?”


Because we do so much of our administration in software, we’ve mostly avoided that problem in recent years. The closest we’ve gotten to scrambling hardware lately was recovering from disk failures after a power outage over the summer. We had to send a lot of “sorry, X is down” emails. I wouldn’t want that to be our approach to managing all servers all the time.

(Of course there are many other alternatives, but running Xen VM’s serves our purposes nicely. It’s also, for many reasons, good practice for our student system administrators.)

*I tweeted about this originally and said we quadrupled the RAM. In fact, a previously-arranged RAM doubling had been specified in the VM’s config file but not implemented. Before we restarted the machine, we decided to boost it even more. Ultimately we quadrupled the double of the previous RAM amount.