I’m making websites!

As the exclamation point indicates, I’m excited to announce this: I’m now making websites again!

A bit over two years ago, I left self-employment as an all-around tech services provider and joined my alma mater, Earlham College. That was a good move. I have built my skills across the board, and having this job has kept my career steady through e.g. COVID.

However, I’ve missed some of the work from those days, as well as the independence. I don’t like having only one income source in a time of high economic unpredictability. I also want to continue expanding my skillset, growing my portfolio, and controlling the course of my own career.

For all these reasons, I’m accepting new projects effective now. You can click here to seen plans and examples or reach out (cearley@craigearley.com) hire me to make a website for you.

My particular passions are making websites for individuals and small businesses (including online stores). Most likely if you’re at a larger scale than that, you have in-house web and sysadmin teams anyway. 🙂 If what I offer is right for you, please reach out. I look forward to hearing from you.

Meet our Terrestrial Mapping Platform!

Just a nice photo from Iceland

I’m excited to share that the Earlham field science program is now sharing the core of our Terrestrial Mapping Platform (TMP)! This is very much a work-in-progress, but we’re excited about it and wanted to share it as soon as we could.

We had to delay the 2020 Iceland trip because of COVID-19. That of course pushed back the implementation and case study component of this project, which was Iceland-centric. But we are moving forward at full speed with everything else. As Earlham has now started the new academic year, we have also resumed work on the TMP.

The project is a UAV hardware-software platform for scientists. It consists of:

  • a consumer-grade drone for capturing images
  • flight plan generation software and application to automate drone flights
  • data analysis workflows for the images – visible light and NIR, assembled into 2D and 3D models

All of this goes toward making science more accessible to a broader range of domain scientists. Archaeologists and glaciologists are our current target cohort, but many more could find use for this work if it’s successful.

We will make all of this accessible in repositories with open licenses on our GitLab instance. Some are already available. Others we will share once we review them for (e.g.) accidentally-committed credentials.

That was all planned, if delayed. We’re also using our extra year of preparation time to make the project better in a few ways:

  • Reevaluating our choice of UAV make and model
  • Prettifying our web presence, which very much includes blog posts like this
  • Reducing the friction and pain points in our current workflow
  • Making our code and infrastructure better in general (I’ve covered my growing emphasis on quality here before)

The team mostly comprises students and faculty (of whom I’m the junior-most). Additionally, there are a few on-site partners in Iceland and innumerable personal supporters who make this possible. We’ll be sharing more at the Earlham Field Science blog as we go. I will undoubtedly share more here as well.

COVID is bad, but we want to make the best of this era. This is one way we’re doing that.

(Disclosure: We received funding for this from a National Geographic grant. None of the views in this blog post or our online presence represents, or is endorsed by, Nat Geo.)

Golang and more: this week’s personal tech updates

First I haz a sad. After a server choke last week, the Earlham CS admins finally had to declare time-of-death on the filesystem underlying one of our widely-used virtual machines. Definitive causes evade us (I think they are lost to history), so we will now pivot to rebuilding and improving the system design.

In some respects this was frustrating and produced a lot of stress for us. On the other hand, it’s a sweet demo of the power of virtualization. The server died, but the hardware underlying it was still fine. That means we can rebuild at a fraction of the cost of discovering, purchasing, installing, and configuring new metal. The problem doesn’t disappear but it moves from hardware to software.

I’ve also discovered a few hardware problems. One of the drones we will take to Iceland need a bit of work, for example. I also found that our Canon camera may have a bad orientation sensor, so the LCD display doesn’t auto-rotate. Discovering those things in February is not fun. Discovering them in May or June would have been much worse.

Happier news: I began learning Go this week. I have written a lot of Java, C, and some Python, but for whatever reason I’ve taken to Golang as I have with no other language. It has a lot of the strengths of C, a little less syntactical cruft, good documents, and a rich developer literature online. I also have more experience.

A brief elaboration on experience: A lot of people say you have to really love programming or you have no hope of being good at it. Maybe. But I’m more partial to thinking of software engineering as a craft. Preexisting passion is invaluable but not critical, because passion can be cultivated (cf. Cal Newport). It emerges from building skills, trying things, perseverance, solving some interesting problems, and observing your own progress over time. In my experience (like here and here), programming as a student, brand-new to the discipline, was often frustrating and opaque. Fast forward, and today I spent several hours on my day off learning Golang because it was interesting and fun. 🤷‍♂️

Your mileage may vary, but that was my experience.

Finally, here are a few articles I read or re-read this week:

Earlham’s four-day weekend runs from today through Sunday. After a couple of stressful weeks, I’m going to take advantage of the remainder of the time off to decompress.

Learning on my own

One of the CS servers choked this week. That made for a stressful recovery, assessment, and cleanup process.

To cut the stress, I put in about an hour each night to some autodidactic computing education. That included a mix of reading and exercises.

In particular, I walked through this talk on compilers.

As I noted in my GitHub repo‘s README, I did fine in my Programming Languages course as a student, but I was never fully confident with interpreters and compilers in practice. People (accurately!) talk about building such programs as “metaprogramming”, but as a student I found they always came across as more handwavey or tautological than meta.

This exercise, which I’d emphasize consists of code built by someone else (Tom Stuart, posted at his site Codon) in 2013 for demo purposes and not by me, was clarifying. Meticulously walking through it gave me a better intuition for interpreters and compilers – which are not, in fact, handwavey or tautological. 🙂 The Futamura projections at the end of the article were particularly illuminating to discover and think through.

I also read some articles.

  • Teach Yourself Programming in Ten Years” (Peter Norvig; re-read)
  • Purported origin of “never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway”. (This was evidently considered an extremely funny joke among 1970s computer geeks.)
  • The performance links in this post.
  • The next language I’d like to explore is Go, and I started this week.

Computing lets you learn on your own. It’s not unique as a field in this way, but I appreciate that aspect of it.

My new laptop

Now that Apple fixed the keyboard, I finally upgraded my Mac.

I am a non-combatant in the OS wars. I have my Mac laptop and a Windows desktop. I have an iPhone and build for Android at work. I run Linux servers professionally. I love everybody.

My little 2015 MacBook Air is a delightful machine. But it wasn’t keeping up with my developer workloads, in particular Android app builds. I bought the $2799 base model MacBook Pro 16″.

I started from scratch with a fresh install. Generally I try to avoid customizing my environments too much, on the principle of simplicity, so I didn’t bother migrating most of my old configs (one exception: .ssh/config).

Instead I’ve left the default apps and added by own one by one. I migrated my data manually – not as daunting as it sounds, given that the old laptop was only 128GB and much of it was consumed by the OS. I closed with an initial Time Machine backup to my (aging) external hard drive.

Now I’ve had a couple of weeks to actually use the MacBook Pro. Scattered observations:

  • WOW this screen.
  • WOW these speakers.
  • WOW the time I’m going to save building apps (more on that later).
  • I’m learning zsh now that it’s the Mac’s default shell.
  • Switching from MagSafe to USB-C for charging was ultimately worth the tradeoff.
  • I was worried about the footprint of this laptop (my old laptop is only 11-inch!), but I quite like it. Once I return to working at my office, I think it will be even better.
  • I am running Catalina. It’s fine. I haven’t seen some of the bad bugs people have discussed – at least not yet.
  • I’m holding on to my old Mac as a more passive machine or as a fallback if something happens to this one.

Only one of those really matters, though.

Much better for building software

The thing that makes this laptop more than a $2799 toy is the boon to my development work. I wanted to benchmark it, not in a strictly scientific way (there are websites that will do that) but in a comparative way in the actually existing use case for me: building Android apps.

The first thing I noticed: a big cut in the time taken to actually launch Studio. It’s an immediate lifestyle improvement.

I invalidated caches and restarted Studio on both machines. The two apps opened at the same time (not optimal performance-wise, but not uncommon when I’m working on these apps intensively).

I then ran and recorded times for three events, on each machine, for both of the apps I regularly build:

  • Initial Gradle sync and build
  • Build but don’t install (common for testing)
  • Build and install

Shock of all shocks, the 2019 pro computer is much better than the 2015 budget-by-Apple’s-standards computer (graphs generated with a Jupyter notebook on the new laptop, smaller bars are better; code here):

Yeah. I needed a new computer. 🙂

I expect 2020 to be a big year for me for a number of reasons I’ll share over time, and my old laptop just couldn’t keep up. This one will, and I’m happy with it.

I sure hope there’s enough to do…

I begin the month-long winter break this weekend. Students and teaching faculty finish about a week later. When classes reconvene in January, we’ll start spending a lot of time on Iceland and its spinoff scientific computing projects.

To lay the groundwork, we’ve spent the last few weeks clearing brush:

  • Updating operating systems and apps on a dozen laptops, handsets, and tablets
  • Syncing accounts with Apple and Google
  • Sitting in on planning/logistics meetings
  • Coordinating with the students who will do most of the actual research and development
  • Producing the list of software improvements we need to make

The last item is the most substantial from the perspective of my colleagues and students in CS. We build a lot of software in-house to collect and visualize data, capture many gigabytes of drone photos, and run datasets through complex workflows in the field.

It takes a lot of work locally to make this succeed on-site. Students have to learn how to both use and develop the software in a short time. Since the entire Iceland team (not just CS students) depends on everything we build, these projects provide real and meaningful stakes.

All of this has come together in the last few weeks in a satisfying way. We’re up to 62 GitLab issues based on our experiences using the software. That’s a good enough list to fill a lot of time in the spring, for both students and faculty.

We’ll hit the ground running in January, when the clock officially begins ticking.

Computing lessons from DNA analysis experiments

I’ve been working with my colleagues in Earlham’s Icelandic Field Science program on a workflow for DNA analysis, about which I hope to have other content to share later. (I’ve previously shared my work with them on the Field Day Android app.)

My focus has been heavily experimental and computational: run one workflow using one dataset, check the result, adjust a few “dials”, and run it again. When we’re successful, we can often automate the work through a series of scripts.

At the same time, we’ve been trying to get our new “phat node” working to handle jobs like this faster in the future.

Definitions vary by location, context, etc. but we define a “phat node” or “fat node” as a server with a very high ratio of (storage + RAM)/(CPU). In other words, we want to load a lot of data into RAM and plow through it on however many cores we have. A lot of the bioinformatics work we do lends itself to such a workflow.

All this work should ultimately redound to the research and educational benefit of the college.

It’s also been invaluable for me as a learning experience in software engineering and systems architecture. Here are a few of the deep patterns that experience illustrated most clearly to me:

  • Hardware is good: If you have more RAM and processing power, you can run a job in less time! Who knew?
  • Work locally: Locality is an important principle of computer science – basically, keep your data as close to your processing power as you can given system constraints. In this case, we got a 36% performance improvement just by moving data from NFS mounts to local storage.
  • Abstractions can get you far: To wit, define a variable once and reuse it. We have several related scripts that refer to the same files, for example, and for a while we had to update each script with every iteration to keep them consistent. We took a few hours to build and test a config file, which resolved a lot of silly errors like that. This doesn’t help time for any one job, but it vastly simplifies scaling and replicability.
  • Work just takes a while: The actual time Torque (our choice of scheduler) spends running our job is a small percentage of the overall time we spend shaping the problem:
    • buying and provisioning machines
    • learning the science
    • figuring out what questions to ask
    • consulting with colleagues
    • designing the workflow
    • developing the data dictionary
    • fiddling with configs
    • testing – over, and over, and over again
    • if running a job at a bigger supercomputing facility, you may also have to consider things like waiting for CPU cycles to become available; we are generally our systems’ only users, so this wasn’t a constraint for us

A lot of this is (for computer scientists, software engineers, etc.) common sense, but taking care to apply that common sense can be critical for doing big interesting work.

The punchline of it all? We managed to reduce the time – walltime, for fellow HPC geeks – required to run this example workflow from a little over 8 hours to 3.5 hours. Just as importantly we developed a bunch of new knowledge in the process. (I’ve said almost nothing here about microbiology, for example, and learning a snippet of that has been critical to this work.) That lays a strong foundation for the next several steps in this project.

If you read all this, here’s a nice picture of some trees as a token of my thanks (click for higher-resolution version):

Image of trees starting to show fall color
Relevance: a tree is a confirmed DNA-based organism.

A tale of two large-ish app updates

This week I spent some time working on Earlham CS’s Field Day Android application. It’s the app used by our student-faculty field science researchers to collect data on trips to, say, a glacier in Iceland. I made two substantial changes.

The first was updating our system dependencies. At the start of the summer, Field Day wasn’t a fully modern application. That’s mostly because its development is contingent on the interest levels of students and faculty who (correctly!) have other priorities during the academic year. We experience our only consistent spikes in development during preparation for a trip to Iceland. Even then, we tend to focus on adding or fixing features, rather than major design choices or boring updates. Whatever their benefits, such changes always risk eating up precious time in the short run.

As a result, we had long neglected to update SDK versions, themes, and other app fundamentals. I wanted to fix that before classes resumed this month.

Not being an Android expert (yet?), I relied on a mix of automated tools in Android Studio, manual code tweaks, and careful testing to push the update process forward. Here’s how I described it in my merge request:

I wanted to make us a “grownup” application, by which I mean that I wanted to move us away from as many deprecated tools and dependencies as possible, as far in advance of a field trip as possible. (EDIT: With one exception: these changes do not attempt to resolve the [looming] Google Drive [API] deprecation.)

To that end, this merge request involves substantial changes to build fundamentals like the Gradle version, as well as some Lint cleanup and general tidying. Much of it was done following a simple pattern:

– run a built-in Android Studio update tool (e.g. “Update to AppCompat”)

– change a bunch of details in the code so it builds

– test on the device

– lather, rinse, repeat

Field Day merge request 9

After some tests by myself and a colleague, I approved the merge.

To reward myself for accomplishing that admittedly tedious process (which followed a long, slow battery testing process), I did something more fun.

For a long time I’d wanted to improve Field Day’s UI to streamline the navigation. I made a batch of changes, then submitted the following merge request:

[Field Day’s original creative developers] created a great design palette for Field Day: fun fonts, bright colors, intuitive icons.

I wanted to keep that but update the navigation to reflect the current understanding of our usage model. To that end, this merge centralizes everything onto one screen, miniaturizes our less-used buttons, and puts database and sensors at the forefront.

No specific activities or fragments other than the main screen (and the deletion of the obsolesced sensor screen) have been changed.

I can foresee a future where we do more data analysis and aggregation through the lab notebook, so I’ve preserved the notebook icon for future use.

Field Day merge request 10

The changes in that request took us from this set of two main screens:

Previous main screen (“Sampling” takes user to the second screen)
Previous second screen, containing our sensor and database features

… to this one screen:

Our most commonly-used buttons are on the main screen and fill the entire screen width.

I again checked with my colleague and then approved the request. I’m now working on other issues and have already found the changes to be substantial boosts to the user experience.

This is a sample of my own personal work, but of course building software is a team sport. And it relies on iteration. The original designers of Field Day – current and former colleagues of mine – did a lot of the heavy lifting over a few years building the core logic and aesthetic of the app. As I made my changes in the last few months, I’ve worked to maintain their original design palette while improving usability, performance, and the underlying data model. It’s a useful, specialized, and dare I say fun application, and I want it to keep getting better.

As a closing note about process, I find it sharpens my skills development when I have to summarize my work into prose, as in these merge requests. Writing them requires more precision than a quick chat in a hallway. That’s to say nothing of possible benefits to future developers trying to retrace changes and intentions.

Simple battery testing for common use cases of an Android application

In developing the Android app we use to collect field data, I recently completed a series of tests. We wanted to determine how worried we should be about the Android “Location” tab’s judgment that Field Day is a “High battery usage” app. We wanted to avoid installing and running any additional apps to do so.

Protocol

The simple (if slow) protocol we developed to meet those requirements goes something like this. It’s best when run on multiple, comparable devices at the same time.

  1. Fully charge the device.
  2. Restart the device.
  3. Close all applications other than the app you’re testing.
  4. Disable sleep, screen turn-off – any battery-saving features that apply to your phone but do not fit the use case of Field Day when we actually run it.
  5. Do any setup tasks – in our case, linking to a remote database and an Arduino sensor platform to collect data for us.
  6. Run it for hours not minutes. In our case this was convenient because our app and platform can run without much user interference and still produce valid data.
  7. Close the app.
  8. Immediately go to the phone settings and find battery info using the built-in Android battery analyzing tools.
    1. Check “Battery” (possibly a submenu) as well as “Location”.
    2. Write down the numbers someplace as data.

I did my day of tests using my cheap second-hand Samsung phone and a Nexus tablet. I connected a Bluetooth sensor platform and collected data from it once every five seconds for five hours (this is of course automated in the app). I kept the screens turned on most of the time.

Findings

You’ll observe below that different Android versions (including the extra stuff a manufacturer installs) cause apps and services to report battery usage somewhat differently. My phone, for example, doesn’t list Field Day itself, but does list screen and battery usage, which are (nearly) 100% attributable to Field Day because of the constraints we imposed on our device uses in the early steps in the protocol. (The Nexus did list Field Day, conveniently.)

Craig’s phone

  • 64% remaining after 5 hours
  • Bluetooth (97%)
    • Time on 5h7m35s
    • CPU total 19 sec
    • Stay awake 1 sec
    • Computed power usage 38232 mAh
  • Screen (2%)
    • Time on 4h 4m 20s
    • Computed power usage 1059 may
    • (adaptive display, mid-to-low brightness)

Nexus

  • 40% remaining after 5 hours
  • Screen 22%
    • Time on 5h 5m 0s
    • Computed power usage 753 mAh
    • (adaptive display, max brightness)
  • Field Day 9%
    • CPU Total 6m 19s
    • CPU Foreground 6m 9s
    • Keep awake 3m 25s
    • GPS 5h 2m 23s
    • WiFi packets received 19
    • WiFi packets sent 30
    • Computed power use 317 mAh

In both cases, the “Location” menu continues to report Field Day as “high battery usage” as this issue reports. In practice, battery usage appears to be what we might expect: bright screens and Bluetooth make your battery work more (“I’m shocked, shocked…”).

This test wasn’t the only one we’ve run, but it was the most systematic. It also tested a relatively power-greedy case – normally, we do not keep the screen on at all when we collect a data stream.

Next steps

I’m going to do another round of tests tomorrow, with the screens off to check the more common usage case. However, based on current observations, the app’s battery usage is in line with what I would expect.

There are pieces of this process we should refine – for example, we should have simply controlled for screen brightness rather than let it be such a dominant player in the first place. But after some discussion, we are confident enough to conclude that our battery usage conforms with what should be expected from an app that heavily uses Bluetooth, GPS, cellular, and/or WiFi. We expect and hope that tomorrow’s screen-off test will confirm this.

GitLab

I have a GitHub account, but most of my coding activity is on Earlham CS’s GitLab instance. That includes most of the code I write for use in production, as well as much of the commentary and communication about it.

Today, for example, I submitted a merge request for some work on the Field Day application, one of my favorite projects and one of ECCS’s most successful multi-year collaborative creations.

Software development like this is only one part of my job, and as you’ll see by the squares on my profile I don’t get to do it every single day. But it’s quite rewarding when I do get to dedicate some time to it.