Meet our Terrestrial Mapping Platform!

Just a nice photo from Iceland

I’m excited to share that the Earlham field science program is now sharing the core of our Terrestrial Mapping Platform (TMP)! This is very much a work-in-progress, but we’re excited about it and wanted to share it as soon as we could.

We had to delay the 2020 Iceland trip because of COVID-19. That of course pushed back the implementation and case study component of this project, which was Iceland-centric. But we are moving forward at full speed with everything else. As Earlham has now started the new academic year, we have also resumed work on the TMP.

The project is a UAV hardware-software platform for scientists. It consists of:

  • a consumer-grade drone for capturing images
  • flight plan generation software and application to automate drone flights
  • data analysis workflows for the images – visible light and NIR, assembled into 2D and 3D models

All of this goes toward making science more accessible to a broader range of domain scientists. Archaeologists and glaciologists are our current target cohort, but many more could find use for this work if it’s successful.

We will make all of this accessible in repositories with open licenses on our GitLab instance. Some are already available. Others we will share once we review them for (e.g.) accidentally-committed credentials.

That was all planned, if delayed. We’re also using our extra year of preparation time to make the project better in a few ways:

  • Reevaluating our choice of UAV make and model
  • Prettifying our web presence, which very much includes blog posts like this
  • Reducing the friction and pain points in our current workflow
  • Making our code and infrastructure better in general (I’ve covered my growing emphasis on quality here before)

The team mostly comprises students and faculty (of whom I’m the junior-most). Additionally, there are a few on-site partners in Iceland and innumerable personal supporters who make this possible. We’ll be sharing more at the Earlham Field Science blog as we go. I will undoubtedly share more here as well.

COVID is bad, but we want to make the best of this era. This is one way we’re doing that.

(Disclosure: We received funding for this from a National Geographic grant. None of the views in this blog post or our online presence represents, or is endorsed by, Nat Geo.)

Golang and more: this week’s personal tech updates

First I haz a sad. After a server choke last week, the Earlham CS admins finally had to declare time-of-death on the filesystem underlying one of our widely-used virtual machines. Definitive causes evade us (I think they are lost to history), so we will now pivot to rebuilding and improving the system design.

In some respects this was frustrating and produced a lot of stress for us. On the other hand, it’s a sweet demo of the power of virtualization. The server died, but the hardware underlying it was still fine. That means we can rebuild at a fraction of the cost of discovering, purchasing, installing, and configuring new metal. The problem doesn’t disappear but it moves from hardware to software.

I’ve also discovered a few hardware problems. One of the drones we will take to Iceland need a bit of work, for example. I also found that our Canon camera may have a bad orientation sensor, so the LCD display doesn’t auto-rotate. Discovering those things in February is not fun. Discovering them in May or June would have been much worse.

Happier news: I began learning Go this week. I have written a lot of Java, C, and some Python, but for whatever reason I’ve taken to Golang as I have with no other language. It has a lot of the strengths of C, a little less syntactical cruft, good documents, and a rich developer literature online. I also have more experience.

A brief elaboration on experience: A lot of people say you have to really love programming or you have no hope of being good at it. Maybe. But I’m more partial to thinking of software engineering as a craft. Preexisting passion is invaluable but not critical, because passion can be cultivated (cf. Cal Newport). It emerges from building skills, trying things, perseverance, solving some interesting problems, and observing your own progress over time. In my experience (like here and here), programming as a student, brand-new to the discipline, was often frustrating and opaque. Fast forward, and today I spent several hours on my day off learning Golang because it was interesting and fun. 🤷‍♂️

Your mileage may vary, but that was my experience.

Finally, here are a few articles I read or re-read this week:

Earlham’s four-day weekend runs from today through Sunday. After a couple of stressful weeks, I’m going to take advantage of the remainder of the time off to decompress.

Improve software performance, both sooner and later

This week I read “How To Be A Programmer”. It’s part of my work to shore up my fundamental computing skills. From a section in “Beginners” called “How to Fix Performance Problems” (emphasis added):

The key to improving the performance of a very complicated system is to analyse it well enough to find the bottlenecks, or places where most of the resources are consumed. There is not much sense in optimizing a function that accounts for only 1% of the computation time. As a rule of thumb you should think carefully before doing anything unless you think it is going to make the system or a significant part of it at least twice as fast.

That struck me for two reasons: One: I’ve reflected in the past on high-performance computing showing exceptions to rules we learn as beginners. Two: just an hour earlier, I’d read Nelson Elhage’s excellent blog post “Reflections on software performance” (emphasis added):

I think [“Make it work, then make it right, then make it fast”] may indeed be decent default advice, but I’ve also learned that it is really important to recognize its limitations, and to be able to reach for other paradigms when it matters. In particular, I’ve come to believe that the “performance last” model will rarely, if ever, produce truly fast software (and, as discussed above, I believe truly-fast software is a worthwhile target). 

One of my favorite performance anecdotes is the SQLite 3.8.7 release, which was 50% faster than the previous release in total, all by way of numerous stacked performance improvements, each gaining less than 1% individually. This example speaks to the benefit of worrying about small performance costs across the entire codebase; even if they are individually insignificant, they do add up. And while the SQLite developers were able to do this work after the fact, the more 1% regressions you can avoid in the first place, the easier this work is.

Software development advice: land of contrasts!

Both approaches have merit. However, from my admittedly limited experience, I’m partial to the latter.

The traditional advice – build it, make it work, then make it fast – works in many cases. It’s a pleasantly simple entry point if you’re just learning to build software. I learned to code that way, and so do many of our students. Both text selections give it credit – “rule of thumb”, “decent default”. But I think its placement in the “Beginner” section is appropriate.

I’m not even at a tech company, but I work on projects where performance matters from start to finish. I’ve also worked on projects where bad performance made the user experience pretty miserable. As Elhage emphasizes in his post, “Performance is a feature”. CS majors learn “big-O” notation for a reason. Everyone likes fast software, and that requires both good design and ongoing optimization.

Compare:

Computing lessons from DNA analysis experiments

I’ve been working with my colleagues in Earlham’s Icelandic Field Science program on a workflow for DNA analysis, about which I hope to have other content to share later. (I’ve previously shared my work with them on the Field Day Android app.)

My focus has been heavily experimental and computational: run one workflow using one dataset, check the result, adjust a few “dials”, and run it again. When we’re successful, we can often automate the work through a series of scripts.

At the same time, we’ve been trying to get our new “phat node” working to handle jobs like this faster in the future.

Definitions vary by location, context, etc. but we define a “phat node” or “fat node” as a server with a very high ratio of (storage + RAM)/(CPU). In other words, we want to load a lot of data into RAM and plow through it on however many cores we have. A lot of the bioinformatics work we do lends itself to such a workflow.

All this work should ultimately redound to the research and educational benefit of the college.

It’s also been invaluable for me as a learning experience in software engineering and systems architecture. Here are a few of the deep patterns that experience illustrated most clearly to me:

  • Hardware is good: If you have more RAM and processing power, you can run a job in less time! Who knew?
  • Work locally: Locality is an important principle of computer science – basically, keep your data as close to your processing power as you can given system constraints. In this case, we got a 36% performance improvement just by moving data from NFS mounts to local storage.
  • Abstractions can get you far: To wit, define a variable once and reuse it. We have several related scripts that refer to the same files, for example, and for a while we had to update each script with every iteration to keep them consistent. We took a few hours to build and test a config file, which resolved a lot of silly errors like that. This doesn’t help time for any one job, but it vastly simplifies scaling and replicability.
  • Work just takes a while: The actual time Torque (our choice of scheduler) spends running our job is a small percentage of the overall time we spend shaping the problem:
    • buying and provisioning machines
    • learning the science
    • figuring out what questions to ask
    • consulting with colleagues
    • designing the workflow
    • developing the data dictionary
    • fiddling with configs
    • testing – over, and over, and over again
    • if running a job at a bigger supercomputing facility, you may also have to consider things like waiting for CPU cycles to become available; we are generally our systems’ only users, so this wasn’t a constraint for us

A lot of this is (for computer scientists, software engineers, etc.) common sense, but taking care to apply that common sense can be critical for doing big interesting work.

The punchline of it all? We managed to reduce the time – walltime, for fellow HPC geeks – required to run this example workflow from a little over 8 hours to 3.5 hours. Just as importantly we developed a bunch of new knowledge in the process. (I’ve said almost nothing here about microbiology, for example, and learning a snippet of that has been critical to this work.) That lays a strong foundation for the next several steps in this project.

If you read all this, here’s a nice picture of some trees as a token of my thanks (click for higher-resolution version):

Image of trees starting to show fall color
Relevance: a tree is a confirmed DNA-based organism.

A tale of two large-ish app updates

This week I spent some time working on Earlham CS’s Field Day Android application. It’s the app used by our student-faculty field science researchers to collect data on trips to, say, a glacier in Iceland. I made two substantial changes.

The first was updating our system dependencies. At the start of the summer, Field Day wasn’t a fully modern application. That’s mostly because its development is contingent on the interest levels of students and faculty who (correctly!) have other priorities during the academic year. We experience our only consistent spikes in development during preparation for a trip to Iceland. Even then, we tend to focus on adding or fixing features, rather than major design choices or boring updates. Whatever their benefits, such changes always risk eating up precious time in the short run.

As a result, we had long neglected to update SDK versions, themes, and other app fundamentals. I wanted to fix that before classes resumed this month.

Not being an Android expert (yet?), I relied on a mix of automated tools in Android Studio, manual code tweaks, and careful testing to push the update process forward. Here’s how I described it in my merge request:

I wanted to make us a “grownup” application, by which I mean that I wanted to move us away from as many deprecated tools and dependencies as possible, as far in advance of a field trip as possible. (EDIT: With one exception: these changes do not attempt to resolve the [looming] Google Drive [API] deprecation.)

To that end, this merge request involves substantial changes to build fundamentals like the Gradle version, as well as some Lint cleanup and general tidying. Much of it was done following a simple pattern:

– run a built-in Android Studio update tool (e.g. “Update to AppCompat”)

– change a bunch of details in the code so it builds

– test on the device

– lather, rinse, repeat

Field Day merge request 9

After some tests by myself and a colleague, I approved the merge.

To reward myself for accomplishing that admittedly tedious process (which followed a long, slow battery testing process), I did something more fun.

For a long time I’d wanted to improve Field Day’s UI to streamline the navigation. I made a batch of changes, then submitted the following merge request:

[Field Day’s original creative developers] created a great design palette for Field Day: fun fonts, bright colors, intuitive icons.

I wanted to keep that but update the navigation to reflect the current understanding of our usage model. To that end, this merge centralizes everything onto one screen, miniaturizes our less-used buttons, and puts database and sensors at the forefront.

No specific activities or fragments other than the main screen (and the deletion of the obsolesced sensor screen) have been changed.

I can foresee a future where we do more data analysis and aggregation through the lab notebook, so I’ve preserved the notebook icon for future use.

Field Day merge request 10

The changes in that request took us from this set of two main screens:

Previous main screen (“Sampling” takes user to the second screen)
Previous second screen, containing our sensor and database features

… to this one screen:

Our most commonly-used buttons are on the main screen and fill the entire screen width.

I again checked with my colleague and then approved the request. I’m now working on other issues and have already found the changes to be substantial boosts to the user experience.

This is a sample of my own personal work, but of course building software is a team sport. And it relies on iteration. The original designers of Field Day – current and former colleagues of mine – did a lot of the heavy lifting over a few years building the core logic and aesthetic of the app. As I made my changes in the last few months, I’ve worked to maintain their original design palette while improving usability, performance, and the underlying data model. It’s a useful, specialized, and dare I say fun application, and I want it to keep getting better.

As a closing note about process, I find it sharpens my skills development when I have to summarize my work into prose, as in these merge requests. Writing them requires more precision than a quick chat in a hallway. That’s to say nothing of possible benefits to future developers trying to retrace changes and intentions.

IDE vs. text editing app vs. editor in a shell

I’ve been doing software development of various kinds for a few years now, using a variety of tools. In my experience, these are the tradeoffs between an integrated development environment (IDE), a standalone text editing app, and a text editor in the terminal. (I haven’t listed any examples here that you need to pay for.)

IDE

Examples: Android Studio, Xcode

An integrated development environment is exactly what it says on the tin: a complete development environment for a project featuring a mix of tools that work together to help you build your application. An IDE’s features go far beyond reading and writing files, and might include…

  • managing your project in its version control system
  • scanning for possible mistakes in code
  • displaying a preview of your application next to the text file that describes it
  • serial port monitoring
  • debugging
  • automated building

IDE’s are powerful and can provide real boosts to productivity and insight. The tradeoff is lock-in: learning one IDE doesn’t mean you automatically know how to use another, and each editor has a learning curve.

In practice that’s usually fine. If you are exclusively, say, an Android app developer, you’ll be well-served to learn Android Studio. It just won’t teach you much about using Eclipse, let alone Xcode.

Text editor, standalone app

Examples: Atom, Notepad++

Standalone text-editing apps can be extremely handy. They’re not as heavy (in terms of system resources and user experience) as an IDE, but they’re great for all the common tasks we’d expect of an editor: find-and-replace, viewing the outline of a project in the filesystem, remote editing by SFTP, and more. Some editing apps, like Atom (which I use), support installing extensions that let you supercharge them to an almost arbitrary degree.

Good text editing apps can do all you need it to do and more. They may be the best choice for many users in many cases. That hasn’t been the case for me, though: I tend to experience them as an unhappy medium between horsepower and simplicity, and I don’t use them as often as the other two categories in this post.

Text editor in the shell

Examples: Vi, Nano

Vim is my fondest friend in the world of dev environments. I learned it within my first semester studying computing. While it has a learning curve, its keyboard shortcuts are simple and powerful. It is a widely-used example of an in-shell text editor – one that you run by typing its name in your shell followed by a space and the name of the file you intend to edit.

vim helloworld.txt     # 2 tokens, no waiting

This category tends to attract strong partisans. Stern computer science people on the Internet often emphasize vi for its near-guaranteed presence on Linux systems, for example. I like vim. Others like nano. Try them, and see what works for you.

There are some great advantages to using an editor in the shell:

  • Simplicity: An in-shell editor is sparse. You don’t get error checking, colorful little GUI buttons, simulators, design previews, or any other IDE feature, but that’s often unnecessary. (In most cases you can add font coloring, which is handy.) What you get instead is the ability to open and examine any file you can imagine – even if it’s a binary file and all you see is gibberish – in a predictable way.
  • Fast: From the POV of the user, an in-shell editor typically loads as fast as you can type. By comparison, IDE’s and editing apps have long load times – and sometimes long read and write times. This speed has the side effect of making shell editors operate similarly no matter your hardware, something that’s not true of either of the other two options.
  • Real big files: It can handle gigantic files better than GUI applications including IDE’s can.
  • Skills that scale: An editor in a shell makes you start learning how to use a shell. Knowing at least the basics of operating at that level is a solid upgrade to your development skillset. (The biggest confounding variable here is the type of shell you work with. I, like most people, just use the Bash shell. Experience could be quite different using ZSH or TCSH.)

Shell editors aren’t for every person working on every project, but I use mine several times daily because “it just works”.

(I’ll throw in a kind word for GNU Emacs here. It can be run as either a shell client or an editing app. I haven’t used it for years, but some people really love it.)

If the environment isn’t an IDE, building and/or running the edited software can be done by running your compiler or interpreter in the terminal. This isn’t as good an approach for something like an Android app, which needs to run on a device or at least a simulator, but it works for many applications.

So which do I pick?

I’m a fan of simple rules for choices like this. When it comes to development environments, try (in this order):

  1. The one everyone else working on the project with you uses. This is how I started working with both vim and Android Studio.
  2. The one you’re most comfortable with.
  3. If you’re early in your dev career and don’t yet have a comfort level with any of these, don’t be afraid to try several or all of them – it’s served me well to be able to shift between them from time to time.

Your mileage may vary on all of this, but these are the patterns in my experience.

Simple battery testing for common use cases of an Android application

In developing the Android app we use to collect field data, I recently completed a series of tests. We wanted to determine how worried we should be about the Android “Location” tab’s judgment that Field Day is a “High battery usage” app. We wanted to avoid installing and running any additional apps to do so.

Protocol

The simple (if slow) protocol we developed to meet those requirements goes something like this. It’s best when run on multiple, comparable devices at the same time.

  1. Fully charge the device.
  2. Restart the device.
  3. Close all applications other than the app you’re testing.
  4. Disable sleep, screen turn-off – any battery-saving features that apply to your phone but do not fit the use case of Field Day when we actually run it.
  5. Do any setup tasks – in our case, linking to a remote database and an Arduino sensor platform to collect data for us.
  6. Run it for hours not minutes. In our case this was convenient because our app and platform can run without much user interference and still produce valid data.
  7. Close the app.
  8. Immediately go to the phone settings and find battery info using the built-in Android battery analyzing tools.
    1. Check “Battery” (possibly a submenu) as well as “Location”.
    2. Write down the numbers someplace as data.

I did my day of tests using my cheap second-hand Samsung phone and a Nexus tablet. I connected a Bluetooth sensor platform and collected data from it once every five seconds for five hours (this is of course automated in the app). I kept the screens turned on most of the time.

Findings

You’ll observe below that different Android versions (including the extra stuff a manufacturer installs) cause apps and services to report battery usage somewhat differently. My phone, for example, doesn’t list Field Day itself, but does list screen and battery usage, which are (nearly) 100% attributable to Field Day because of the constraints we imposed on our device uses in the early steps in the protocol. (The Nexus did list Field Day, conveniently.)

Craig’s phone

  • 64% remaining after 5 hours
  • Bluetooth (97%)
    • Time on 5h7m35s
    • CPU total 19 sec
    • Stay awake 1 sec
    • Computed power usage 38232 mAh
  • Screen (2%)
    • Time on 4h 4m 20s
    • Computed power usage 1059 may
    • (adaptive display, mid-to-low brightness)

Nexus

  • 40% remaining after 5 hours
  • Screen 22%
    • Time on 5h 5m 0s
    • Computed power usage 753 mAh
    • (adaptive display, max brightness)
  • Field Day 9%
    • CPU Total 6m 19s
    • CPU Foreground 6m 9s
    • Keep awake 3m 25s
    • GPS 5h 2m 23s
    • WiFi packets received 19
    • WiFi packets sent 30
    • Computed power use 317 mAh

In both cases, the “Location” menu continues to report Field Day as “high battery usage” as this issue reports. In practice, battery usage appears to be what we might expect: bright screens and Bluetooth make your battery work more (“I’m shocked, shocked…”).

This test wasn’t the only one we’ve run, but it was the most systematic. It also tested a relatively power-greedy case – normally, we do not keep the screen on at all when we collect a data stream.

Next steps

I’m going to do another round of tests tomorrow, with the screens off to check the more common usage case. However, based on current observations, the app’s battery usage is in line with what I would expect.

There are pieces of this process we should refine – for example, we should have simply controlled for screen brightness rather than let it be such a dominant player in the first place. But after some discussion, we are confident enough to conclude that our battery usage conforms with what should be expected from an app that heavily uses Bluetooth, GPS, cellular, and/or WiFi. We expect and hope that tomorrow’s screen-off test will confirm this.

A -> AB -> B

I was reading a recent Rachel By The Bay post in my RSS reader and this struck me:

Some items from my “reliability list”

It should not be surprising that patterns start to emerge after you’ve dealt with enough failures in a given domain. I’ve had an informal list bouncing around inside my head for years. Now and then, something new to me will pop up, and that’ll mesh up with some other recollections, and sometimes that yields another entry.

Item: Rollbacks need to be possible

This one sounds simple until you realize someone’s violated it. It means, in short: if you’re on version 20, and then start pushing version 21, and for some reason can’t go back to version 20, you’ve failed. You took some shortcut, or forgot about going from A to AB to B, or did break-before-make, or any other number of things.

That paragraph struck me because I’m about one week removed from making that very mistake.

Until last week, we’d been running a ten-year-old version of the pfSense firewall software on a ten-year-old server (32-bit architecture CPU! in a server!). I made a firewall upgrade one of our top summer priorities.

The problem was that I got in a hurry. We tried to upgrade without taking careful enough notes about how to reset to our previous configuration. We combined that with years’ worth of lost knowledge about the interoperability of the Computer Science Department’s subnets with the Earlham ITS network. That produced a couple of days of downtime and added stress.

We talked with ITS. We did research. I sat in a server room till late at night. Ultimately we reverted back to the old firewall, allowing our mail and other queues to be processed while we figured out what went wrong in the new system.

The day after that we started our second attempt. We set up and configured the new one alongside the old, checking and double-checking every network setting. Then we simply swapped network cables. It was almost laughably anticlimactic.

In short, attempting to move directly from A to B generated hours of downtime, but when we went from A to AB, and then from AB to B, it was mere seconds.

We learned a lot from the experience:

  1. The A->AB->B pattern
  2. ECCS and ITS now understand our network connections much more deeply than we did three weeks ago.
  3. Said network knowledge is distributed across students, staff, and faculty.
  4. We were vindicated in our wisest decision: trying this in July, when only a handful of people had a day-to-day dependence on our network and we had time to recover.

A more big-picture lesson is this: We in tech often want to get something done real fast, and it’s all too easy to conflate that with getting it done in a hurry. If you’re working on something like this, take some time to plan a little bit in advance. Make sure to allow yourself an A->AB->B path. A little work upfront can save you a lot later.

Or, as one mentor of mine has put it in the context of software development:

Days of debugging can save you from hours of design!

GitLab

I have a GitHub account, but most of my coding activity is on Earlham CS’s GitLab instance. That includes most of the code I write for use in production, as well as much of the commentary and communication about it.

Today, for example, I submitted a merge request for some work on the Field Day application, one of my favorite projects and one of ECCS’s most successful multi-year collaborative creations.

Software development like this is only one part of my job, and as you’ll see by the squares on my profile I don’t get to do it every single day. But it’s quite rewarding when I do get to dedicate some time to it.

Learn the rules. And their exceptions.

As I learn more about how to develop software and manage systems, I’m struck that there are many things we learn at some point – only to discover a pile of exceptions later.

A good example: one of the first things you learn as a coder, either self-taught or in an intro course in college, is to make every function small, readable, easily-testable, do as little as possible, and return a result predictably.

So now we know how to code. We just write a lot of small functions and duct tape them together call them in a sensible order.

Of course, life’s not that simple.

Henry Neeman in Supercomputing in Plain English notes that in high-performance computing (HPC) applications, you want a function to do as much as possible so you can stuff it into a loop, which many compilers can optimize nicely:

How many of you love loops? Hate loops? Do not have an emotional reaction to loops? Okay I want you to love loops.

[In many programming contexts,] we write little routines on little pieces of data because it’s easier to debug and – after all – bugs are bad, right? We don’t want a lot of bugs.

The problem with that approach is then you don’t have enough for the compiler to chew on and the hardware to chew on, and so you get terrible performance.

So the way you want to write code to get good performance is literally the exact opposite of what we were taught when we were wee lads and lasses in our first programming course. You want to write big routines on big chunks of data.

It’s still probably best, in many cases, to develop small bits of code that are easily tested and iterated. (Most applications are not high-performance computing applications.) But it’s not always the right idea.

The takeaway for me is that domain knowledge is as important for building software as good intuition about coding best practices, precisely because “best practices” are downstream of the type of problem you’re trying to solve.