What’s next

Some personal news… 🙂

I have been at Earlham College for almost seven years, including my time as a student and as CS faculty. Today is my last day there.

It’s been an incredible place to grow as a person, deepen my skills, collaborate with talented people from all walks of life, and try to make the world a little bit better. I’ve seen a few generations of the community cycle through and watched us withstand everything up to and including a literal pandemic. I capped it with the trip of a lifetime, spending a month doing research in Iceland – on a project I hope to continue working on in the future.

To the Earlham Computer Science community in particular I owe a big thanks. I have had a supportive environment in which to learn and grow for virtually the entirety of those years. The value they’ve added to my life can’t be quantified. I am deeply grateful.

What’s next?

I am elated to announce that in mid-September I will go to work as a Research Computing HPC Cluster Administrator at the University of Colorado Boulder! I’m excited to take the skills I’ve built at Earlham and apply them at the scale of CU Boulder. Thanks to the many people who’ve helped make this opportunity possible.

Highlights of an amazing trip

Today is the last day most of us are in Iceland for this trip. As I started this post, we were completing a tour of the Golden Circle after a few days in beautiful Reyjkavik. Now we are preparing for departure.

Our view of the volcano

I wanted to post some of the highlights of our trip. There’s a rough order to them, but don’t take the numbering too seriously – it’s been a great experience all-around. Without further ado:

  1. The volcano is truly incredible. It was not uncommon for people to spontaneously shout “Wow!” and “Oh my god!” as the lava burst up from the ground.
  2. We woke up every day for a few weeks with a view of a fjord.
  3. We did a glacier hike on Sólheimajökull, with two awesome guides.
  4. This was a historically successful round of data collection, both on the drone side and on the biology side. We’ll write and share a lot more about this in the next few months.
  5. We shared space with the group of phenomenal students from the University of Glasgow. We also collaborated with them on multiple occasions, learning a lot about different ways to study wildlife and local sites.
  6. THE FOOD – you probably don’t associate Iceland with food culture (I certainly didn’t), but our meals were delicious.
  7. The architecture and decorations are so distinctly Icelandic.
  8. Amazing photography and video – in high quality and high quantity.
  9. Walking along the boundary between the North American and European plates.
  10. Guided tour from our Skalanes hosts – who incidentally are awesome people – of a stretch of eastern Iceland.
Getting the rundown about glaciers at Solo

Some of my personal honorable mentions include:

  • Trail running at Skalanes is breathtaking.
  • Blue glacier ice is real neat.
  • The National Museum of Iceland is fascinating and well-done.
  • Rainbow roads in both Seyðisfjörður and Reykjavik highlight what a welcoming place this country is – also perfect reminders of Pride Month in the U.S.!
  • My first-in-my-lifetime tour of a beautiful country happened alongside people I admire who teach me things every single day. What more could I ask for?
A drone photo of the coast by the fjord

If you haven’t already, check out this interview with Charlie and Emmett, conducted by Cincinnati Public Radio.

Davit and Tamara flying

In addition to our success this year, we’ve also set up some great new opportunities for future years. With our long-time friend and collaborator Rannveig Þórhallsdóttir, we’ve added the cemetery in Seyðisfjörður to our list of sites to survey. We believe there may be historically-significant artifacts to be found there, and our drone work lends itself well to finding out.

The fjord at Skalanes

Finally, here’s the trip by the numbers:

  • 7 Earlhamites
  • 26 days
  • 183 GB of initial drone images and initial assemblies
  • 2 great hosts at Skalanes
  • 6 outstanding co-dwellers
  • 4 guides at 2 sites
  • 1 perfect dog
  • N angry terns
  • 1 amazing experience
Admiring the view

And that’s a wrap. Hope to see you again soon, Iceland!

Cross-posted at the Earlham Field Science blog.

Flying cameras are good

We recently chose not to use ground control points (GCP’s) as part of our surveying work. This is a departure from standards and conventions in the near-Earth surveying space. However, we believe we have made a sound decision that will support equally effective and more time and cost-effective research. In this post, I’ll explain that decision.

The short version: drone imagery and open-source assembly software (e,g. OpenDroneMap) are now so good that, for our purposes, GCP’s have no marginal benefit.

We have high-quality information about our trial area from an established authority – the Cultural Heritage Agency of Iceland. Their 2007 report of finds is the basis of our trial runs here at Skalanes. Surveying these predefined areas, we’ve now flown multiple flights, gathered images, and then run three assembles with OpenDroneMap.

Here’s a simple run over the area with no GCP’s:

Here’s a run over the area with GCP’s, adding no location metadata other than the craft’s built-in GPS coordinates (you’ll note that the ground footprint is slightly different, but the roundhouse in the middle is the key feature):

We also manually geocoded the GCP’s for one run.

In the end, we observed no meaningful difference between an assembly with GCP’s and an assembly without them. Adding the images as raster layers to a QGIS project confirmed this to our satisfaction:

With GCP:

Without GCP:

In summary, ground control points just don’t help us much compared to just taking a bunch of good photos and using high quality software to assemble them. They also cost us in portability: even four GCP’s are difficult to carry, occupying significant space in airport luggage and weighing down walks in the field. For scientists interested in doing work over a large area, potentially multiple times, that inconvenience is not a trivial cost.

The ODM assemblies are outstanding by themselves. We have good technology and build on the work of a lot of brilliant people. That frees us to be more nimble than we might have been before.

It wouldn’t be a post by me if it didn’t end with a cool picture. Here’s a drone image from a cliff near the house where we’re staying:

Cross-posted at the Earlham Field Science blog.

Awe in Iceland

The Greater Good Institute at Berkeley considers awe one of the keys to well-being:

Awe is the feeling we get in the presence of something vast that challenges our understanding of the world, like looking up at millions of stars in the night sky or marveling at the birth of a child. When people feel awe, they may use other words to describe the experience, such as wonder, amazement, surprise, or transcendence.

That’s the feeling I have at least once a day, every day, here in Iceland.

And it’s difficult to write a blog post about awe. Almost by definition, it’s an emotion that defies easy explanation. It has a mystique that risks being lost in the translation to plain language.

But if I can’t describe the feeling, I can describe why I’m having it.

Unique among my traveling companions, this is my first-ever trip out of my country of origin (🇺🇸) The sliver of gray in this image is the first thing I ever saw of a country not my own:

When we arrived, I got a passport stamp and exchanged currency – both brand new experiences. However mundane, they were novel for me and began waking me up to the new world I’d entered.

Our first few days were chilly, windy, and rainy. I was much happier about this than were my traveling companions. If our weather wasn’t pleasant, it was nonetheless exactly the immersive experience I was hoping for when I signed up for this trip.

In those first few days, I got to see this amazing waterfall:

I got to participate in collecting soil samples at a glacier —

Solo!

— and in howling wind on the side of a moraine:

The right side of the moraine was calm and quiet. The left was much less so.

For good measure, I saw floating blue ice for the first time:

All this was great, and to me they made this trip worth the months of planning and days of travel difficulties it took to get here.

Then we got to Skalanes, where I’m writing this post, and its landscapes exist on a whole other level. Here are ten views here, drawn almost at random from my photos:

This is a country that absolutely runs up the score on natural beauty.

I’ve taken hundreds of pictures here and they’re all amazing – but none does justice to actually being here. That combination is the signature of an awe-inspiring experience.

Awe puts us in touch with something above and beyond our daily worldly experience – call it the divine, the sublime, whatever speaks to you. It’s an experience you can reproduce if you try, but I believe it connects most deeply when it emerges organically from the world you enter. That’s what’s happened to me here.

It is remarkable that this is what we get to do for work, and I am so glad we have some more time to spend here in this awesome country.

Cross-posted at the Earlham Field Science blog.

“A lot of people” – a brief note

Pet peeve: “A lot of people say X” or similarly “A lot of people do Y”.

There’s a lot of people in general: ~330,000,000 in America alone. If you look hard enough, you can find *somebody* who will say/do almost literally anything.

If 1% of Americans believe something, that’s over 3 million believers. That’s both a huge number of people and hardly anybody, depending on your context.

I find this usually isn’t a problem in writing, where with minimally competent phrasing/structure the context is clear. During chit-chat or as used by the likes of Fox News, it can be (often purposely) obtuse.

I like us to try to be specific, is what I’m saying. 🙂

Originally a Twitter thread.

Iceland is on track for summer

Discussions are ongoing about the viability of summer travel given the pandemic. However, as my colleague Charlie has blogged recently, we are “acting as if”. As such, we are trying to maintain our original calendar.

Lo and behold, we have:

Imagine a “You are here” arrow by Spring 1

Here’s the full breakdown of that schedule and our progress:

Our plan for the fall was to find and test alternative UAV’s. This proved prudent, as the federal government banned DJI craft late last year. We are happy with both the Parrot and the Skydio craft, for different reasons which we’ll undoubtedly cover here on this blog in the future.

December and January, which were effectively a long winter break for a subset of us, were dedicated to testing the craft, capturing initial video, and possibly beginning development. This was a success as well. Additionally we have begun spinning up a more sophisticated web presence for the stories we’re telling – changes we will be prepared to publish soon.

We’ve now started the calendar for the spring, term 1 of 2. We are moving into scaling up our operation of the craft and developing software to automate that work. It’s a tough problem but one we can solve in the time we have.

We’re optimistic about our ability to meet the moment. If the world continues to make progress on COVID-19, we should be in shape to have a successful research trip.

2020 in review

You don’t need me to tell you 2020 was a bad year. Others will write about the details that apply nationally and globally, so I’m going to jump right into my own retrospective.

The 2020 wallowing

I was planning on a trip to Iceland followed by a new job with room for advancement in 2020. Instead I stayed at my current job (a good job!), made an attempt at a side hustle that has so far largely fizzled, and was obviously not able to go to Iceland. I didn’t visit family in Montana for Thanksgiving or Christmas. At work, I made a few dumb mistakes (we did rebound in each case, happily). It was, on net, a rough year.

That’s about all I have to say about that. I don’t want to wallow too much, but I also don’t want to go further without acknowledging the struggle.

The better stuff

All that said, the rest of this post summarizes my accomplishments for the year. I write this to remind myself that even though it didn’t generate those external signals, I still did a lot. I advanced my skillset, did my job well, and patched through the year.

Accomplishments:

  • Got out of bed every day and went back to sleep every night
  • Kept Earlham CS running through the pandemic, student dispersal, lockdown, and restricted return
  • Modernized our systems engineering infrastructure with better monitoring, solid backups, improved responsiveness to inquiries, and higher availability – still a long way to go, but we’re so much better than we were a year ago
  • For each error I made, rebounded and learned a lesson
  • Migrated our cluster infrastructure from Torque to Slurm successfully
  • Learned a bunch of low-level details about filesystems, SELinux, and more en route to improving overall quality
  • Took 10,000 steps most days and got outside frequently
  • Provided a lot of internal tech support, engineering, feedback, and project contributions to the projects associated with the Iceland program
  • Made checking LinkedIn a regular part of my routine, though I should use it more socially in the new year
  • Purely as a hobby, learned a ton about video and audio production

Casual observation: I posted a lot in February, and my tech achievements primarily happened over the summer.

I also want to dedicate a section to expanding my horizons. I couldn’t do it with travel, but there were a lot of new things I got the chance to explore this year:

  • On Spotify, listened to 1,408 new artists and 366 new genres (“genre” seems like a nebulous category, but I am taking the win)
  • Watched a lot of new movies, including the complete Hayao Miyazaki filmography
  • Learned to make curry! (this winter squash red Thai curry recipe is great)
  • Baked a pie – apple – for the first time, at Christmas
  • Grew my hair long for the first time in my life (it’s still growing actually – not getting a professional haircut during a pandemic)
  • Visited and walked new hiking trails

That was 2020 for me.

What’s next?

I can’t guarantee 2021 will be better than this year. However, I do have some broad intentions around a theme for the new year. I will do all that is in my control to make the next year better, and I hope you join me.

Extremely preliminary notes about the Parrot Anafi

After three extremely short flights, these are my notes about the Parrot Anafi drone.

We’re experimenting with different UAV’s as part of the Iceland terrestrial surveying program (we’re being optimistic about travel in 2021…). These are some notes with my initial observations.

This is based on the base case: taking the craft out, taking off, flying for at most a few minutes, and touching back down. As such, don’t take a single word of this as gospel – it’s just preliminary opinions for the historical record. 🙂

Short version of the review: holy portability! One thing I don’t like about the DJI Phantoms is that they are so heavy (both the craft and the RC-tablet unit). If it’s a pain here on-campus, where trips are short, I imagine it’s a pain in the field. The Anafi is ludicrously lightweight and doesn’t feel like a chore to carry around.

Video quality on the built-in camera is fantastic (4K etc.).

It’s not a perfectly seamless integration with our existing workflows. Within our group, for example, we usually use tablets, which are handy for their big screens. The Anafi seems built around the assumption of a phone. That’s true all the way down to the RC unit being designed to accommodate a phone but not a tablet. That’s different, but if we can only get this app for phones I am not necessarily sad about it.

There are many X factors I haven’t yet been thorough enough to review. For example: battery life, stability in breezes (heavy winds make most UAV’s hard to use), and the software/developer ecosystem.

These have been my extremely preliminary notes about the Parrot Anafi. It’s not even close to a comprehensive evaluation of everything we care about. Still, those usability factors are important if this is going to scale and be useful for others. So for now, I’m impressed.

CentOS 8 is going away

CentOS 8 is going away at the end of 2021 (emphasis added):

The future of the CentOS Project is CentOS Stream, and over the next year we’ll be shifting focus from CentOS Linux, the rebuild of Red Hat Enterprise Linux (RHEL), to CentOS Stream, which tracks just ahead of a current RHEL release. CentOS Linux 8, as a rebuild of RHEL 8, will end at the end of 2021. CentOS Stream continues after that date, serving as the upstream (development) branch of Red Hat Enterprise Linux.

We’re just going to walk into the ocean now…

Losing an HPC-friendly, enterprise-grade, stable, free operating system throws a wrench into our activities. We run several small CentOS clusters, though fortunately they all run CentOS 7 (maintenance updates till 2024). We have time to respond, but it will be extra work when it comes time to upgrade.

Here’s an aggregation of observations from the last couple of days:

  • The comments on that blog post speak bluntly.
  • The Beowulf email list is full of discussion already.
  • I see a lot of chatter now about Ubuntu and Debian. Both would be viable distros for the few CentOS 8 hosts we currently run. They may or may not fit cluster upgrade needs a few years from now.
  • Others have chimed in to mention Oracle Enterprise Linux (ehhhhh), a revival of Scientific Linux, NixOS, and GuixOS.
  • This replacement project, Rocky Linux, is in development here or maybe here (hard for me as an outsider to tell). It may go somewhere – or maybe not. I’ve seen others float their own forks as well. Maybe one will get traction, but who can say?
  • CentOS 8 was quite a bit different from CentOS 7. Hard not to feel frustrated at learning a new system just to see its lifespan cut short by eight years.
  • I put a note about this in our issue tracker today. It reads: “The path is straightforward and requires less data copying than some past migrations. We just need to have a good answer on the OS of choice. I imagine the community will converge on either a ‘correct’ answer or a few good answers relatively soon, but I don’t think we’re there yet.”
  • In short, for us: This will be fine, but it will also be annoying.

This does change our strategy for the next few months. We need a new external SSH server, for example, and it can’t reasonably be CentOS 8 anymore. Down the road, we will have cluster systems to upgrade, and I still have no idea what that looks like.

A 2020 success story from Earlham CS

I’m proud of something from this year – a real 2020 success story.

Earlham, winter 2020; 2020 success story

To give some backstory: January-March were maybe the roughest three months of my tech life to date. We had a cascade of server hardware failures that induced a lot of downtime. Total catastrophe. I’m grateful for my institution’s patience.

After a lot of extra hours in windowless rooms working on it, we did resolve those problems. We diagnosed the root causes and took steps to prevent similar issues in the future. I also learned a lot. (Some of the lessons from those days continue to guide us, and they’ve been imprinted on me forever.)

The very next day the March lockdowns started and the College sent everyone away.

We went all-remote for the rest of spring and shifted into hybrid mode for the fall. That increased the dependency on system availability. Naturally, I was uneasy about that after the stress of the spring term. I directed myself and the CS admin students to focus on uptime, iterative improvement, and minimal disruption.

What makes me proud is this: it worked.

Since resolving those issues in the winter and spring, we’ve been stable. Individual services and hosts have had issues, of course. Some of those issues took significant time and energy, and we’re still not perfect (probably never will be!). There is always more to fix, more to improve, more to automate, more to introduce.

But systematically we’ve operated without unplanned interruption since March.

We’ve faced uncertainty after uncertainty in 2020. But my colleagues and students have been able to count on our systems working. We’re not a giant shop here, but we have kept up with the changing times.

There it is: one clear 2020 success story. Engineering this success was a collaborative effort to which I’m just one contributor, but I am proud of it.