Highlights of an amazing trip

Today is the last day most of us are in Iceland for this trip. As I started this post, we were completing a tour of the Golden Circle after a few days in beautiful Reyjkavik. Now we are preparing for departure.

Our view of the volcano

I wanted to post some of the highlights of our trip. There’s a rough order to them, but don’t take the numbering too seriously – it’s been a great experience all-around. Without further ado:

  1. The volcano is truly incredible. It was not uncommon for people to spontaneously shout “Wow!” and “Oh my god!” as the lava burst up from the ground.
  2. We woke up every day for a few weeks with a view of a fjord.
  3. We did a glacier hike on Sólheimajökull, with two awesome guides.
  4. This was a historically successful round of data collection, both on the drone side and on the biology side. We’ll write and share a lot more about this in the next few months.
  5. We shared space with the group of phenomenal students from the University of Glasgow. We also collaborated with them on multiple occasions, learning a lot about different ways to study wildlife and local sites.
  6. THE FOOD – you probably don’t associate Iceland with food culture (I certainly didn’t), but our meals were delicious.
  7. The architecture and decorations are so distinctly Icelandic.
  8. Amazing photography and video – in high quality and high quantity.
  9. Walking along the boundary between the North American and European plates.
  10. Guided tour from our Skalanes hosts – who incidentally are awesome people – of a stretch of eastern Iceland.
Getting the rundown about glaciers at Solo

Some of my personal honorable mentions include:

  • Trail running at Skalanes is breathtaking.
  • Blue glacier ice is real neat.
  • The National Museum of Iceland is fascinating and well-done.
  • Rainbow roads in both Seyðisfjörður and Reykjavik highlight what a welcoming place this country is – also perfect reminders of Pride Month in the U.S.!
  • My first-in-my-lifetime tour of a beautiful country happened alongside people I admire who teach me things every single day. What more could I ask for?
A drone photo of the coast by the fjord

If you haven’t already, check out this interview with Charlie and Emmett, conducted by Cincinnati Public Radio.

Davit and Tamara flying

In addition to our success this year, we’ve also set up some great new opportunities for future years. With our long-time friend and collaborator Rannveig Þórhallsdóttir, we’ve added the cemetery in Seyðisfjörður to our list of sites to survey. We believe there may be historically-significant artifacts to be found there, and our drone work lends itself well to finding out.

The fjord at Skalanes

Finally, here’s the trip by the numbers:

  • 7 Earlhamites
  • 26 days
  • 183 GB of initial drone images and initial assemblies
  • 2 great hosts at Skalanes
  • 6 outstanding co-dwellers
  • 4 guides at 2 sites
  • 1 perfect dog
  • N angry terns
  • 1 amazing experience
Admiring the view

And that’s a wrap. Hope to see you again soon, Iceland!

Cross-posted at the Earlham Field Science blog.

Flying cameras are good

Update: We have learned! And we no longer agree with this post! GCP’s remain critical for deriving elevation. The cameras are not yet ready to replace that kind of precision. Always something you didn’t realize at first glance. Post preserved for posterity and because lots of it is still perfectly valid.

We recently chose not to use ground control points (GCP’s) as part of our surveying work. This is a departure from standards and conventions in the near-Earth surveying space. However, we believe we have made a sound decision that will support equally effective and more time and cost-effective research. In this post, I’ll explain that decision.

The short version: drone imagery and open-source assembly software (e,g. OpenDroneMap) are now so good that, for our purposes, GCP’s have no marginal benefit.

We have high-quality information about our trial area from an established authority – the Cultural Heritage Agency of Iceland. Their 2007 report of finds is the basis of our trial runs here at Skalanes. Surveying these predefined areas, we’ve now flown multiple flights, gathered images, and then run three assembles with OpenDroneMap.

Here’s a simple run over the area with no GCP’s:

Here’s a run over the area with GCP’s, adding no location metadata other than the craft’s built-in GPS coordinates (you’ll note that the ground footprint is slightly different, but the roundhouse in the middle is the key feature):

We also manually geocoded the GCP’s for one run.

In the end, we observed no meaningful difference between an assembly with GCP’s and an assembly without them. Adding the images as raster layers to a QGIS project confirmed this to our satisfaction:

With GCP:

Without GCP:

In summary, ground control points just don’t help us much compared to just taking a bunch of good photos and using high quality software to assemble them. They also cost us in portability: even four GCP’s are difficult to carry, occupying significant space in airport luggage and weighing down walks in the field. For scientists interested in doing work over a large area, potentially multiple times, that inconvenience is not a trivial cost.

The ODM assemblies are outstanding by themselves. We have good technology and build on the work of a lot of brilliant people. That frees us to be more nimble than we might have been before.

It wouldn’t be a post by me if it didn’t end with a cool picture. Here’s a drone image from a cliff near the house where we’re staying:

Cross-posted at the Earlham Field Science blog.

Awe in Iceland

The Greater Good Institute at Berkeley considers awe one of the keys to well-being:

Awe is the feeling we get in the presence of something vast that challenges our understanding of the world, like looking up at millions of stars in the night sky or marveling at the birth of a child. When people feel awe, they may use other words to describe the experience, such as wonder, amazement, surprise, or transcendence.

That’s the feeling I have at least once a day, every day, here in Iceland.

And it’s difficult to write a blog post about awe. Almost by definition, it’s an emotion that defies easy explanation. It has a mystique that risks being lost in the translation to plain language.

But if I can’t describe the feeling, I can describe why I’m having it.

Unique among my traveling companions, this is my first-ever trip out of my country of origin (🇺🇸) The sliver of gray in this image is the first thing I ever saw of a country not my own:

When we arrived, I got a passport stamp and exchanged currency – both brand new experiences. However mundane, they were novel for me and began waking me up to the new world I’d entered.

Our first few days were chilly, windy, and rainy. I was much happier about this than were my traveling companions. If our weather wasn’t pleasant, it was nonetheless exactly the immersive experience I was hoping for when I signed up for this trip.

In those first few days, I got to see this amazing waterfall:

I got to participate in collecting soil samples at a glacier —

Solo!

— and in howling wind on the side of a moraine:

The right side of the moraine was calm and quiet. The left was much less so.

For good measure, I saw floating blue ice for the first time:

All this was great, and to me they made this trip worth the months of planning and days of travel difficulties it took to get here.

Then we got to Skalanes, where I’m writing this post, and its landscapes exist on a whole other level. Here are ten views here, drawn almost at random from my photos:

This is a country that absolutely runs up the score on natural beauty.

I’ve taken hundreds of pictures here and they’re all amazing – but none does justice to actually being here. That combination is the signature of an awe-inspiring experience.

Awe puts us in touch with something above and beyond our daily worldly experience – call it the divine, the sublime, whatever speaks to you. It’s an experience you can reproduce if you try, but I believe it connects most deeply when it emerges organically from the world you enter. That’s what’s happened to me here.

It is remarkable that this is what we get to do for work, and I am so glad we have some more time to spend here in this awesome country.

Cross-posted at the Earlham Field Science blog.

Extremely preliminary notes about the Parrot Anafi

After three extremely short flights, these are my notes about the Parrot Anafi drone.

We’re experimenting with different UAV’s as part of the Iceland terrestrial surveying program (we’re being optimistic about travel in 2021…). These are some notes with my initial observations.

This is based on the base case: taking the craft out, taking off, flying for at most a few minutes, and touching back down. As such, don’t take a single word of this as gospel – it’s just preliminary opinions for the historical record. 🙂

Short version of the review: holy portability! One thing I don’t like about the DJI Phantoms is that they are so heavy (both the craft and the RC-tablet unit). If it’s a pain here on-campus, where trips are short, I imagine it’s a pain in the field. The Anafi is ludicrously lightweight and doesn’t feel like a chore to carry around.

Video quality on the built-in camera is fantastic (4K etc.).

It’s not a perfectly seamless integration with our existing workflows. Within our group, for example, we usually use tablets, which are handy for their big screens. The Anafi seems built around the assumption of a phone. That’s true all the way down to the RC unit being designed to accommodate a phone but not a tablet. That’s different, but if we can only get this app for phones I am not necessarily sad about it.

There are many X factors I haven’t yet been thorough enough to review. For example: battery life, stability in breezes (heavy winds make most UAV’s hard to use), and the software/developer ecosystem.

These have been my extremely preliminary notes about the Parrot Anafi. It’s not even close to a comprehensive evaluation of everything we care about. Still, those usability factors are important if this is going to scale and be useful for others. So for now, I’m impressed.

Meet our Terrestrial Mapping Platform!

Just a nice photo from Iceland

I’m excited to share that the Earlham field science program is now sharing the core of our Terrestrial Mapping Platform (TMP)! This is very much a work-in-progress, but we’re excited about it and wanted to share it as soon as we could.

We had to delay the 2020 Iceland trip because of COVID-19. That of course pushed back the implementation and case study component of this project, which was Iceland-centric. But we are moving forward at full speed with everything else. As Earlham has now started the new academic year, we have also resumed work on the TMP.

The project is a UAV hardware-software platform for scientists. It consists of:

  • a consumer-grade drone for capturing images
  • flight plan generation software and application to automate drone flights
  • data analysis workflows for the images – visible light and NIR, assembled into 2D and 3D models

All of this goes toward making science more accessible to a broader range of domain scientists. Archaeologists and glaciologists are our current target cohort, but many more could find use for this work if it’s successful.

We will make all of this accessible in repositories with open licenses on our GitLab instance. Some are already available. Others we will share once we review them for (e.g.) accidentally-committed credentials.

That was all planned, if delayed. We’re also using our extra year of preparation time to make the project better in a few ways:

  • Reevaluating our choice of UAV make and model
  • Prettifying our web presence, which very much includes blog posts like this
  • Reducing the friction and pain points in our current workflow
  • Making our code and infrastructure better in general (I’ve covered my growing emphasis on quality here before)

The team mostly comprises students and faculty (of whom I’m the junior-most). Additionally, there are a few on-site partners in Iceland and innumerable personal supporters who make this possible. We’ll be sharing more at the Earlham Field Science blog as we go. I will undoubtedly share more here as well.

COVID is bad, but we want to make the best of this era. This is one way we’re doing that.

(Disclosure: We received funding for this from a National Geographic grant. None of the views in this blog post or our online presence represents, or is endorsed by, Nat Geo.)

I sure hope there’s enough to do…

I begin the month-long winter break this weekend. Students and teaching faculty finish about a week later. When classes reconvene in January, we’ll start spending a lot of time on Iceland and its spinoff scientific computing projects.

To lay the groundwork, we’ve spent the last few weeks clearing brush:

  • Updating operating systems and apps on a dozen laptops, handsets, and tablets
  • Syncing accounts with Apple and Google
  • Sitting in on planning/logistics meetings
  • Coordinating with the students who will do most of the actual research and development
  • Producing the list of software improvements we need to make

The last item is the most substantial from the perspective of my colleagues and students in CS. We build a lot of software in-house to collect and visualize data, capture many gigabytes of drone photos, and run datasets through complex workflows in the field.

It takes a lot of work locally to make this succeed on-site. Students have to learn how to both use and develop the software in a short time. Since the entire Iceland team (not just CS students) depends on everything we build, these projects provide real and meaningful stakes.

All of this has come together in the last few weeks in a satisfying way. We’re up to 62 GitLab issues based on our experiences using the software. That’s a good enough list to fill a lot of time in the spring, for both students and faculty.

We’ll hit the ground running in January, when the clock officially begins ticking.

Going to Iceland

The title is the tl;dr. (!)

Barring a catastrophe (a non-trivial assumption!), I’m traveling to Iceland with the field science program at Earlham in June 2020. I’ll be one of the faculty members on the student-faculty research team for the annual expedition. It’s a thrilling opportunity.

I’ve been working toward this for a while. I’ve acted as “ground control” for several summers, both as a student and in my current role as a member of the CS faculty. Between trips, I’ve been part of the team that’s engineering and coding software to do fascinating things:

  • DNA analysis
  • in-field mobile data collection
  • drone flight planning
  • image analysis

But for a variety of reasons, it’s never been feasible for me to take the trip. Finally, the path seems clear.

Of course, anything could happen. This could fall through, or it could turn out amazingly well. Wherever it ultimately falls on that spectrum, I’ll spend the next few months wrangling various software projects, mentoring students, and assisting the official leaders of the trip as we prepare to go do science. My focus will be on building software, but that will be just one task among many.

Text is not a perfect medium for communicating emotion, but I’m quite excited. I wanted to flag this as a personal and professional milestone. I’m certain to be posting more about it over time.

Computing lessons from DNA analysis experiments

I’ve been working with my colleagues in Earlham’s Icelandic Field Science program on a workflow for DNA analysis, about which I hope to have other content to share later. (I’ve previously shared my work with them on the Field Day Android app.)

My focus has been heavily experimental and computational: run one workflow using one dataset, check the result, adjust a few “dials”, and run it again. When we’re successful, we can often automate the work through a series of scripts.

At the same time, we’ve been trying to get our new “phat node” working to handle jobs like this faster in the future.

Definitions vary by location, context, etc. but we define a “phat node” or “fat node” as a server with a very high ratio of (storage + RAM)/(CPU). In other words, we want to load a lot of data into RAM and plow through it on however many cores we have. A lot of the bioinformatics work we do lends itself to such a workflow.

All this work should ultimately redound to the research and educational benefit of the college.

It’s also been invaluable for me as a learning experience in software engineering and systems architecture. Here are a few of the deep patterns that experience illustrated most clearly to me:

  • Hardware is good: If you have more RAM and processing power, you can run a job in less time! Who knew?
  • Work locally: Locality is an important principle of computer science – basically, keep your data as close to your processing power as you can given system constraints. In this case, we got a 36% performance improvement just by moving data from NFS mounts to local storage.
  • Abstractions can get you far: To wit, define a variable once and reuse it. We have several related scripts that refer to the same files, for example, and for a while we had to update each script with every iteration to keep them consistent. We took a few hours to build and test a config file, which resolved a lot of silly errors like that. This doesn’t help time for any one job, but it vastly simplifies scaling and replicability.
  • Work just takes a while: The actual time Torque (our choice of scheduler) spends running our job is a small percentage of the overall time we spend shaping the problem:
    • buying and provisioning machines
    • learning the science
    • figuring out what questions to ask
    • consulting with colleagues
    • designing the workflow
    • developing the data dictionary
    • fiddling with configs
    • testing – over, and over, and over again
    • if running a job at a bigger supercomputing facility, you may also have to consider things like waiting for CPU cycles to become available; we are generally our systems’ only users, so this wasn’t a constraint for us

A lot of this is (for computer scientists, software engineers, etc.) common sense, but taking care to apply that common sense can be critical for doing big interesting work.

The punchline of it all? We managed to reduce the time – walltime, for fellow HPC geeks – required to run this example workflow from a little over 8 hours to 3.5 hours. Just as importantly we developed a bunch of new knowledge in the process. (I’ve said almost nothing here about microbiology, for example, and learning a snippet of that has been critical to this work.) That lays a strong foundation for the next several steps in this project.

If you read all this, here’s a nice picture of some trees as a token of my thanks (click for higher-resolution version):

Image of trees starting to show fall color
Relevance: a tree is a confirmed DNA-based organism.

A tale of two large-ish app updates

This week I spent some time working on Earlham CS’s Field Day Android application. It’s the app used by our student-faculty field science researchers to collect data on trips to, say, a glacier in Iceland. I made two substantial changes.

The first was updating our system dependencies. At the start of the summer, Field Day wasn’t a fully modern application. That’s mostly because its development is contingent on the interest levels of students and faculty who (correctly!) have other priorities during the academic year. We experience our only consistent spikes in development during preparation for a trip to Iceland. Even then, we tend to focus on adding or fixing features, rather than major design choices or boring updates. Whatever their benefits, such changes always risk eating up precious time in the short run.

As a result, we had long neglected to update SDK versions, themes, and other app fundamentals. I wanted to fix that before classes resumed this month.

Not being an Android expert (yet?), I relied on a mix of automated tools in Android Studio, manual code tweaks, and careful testing to push the update process forward. Here’s how I described it in my merge request:

I wanted to make us a “grownup” application, by which I mean that I wanted to move us away from as many deprecated tools and dependencies as possible, as far in advance of a field trip as possible. (EDIT: With one exception: these changes do not attempt to resolve the [looming] Google Drive [API] deprecation.)

To that end, this merge request involves substantial changes to build fundamentals like the Gradle version, as well as some Lint cleanup and general tidying. Much of it was done following a simple pattern:

– run a built-in Android Studio update tool (e.g. “Update to AppCompat”)

– change a bunch of details in the code so it builds

– test on the device

– lather, rinse, repeat

Field Day merge request 9

After some tests by myself and a colleague, I approved the merge.

To reward myself for accomplishing that admittedly tedious process (which followed a long, slow battery testing process), I did something more fun.

For a long time I’d wanted to improve Field Day’s UI to streamline the navigation. I made a batch of changes, then submitted the following merge request:

[Field Day’s original creative developers] created a great design palette for Field Day: fun fonts, bright colors, intuitive icons.

I wanted to keep that but update the navigation to reflect the current understanding of our usage model. To that end, this merge centralizes everything onto one screen, miniaturizes our less-used buttons, and puts database and sensors at the forefront.

No specific activities or fragments other than the main screen (and the deletion of the obsolesced sensor screen) have been changed.

I can foresee a future where we do more data analysis and aggregation through the lab notebook, so I’ve preserved the notebook icon for future use.

Field Day merge request 10

The changes in that request took us from this set of two main screens:

Previous main screen (“Sampling” takes user to the second screen)
Previous second screen, containing our sensor and database features

… to this one screen:

Our most commonly-used buttons are on the main screen and fill the entire screen width.

I again checked with my colleague and then approved the request. I’m now working on other issues and have already found the changes to be substantial boosts to the user experience.

This is a sample of my own personal work, but of course building software is a team sport. And it relies on iteration. The original designers of Field Day – current and former colleagues of mine – did a lot of the heavy lifting over a few years building the core logic and aesthetic of the app. As I made my changes in the last few months, I’ve worked to maintain their original design palette while improving usability, performance, and the underlying data model. It’s a useful, specialized, and dare I say fun application, and I want it to keep getting better.

As a closing note about process, I find it sharpens my skills development when I have to summarize my work into prose, as in these merge requests. Writing them requires more precision than a quick chat in a hallway. That’s to say nothing of possible benefits to future developers trying to retrace changes and intentions.