When building software for large datasets or HPC workflows, we talk a lot about the trip cost versus the item cost.
The item cost is the expense (almost always measured in time) to run an operation on a single unit of data – one member of a set, for example. The trip cost is the total expense of running a series of operations on some subset (possibly the whole set) of the data. The trip cost incorporates overhead, so it’s not just N times the item cost.
This is a key reason that computers, algorithms, and data structures that support high-performance computing are so important: by analyzing as many items in one trip as is feasible, you can often minimize time wasted on unnecessary setup and teardown.
Thus trip cost versus item cost is an invaluable simplifying distinction. It can clarify how to can make many systems perform better.
Yes, Virginia, there is a trip cost
Christmas trees provide a good and familiar example.
Let’s stipulate that you celebrate Christmas and that you have a tree. You’ve put up lights. Now you want to hang the ornaments.
The item cost for each of the ornaments is very small: unbox and hang the ornament. It takes a couple of seconds, max – not a lot, for humans. It also parallelizes extremely well, so everyone in the family gets to hang one or more ornaments.
The trip cost is at least an order of magnitude (minutes rather than seconds) more expensive, so you only want to do it once:
- Find the ornament box
- Bring the box into the same room as the tree
- Open the box
- Unbox and hang N ornaments
- Close the box
- Put the box back
Those overhead steps don’t parallelize well, either: we see no performance improvement and possibly a performance decline if two or more people try to move the box in and out of the room instead of just one.
It’s plain to see that you want to hang as many ornaments as possible before putting away the ornament box. This matches our intuition (“let’s decorate the tree” is treated as a discrete task typically completed all in one go), which is nice.
Whether Christmas is your holiday or not, I wish you the best as the year draws to a close.