By Walt Hickey
Welcome to the Numlock Sunday edition.
This week, I spoke to Alex Davies who is out with a new book, Kobuk the Destroyer and Other Tales from the Wild Unseen World of Test Engineering. He’s previously written for Wired and a book about the creation of self-driving cars, and now he’s back and writing about the world of testing.
We talked about the early testing of airplanes, including the sacrifice of many chickens, and the bear that inspired the title Kubok the Destroyer. The book is full of great anecdotes that thread the line between ridiculously creative but also bowing to politics and insurance.
You can find the book anywhere you get books, in any format that you read books.
This interview has been condensed and edited.
Alex, thank you so much for coming back on. It is delightful to have you.
Hey. Thanks for having me back.
You are the author of a brand new book called Kobuk the Destroyer. It is all about the testing industry and how much effort goes into product engineering and product testing. And it is one of the most fascinating dives into an industry that I’ve put very little thought into that I’ve ever read and I’ve really, really enjoyed it. Do you want to talk a little bit about what got you on this specific topic?
Yeah, so a lot of my interest in test engineering was born out of my time as a reporter for Wired magazine where I covered the transportation field. And I spent a lot of my time writing about new products, new cars, new kinds of flying cars, hyperloops and all sorts of technologies. I found the best way to get them off their prescribed talking points was to ask them about the development of the thing, of the ideas that didn’t work, of the designs they thought would be the answer but that they had to adjust on the fly. And that got them typically talking about testing because testing is the work that takes you from invention to product.
The more I heard about testing, the more I would ask about testing. I’d go into interviews thinking, “I’m really interested in this product. I really wonder how they tested it.” As I did that, I started to see more and more that this is a whole world. This is a whole way of thinking. As I thought more and more deeply about it, I realized that testing is the work that underpins basically everything we buy, write in, use, interact with. Basically every product has been tested in some way, whether very well or inadequately. That dictates how we interact with it, and our lives are somewhat governed by the stuff around us. And that made me think it’s worth taking a proper deep look at this work.
You’ve been on, and we’ve interviewed you before. You’ve written a book before particularly about self-driving cars. And I really, really love that one because it was all about the early, early, early, pre-alpha testing of how you put a go-kart in the desert that has some self-driving system on it. It is this “fail fast, break things, learn a lot in the process” stuff. I really enjoyed the similarities in this book, where it was just like, “Millions of people fly every day, and in order to make sure that the process is safe, an inordinate amount of testing has had to be done, whether that’s bending the aircraft wings back or launching, I believe, a massive gelatin that vaguely resembles a bird at an engine.”
Yeah. And they actually use real birds. They shoot these gelatin birds two miles an hour into the jet engine. And that’s to make sure that the jet engine can keep running or shut down safely in the event of a bird strike. But the actual certification tests require you to use a real once living bird. And it’s typically a chicken.
Famously found at 40,000 feet.
Exactly. And that’s one of the things that I really learned as I dove into this world is it’s not just all really impressive creativity and rigor. It’s also a lot of trade-offs and negotiations, because the two key tenants of testing are what I call similarity and workability. Similarity is creating a test that mimics as closely as possible the real world. And workability making a test that’s reproducible, observable, practical. That’s like any scientific experiment, you want to be able to get reliable, consistent, good data from it. And that’s what workability is for.
Those things are in inherent tension because the more like the real-world you get, the less controllable it is. The more controllable the test is, the less it’s like the real world. And so the FAA tests for birds, what’s called the bird ingestion test, uses chickens because its standard says, “Look, we’re testing for a four-pound bird. It’s easy to get a four-pound chicken.” They’re very available. You can easily buy them plucked and frozen. You just thaw them and stick them in the cannon.
But as you noted, birds don’t fly. The FAA has never recorded a bird on chicken encounter. It has recorded more planes hitting bearded seals than planes hitting chickens. One, by the way, one bearded seal. One poor bearded seal.
So there’s this tension in the test, and there are other little holes you can poke in it. As we noted, they use these gelatin blobs a lot of the time in R&D. And those blobs are good because they mimic the mass of a chicken, but they’re not a chicken, right? They even have pretty sophisticated blobs that inside them have things that mimic the bones and the tissues. But at the end of the day, it’s that thing. It’s a more controllable thing because each one is exactly the same as the next. So if you’re testing different iterations of an engine, it’s going to give you good, reliable data. But, again, you’re not using a real bird.
Then there’s the fact that we only test up to four-pound birds, whereas geese are one of the major problems. Geese are what took down Captain Sully’s flight with the miracle on the Hudson. He flew into a flock of geese. Geese are like 10 pounds.
So I talked to a risk epistemologist about this, and this is a guy with the perfectly fitting name of John Downer, who dove into this work and pulled a lot of these lessons out and, to be fair, drafting off of his research. What he told me was that it’s not that jet engines aren’t safe, right? The miracle on the Hudson flight, those jet engines did what they needed to do, which was shut down safely and gave Sully the opportunity to land his plane safely. They didn’t explode. And it’s not that planes are less safe than we think they are, but it is that maybe the test isn’t telling us quite what we think it’s telling us. And that’s a thing that’s worth keeping in mind.
It’s true of basically all testing. You have to think really carefully about what I am actually learning here. And that’s a lot of what I wrote the book to figure out.
Yeah, I liked how you also go into some of the history of different testing and how the testing itself can evolve as the testing itself is tested. Are there any memorable tests or examinations of various different kinds of tech that have evolved over the course of the life of the product as we’ve come to understand more about whether it’s the product itself or what can go wrong with it?
Yeah, so let’s stick with airplanes because it’s a really good example. A new airplane model goes through years and years of incredibly intense testing. It’s brought to the coldest places in the world. It’s brought to the hottest places in the world. It’s not just birds they’re firing into jet engines, it’s sand. And it’s not just sand, it’s sand from different deserts. That jet engine can handle taking off from Dubai and taking off from Phoenix.
The level of this is incredible. But when aviation was first becoming a thing at the start of the 20th century, nobody had the idea how to do any of this. They tried to figure out wing strength by just throwing sandbags on the wings while the plane was on the ground. They were like, “All right, it can take this weight and the plane’s about this weight. So I’ll figure it out.”
Some of the best early work comes from Jimmy Doolittle, who later became famous during World War II because he led the first bombing raid over Tokyo. And if you remember the movie Pearl Harbor with Josh Hartnett and Ben Affleck, Jimmy Doolittle is played by Alec Baldwin in that. But when he was a younger guy in the 20s, he was a great pioneering pilot. He was actually one of the first graduates of MIT’s aeronautical engineering program. For his master’s thesis, he said, “The way we’re testing, we’re load-testing planes, which is essentially figuring out how many G-forces they can pull before the wings crack, with sandbags.” What people do who don’t even put the bags, they just put sand on the plane and move it around with rakes or don’t do any testing at all, which is true of some of these people who just build a plane and fly it.
Doolittle said, “I want to create for my master’s thesis a new way of testing airplanes.” So he got a Fokker plane and he created this regimen doing all of these tricks, essentially barrel rolls and inverted flying and he used accelerometers in his plane, which people didn’t really think about using for anything at the time. He created a regimen in which he figured out that the plane he was flying, which was advertised to be able to pull up to something like nine times its own weight, was really only able to pull about seven.
But the big takeaway was less the results of that specific plane and more this way of thinking about how do you actually test the thing? You’ve got to actually fly it. You actually have to mimic the forces that it’s going to undergo and get good measurable data. Not many people can say “My master’s thesis became a landmark paper in the world of aviation safety,” but Jimmy Doolittle could.
That’s amazing. Obviously, it’s funny. I think that tension is a key theme of the book. Can the aluminum tension hold over the course of the life of a product? But also, there’s a ton of tension between various different forces in the testing world. It’s a pervasive theme, I would say, in each of these chapters. Manufacturers and the organizations that test their equipment have a collaborative but also, to an extent, adversarial relationship. One of them is potentially going to destroy the ability of the other to make a livelihood off of the thing that they’ve created. What are some examples from the book about this tension coming into play?
Yeah, I talked to one guy named Michael Peck, who runs an actual reliability engineering lab where people can bring their products and he’ll test them. The thing he told me is that companies hate their own reliability engineers. Because they are the people saying that you can’t sell this thing or you have to fix this thing. They’re basically adding cost and complication to a product to make it safer.
My book follows the arc of the 20th century, largely, and comes into the 21st. But a lot of what we’re looking at is how tort law evolved and how the legal system essentially demanded safer products for American people. One is the lawnmower standard. The Consumer Product Safety Commission was created in the ’70s. This came out of Lyndon Johnson’s Great Society. And it was a time when someone said, “Hey, what about if we have products that don’t definitely maim you?”
A lot of people were losing fingers to lawnmowers because lawnmowers have chutes and they would get clogged. You could turn off the engine, but it was a pain. So, people were just reaching in and being like, “I’m not going to touch the blade, I just need to clog this thing,” and off goes a finger. A lot of people were being hurt by this. The obvious answer that the CPSC came up with was you have to add a dead man’s switch to the lawnmower. If you take your hand off control to reach into the chute, it has to automatically stop. But then they got into this huge debate over how to make that happen. How quickly does it have to stop? Does it have to stop in one second? Does it have to stop in five seconds? And they did all of these tests.
Ultimately, the CPSC came up with a three-second limit. It did all of these human factor studies where it brought in test subjects and had them work fake lawnmowers and see how long it took them to take their hand off and go down here. What happened was the CPSC comes out and says, “Three seconds, that’s how fast it has to work.” Then consumer advocates come out and argue and file lawsuits and say “It needs to be faster than that, it needs to be a second and a half.” Then the industry comes out and says, “This is too hard to accomplish. It needs to be five seconds.” That’s a lot of the negotiation, you see.
And it’s the same thing like when the National Highway Traffic Safety Administration, which governs car safety, creates a new crash standard. I looked at one example, which was created in the ’80s and ’90s, of side impact, basically getting T-boned, and the level of protection they wanted to offer car occupants. That came out in G-forces. Basically, they said, “We’re thinking about limiting G-forces to the people inside the car, to somewhere between 80 and 115 Gs.” They had numbers on this end of the scale that will save this many lives, so they said 80 to 115. Consumer advocates would say 70. Industry comes and doesn’t propose a number. The car industry unanimously said, this shouldn’t be a rule at all. But still, NHTSA compromised and came up with a level, a limit of 85.
That’s why I think standards are a really important part of this book. Because you see a warranty, you see a standard, and it can almost feel like it’s just preordained, or like this is definitely safe enough, but it’s always a negotiation. It’s a very human work of saying, “This is safe enough. This is reliable enough.”
There’s only so much that science can say. It is actually more of a political or legal issue. I was very intrigued by the fact that a lot of the origin of a lot of these laboratories was like the insurance industry or the casualty insurance and that kind of stuff. We’re attempting to use science to solve an actuarial problem that is ultimately going to be a political question.
Yes. The Underwriters Laboratories, which is the first company that turned testing into a business, was born out of insurance questions. Its founder, this guy, W.H. Merrill, was brought into the Chicago World’s Fair in the 1890s, where they were going to show off electricity. And electricity was the huge thing. It was all the rage, and there was going to be a palace of electricity. The people insuring the fair were like, “This is going to light everything on fire, and we’re not going to give you coverage.” So the fair brought in Merrill, who had just graduated from MIT’s brand-spanking new electrical engineering program, to come in and make sure everything was actually safe. And he came out of that with the idea of like, there’s a business here. And that’s why it’s called Underwriters Laboratories, because it’s labs-based and going to give insurance companies confidence that they can insure actual things. That’s how it started off.
Fascinating stuff, man. Really, it’s a very exciting book. The number of charming anecdotes I’ve come away from this book with are voluminous. There’s so much cool stuff in here. I do want to finish by talking about the eminent entity in the title of the book, Kobuk, and the magnificent talents of this particular employee of the testing industry.
I’ll start with the story of Kobuk. Before there was a bear named Kobuk the Destroyer, there was a bear named Kobuk, and before there was a bear named Kobuk, there was an unnamed bear in the Alaskan wilderness who lived with his mom and his two cub siblings. Kobuk’s mom was a classic problem bear. Problem bears is our euphemism for bears who are very motivated to get human food because they’ve developed a taste for human food. And that taste for human food brings them into conflict with humans, conflict that can be dangerous for people and for bears. Kobuk’s mom was a classic problem bear, and she led her cubs onto an Alaskan rancher’s land and began killing the free-range chickens for a meal. The rancher came out and shot and killed Kobuk’s mother and also killed one of the cubs. But Kobuk and his surviving sister, Nikina, weren’t killed. They were granted a life of leisure, and they were granted a new life at a bear sanctuary outside of Yellowstone National Park in Montana. This was a sanctuary for problem bears where they got to live in peace rather than in danger and couldn’t hurt people. But they also had to sing for their supper.
The best way to stop a bear becoming a problem bear is to keep it out of human food in the first place. That’s why if you go backpacking, you often have to use a bear canister that you put your food in that is basically impossible for a bear to open. If you go to a national park or an area with bears, you tend to find those garbage cans with the little metal lid over the latch, and you have to slide your hand all the way in there to pull the latch.
If you look carefully at those products, you might notice that there’s a little seal that says Certified Bear Resistant. They’re certified by bears at this sanctuary where if you’re a manufacturer and you’ve got a new cooler or bear canister or garbage can you think is going to keep bears out, you bring it there, you fill it with fish and honey and all sorts of things bears like, and you hand it to a bear. The bear gets up to one hour to break into that product. And if they can’t break in, it’s Certified Bear Resistant. If they do break in, one, they get to eat what’s inside, and two, you, the manufacturer, have to go back to the drawing board and figure out what went wrong. And Kobuk was really good at this. I talked to the guy who worked with Kobuk, and he was like, “That bear, man, he was so smart. He could get into anything. He could get into one of those garbage cans because you think, well, my whole paw doesn’t fit in here, but my claw does. And if I twist my body 180 degrees around and bend my arm back behind me, I can move it forward and get my claw in far enough to pop it open.”
He was called the destroyer because he destroyed egos. He destroyed human egos because all these manufacturers would think, “This thing’s great, I figured out everything a bear can do,” and nope, not Kobuk.
It’s such a great story. I have heard before that there’s a considerable overlap between the smartest bear and the dumbest human, which is one reason that there’s so much difficulty sometimes with these trash cans. But it was just so great hearing that the actual bleeding edge of bear testing technology is this bear, Kobuk.
Before they had this program, which has been around for 20, 25 years, they used mechanical forces to recreate bear strength. They’d have a big block with a metal spike on it, and they’d have hydraulic things to mimic a bear’s strength. But it’s not enough to mimic a bear’s strength. You have to mimic a bear’s intelligence, and the best way to do that is with a bear.
The book is called Kobuk the Destroyer and Other Tales from the Wild Unseen World of Test Engineering. Alex, this was so much fun. I really, really enjoyed it. Thank you for writing it and thanks for talking to us.Why don’t you tell folks a little bit about it and where they can find it to purchase?
So you can find it at your local bookstore. You can find it on Amazon. It’s published by W.W. Norton. You can get it in hardcover. You can get it in e-book. You can get it in audiobook. And before too long, it’ll be out in paperback. But I suggest not waiting.
Amazing, all right. That’s going to do it, dude. Thank you very much for coming on.
Yeah, thanks for having me. This was fun. It’s always fun to talk about it, especially now that it’s done and I don’t have to keep working on it.
Edited by Crystal Wang
If you have anything you’d like to see in this Sunday special, shoot me an email. Comment below! Thanks for reading, and thanks so much for supporting Numlock.
Thank you so much for becoming a paid subscriber!
Send links to me on Twitter at @WaltHickey or email me with numbers, tips or feedback at walt@numlock.news.