KEVIN ESVELT: I made a discovery six years ago that was literally unimagined by any human being at that time – not in science fiction, not in any form of literature, not in any scientific journals.

We can do things that our ancestors would have considered godlike.

The next day, I woke up absolutely terrified.

Is this the sort of thing that I can tell anyone about? What are the odds that someone else will discover it? Are they likely to realize the implications? How would they talk about it? Could it be misused? In what ways? How seriously?

If you can edit a creature to solve a problem, you can edit a creature to create a problem.

If you’re building something that you think might autonomously spread beyond your control, don’t do that if it might do so should it escape the laboratory.

If you give anyone the power to unilaterally change the configuration of the universe at scale, then as soon as that capability becomes available to everyone, we’re all in deep, deep trouble.

CATERINA FAKE: That was scientist Kevin Esvelt. He discovered a technique for engineering new traits and then forcing them to spread through species in the wild. The technique is called a gene drive and it gives humans a power we’ve never had before: to change the DNA of entire species in nature.

A successful gene drive could prevent mosquitoes from transmitting malaria. Think of the lives you could save. But do humans really have the right to impose our own outcomes on every other species?

“We are as Gods, and might as well get good at it.” Stewart Brand wrote that in the very first issue of the Whole Earth Catalog in 1968. “We. Are. As. Gods.” And nothing we’ve discussed on this show so far, speaks to this as strongly as what you’re about to hear.

[THEME MUSIC]

FAKE: I’m Caterina Fake, and we know that the boldest new technologies can help us flourish as human beings – or destroy the very thing that makes us human. It’s our choice.

A bit about me: I co-founded Flickr, and helped build companies like Etsy, Kickstarter, and Public Goods from when they first started. I’m now an investor at Yes VC, and your host.

Let me start with a question: If you could get rid of malaria – which is responsible for hundreds of thousands of deaths every year – would you do it? Any humane person would say “yes”.

Every two minutes a child dies of malaria. That’s a lot of kids. That’s a lot of broken hearts. But what if the only way to save those kids was to actually change the DNA of an entire species – would you do it? Are we willing to make a radical change to the world? Play God? That’s the crux of the question.

On today’s show: Kevin Esvelt from the MIT Media Lab. He’s one of the leading scientists with the power to create those future creatures. And he takes the task as seriously as you might hope.

The technique he is proposing is called a gene drive and it gives humans a power we’ve never had before: to change the DNA of entire species in the wild.

The gene drive is made possible by the breakthrough in DNA editing called CRISPR. CRISPR made it super easy to edit the DNA of a lab animal, in order to bring out specific traits. But a gene drive takes this ability out of the lab and into the wild. It allows scientists to introduce a new trait to a lab animal, and then make sure it’s passed down through an entire species, for all time.

No one has yet released a gene drive into nature – and there are sharp U.N. restrictions around them. But Kevin is among the handful of scientists working toward the first field trials of a gene drive. This month, he announced he’s developed a way to make gene drives safer.

But there’s no guarantee that a rogue scientist – or even a well-intentioned one – might release a gene drive. And that would have consequences, expected or not. It’s hard to over-state the scope or newness of this power.

ESVELT: I made a discovery six years ago that was literally unimagined by any human being at that time – not in science fiction, not in any form of literature, not in any scientific journals. No one even imagined that we might be able to readily edit entire species, that potentially a single individual could single handedly genetically engineer a whole species.

FAKE: This is what’s so powerful and startling about gene drive. A single person can actually change the trajectory of a species. You can change what a mouse is, forever. What a mosquito is, forever. It’s one thing to change the DNA of one mouse in a lab. But changing the DNA of all wild creatures in perpetuity? On and on into the future, throughout all ages, world without end? That’s God stuff. That’s God.

And I think that’s why this particular ability raises such alarm amongst so many people – even the most passionately pro-science among us.

BARATUNDE THURSTON: I don’t want to be loose with the word “insane,” but this might be insane. It’s certainly an extraordinary idea. The scale of unanticipated consequences for editing a species is so beyond anything. It’s just like, what are you doing? What are we doing?

FAKE: That was Baratunde Thurston, whose career sits at the intersection of comedy and technology. We’ll hear more from Baratunde in the second half of this show, when we’ll dig into the big questions that surround the gene drive. But we’re going to start with Kevin, and the problem he’s trying to solve.

Kevin discovered a passion for genetics early.

ESVELT: When I was in sixth grade, I had the incredible good fortune to have amazing parents who took me to the Galapagos Islands, and it’s a remarkable place. You have, of course, the turtles, the iconic turtles.

You also have creatures like marine iguanas, which Darwin called “imps of darkness”. And they have these webbed feet and long tails adapted for swimming. You also have these birds, called boobys. And there’s blue footed ones and red footed ones. And you wonder: How did they end up with blue and red feet?

It was just the most magnificent and magical experience, seeing all these incredible creatures. And I was just captivated.

FAKE: Kevin started reading The Origin of Species by Charles Darwin. He was fascinated. But he had some questions.

ESVELT: If evolution can make it a blind process, is there a way that we could learn how to do the same thing, make equally magnificent creatures?

FAKE: Inspired by the Galapagos, Kevin knew in 1994 – at the age of 12 – that his future lay in genetic engineering. But he had no way of knowing just how much the field itself would evolve in the years ahead.

If you’re not a genetic engineer, you might find it useful to brush up on the science here. So let’s talk genetics for a minute. We all know that strands of DNA are the building blocks of life. All our traits are embedded in our DNA. When two creatures reproduce, the child inherits DNA from both parents.

Over time, evolution favors the traits that give that animal an edge. This is called natural selection; and it’s what allowed giraffes to evolve their tall skinny necks, and humans to evolve our big, heavy brains.

But us humans – thanks to our big brains – are always tampering with natural selection. We’ve been changing the species around us for all of human history in all kinds of ways. Want a fast horse? Pick the fastest female and the fastest male and mate them, over and over again.

ESVELT: There’s all sorts of examples in which we’ve used directed evolution – selective breeding – as the magician’s wand, just to turn life into whatever form or character that we choose.

FAKE: It’s how we turned wolves into chihuahuas. It’s how ancient Mexicans turned a wild grass into corn on the cob.

ESVELT: Humanity turned this spindly little Mexican weed that has kernels that normally pop off and blow away in the wind, into these magnificent ears of corn, one of which is big enough to almost give you a full meal. I mean, that’s incredible.

FAKE: Selective breeding is incredible. But it takes a long time. In the 20th century, we started changing other species faster. We discovered DNA, and found ways to select the traits we wanted, like, say, sweeter strawberries. Then we found a way to create new DNA – and therefore new kinds of species, like crops that are resistant to disease. These crops are often called GMOs, or genetically modified organisms.

Then everything changed. We discovered something called CRISPR (which is spelled C-R-I-S-P-R, by the way). I like to think that they were following the lead of Flickr be taking out the E. CRISPR made genetic engineering even faster.

You can think of CRISPR as a tiny, tiny scalpel: It allows scientists to make changes in DNA precisely and easily. It’s hard to overstate this.

ESVELT: Before CRISPR, our default expectation in biology is that you run an experiment and there’s a less than 50% chance that it’s going to work. CRISPR almost always works, and it works in almost every species.

FAKE: So we’ll pick up Kevin’s story in 2014. The kid who loved the Galapagos has gone to Harvey Mudd College, and studied under renowned geneticist George Church at Harvard. He’s now a professor at the MIT Media Lab. CRISPR has recently been discovered – and it changed everything. But weirdly, Kevin felt bored. The work ahead seemed obvious to him. He was contemplating a possible career move one cold Spring morning.

ESVELT: I was walking to work through the Emerald Necklace, which is this series of parks in Boston. And there is this stream and a bridge going over it. And that day, I happened to see a turtle. And for some reason I decidedly wondered, “You know, we’ve been using CRISPR to edit everything.” So I wondered, “Would we ever edit this turtle?”

And I thought, “Well probably not. Because even if we could edit one in the laboratory, natural selection would remove that change if we tried to release that turtle in the wild.” And then I thought, “Well, but what if we made genome editing happen every generation?”

FAKE: Kevin was struck with an ingenious idea: What if you could not only edit a turtle’s genome, but also encode the tool that does the editing? This editing tool would be inherited by the next generation of turtles, who would spread it to the next, eventually changing the entire species.

Kevin was enthralled with the possibilities, and ran back to his lab to research. He found that the gene drive actually exists in nature. And that a scientist named Austin Burt had proposed a similar system in 2003, before CRISPR. Kevin built on Austin’s work – and started thinking about malaria.

ESVELT: Imagine in the long run, instead of having to suppress mosquito populations, we edit them so they don’t like the smell of humans. My first thought was, “Wow, that’s amazing. This is the key. This could save millions of lives, and malaria is just the start.”

But the next day, I woke up absolutely terrified. Because here is something that no one imagined. It had finally sunk in: Anyone could do this, in any species. And if you can edit a creature to solve a problem, you can edit a creature to create a problem.

FAKE: The full spectrum of doomsday scenarios flashed before his eyes.

ESVELT: Is there a chance of an accident? Because it shouldn’t take that many to start spreading in the wild if you happen to release it. How many laboratories actually are careful about containment and safeguards? Was CRISPR-based gene drive a potential bioweapon?

FAKE: Do you still have that reaction? Like do you still feel this sense of impending doom?

ESVELT: On thinking about it a lot, seems that the technology gods had mercy on me, in the sense that it’s not all that dangerous. But when you begin thinking that way, of: Should this invention be made public? Is this the sort of thing that I can tell anyone about? What are the odds that someone else will discover it? Are they likely to realize the implications? How would they talk about it? Could it be misused? In what ways? How seriously? And what we’re talking about here is a way for individuals to create constructs that will spread autonomously, without any human input.

FAKE: Right, that’s completely new.

ESVELT: You can imagine we might be able to make something that could spread on its own, but it’s always been unlikely before now. But CRISPR-based gene drive was arguably the first technology in human history that could, once created, spread without the benefit of humans. That has never happened before.

FAKE: I thought this might be the moment when Kevin realized how dire the consequences might be, like when the physicists working on The Manhattan Project said they’d become “the destroyer of worlds.”

FAKE: And when do you get to your like, you know, Robert Oppenheimer, “We are become death” moment?

ESVELT: I thankfully never got that far, because I realized that anything that is slow and obvious and easy to block isn’t much of a threat. But I do think a lot about Oppenheimer through a different quote: “When you see something that is technically sweet, you go ahead and do it. And you argue about what to do about it only after you have had your technical success.”

FAKE: Achieve sweet technical success – and worry about the consequences later. This reminds me of a lot of things you hear in Silicon Valley today. But Kevin has a different view.

ESVELT: We decided to tell the world before we ever tested in the lab, because we were reasonably confident it wasn’t all that hazardous, but we wanted to establish this precedent whereby no one should do this kind of work without telling other people what they were thinking of doing first.

FAKE: Kevin made a monumental decision here: Gene drive would not be a scientific secret. It would be a public conversation from the very beginning.

ESVELT: If you don’t, you’re denying people who share the environment the opportunity to have any kind of voice in research intended to change it.

FAKE: Kevin is one of the most thoughtful scientists I’ve talked to. He sees both the potential and peril of designing these future creatures. He created his “scary list” of possible outcomes, and started taking it out to the world.

And this seems the right time to pause, and explain a bit more about how a gene drive works. We’ve established that CRISPR is a tiny scalpel that cuts DNA precisely and easily. Now I’d like you to think of dominoes.

Imagine a million dominoes, precisely positioned one right after the other. They stand in long, straight lines like soldiers, and branch off into intricate spirals and branches. Every domino is exactly positioned so that if you topple one, it will cause a chain reaction, taking down every domino in its path. You know: the domino effect.

Once the dominoes are set and the first domino is pushed, you cannot control which ones topple. It’s the DNA of the domino pattern, its immutable future… Unless you have gene drive.

Imagine a row of dominoes leading to a split-off. One curves to the right, the other curves to the left. Normally, both sides would topple. But what if you could always – and I mean always – make the dominoes topple to the left. You’ve given the left path an advantage.

This happens again and again. The dominoes can topple only to the left. That’s gene drive, using CRISPR to give one side, or trait, an overwhelming biological advantage, generation after generation. Pre-determining the path with a super dominant trait.

But what if you want it to stop? What if you want those dominoes to return to their original pattern?

Let’s say after five split-offs, after five times toppling to the left and only to the left, we want the dominoes to go to the right and resume their patterns without any of our interference. We place a stopper on that left hand path and the dominoes are forced to go to the right. At the next split-off, they topple seamlessly to both sides.

This is called the daisy drive, another one of Kevin’s innovations. The daisy drive ensures that after a certain number of generations, the gene drive cuts out, the trait hacking stops, and nature’s rule of recessive and dominant genes is once more the law of the land.

This is crucial because it means that gene drive can’t get out of control – and stay out of control. We’re not going to change an animal permanently, forever and ever. We won’t have to watch the dominoes fall, helpless to do anything, wishing we had never made them go left.

And now, back to Kevin… He wanted to use gene drive to solve a local problem alongside a local community. He chose Lyme disease, because it’s a huge problem in New England, infecting up to 600,000 people a year, many of whom go undetected and untreated. People in New England – and especially parents in New England – are terrified of Lyme disease.

ESVELT: It’s particularly bad on the islands of Nantucket and Martha’s Vineyard, off the coast of Massachusetts. But the ticks aren’t born infected with Lyme disease, they get it when they bite an infected animal. And the best reservoir of disease, we call it, is the white-footed mouse.

FAKE: So the white-footed mouse is the original source of Lyme disease before it gets to the ticks, and then to humans. Kevin believes he can address this problem by editing the DNA of mice and preventing them from passing the disease to ticks. But there are two paths: one is using gene drive and one without.

Kevin took these choices to the experts: the locals of Nantucket. His forum: A classic New England town hall meeting.

ESVELT: I was pretty nervous because I’d given quite a few scientific talks but I don’t think I’d really addressed the community before.

I talked about why we were there talking to the community and emphasizing that we hadn’t done anything yet – and part of the reason was that we wanted them to decide if they were interested in any of these at all. And if not, then we would just walk away.

I was able to talk to probably half of the people who were there afterwards. Everyone hate, hate, hate, hate, hated Lyme disease, and they were incredibly interested in a possible solution.

FAKE: The residents in Nantucket and Martha’s Vineyard both opted to pursue the project – but not, as of yet, the gene drive.

I admire the way Kevin is out in front of this question. This is not a defensive conversation. He’s playing offense on this. He’s going out into the world with what I call the “scary list” and saying, “Hey friends, colleagues, co-inventors, neighbors, let’s all exercise caution here.”

And Kevin needs to do this. Because the stakes are extremely high. Kevin believes – as do many scientists – that a gene drive is our best possible tool for preventing malaria and other diseases that kill people every day. Those 10 kids who died since you started listening to this episode? They could still be with their parents. They could grow up and go to school and make a difference in the world. And that’s a powerful motivation to keep developing this technology.

But if Kevin makes a mistake? If he unleashes a gene drive that causes irreparable harm to an environment? Or worse – if he gives the idea of a gene drive to another scientist with evil intentions? It could conceivably, no joke, unleash a chain reaction that ends civilization as we know it.

Welcome to the Should This Exist? workshop. Here, Kevin and I will work through some of the thorniest human questions raised by the gene drive. I’ve gathered provocative ideas from a handful of smart, creative experts. We asked them to push our thinking about both possibilities and pitfalls.

I wanted to start with a thought that may have already crossed your mind: The complexity of nature is beyond our understanding. We just don’t know the whole story of how species evolved, or how they’re interdependent.

Our first expert brings this idea to life. Baratunde Thurston is a comedian at the intersection of comedy and technology. He’s the host of the podcasts “#TellBlackStories” and “Spit.”

THURSTON: Just for the record: I’m against malaria. I think if you examine my public record, I’ve always been against malaria. I’m not a fan of Lyme disease; like when Lyme disease shows up, I’m never applauding. And I do want to reduce the amount of human suffering in the world.

But I just have some obvious caution around the idea of the chain of consequences from editing a species. One of the things that jumped out to me with the Lyme disease example is that there’s this chain, like an interspecies chain of events. Like, we can’t know what else the impact of messing with these mice will be.

So does it do something else to them? And then we’re going to have to edit the species again for that? And so then you’re editing to correct your edits? So when we got rid of the Lyme disease from the mice, it turns out that made the mice poop, like a lot more. So now we’ve got to go in and the mice poop is carrying something else, and it’s messing with the underlying flora of the forest – and that’s affecting like plant growth rates. So we’re going to go back, re-edit the mice so that we control the poop valve. Well it turns out, when we control the poop valve on the mice…

And it just like it feels like it could never end. It would be like this hall of genetic mirrors. Nothing humans have done to this point in history has ever been perfect every time. So it’s a fair bet to say they we’re going to mess this up somehow. And what I just want to know is: What’s the back-up plan?

ESVELT: Well, actually, I’ll take the mouse poop over John Oliver’s, “And then all of the seagulls are dead” limerick.

FAKE: If you missed the episode of John Oliver when he talked about gene editing, let me give you a reprise. The limerick was:

There once was a man from Nantucket,

Who gathered some mice in a bucket.

He altered those mice,

Engineered with a splice,

And now all the seagulls are dead.

I agree: Let’s take the mouse poop over the seagull carcasses.

ESVELT: But more seriously, this gets back to this unintended consequences, and how can we know what they are? Anyone who engineers biology understands that you’re tinkering with a system that was not designed by people to be engineered by people.

When we build something from scratch, we design it so that there’s as few interacting parts as possible. Evolution doesn’t work like that – when you tweak something, it’s going to affect a bunch of other things. And when you work it for long enough, in that kind of environment, you start to develop a few rules of thumb.

I have two general rules for engineering evolved complex systems that you don’t understand. Rule number one is you make the smallest possible change that you think can solve the problem – and that’s just because you don’t know what else you might be changing. You can safely assume it’s likely to be something else, but you don’t know what it is. So make the change as small as you can. And rule number two is you start small and local and see what happens before you scale up.

So what we’re doing on Nantucket in the Vineyard is we are working with the communities. If you have a clever idea for how to change the world, you should find a representative community that might be interested in using it to solve some problem you think they have. You should go there and ask them.

FAKE: In some ways, his town hall meeting served the same purpose we’re trying to serve here with the Should This Exist? workshop – creating a new conversation between inventors and the world.

And one of the questions Kevin fielded in the town hall meeting left a deep impression on him. It came up during the Q-and-A at the end of the meeting. I asked him to share it here.

ESVELT: A woman came up and she said, “You know, I don’t have a PhD, but I was wondering if you’re making the mice tick-resistant so the ticks fall off, in the year when you release them, doesn’t that mean that a bunch of these ticks that would have bitten mice are gonna try, fall off, and then would they try to bite a human? Is that going to increase the risk of human infection?”

And I said, “You know, we never thought of that.”

And the actual answer seems to be no, it wouldn’t. And the reason is there’s still just as many wild mice that are good hosts for ticks as there were before.

But here’s an example of something that we had never considered – and might not have considered – had not someone else brought it up.

It’s also worth noting that that’s something that we bloody well should have thought of.  There’s no excuse for us not thinking of that. But again, we’re only human. And part of this is humility. So that’s why we do it. Because local people often know more than we do, regardless of their technical background.

FAKE: The important thing is that regular people need to be involved in this conversation, not just scientists or technologists.

Our next idea comes from Janna Levin, who is a regular on this show and a professor of astrophysics at Columbia. Janna also brought up the ecological cause and effect of gene drive and zeroed in on two potential principles guiding its use: uncertainty and consent.

JANNA LEVIN: The threshold of certainty is a problem in all biomedical fields. It just is. Original vaccinations killed a couple hundred thousand people. And I think it’s a risk analysis that people have to do when you’re introducing it ecologically. We just don’t understand, we don’t understand the whole tower of things stacked on top of each other.

And so that’s what I would worry about: Could we ever, ever really understand the implications or is it just grounded from the start? Because I do think to be uncertain at all might be totally unacceptable.

If we start introducing this into the ecology and we don’t understand the implications, there’s a consent issue about the ramifications. It’s just something I think it’s going to require a massive undertaking and analysis before we can have something equivalent to informed consent for the planet, as opposed to just informed consent for an individual.

ESVELT: This is one we go over and over again. And I think when you develop a technology, you are absolutely morally responsible for any and all consequences of its use, including the unintended consequences. For a technology like gene drive, that’s going to depend entirely on the specific case: What alteration are you making to which organism and where? It’s not something that you can ever generalize.

But you are also morally responsible for the continued consequences of any problems that you could have solved and chose not to because you were afraid of the unintended consequences of acting.

FAKE: It’s a subtle point Kevin’s making – and so important. He feels the weight of responsibility for what might go wrong, should he make a mistake. And also for the things that might NOT go right – should he fail to persuade the public that this experimentation is worth it.

We often ask guests on this show to imagine future headlines – positive and negative. Kevin did this spontaneously, as he imagined the kind of mistake that would lose public trust.

ESVELT: I can imagine the headlines, especially if it’s accidental: “Scientists accidentally turn entire species into GMOs. Is CRISPR to blame?” That would not be good for gene drive. That would not be good for any of the animals that might be helped. That would not be good for the environment – and that would most certainly not be good for children dying of malaria or any of these other diseases. That has an expected cost in children’s lives.

FAKE: Most of the guests on this show have to be nudged a bit to think through all the negative scenarios that might arise from their technology. But Kevin goes straight there. He’s pre-imagining all the negative outcomes. Both to avoid them and avoid the public outcry that can accompany them – and shut research down. So you can see where I was heading as I asked him this next question.

FAKE: So how would gene drive potentially be used as a weapon in the hands of a bad actor?

ESVELT: Why are we deliberately discussing ways of making more effective weapons about technologies on the air? Sorry, this is my whole point, right? You don’t run your red team simulations in the open. You don’t have your black hat types tell you all of the different avenues of attack they’re exploring. Only the people who are wearing white hats at the time get to know what the black hats are doing. You don’t disclose what the black hats are doing.

FAKE: Point taken. Kevin and I went on to talk about this strategy: Imagine everything bad that can happen… but keep those ideas to yourself.

ESVELT: I would be much more comfortable if both the scientific and the technology communities recognize that once you tell the world, it’s out there. The idea has spread autonomously, and no one can take it back. We never want to be in a situation where someone like Ted Kaczynski could come up with, could have access to a technology that would bring down civilization. So we cannot afford to give everyone the power to do that.

FAKE: How did you set up your team so that you have the white hats, and then you have the black hats? You have people who are sitting there thinking through the doomsday scenarios and the worst possible outcomes. Do you kind of like invite in a cadre of paranoid pessimists?

ESVELT: That’s my role. I am a techno-optimist, but I am a social pessimist. That is, I’m very optimistic that we will be able to invent increasingly powerful technologies. I’m just not so optimistic about our wisdom in deciding whether, when, and how to use them. And I’m very much not optimistic about the ability of our institutions to help us make those kinds of decisions.

FAKE: It’s a sentiment that Baratunde shares. And among those suspicious institutions? Baratunde points to corporate America.

THURSTON: Who has an interest in shifting the direction of species? Is there a way to control this so that it’s only or mostly for the public interest? Like if you give this to an oil company or Walmart? What would Walmart do with this? I don’t even… Even my twisted brain doesn’t want to go there and imagine what Walmart would do with genetic editing. We don’t want that world. So we just need some protocols in place to try to manage access and to have some accountability around it, because this will get loose.

ESVELT: Well, I personally, have called for all CRISPR-based dene drive applications to be developed in a non-profit manner. When you just look at the difference between the non-profit lead development, and for-profit in general, you get 20-something percent change in public support. Twenty percent! That is huge.

So I called for all gene drive applications to be non-profit for the foreseeable future. Because if we don’t, if someone develops it to make themselves a profit for something – even if it’s just to control some agricultural pest or whatever – doesn’t matter. The technology as a whole is less likely to win public support. So children’s lives are literally riding on the question of whether scientists can restrain themselves from launching for-profits using gene drive.

FAKE: This point seems so important to me: Businesses have no place in developing gene drive. We just can’t afford to have profit be a motivation when we tamper this powerfully – and this permanently – with the environment. Of course, scientists have motivations of their own. And Kevin is quick to point out the ways in which science today is broken.

ESVELT: When you have academic researchers doing “blue sky” out there, they need to build a new tool to accomplish whatever their study is, they don’t tell anyone what they’re doing. No one who is outside of that laboratory even knows that it exists until it works – and that’s not how most technology development is done.

You typically have, at least, other members of a team who often have different forms of expertise. Often you’ll have management at least knows what’s going on, why the research is going about.

Academia: you have none of that. The culture evolved such that you get credit for doing it. The first one to get it working gets all the credit, and everybody else gets none. You get your paper in Nature, and you get all of the glory. You know, all of your colleagues will have seen, “Oh, you got a paper in Nature. That was really …” Even if they haven’t read the details, they still will recognize that you got a paper in Nature.

Science has its own sort of cult of celebrity. So we end up in this system whereby you should have collaborations between the people with the ideas and the people who are really good at getting stuff working; but in practice, we don’t. And everyone keeps what they’re working on a secret from absolutely everybody else because they are deathly afraid of being scooped, which is what we call it when you wake up, check your feed of articles published in the day’s journals, and you see that someone else has just laid out…

FAKE: Scooped you.

ESVELT: Exactly what you were planning on publishing in the near future yourself.

FAKE: Battle for prestige.

ESVELT: That’s what it is, because… I mean most, at least, academic researchers, we’re obviously not in this for the money. We’re in this for the freedom to do what we think is fascinating. Most of them.

FAKE: You know what’s really interesting is that in the very early days of technology, there is this very strong desire to be what is called first to market, first to market. And kind of in a similar way, you know, kind of having your paper appear in Nature or other journals.

But what was subsequently discovered was that your technology did not necessarily succeed because it was first to market. And that often the second to market or the third to market or the person who actually perfected some other aspect of the technology actually had an advantage over you.

ESVELT: George Church always says that you don’t want to be the first into a space, you want to be the last.

FAKE: Yes.

ESVELT: Because if you’re the last, that means that you solved the problem so thoroughly that there was no point for anyone else to ever go there again.

FAKE: Giving up funding, glory, acclaim? Is that the way to ensure this science follows a brighter and more humanistic path? Joi Ito, Director of the MIT Media Lab, thinks the real issue lies somewhere else.

JOI ITO: Gene edit is an example of a category of technologies that will have both positive and negative impacts; and how thoughtful we are on trying to come up with measures for understanding the second order effects and also building in safeguards is really important. And I think the problem is that whether you’re in academia or in business, we’re not really rewarded for the precautionary stuff enough.

FAKE: Can we reward scientists for the precautionary stuff? I really like this idea of Joi’s. And asked Kevin: What would it take to make that a reality?

ESVELT: Changing the incentives for scientists to encourage caution and doing the right thing early requires different institutions than we have. The reason why I can do this is because I am at the MIT Media Lab, and my tenure case will depend at least as much on actual impacts on the world through projects, through things like calls for caution, through the development of safeguards – as it will on the number of papers I publish in Nature and Science. That is not true for virtually any of my colleagues. It’s just not.

I can be rewarded by going to a community and saying, “Hey, are you interested in this technology before we develop it?” That was worth something to me because I am here. That would not be worth anything to a young faculty member at a typical bioengineering department. And that is a problem.

At the Media Lab we’re encouraged to make a difference through any sphere of activity, doesn’t have to just be through publication in scientific journals.

FAKE: Is that – that’s unusual.

ESVELT: That’s profoundly unusual. The most important thing that we can do is use technologies like this as levers with which to change our institutions, to change the incentives and reward structures for the people who develop new technologies, who make new scientific discoveries enabling new technologies.

Because the future of our civilization will primarily be determined by the technologies that we choose to invent – and the wisdom with which we decide whether, when, and how to deploy them.

FAKE: I really think there’s a lot to learn from this whole conversation. And if you could distill advice for entrepreneurs and technologists to learn from the scientific community and some of the ideas that you have about information hazard and resident pessimists and other practices that you’ve built into your research and your kind of the way that you proceed in the world, we would love to hear it, and it may benefit all of us.

ESVELT: I’ve got to say: “Move fast and break things”? Not my favorite saying from the region of the world commonly viewed as being tasked with advancing increasingly powerful technologies.

FAKE: If you can’t quite place that quote, “Move fast and break things” was the early mantra of Mark Zuckerberg at Facebook.

ESVELT: I also don’t want to say that the scientific community has any special wisdom of over this. I don’t necessarily think that we do, with the exception of the community of physicists who take this, I think, a little bit more seriously because they have, you might say sinned. Physics sinned.

They conceived of an incredibly powerful technology and they made the mistake of telling the politicians about it. And they arguably had some good reasons for doing that. But on the other hand, we came all too close to massive nuclear war on a couple of occasions, and we should all be grateful to the individuals who stepped back from that.

If you wanted to pick one person who has saved more lives than any other person: Stanislav Petrov. We owe our lives to him.

FAKE: Stanislav Petrov, by the way, was a Soviet lieutenant who received a false report that the US had fired six missiles on the USSR. Had he followed military protocol, he would have fired retaliatory nuclear missiles at the United States, starting a nuclear war. His suspicion that the report was false, saved millions and millions of lives. Nuclear technology could have destroyed us. It didn’t. And so we press on.

ESVELT: So we need to be humble about what we can and cannot foresee, but we need to recognize that the future is in our hands, because the future belongs to technology.

FAKE: “The future belongs to technology.” We know this. And I love what Kevin shared next about the path ahead.

ESVELT: I view us as ascending the tree of knowledge. And we have no choice but to continue our ascent because the region of the tree that we’ve explored is not sufficient to support all of us. Civilization is not sustainable. We are using too many resources.

So we have no choice but to continue our ascent. But we are terrible at locking away an advance once we discover it. If it looks useful to anyone, we will make use of it. So the only way that we can prevent ourselves from going down a hazardous looking branch is by noticing that it looks hazardous.

And how confident are we that those branches will bear our weight? How confident are we that those fruits are not poisoned? It just requires one fruit that is sufficiently toxic to cause us to fall. And so that’s what I’m worried about.

We need to grow. We need to change morally. We need to become angels, if we’re going to wield the powers of the angels. The world is dramatically better than it has ever been in all of history. That is a gift that our ancestors have bequeathed unto us. And I see us as having a duty to continue that. But we can only do so by continuing to invent. And every new invention means we’re climbing a little bit higher in the tree of knowledge.

FAKE: This has been a wonderful conversation. You’ve raised so many potential issues and I think that you have a very clear sighted view of the future, the potential upsides, the potential downsides. Tremendous admirer.

ESVELT: It’s very kind of you to say so, but I spend most of my time worrying about I’m missing and what could go horribly wrong.

FAKE: Well, as a resident pessimist, it comes with the job.

FAKE: We started this episode with a quote from Stewart Brand: “We are as Gods, and might as well get good at it.” Stewart is a genius, but I think he got this one wrong.

We should acknowledge that we are not as Gods, and should be humble about it. We have, as sociobiologist E.O.Wilson has said, “paleolithic emotions, medieval institutions, and Godlike technology.”

“We are as gods” is an expression of our power, but as Kevin has pointed out, we can act as both angels, and as devils. We need to assume the mantle of the angels, grow our moral compass, and sometimes deliberately curtail our power when we see it go beyond our human limitations.  

This is what was so compelling about talking to Kevin about the gene drive. He takes on the moral responsibility for his discoveries and inventions – not only for his own work, but for the work of other scientists, and he gets regular people involved in the conversation.

Our decision on gene drive will shape our world for centuries to come, and we want to steer it towards its best outcomes for humanity. Should this exist?