Since there’s a lot of AI slop and doom chatter all over the news, podcasts, and social media these days, I figured I’d revisit Bill Joy in April 2000 for some context. I remember that Joy’s massive article “Why the Future Doesn’t Need Us” hit pretty hard twenty-six years ago not only because of the content but also because of who wrote it. Joy was a cofounder of Sun Microsystems, he helped build the internet, and here he was warning that three technologies might eventually lead to human extinction. The technologies he cited that could end us all include robotics, genetic engineering, and nanotechnology. And his core argument was actually pretty simple. These are not like past technologies. These new things can potentially replicate themselves, and that’s the bit that would change everything if they were used as weapons or simply were let lose by accident.
I first read Joy’s piece at a coffee shop in Cupertino, California across the street from Sun where I worked in software systems marketing. The article was widely read at Sun and also across Silicon Valley and resulted in discussions about Joy and his analysis for months. Many people just called him crazy. But that only demonstrates to me that those who made such flippant statements never read his article. Others knew better, though. They knew full well that Joy was documenting in detail the very real risks of rapidly developing technology without considering the consequences. That is, of course, the standard and pervasive culture of Silicon Valley. The valley may be big, but it is remarkably insular. I didn’t know Joy at the time I read his article, but I went on to meet him several times and worked closely with his teams promoting projects like SPARC, Solaris, Java, Jini, and later on JXTA. I didn’t really know him well, but he was always friendly and professional to me. He was quiet, too, and I always found him a serious thinker who obviously knew far more than he ever expressed. Sun was filled with such characters. They all fascinated me to no end.
The timing of the Joy article is interesting. He said he had been working on the essay since his 1998 conversation with Ray Kurzweil and continued to revise drafts through 1999. When Wired published the piece in April 2000, the tech world was at its peak. The NASDAQ hit a high of 5,048 in March of 2000 just a few weeks before the article dropped. And at that time Sun’s stock reached $250 a share, which gave the company a market cap around $200 billion. That was a significant achievement for 2000. Sun was one of the hottest companies in the valley back then, and it was quite a wild experience working there. The place was buzzing with activity. I loved it. Many of us did. So, it was into that environment of overt tech optimism running at manic levels that Joy published his thoughts about our potentially perilous future.
Andy Bechtolsheim, Vinod Khosla, Scott McNealy, Bill Joy at the Sun Reunion in Silicon Valley in October 2019. Photo by Jim Grisanzio.
Boom!
Then everything blew up. The bubble burst. By mid-April 2000 the NASDAQ suffered its worst week in history, falling more than 25 percent. Companies started dumping workers like I’ve never seen before. Joy’s dark warnings about unchecked technology landed precisely as that optimism crashed. He obviously couldn’t have seen the future, but his timing was remarkable.
Joy’s warnings took on an even darker tone the following year. After publishing the Wired article, he signed a book contract to expand on the article. He moved into a hotel room in New York City and surrounded himself with gloomy books on plagues and nuclear bombs and other such material he was studying on the future risks of technology. Then on September 11, 2001 came the terrorist attacks we have come to know as 9/11. I knew a few people from Sun who worked at the corporate building in New York City, but I did not know Joy was there at the time. He said he stood in the streets with everyone else and watched the impossible happen in real life. The next morning he walked out into the city streets past a long line of sanitation trucks parked on Houston Street ready to haul away the rubble. Everything below 14th Street was closed, he said. “It was quite a compelling experience,” he said in a TED Talk, “but not really, I suppose, a surprise to someone who had his room full of the books I was reading. I was not surprised that it happened at all.”
Joy eventually abandoned the book project. I point this out just as an aside since the event occurred shortly after he published the article I am writing about here in this post. Still, it does reflect the feeling of the times. How much had changed in Silicon Valley and the United States in just one year.
Anyway, back to the article.
Who Is Bill Joy?
Joy wasn’t a fringe thinker or outsider. He was born in 1954 in Farmington Hills, Michigan, and he was a child prodigy who started school early. One time his father took him to the local elementary school principal’s office when he was only three years old. Joy then promptly sat on the principal’s lap and read him a story. He later excelled in math and graduated high school at 16. He loved books and thinking and that became his escape. He also loved science fiction. He devoured Heinlein’s “Have Spacesuit Will Travel” and Asimov’s “I, Robot” with its Three Laws of Robotics. He wanted to be a ham radio operator, which were the Internet hackers of their day, but couldn’t afford the equipment. On TV Star Trek inspired his imagination every Thursday night when his parents went bowling. GeneRoddenberry’s “The Prime Directive” clearly resonated with Joy. You can see that ethic woven into his writing thereafter.
At Berkeley in the 1970s, he created the vi text editor, which, to his surprise, was still widely used more than twenty years later and some hard core developers still use it even now. He also developed the Berkeley version of the Unix operating system. When the other founders of Sun Microsystems (Andy Bechtolsheim, Vinod Khosla, and Scott McNealy) invited him to join them, he participated in the creation of advanced microprocessor technologies and Internet technologies such as Java and Jini. As codesigner of three microprocessor architectures — SPARC, picoJava, and MAJC — he helped drive innovations that shaped modern computing.
By the time he wrote his famous Wired essay, Joy was only 45 years old and at the peak of his influence among developers in Silicon Valley. But Joy was far more then just a coder. He was well connected to the broader scientific community. That is what made the article so jarring. He was not an uninformed critic glancing in from outside with yet another opinion. He was a core architect of the digital age expressing deep doubts about where his own work was leading. His self-reflection was pervasive during this time in his writings and conference presentations.
The Kurzweil Meeting
Joy’s concern seemed to begin at George Gilder’s Telecosm conference in 1998 when he met Ray Kurzweil, who was an inventor and futurist. Kurzweil talked about how the rate of technological improvement was accelerating and also how humans may merge with robots or download their consciousnesses to achieve near immortality. I remember attending several talks on this topic of immortality when I moved to Silicon Valley. It always sounded so silly to me. I wondered how such smart people could take that stuff so seriously. But even now some people in these circles talk about downloading themselves. It still sounds silly. The other bits, though, about intelligent robots and genetic engineering were much more reasonable given my own experience working in the biotech industry. Where to draw the line, however, sometimes really is not clear. Joy had heard such talk before and always felt sentient robots were science fiction. But hearing it from someone he respected changed things. Kurzweil gave him a preprint of “The Age of Spiritual Machines,” which outlined a utopian future where humans gained near immortality by becoming one with robotic technology. I don’t know how far Joy goes with respect to robotic sentience, but it’s clearly more than I’m willing to accept.
Nevertheless, Joy’s unease intensified after reading the book. He felt sure Kurzweil was understating the dangers. Then he found a passage in the book describing a dystopian future where machines become so capable that humans depend on them completely. The passage argued that we would not consciously hand over control to the bots. Instead, “the human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines’ decisions.” In other words, we would become dependent gradually. I can surely see that as a potential reality, no question about it.
But that passage came from Ted Kaczynski, the Unabomber! Joy admits this realization was uncomfortable to say the very least since he was taking a point from a terrorist seriously. Many people have said the same thing after reading Kaczynski’s words. Kaczynski’s bombs had killed three people and wounded many others. One bomb gravely injured David Gelernter, one of Joy’s colleagues and friends. But Joy felt compelled to confront the argument because, however uncomfortable, he saw merit in that single passage about unintended consequences.
The Self-Replication Problem
Joy’s central concern circles around one key difference between powerful 21st-century technologies and those of the 20th-century. Nuclear weapons required huge facilities and rare materials. But genetic engineering, nanotechnology, and robotics, what Joy called GNR technologies, require less infrastructure and can potentially make copies of themselves. A bomb explodes once, but a self-replicating machine does not stop. And that could be a serious problem if something goes wrong.
This matters because knowledge spreads freely. You cannot control ideas at all like you may be able to control uranium. Once people know how to genetically engineer bacteria or design tiny self-replicating machines, that knowledge exists in the world and will move rapidly. A small group, even one person, could potentially cause massive harm. Joy calls this “knowledge-enabled mass destruction.” As he wrote, “I think it is no exaggeration to say we are on the cusp of the further perfection of extreme evil, an evil whose possibility spreads well beyond that which weapons of mass destruction bequeathed to the nation-states, on to a surprising and terrible empowerment of extreme individuals.”
He made the point even sharper later. The real danger, he said, is no longer nation-states but individuals or small groups now empowered with “pandemic power.” These new digital, self-replicating technologies give extreme individuals the kind of destructive capability once reserved only for governments. That shift changes everything because the ramifications of mistakes or ill intent can’t be calculated or controlled.
Joy learned about complex systems and non-linear systems from physicists Stephen Wolfram and Brosl Hasslacher in the early 1980s. These are systems where small changes move in unpredictable ways and where feedback loops create unexpected outcomes. Later, conversations with Danny Hillis, biologist Stuart Kauffman, and Nobel laureate Murray Gell-Mann deepened his understanding. Hasslacher and Mark Reed also gave him insight into molecular electronics, which is the manipulation of matter at the atomic and molecular level where individual atoms replace transistors. When you get to this point in Joy’s article you can’t help but realize that he’s well beyond just a smart software developer who happened to strike it rich by helping found a successful tech company.
Joy knew that the merger of computers and physical sciences was creating enormous power, which up to that point others hadn’t expressed such thoughts in the popular tech media. By 2030, Joy calculated, we would likely build machines a million times as powerful as personal computers of 2000. That is enough computing power to make the scenarios that worried him technically possible. As he wrote, “But now, with the prospect of human-level computing power in about 30 years, a new idea suggests itself: that I may be working to create tools which will enable the construction of the technology that may replace our species. How do I feel about this? Very uncomfortable.” He admitted he had once been too optimistic about nanotechnology. Having struggled his entire career to build reliable software systems, it seemed to him more than likely that this future would not work out as well as some people might imagine.
Three Scenarios
Joy walks through what could actually happen with these technologies. What’s interesting is that evolution itself would be one of the driving forces moving these technologies from a positive outcome to something very negative.
One — Robots might simply out-compete us for resources the way better-adapted species have always displaced others. We would not need a robot uprising. Hans Moravec argued that “in a completely free marketplace, superior robots would surely affect humans as North American placentals affected South American marsupials.” Economic forces alone could push us aside. Also, the dream of robotics includes downloading our consciousnesses into machines. But Joy questions whether a downloaded consciousness would be human in any meaningful sense. The robots would not be our children, and on that path our humanity might be lost entirely. This is one of the reasons why I take Joy seriously. He sees the obvious problem with the downloading issue in a way that others in this field simply do not.
Two — Genetic engineering gives us power to create devastating plagues, either by accident or intention. Joy calls this the “White Plague” scenario, which references a Frank Herbert novel where a molecular biologist weaponizes his knowledge. We now know these profound changes in biological sciences are imminent and will challenge all our notions of what life is, and Joy points out that the public remains skeptical even by the standards of 2000.
Three — Nanotechnology could produce “gray goo,” which are self-replicating nanobots that consume the biosphere. Eric Drexler warned that “tough omnivorous ‘bacteria’ could out-compete real bacteria. They could spread like blowing pollen, replicate swiftly, and reduce the biosphere to dust in a matter of days.” Joy notes grimly, “Gray goo would surely be a depressing ending to our human adventure on Earth, far worse than mere fire or ice, and one that could stem from a simple laboratory accident. Oops.”
The Manhattan Project Parallel
Joy uses the atomic bomb as his template. After Hiroshima and Nagasaki, physicists were shocked by what they created. Oppenheimer later said the physicists “have known sin.” There was a real opportunity through the Acheson-Lilienthal report and Baruch Plan to prevent a nuclear arms race by internationalizing nuclear power. But it failed because political distrust and competitive pressure got in the way. Within years, the Soviets had the bomb, and the arms race was on.
Freeman Dyson captured the moment: “The glitter of nuclear weapons. It is irresistible if you come to them as a scientist. To feel it is there in your hands, to release this energy that fuels the stars, to let it do your bidding. To perform these miracles, to lift a million tons of rock into the sky. It is something that gives people an illusion of illimitable power, and it is, in some ways, responsible for all our troubles, this, what you might call technical arrogance, that overcomes people when they see what they can do with their minds.”
Up to that point, everyone feared nuclear bombs as the ultimate expression of madness. But Joy fears we are repeating this pattern with even more dangerous technologies, and the commercial incentives for their production are enormous. Nations compete. Corporations compete. Researchers want breakthrough innovations. The momentum builds, and pretty soon it is almost impossible to stop. We are being propelled into the new century with no plan, no control, no brakes. However, the driver is not military necessity this time but instead private sector economic gain and competitive pressure. This is human nature, though. Our history is literally filled with these processes. It’s who we are.
The Relinquishment Argument
This is the part that bothered many people. Joy argues for “relinquishment,” or the voluntary decision not to pursue certain lines of knowledge or technology because they are too dangerous. This goes against everything we believe about the value of knowledge and open inquiry, especially within the walls of the Silicon Valley and across the scientific community generally.
That may be why Joy’s article struck such a nerve. The general population is used to various power centers attempting to curtail their freedoms for whatever reason of the day. But regular people are largely powerless to do much about it beyond voting, which is time consuming, or protesting, which brings it’s own personal risks. The scientific and technological elite, however, is something different. They are one of the power centers themselves, and here was one of their own, a high profile one at that, advocating for relinquishment.
But Joy asks, what if unlimited pursuit of knowledge puts us all in mortal danger? He points out that we have done this before. At a 1989 nanotechnology conference, Joy said, “We cannot simply do our science and not worry about these ethical issues.” The United States unilaterally abandoned biological weapons development because the logic was clear. These weapons were easy to replicate and could easily end up in the wrong hands. We would be more secure if nobody developed them, and we embodied this in the 1972 Biological Weapons Convention and 1993 Chemical Weapons Convention.
Joy quotes Thoreau: “We do not ride on the railroad. It rides upon us.” Then he asks directly, “The question is, indeed, Which is to be master? Will we survive our technologies?”
This does not mean stopping all research. It means being strategic about which lines of inquiry we pursue and which we intentionally avoid. It means international agreements, verification systems, and scientists adopting ethical codes like the Hippocratic oath. It requires transparency and cooperation.
Personal Responsibility
Joy writes honestly about his own sense of responsibility: “I feel, too, a deepened sense of personal responsibility, not for the work I have already done, but for the work that I might yet do, at the confluence of the sciences.” He was not speaking as an outside observer but as someone who helped create the technologies that might enable the dangers he feared.
He finds hope in the Dalai Lama’s “Ethics for the New Millennium,” which argues that the most important thing is to conduct our lives with love and compassion for others, and that our societies need a stronger notion of universal responsibility. Neither material progress nor the pursuit of knowledge is the key to happiness. We need to find alternative outlets for our creative forces beyond the culture of perpetual economic growth.
In the TED Talk he gave six years after publishing his article, Joy made the point even clearer. The solution, he said, cannot be technology alone. We need both better public policy and deeper moral progress. He spoke of the need for “the head and the heart,” echoing Russell and Einstein. He argued that scientists, technologists, and businessmen must be held personally accountable under the law for the consequences of their inventions. Today they face no such responsibility. That, he believed, has to change. What’s striking hearing him articulate these perfectly reasonable points is that they aren’t taken seriously at all. many thoughtful people say the same things at conferences or in political speeches. We all clap and support the concepts. Yet, very little actually changes. Or at least the changes take so long and occur in such small steps that we’re left unsatisfied in the present moment.
Was Joy Right?
Joy’s article was unique for its time in the popular press. When it appeared on Wired’s cover in April 2000, it created quite the rumble in tech circles. Wired had been a cheerleader for the digital age for nearly a decade. Its shift from cheering to warning marked an important and surprising moment in the digital zeitgeist. Also, Bill Joy was not some alarmist outsider. He was one of the architects. His warning came from inside the cathedral and it certainly resonated. At the time, I was working in marketing at Sun, and one of my jobs was to promote Sun’s technologies to the media. In press interviews during this period reporters would always bring up Joy’s article even if the meeting was booked on another issue. We had to draft briefing and messaging documents for prepare executives, managers, and engineers that we brought into all interviews because we knew Joy’s article would alway come up in the discussion.
The article presaged much of what we are experiencing now but not necessarily in the ways Joy anticipated. His specific predictions about nanotechnology have not materialized. The gray goo scenario is now considered flawed and implausible. Most scientists believe built-in limitations make runaway nanotechnology highly improbable.
But Joy’s underlying concerns were prescient. He worried about knowledge-enabled destruction, powerful technologies becoming widely available, and complex systems we do not fully understand. All of these have become more relevant as artificial intelligence has advanced today far faster than most people expected in 2000. Interestingly, Joy only explicitly mentioned artificial intelligence once in his article, possibly because he was writing at the tail end of the second “AI winter.” Yet his concerns about self-replicating technologies and systems beyond human control have found new relevance with modern AI these days.
The Sun Sets
After leaving Sun in 2003, Joy moved into venture capital and focused on green energy investments at Kleiner Perkins until 2014. Now 71, he works as principal investigator and chief scientist at Water Street Capital. But despite the renewed relevance of his warnings with the rise of AI, Joy has remained largely silent regarding the public debate he started 26 years ago. It is almost as if he said what he wanted to say in 2000 and we’re still digesting his thoughts all these years later.
He didn’t, however, stay stuck in doom or alarm. In the years after the Wired article he actively tried to move toward better outcomes. He joined Kleiner Perkins specifically to invest in solutions. He backed innovations in education, new materials for the environment, and a major $200 million biodefense fund aimed at closing the gaps that could lead to a pandemic. He came to believe we cannot solve the management of dangerous technology with more technology alone. Instead we need better policy, markets that price in the true cost of catastrophe, and a deeper moral shift.
Joy put it simply in that later talk at TED. “We can’t pick the future, but we can steer the future.” Over the years technologies have changed, but the fundamental challenge he identified in 2000 remains relevant. Figuring out how to pursue knowledge and innovation while maintaining enough wisdom and caution to survive the unintended consequences seems to be a question others should carefully consider today.
In his article, Joy compared coding to Michelangelo releasing statues from marble. He described his software engineering in a similar way with those ecstatic moments when the code emerged from his imagination as if it were already waiting in the machine to be freed. He ended his essay with that same image. After eighteen pages exploring multiple scientific disciplines and warning about existential dangers from the exploitation of technology, he wrote, “I am up late again, it is almost 6 am. I am trying to imagine some better answers, to break the spell and free them from the stone.”
Twenty-six years later, we are still up late too. We’re still searching for those answers, as well. We’re still trying to release better possibilities from the block of marble that is our technological future. And now we go forward in the wild world of AI.
Good luck to us all.
