How Innovation Works

HOW INNOVATION WORKS

Matt Ridley

Copyright

4th Estate

An imprint of HarperCollinsPublishers

1 London Bridge Street

London SE1 9GF

www.4thEstate.co.uk

This eBook first published in Great Britain by 4th Estate in 2020

Copyright © Matt Ridley 2020

Cover image © Shutterstock

Matt Ridley asserts the moral right to be identified as the author of this work

A catalogue record for this book is available from the British Library

All rights reserved under International and Pan-American Copyright Conventions. By payment of the required fees, you have been granted the non-exclusive, non-transferable right to access and read the text of this e-book on-screen. No part of this text may be reproduced, transmitted, down-loaded, decompiled, reverse engineered, or stored in or introduced into any information storage and retrieval system, in any form or by any means, whether electronic or mechanical, now known or hereinafter invented, without the express written permission of HarperCollins

Source ISBN: 9780008334819

Ebook Edition © June 2020 ISBN: 9780008334826

Version: 2020-06-16

Dedication

For Felicity Bryan

CONTENTS

Cover

Title Page

Copyright

Dedication

Introduction: The Infinite Improbability Drive

1. Energy

Of heat, work and light

What Watt wrought

Thomas Edison and the invention business

The ubiquitous turbine

Nuclear power and the phenomenon of disinnovation

Shale gas surprise

The reign of fire

2. Public health

Lady Mary’s dangerous obsession

Pasteur’s chickens

The chlorine gamble that paid off

How Pearl and Grace never put a foot wrong

Fleming’s luck

The pursuit of polio

Mud huts and malaria

Tobacco and harm reduction

3. Transport

The locomotive and its line

Turning the screw

Internal combustion’s comeback

The tragedy and triumph of diesel

The Wright stuff

International rivalry and the jet engine

Innovation in safety and cost

4. Food

The tasty tuber

How fertilizer fed the world

Dwarfing genes from Japan

Insect nemesis

Gene editing gets crisper

Land sparing versus land sharing

5. Low-technology innovation

When numbers were new

The water trap

Crinkly tin conquers the Empire

The container that changed trade

Was wheeled baggage late?

Novelty at the table

The rise of the sharing economy

6. Communication and computing

The first death of distance

The miracle of wireless

Who invented the computer?

The ever-shrinking transistor

The surprise of search engines and social media

Machines that learn

7. Prehistoric innovation

The first farmers

The invention of the dog

The (Stone Age) great leap forward

The feast made possible by fire

The ultimate innovation: life itself

8. Innovation’s essentials

Innovation is gradual

Innovation is different from invention

Innovation is often serendipitous

Innovation is recombinant

Innovation involves trial and error

Innovation is a team sport

Innovation is inexorable

Innovation’s hype cycle

Innovation prefers fragmented governance

Innovation increasingly means using fewer resources rather than more

9. The economics of innovation

The puzzle of increasing returns

Innovation is a bottom-up phenomenon

Innovation is the mother of science as often as it is the daughter

Innovation cannot be forced upon unwilling consumers

Innovation increases interdependence

Innovation does not create unemployment

Big companies are bad at innovation

Setting innovation free

10. Fakes, frauds, fads and failures

Fake bomb detectors

Phantom games consoles

The Theranos debacle

Failure through diminishing returns to innovation: mobile phones

A future failure: Hyperloop

Failure as a necessary ingredient of success: Amazon and Google

11. Resistance to innovation

When novelty is subversive: the case of coffee

When innovation is demonized and delayed: the case of biotechnology

When scares ignore science: the case of weedkiller

When government prevents innovation: the case of mobile telephony

When the law stifles innovation: the case of intellectual property

When big firms stifle innovation: the case of bagless vacuum cleaners

When investors divert innovation: the case of permissionless bits

12. An innovation famine

How innovation works

A bright future

Not all innovation is speeding up

The innovation famine

China’s innovation engine

Regaining momentum

Afterword

Sources and further reading

Index

Acknowledgements

About the Author

By the same author

About the Publisher

INTRODUCTION

The Infinite Improbability Drive

Innovation offers the carrot of spectacular reward or the stick of destitution.

JOSEPH SCHUMPETER

I am walking along a path on the Inner Farne, an island off the coast of north-east England. By the side of the path, amid the sea-campion flowers, sits a female eider duck, dark brown and broody, silently incubating her clutch of eggs. I stoop to take a picture of her with my iPhone from a few feet away. She is used to this: hundreds of visitors come here every day in summer and many will take her picture. For some reason, an idea pops into my head as I click: a riff on the second law of thermodynamics based on a remark by my friend John Constable. The idea is this: the electricity in the iPhone’s battery and the warmth in the eider duck’s body are doing roughly the same thing: making improbable order (photographs, ducklings) by expending or converting energy. And then I think that the idea I’ve just had itself, like the eider duck and the iPhone, is also an improbable arrangement of synaptic activity in my brain, also fuelled by energy from the food I have recently eaten, of course, but made possible by the underlying order of the brain, itself the evolved product of millennia of natural selection acting on individuals, each of whose own improbabilities were sustained by energy conversion. Improbable arrangements of the world, crystallized consequences of energy generation, are what both life and technology are all about.

In Douglas Adams’s The Hitchhiker’s Guide to the Galaxy, Zaphod Beeblebrox’s starship Heart of Gold – a metaphor for wealth – is powered by a fictional ‘infinite improbability drive’. Yet a near-infinite improbability drive does indeed exist, but here on Planet Earth, in the shape of the process of innovation. Innovations come in many forms, but one thing they all have in common, and which they share with biological innovations created by evolution, is that they are enhanced forms of improbability. That is to say, innovations, be they iPhones, ideas or eider ducklings, are all unlikely, improbable combinations of atoms and digital bits of information. It is astronomically improbable that the atoms in an iPhone would be neatly arranged by chance into millions of transistors and liquid crystals, or the atoms in an eider duckling would be arranged to form blood vessels and downy feathers, or the firings of neurons in my brain would be arranged in such a pattern that they can and sometimes do represent the concept of ‘the second law of thermodynamics’. Innovation, like evolution, is a process of constantly discovering ways of rearranging the world into forms that are unlikely to arise by chance – and that happen to be useful. The resulting entities are the opposite of entropy: they are more ordered, less random, than their ingredients were before. And innovation is potentially infinite because even if it runs out of new things to do, it can always find ways to do the same things more quickly or for less energy.

In this universe it is compulsory, under the second law of thermodynamics, that entropy cannot be reversed, locally, unless there is a source of energy – which is necessarily supplied by making something else even less ordered somewhere else, so the entropy of the whole system increases. The power of the improbability drive is therefore limited only by the supply of energy. So long as human beings apply energy to the world in careful ways, they can create ever more ingenious and improbable structures. The medieval castle at Dunstanburgh I can see from the island is an improbable structure, and its partial ruin after 700 years is more probable, more entropic. The castle in its prime was the direct consequence of the expenditure of lots of energy, in this case mainly in the muscles of masons who were fed with bread and cheese that was made from wheat and grass that was grown in sunlight and eaten by cows. John Constable, a former Cambridge and Kyoto academic, points out that the things we rely on to make our lives prosperous are

all of them, without exception, physical states far from thermodynamic equilibrium, and the world was brought, sometimes over long periods of time, into these convenient configurations by energy conversion, the use of which reduced entropy in one corner of the universe, ours, and increased it by an even larger margin somewhere else. The more ordered and improbable our world becomes, the richer we become, and, as a consequence, the more disordered the universe becomes overall.

Innovation, then, means finding new ways to apply energy to create improbable things, and see them catch on. It means much more than invention, because the word implies developing an invention to the point where it catches on because it is sufficiently practical, affordable, reliable and ubiquitous to be worth using. The Nobel Prize-winning economist Edmund Phelps defines an innovation as ‘a new method or new product that becomes a new practice somewhere in the world’. In the pages that follow I will trace the path of ideas from the invention to the innovation, through the long struggle to get an idea to catch on, usually by combining it with other ideas.

And here is my starting point: innovation is the most important fact about the modern world, but one of the least well understood. It is the reason most people today live lives of prosperity and wisdom compared with their ancestors, the overwhelming cause of the great enrichment of the past few centuries, the simple explanation of why the incidence of extreme poverty is in global freefall for the first time in history: from 50 per cent of the world population to 9 per cent in my lifetime.

What made most of us, not just in the West but in China and Brazil too, unprecedentedly rich, so the economic historian Deirdre McCloskey says, was ‘innovationism’: the habit of applying new ideas to raising living standards. No other explanation of the great enrichment of recent centuries makes any sense. Trade had been expanding for centuries, and colonial exploitation with it, and these alone were unable to give anything like the order of magnitude of improvement in incomes that happened. There was no sufficient accumulation of capital to make such a difference, no ‘piling of brick on brick, or bachelor’s degree on bachelor’s degree’ in McCloskey’s words. There was no sufficiently great expansion in the availability of labour. Nor was the scientific revolution of Galileo and Newton responsible, for most of the innovations that changed people’s lives at least at first owed little to new scientific knowledge and few of the innovators who drove the changes were trained scientists. Indeed many, such as Thomas Newcomen, the inventor of the steam engine, or Richard Arkwright of the textile revolution, or George Stephenson of the railways, were poorly educated men of humble origins. Much innovation preceded the science that underpinned it. The Industrial Revolution therefore was in effect, as Phelps has argued, the emergence of a new kind of economic system that generated endogenous innovation as a product in itself. I will argue that some machines themselves made this possible. A steam engine proved to be ‘autocatalytic’: it drained the mines, which cut the cost of coal, which made the next machine cheaper and easier to make. But I am getting ahead of myself.

The word ‘innovation’ is invoked with alarming frequency by companies trying to sound up to date but with little or no systematic idea about how it occurs. The surprising truth is that nobody really knows why innovation happens and how it happens, let alone when and where it will happen next. One economic historian, Angus Maddison, wrote that ‘technical progress is the most essential characteristic of modern growth and one that is most difficult to quantify or explain’; another, Joel Mokyr, said that scholars ‘know remarkably little about the kind of institutions that foster and stimulate technological progress’.

Take sliced bread, for example. Best thing since, and all that. Looking back it is obvious that somebody would invent a way of automatically pre-slicing bread to make uniform sandwiches. It is fairly obvious that this would probably happen in the first half of the twentieth century when electrical machines were all the rage for the first time. But why 1928? And why in the small town of Chillicothe, in the middle of Missouri? Lots of people tried to make bread-slicing machines, but they either worked poorly or they led to stale bread because it was not well packaged. The person who made it work was Otto Frederick Rohwedder, who was born in Iowa, was educated as an optician in Chicago and set up shop as a jeweller in St Joseph, Missouri, before moving back to Iowa determined – for some reason – to invent a bread slicer. He lost his first prototype in a fire in 1917 and had to start all over again. Crucially he realized that he must invent automatic packaging of the bread at the same time lest the slices go stale. Most bakeries were not interested, but the Chillocothe bakery, owned by one Frank Bench, was and the rest is history. What was special about Missouri? Beyond a general mid-twentieth-century American affection for innovation and the means to make it happen, the best guess is that it was a slice of random luck. Serendipity plays a big part in innovation, which is why liberal economies, with their free-roving experimental opportunities, do so well. They give luck a chance.

Innovation happens when people are free to think, experiment and speculate. It happens when people can trade with each other. It happens where people are relatively prosperous, not desperate. It is somewhat contagious. It needs investment. It generally happens in cities. And so on. But do we really understand it? What is the best way to encourage innovation? To set targets, direct research, subsidize science, write rules and standards; or to back off from all this, deregulate, set people free; or to create property rights in ideas, offer patents and hand out prizes, issue medals; to fear the future; or to be full of hope? You will find champions of all these policies and more, fervently arguing their cases. But the striking thing about innovation is how mysterious it still is. No economist or social scientist can fully explain why innovation happens, let alone why it happens when and where it does.

In this book I shall try to tackle this great puzzle. I will do so not by abstract theorizing or argument alone, though there will be some of both, but mainly by telling stories. Let the innovators who turned their (or other people’s) inventions into useful innovations teach us, by the examples of their successes and failures, how it happened. I tell the stories of steam engines and search engines, of vaccines and vaping, of shipping containers and silicon chips, of wheeled suitcases and gene editing, of numbers and water closets. Let’s hear from Thomas Edison and Guglielmo Marconi, from Thomas Newcomen and Gordon Moore, from Lady Mary Wortley Montagu and Pearl Kendrick, from Al Khwarizmi and Grace Hopper, from James Dyson and Jeff Bezos.

I cannot hope to document every important innovation. I have omitted some very important and well-known ones for no particular reason: the automation of the textile industry, for example, or the history of the limited company. I have left out most innovation in art, music and literature. My main examples are drawn from the worlds of energy, public health, transport, food, low technology, and computers and communications.

Not all the people whose stories I tell are heroes; some are frauds, fakers or failures. Few worked alone, for innovation is a team sport, a collective enterprise, far more than is generally recognized. Credit and authorship are confused and mysterious if not downright unfair. Yet unlike most team sports innovation is not usually a choreographed, planned or managed thing. It cannot be easily predicted, as many a red-faced forecaster has discovered. It runs mostly on trial and error, the human version of natural selection. And it usually stumbles on great breakthroughs when looking for something else: it is heavily serendipitous.

I will plunge back in time to the very start of human culture to try to understand what triggered innovation in the first place and why it happens to people but not to robins or rocks. Chimpanzees and crows do innovate, by developing and spreading new cultural habits, but very occasionally and rather slowly; most other animals not at all.

In the ten years since I published The Rational Optimist, arguing unfashionably that the world has been, is, and will go on getting better, not worse, human living standards have grown rapidly higher for nearly everybody. I finished that book as the world was plumbing the depths of a terrible recession, but the years since have been ones of faster economic growth for much of the poor of the world than ever before. The income of the average Ethiopian has doubled in a decade; the number of people living in extreme poverty has dipped below 10 per cent for the first time in history; malaria mortality has plummeted; war has ceased altogether in the western hemisphere and become much rarer in the Old World, too; frugal LED lights have replaced both incandescent and fluorescent bulbs; telephone conversations have essentially become free on Wi-Fi. Some things have got worse, of course, but most trends are positive. All this is due to innovation.

The chief way in which innovation changes our lives is by enabling people to work for each other. As I have argued before, the main theme of human history is that we become steadily more specialized in what we produce, and steadily more diversified in what we consume: we move away from precarious self-sufficiency to safer mutual interdependence. By concentrating on serving other people’s needs for forty hours a week – which we call a job – you can spend the other seventy-two hours (not counting fifty-six hours in bed) drawing upon the services provided to you by other people. Innovation has made it possible to work for a fraction of a second so as to be able to afford to turn on an electric lamp for an hour, providing the quantity of light that would have required a whole day’s work if you had to make it yourself by collecting and refining sesame oil or lamb fat to burn in a simple lamp, as much of humanity did in the not so distant past.

Most innovation is a gradual process. The modern obsession with disruptive innovation, a phrase coined by the Harvard professor Clayton Christensen in 1995, is misleading. Even when a new technology does upend an old one, as digital media has done to newspapers, the effect begins very slowly, gathers pace gradually and works by increments, not leaps and bounds. Innovation often disappoints in its early years, only to exceed expectations once it gets going, a phenomenon I call the Amara hype cycle, after Roy Amara, who first said that we underestimate the impact of innovation in the long run but overestimate it in the short run.

Perhaps the most puzzling aspect of innovation is how unpopular it is, for all the lip service we pay to it. Despite the abundant evidence that it has transformed almost everybody’s lives for the better in innumerable ways, the kneejerk reaction of most people to something new is often worry, sometimes even disgust. Unless it is of obvious use to ourselves, we tend to imagine the bad consequences that might occur far more than the good ones. And we throw obstacles in the way of innovators, on behalf of those with a vested interest in the status quo: investors, managers and employees alike. History shows that innovation is a delicate and vulnerable flower, easily crushed underfoot, but quick to regrow if conditions allow.

This strange phenomenon of innovation, and the resistance to it, was eloquently celebrated more than three centuries ago, before the start of the great enrichment, by an innovator – though he would not have used that word. William Petty went from being a teenage cabin boy on a ship who was marooned on a foreign shore with a broken leg, to getting a Jesuit education and becoming secretary to the philosopher Thomas Hobbes. Then, following a spell in Holland, he began a career as a physician and scientist, before emerging as a merchant, an Irish land speculator, a Member of Parliament, then a wealthy and politically influential pioneer of the study of economics. He was a better innovator than inventor. Early in his career, while a professor of anatomy in Oxford in 1647, Petty invented and patented a double-writing instrument – by which he could produce two copies of the first chapter of Hebrews in one go, in fifteen minutes – as well as a scheme for making a bridge with no supports on the river bed, and an engine for planting corn. None of them seemed to catch on. With feeling, Petty later wrote this lament about the lot of the inventor, in 1662:

Few new inventions were ever rewarded by a monopoly; for although the inventor, oftentimes drunk with the opinion of his own merit, thinks all the world will encroach and invade upon him, yet I have observed that the generality of men will scarce be hired to make use of the new substances which themselves have not thoroughly tried, and which length of time hath not vindicated from latent inconvenience, so as when a new invention is first propounded in the beginning every man objects and the poor inventor runs the gauntloop of all petulant wits, every man finding his several flaw, no man approving it unless mended according to his own device. Now, not one of a hundred outlives this torture, and those that do are at length so changed by the various contrivances of others, that not any one man can pretend to the invention of the whole, nor well agree about their respective share in the parts. And moreover this commonly is so long adoing, that the poor inventor is either dead, or disabled by the debts contracted to pursue his design; and withal railed upon as a projector or worse, by those who joined their money in partnership with his wit; so as the said inventor and his pretences are wholly lost as vanished.