Scientific Methods/Chapter 8

by Richard D. Jarrard

Chapter 8: Insight

A non-free image has been removed from this page.

“To see a World in a Grain of Sand
And a heaven in a Wild Flower
Hold Infinity in the palm of your hand
And Eternity in an hour.”
[Blake, ~1803]

Rain fell in the mountains, and a brook was born. The young brook splashed and slithered over rocks and under branches, meeting other brooks and merging with them, growing and slowing. No longer a brook, a powerful river emerged from the mountains in a final waterfall, which encountered the desert.

“You cannot pass,” said the desert. But nothing had ever stopped the river, so it flowed forward, now out across the desert. The desert passively soaked up the river’s water, stopping the river’s advance. “You cannot pass,” said the desert.

The river had learned persistence. It continued to flow out into the desert, expecting eventually to win passage. But the desert was no stranger to persistence.

The river accumulated floating debris at the mouth of the waterfall, forming a temporary dam, building up a huge backlog of water, then bursting upon the desert. The desert seemed to be overwhelmed by the torrent, but only temporarily.

The river tried to avoid the desert, skirting it by flowing along the base of the mountain’s foothills. The desert found and drank the river water.

The river felt defeated. Persistence, power and avoidance had always succeeded before but had failed to overcome this obstacle. Everything the river had ever been was for naught - its youthful brooks, later streams, and final river strength - in the face of this obstacle. Everything? Or was there a time before the brooks?

The river gave itself to the wind. And the water that fell over that final waterfall never touched the desert floor. The water was swept up and evaporated, carried by the wind out of reach of the desert, across the desert, to mountains beyond.

Rain fell in the mountains, and a brook was ‘born’.

[loosely based on a Sufi teaching story, e.g., Shah, 1970] ***

Role of Insight in Science

Hypothesis and observation, or theory and empiricism, are only two of the three essential ingredients of modern scientific method. The third pillar of wisdom is insight -- the sudden transcendence of obstacles by a new perspective, like the river transforming to vapor and crossing the desert. Insight changes the counterpoint of hypothesis and data into the upward spiral of hypothesis, data, insight, new hypothesis, different data, . . .

Other terms are used synonymously with ‘insight’: illumination, intuition, serendipity, scientific hunch, revelation, inspiration, enlightenment, sudden comprehension, guess, and discovery. Most of these terms have such a heavy connotation of either religious, psychological, or everyday secular meaning, however, that they are somewhat distracting to use in the current scientific discussion.

Insight brings joy to science. Without this thrill, many of us would not be scientists.

In their excellent and still timely article on the role of ‘scientific hunch’ in research, Platt and Baker [1931] define a scientific hunch as “a unifying or clarifying idea which springs into consciousness suddenly as a solution to a problem in which we are intensely interested. . . A hunch springs from a wide knowledge of facts but is essentially a leap of the imagination, in that it goes beyond a mere necessary conclusion that any reasonable man must draw from the data at hand. It is a process of creative thought.” This is not deduction, but induction -- and sometimes induction totally unwarranted from the available evidence. Sometimes it is a solution to a minor technical problem, and sometimes an insight so fundamental that we can never again see the world in the old way.

Helmholtz [1903], Wallas [1926], Platt and Baker [1931], Sindermann [1987] and others concisely describe scientific method as consisting of four stages: preparation, incubation, illumination, and verification. I agree that these four stages are real, and we will come back to them soon in considering how insight can be encouraged or hampered. These four terms betray a strong bias, however, toward casting illumination as the central and most important aspect of science, with the other stages serving only a supporting role. Such a view is by no means universal. Indeed, when 232 scientists replied to a questionnaire concerning insight in science, 17% said that scientific revelations or hunches never help them find a solution to their problems [Platt and Baker, 1931]. I suspect that they rely on insight as much as I do, but they dislike the connotations of the words ‘revelation’ and ‘hunch’, and they prefer to think of science as more rational than those terms imply. Possibly also, they shy from insight's ‘nonscientific’ characteristics: it is nonreproducible, nonquantifiable, unpredictable, unreliable, and sometimes almost mystical.

Scientists’ reliance on insight is incredibly diverse, partly because of variations in ability but also largely because of value judgments concerning rational data-gathering versus irrational insight. Some get ideas and experiment to test their ideas, some prefer to test others’ hypotheses, and some try to gather data until an answer emerges as virtually proved. A few types of research claim to thrive on minimal insight. For example, C.F. Chandler said that one could solve any problem in chemical research by following two simple rules: “To vary one thing at a time, and to make a note of all you do” [Platt and Baker, 1931]. To many scientists, such an approach is either infeasible or boring.

The four stages of research occupy unequal proportions of our research time. We might wish that insight were 25% of the job volumetrically as well as conceptually, but Thomas Edison’s generalization is probably more accurate: “science is 99% perspiration and 1% inspiration.” At least that is what a former advisor told me when, as a new graduate student, I showed little enthusiasm for

spending countless hours doing routine measurements for his project.

“Before illumination, carry water, chop wood.
After illumination, carry water, chop wood.” [Zen saying]

The content of the preparation and verification stages is primarily routine, straightforward, and mechanical, requiring different skills than are needed for the insight stage. Courses and books (including this one) usually devote much more attention to these skills than to techniques for enhancing insight.

The advertising industry often uses a technique known as anxiety/relief: first create an anxiety, and then offer your product as a potential relief. Today part of the success of this technique is attributable to its strongly conditioned pattern. Why do people like myself get addicted to high anxiety jobs? Perhaps it is because the solutions, when found, are that much sweeter. Problem solving may fulfill a similar role in science, as a non-threatening pattern of anxiety and relief. And the intensity of the thrill of insight may depend partly on the duration and intensity of the quest that preceded it. ***

Characteristics of Insight

Insight occupies a continuum from conscious to unconscious, from minor problem-solving to mystical experience. Always it involves a leap beyond the available evidence, to unforeseen paths. Almost always it brings a sense of certainty, a dangerous conviction of the truth of the insight.

Poincaré [1914] describes insight’s “characteristics of conciseness, suddenness and immediate certainty.” Another typical characteristic is joy or exhilaration. We will return to the characteristic of immediate certainty in a later section on insight pitfalls. The following descriptions of insights illustrate both their variety and some of their common elements:

“He who has once in his life experienced this joy of scientific creation will never forget it; he will be longing to renew it.” [Kropotkin, 1899]

“The joy of discovery is certainly the liveliest that the mind of man can ever feel.” [Bernard, 1865]

“It came to me in a dream and it’s money in the bank. It’s so simple it’s ridiculous. . . Read it and weep.” [1990 fax from a colleague who is an electronics technician, describing a new equipment design]

Alfred Russel Wallace [1853], who discovered evolution independently of Charles Darwin, described his walks in the Welsh countryside: “At such times I experienced the joy which every discovery of a new form of life gives to the lover of nature, almost equal to those raptures which I afterwards felt at every capture of new butterflies on the Amazon.”

Albert Einstein, in a 1916 letter [cited by Hoffmann, 1972], described his discovery and confirmation of general relativity after an 11-year search: “Imagine my joy at the feasibility of the general covariance and at the result that the equations yield the correct perihelion motion of Mercury. I was beside myself with ecstasy for days.”

Of course, insight is neither limited to science nor always best described by scientists:

Author Thomas Wolfe [1936] described his creation of three books as follows: “It was a progress that began in a whirling vortex and a creative chaos and that proceeded slowly at the expense of infinite confusion, toil, and error toward clarification and the articulation of an ordered and formal structure. . . With a kind of hurricane violence that could not be held in check, . . . the storm did break . . . It came in torrents, and it is not over yet.”

“The flame of conception seems to flare and go out, leaving a man shaken, and at once happy and afraid. There’s plenty of precedent of course. Everyone knows about Newton’s [apocryphal] apple. Charles Darwin said his Origin of Species flashed complete in one second, and he spent the rest of his life backing it up; and the theory of relativity occurred to Einstein in the time it takes to clap your hands. This is the greatest mystery of the human mind -- the inductive leap. Everything falls into place, irrelevancies relate, dissonance becomes harmony, and nonsense wears a crown of meaning.” [writer John Steinbeck, 1954; cited by Calvin, 1986]

***

Conditions Favoring Insight

Perhaps the most valuable result of Platt and Baker’s [1931] survey of scientists was its recognition that certain conditions favor achievement of insight:

• Define the problem. The more specific one can be in identifying the paradox or problem, the better is one’s chance of success. Describing the problem to others sometimes helps, because it forces the researcher to define the problem simply. Sometimes one can solve the larger problem piecemeal by obtaining confident solutions for components of the problem. Yet discrepant observations must not be overlooked. Do the partial solutions suggest that other facts are needed, do they suggest analogies, or do they have an impact on other partial solutions or facts? An exam-taking strategy can be useful here: start with the easiest problems, then work up to the harder ones. This strategy helps build momentum and confidence and it avoids overwhelming the researcher with the magnitude of the problems.

• Complete the initial stage of preparation. Killeffer [1969] calls this step accumulation, emphasizing the role of accumulating needed facts. One cannot expect to solve the problem unless the relevant information is available and comprehended. Furthermore, the facts must be organized. Indeed, the juxtaposition of certain facts can provide the mental connection needed for insight, so it may be worthwhile to try arranging the facts in different ways. Sketching or outlining the relationships may help. Mental images may help. Many scientists find that writing a scientific paper triggers insights, because it forces us to organize data, assumptions, and inferences much more systematically than we do mentally. Sometimes one of our assumptions is the obstacle to insight; deliberately listing and challenging all assumptions may help. In summary, preparation includes accumulation, comprehension, evaluation, and organization of data, assumptions, and inferences.

• Desire a solution. Having a personal stake in a problem can help or hinder insight; usually it is a strong asset. Preoccupation with the quest keeps the problem churning through one’s conscious thoughts and subconscious, providing the needed stage of incubation. Desire for a solution becomes counterproductive if it leads to distracting worry and anxiety. Thus some researchers are more successful in achieving insights concerning other people’s problems than in solving their own problems.

“The unconscious work goes on only over problems that are important to the waking mind, only when the mind’s possessor worries about them, only when he cares, passionately.” [Gerard, 1946]

• Relax and temporarily abandon the problem. Insight can be fostered as easily as this: simply pause for thought whenever you encounter anomalous data in your research or reading. Even more conducive conditions are the combination of mental relaxation with either physical relaxation or mild exercise [Platt and Baker, 1931]: walking on a beach or in the forest or between work and home, taking a bath, relaxing in bed just before falling asleep or just after awakening. Receptivity is needed to achieve the goal. Abel [1930] said:

“It is an old saying ever since Archimedes [with the cry, ‘Eureka!’] solved the problem of specific gravity in his bath tub. . . that discoveries are not made in the laboratories but in the bath tub, or during an afternoon or evening walk as in the case of Helmholtz, or in the watches of the night when the puzzled brain will not quiet down.”

Charles Darwin [1876] described his discovery of evolution by natural selection as follows: “I can remember the very spot in the road, whilst in my carriage, when to my joy the solution occurred to me.”

“Did not one of the great masters attain enlightenment upon hearing the splash of his own turd into the water?” [Matthiessen, 1978] I don’t know, but I doubt it.

“The fact that the attack is seemingly unsuccessful shows that something is wrong. Sometimes merely more information is required. Often, however, the difficulty arises from an incorrect interpretation of the facts at hand. . . In taking up any problem after a period of rest, we have the chance of leaving behind an erroneous point of view and of seizing upon one more fruitful.” [Platt and Baker, 1931]

“The archer hitteth the mark partly by pulling, partly by letting go.” [ancient Egyptian saying, cited by Leuba, 1925]

Apparently the subconscious is set working on a problem by our conscious thought and desire for a solution. It keeps working on the problem, trying out possible patterns even when (or especially when) the conscious mind has relaxed and stopped feeding it a variety of distracting extraneous facts.

As spring comes to the Arctic, the icebound rivers appear to be immune to the warming. Invisibly but pervasively, the ice slowly succumbs to spring’s warmth. Without warning, in a few deafening seconds all of the river ice breaks up and begins to flow. The pace of insight is like this breakup.

Additional circumstances, related to the four above, also favor insight. For example, Beveridge [1955] emphasizes the value of discussing ideas with other people. They have different perspectives, and one may benefit from those perspectives or from combining one’s knowledge with theirs. Their questions, as well as our need to frame answers in the context of their backgrounds, may force us out of the rut of established thought patterns and into a more fruitful perspective. They or we may spot faulty reasoning, during the explanation of things normally take for granted. Perhaps the discussion with others will not lead directly to a solution, but it will increase enthusiasm or at least decrease discouragement at how intractable the problem seems to be. Discussions are most likely to encourage insight if they are carried out in a relaxed and friendly, rather than highly critical and defensive, atmosphere.

The Incomplete Guide to the Art of Discovery, by Oliver [1991], suggests that the best way to foster insight is to “try to become associated with the fresh new observations. That is where the discoveries are most likely.” Oliver emphasizes that almost every really novel kind of observation brings surprises and enhances understanding. “We need only to recognize an important unexplored frontier and then plan and carry out a sound program of observation of that frontier.” Simple!

Most insights illuminate merely the central idea, then the mind rapidly grasps all of the details and implications [Platt and Baker, 1931]. At other times, insights can be partial or fleeting. Many scientists find that it is valuable to jot down such ideas for further consideration later; a pad and pencil near the bed can be helpful.

***

Obstacles to Insight

Some obstacles to insight are obvious; others are more insidious, masquerading as an essential part of scientific activity:

• Distractions -- particularly unpleasant distractions such as domestic or business worries, anxiety, and fatigue -- destroy the receptivity needed for insight. I have seen anxiety over possible layoffs cut worker productivity by about 50% and cut discoveries by nearly 100%, although management expected that 10% layoffs would cause only 10% reduction in overall productivity.

In the years 1665-1666, plague in England forced the closing of Cambridge University, so Isaac Newton went home to the village of Woolsthorpe. There, in this brief time, he developed the calculus, discovered the relationship of color to light, and laid the foundation for his later elucidation of the laws of gravitation and dynamics. [Hoffmann, 1972]

Albert Einstein [1879-1955], whose physics eventually superseded Newton’s dynamics, said that the ideal job for a theoretical physicist is to be a lighthouse keeper. In 1933, living in the relatively isolated village of Cromer in England, he said, “I have wonderful peace here; only now do I realize how driven I usually am.” On another occasion he expressed similar thoughts about the same location: “I really enjoy the quiet and solitude here. One can think much more clearly, and one feels incomparably better.” Yet his most productive period for insights was 1905, during which he worked full-time at the Patent Office.

Pleasant distractions, such as excitement or preoccupation with something other than the immediate problem, can be more insidious but just as inimical to research success and insight. Minor problems, experimental techniques, and equipment modifications are visible and readily attacked forms of problem solving, but also distractions from the main research thrust. Particularly dangerous is the black hole of computers: web crawling, software collection, and software usage can begin as a fascinating and justifiable diversion, then become a time-sink that eclipses their contribution to the primary research objective. • Interruptions are probably the most disruptive type of distraction. Even the expectation of possible interruption is counterproductive to insight, because it prevents total immersion in the problem. Perhaps the most potent step that one can take toward enhancing both productivity and insights is to allot an interruption-free portion of each day or week to thinking about, rather than ‘doing’, science. Platt and Baker [1931] received many comments such as these on the problem of interruptions:

“As an example of the benefit due to freedom from interruptions try going to the laboratory on a holiday. Note how easily many formerly complicated problems straighten themselves out, how smoothly the mind works, and how much work is accomplished with little effort.”

“Any employer of my services who wanted creative thinking oftener THAN ONCE A DAY, SHOULD RELIEVE ME OF MY ADMINISTRATIVE WORK, otherwise I might describe myself as a hard worker during the day on the mechanics of the job and a creative thinker at night on my own time.”

• Conditioned thinking can prevent a person from adopting the new perspective that may be needed to solve a problem. A common response in the business world is to ask employees to “think outside the box”. In contrast, the zoo mammal, when moved to a larger cage, continues to pace an area similar to that of the old cage [Biondi, 1980].

Beveridge [1955] suggests several ways to break free from conditioned thinking. Set the problem aside for a while then resume; as discussed in the previous section, temporary abandonment helps by allowing the established thought pattern to fade, perhaps permitting a new one to start. More drastically, one may need to start over from the beginning with a very different approach. Talking over the puzzle with others or writing up the project can provide the new perspective. Reading related papers, or even potentially relevant papers on different subjects, at least will drive a wedge between conditioned thinking and the problem. They also may evoke a useful analogy. The value of abandoning conditioned thinking is the lesson of the Sufi story, The River, which began this chapter.

***

More dangerous than the factors preventing insight is excessive confidence in one’s insight. An almost universal characteristic of insight is the conviction of truth. Unlike the scientist’s normal attitude that hypotheses can be disproved but not proved, the flash of insight often is accompanied by a certainty that the discovered pattern is so elegant that it must be true. This certainty is a scientific pitfall that can undermine the objective undertaking of the fourth stage of research: verification. When Platt and Baker [1931] polled scientists and asked whether they had ever had a revelation that turned out to be wrong, only 7% said that their insights were always correct.

Part of this conviction of truth may be attributable to the sudden breakthrough of pattern recognition. The greater the breadth of the pattern and its apparent ability to account for disparate observations, the greater the conviction of truth. Yet cold analysis may reveal fatal flaws in the insight.

Last winter my scientist wife and I talked often about her unexplained research results. She tested and rejected many hypotheses. Then I had an exhilarating insight into what the ‘true’ explanation was. I explained my complex model to her, as well as the surprising and therefore diagnostic results that my model predicted for two experiments that she had not done yet. She pointed out that the model was contrary to conventional theory; I agreed with a smile and with unshaken conviction of my

model’s accuracy. Although she was dubious of the model, she undertook the two experiments. One result fit my predictions and one contradicted them, and today only a dim echo of my model is accepted by either of us. Yet my exhilaration at discovering the model was not balanced by a corresponding disappointment at seeing the model proved wrong, perhaps because my emotional involvement with the problem was an intrigued outsider’s interest rather than an anxiety of frustrated scientific progress.

About a month after my model failed, my wife was continuing the experiments at another lab and called me to say: “We’ve always been thinking of the energy barriers as peaks. What if they are troughs instead?” Immediately I felt that she was right, that she had solved the problem. Her answer was so much simpler than mine had been. With this new perspective, we were amazed that we all had been obtuse for so long. Of course, not everyone is as certain as we are that she is right.

“When you have at last arrived at certainty, your joy is one of the greatest that can be felt by a human soul.” [Pasteur, 1822-1895,b].

***

The Royal Way

Success or failure in reaching insight often depends on the path followed. Once the goal is achieved, however, the path becomes irrelevant to the evaluation of that insight.

“But any pride I might have felt in my conclusions was perceptibly lessened by the fact that I knew that the solution of these problems had almost always come to me as the gradual generalization of favourable examples, by a series of fortunate conjectures, after many errors. I am fain to compare myself with a wanderer on the mountains, who, not knowing the path, climbs slowly and painfully upwards, and often has to retrace his steps because he can go no farther -- then, whether by taking thought or from luck, discovers a new track that leads him on a little, till at length when he reaches the summit he finds to his shame that there is a royal way, by which he might have ascended, had he only had the wits to find the right approach to it. In my works, I naturally said nothing about my mistakes to the reader, but only described the made track by which he may now reach the same heights without difficulty.” [Helmholtz, 1891]

Between 1899 and 1904 French mathematician Henri Poincaré considered many of the same factors, including in 1904 the same term ‘principle of relativity’, that Albert Einstein brought together in his 1905 paper on special relativity. Yet Poincaré was unable to reach the same insight first. Poincaré says in his 1911 letter of reference for Albert Einstein:

“I do not mean to say that all these predictions [by Einstein] will meet the test of experiment when such tests become possible. Since he seeks in all directions, one must, on the contrary, expect the majority of the paths on which he embarks to be blind alleys. But one must hope at the same time that one of these directions he has indicated may be the right one, and that is enough. This is exactly how one should proceed. The role of mathematical physics is to ask questions and only experiment can answer them.” [cited by Hoffmann, 1972]

Concerning his 1915 discovery of general relativity, Albert Einstein [1879-1955] said:

“In the light of knowledge attained, the happy achievement seems almost a matter of course, and any intelligent student can grasp it without too much trouble. But the years of anxious searching in the dark, with their intense longing, their alternations of confidence and exhaustion, and the final emergence into the light -- only those who have themselves experienced it can understand that.”

Helmholtz would have understood it. ***

How Does Insight Work?

Insight is the least controllable aspect of scientific research. It can be encouraged, however, by immersion in an examination of all relevant evidence, followed by relaxation and temporary abandonment of the problem. Furthermore, we know that conditions such as interruptions can prevent insight. But what is the mechanism of insight? What marriage between data and pattern recognition is performed in the brain, resulting in the birth of insight? I don’t know, but I think we have seen some clues.

J.E. Teeple, a respondent to Platt and Baker’s [1931] questionnaire, may have hit upon the most decisive element, concentration:

“It is this deep concentration that is the most valuable asset in the solution of any problem. We speak of thinking and try to divide it into conscious, subconscious, and completely unconscious, which I think is an error. In deep concentration on any subject you are not only unconscious that you are thinking but you are unconscious of everything else around you.”

Imagine substituting the word ‘concentration’ for ‘insight’ in the previous sections on conditions favoring insight and obstacles to insight; usually the discussions still would be valid. It appears that concentration is a necessary but not sufficient condition for insight.

Another clue to the mechanism of insight may come from the relationships of data and hypothesis generation to insight. We usually feel that there are too few data to force a conclusion, or more likely not enough data of the needed type. In contrast, the geometric expansion of science in this century often creates the converse problem: there are too many data of too many relevant but somewhat different types to grasp and consider simultaneously. One is left with the vague hunch that the answer is hidden somewhere in the masses of data; perhaps, some filter or new perspective is needed to extract the key observations and their relationships.

Do we achieve insight through subconscious processing of all possible permutations of the evidence, or of only a subset? Consider the following two contrasting viewpoints on hypothesis generation:

“Mathematical creation does not consist in making new combinations with mathematical entities already known. Anyone could do that, but the combinations so made would be infinite in number and most of them absolutely without interest. . . The true work of the inventor consists in choosing among these combinations so as to eliminate the useless ones, or rather to avoid the trouble of making them.” [Poincaré, 1905]

“The effort to solve a problem mentally is a constant series of trials and errors. The mind in searching for a solution considers in rapid succession a long series of conceivable answers, each of which is almost instantly rejected on account of some

obvious objection. Finally in this process of trial and rejection we more or less accidentally stumble upon an answer to which the objection is not so obvious. The smooth course of trial and rejection is brought to a halt. Our attention is arrested.” [Platt and Baker, 1931]

“Discovery is something a computer (if constructed and programmed well enough) could do, and do as well (even better) than any human who ever lived.” [Jason, 1989]

Insight spans, I suspect, a continuum from conscious to unconscious. Platt and Baker [1931] may be partly right, if much of the filtering of ideas occurs either subconsciously, on the fringe of consciousness, or in such a brief conscious flash that we are barely aware of it. The less absurd ideas require a little more conscious focus before they can be discarded. Yet Poincaré grasps a central point, missed by Platt and Baker, that the successful scientist owes as much to excluding broad regions from trial-and-error evaluation as to the evaluation itself. A better-trained computer is not the solution.

The approach of considering all possible data permutations is hopeless. What is needed is a leap of insight to the crux. As in chess, the best player does not simply examine all permutations methodically; instead the master visualizes patterns and focuses in on a small subset of the possibilities. I think that the pitfall of conditioned thinking offers a useful perspective: rather than systematically scanning a huge number of possible explanations, scientific thoughts get trapped among a few patterns, like a song that you cannot get out of your head. Relaxation and temporary abandonment may work because the problem continues to pop unbidden into the fringe of consciousness, interspersed with seemingly unrelated thoughts, until suddenly the mind sees the problem in the context of a potentially explanatory pattern.

From a neurobiologist’s perspective, the brain has a vast number of schemata-templates [Calvin, 1986]. Each schema is a neural pattern or pathway, formed and reinforced by electrical current flow. Each schemata-template is triggered whenever we see or experience something that seems to fit the pattern. Boyd [1985] describes hypothesis creation as “finding new combinations of previously understood ideas and concepts.” If schemata are reinforced via electrical current flow in the brain, could insight be a sudden flow in parallel of schemata that had never previously flowed simultaneously?

“It is a wondrous thing to have the random facts in one’s head suddenly fall into the slots of an orderly framework. It is like an explosion inside. . . I think that I spend half my time just talking and listening to people from many fields, searching together for how [plate tectonics] might all fit together. And when something does fall into place, there is that mental explosion and the wondrous excitement. I think the human brain must love order.” [marine geologist Tanya Atwater in 1981, during the period in which the new paradigm of plate tectonics was revolutionizing geology; cited by Calvin, 1986]

Even if the idea of insight as schema generation is correct, its practical usefulness may be small. It does, however, reveal a potential problem: those insights are favored that are similar to ideas and concepts that are already established. Breakthroughs to a radically new perspective are not fostered. Contrast these incremental advances with the following:

“In each of the 1905 papers, Einstein has totally transcended the Machian view that scientific theory is simply the ‘economical description of observed facts.’ None of these theories, strictly speaking, begins with ‘observed facts’. Rather, the theory tells us what we should expect to observe.” [Bernstein, 1982]

***

Alternative Paths to Insight

The preceding sections give the misleading impression that insight only follows a prolonged search for the explanation to one’s observations. Other sources of insight, however, can be just as fruitful. Chance can play a prominent role in discovery. Many breakthroughs are by amateurs, or at least by those with scanty experience of the relevant evidence. And one of the most powerful ways of achieving an insight is to borrow from another field.

Unexpected Results

Chance makes an influential, yet often overlooked, contribution to discovery. For example, Roentgen’s discovery of X-rays began with an accident: photographic plates left near a discharge tube were inexplicably blackened. When Alexander Fleming noticed a strange mold growing on his culture dish, he isolated it, purified it, and discovered penicillin, the first antibiotic.

Rather than providing a variation on existing themes, chance discovery can lead to a totally new perspective. One can seek insight but cannot seek chance discovery. One can, however, open oneself to this type of discovery [Beveridge, 1955]:

Be alert for any unexpected result. Resist the temptation to rationalize away or discard them. Remember that observations that do not fit predictions, though often ignored, sometimes are responsible for the new paradigm. “Remain alert and sensitive for the unexpected while watching for the expected” [Beveridge, 1955]. Try novel procedures, to increase the likelihood of encountering surprises.

In seeking insight from unexpected results, we may encounter pitfalls instead. It is easy to become distracted and pulled in a new direction by every unexpected result, so that one seldom completes a suite of experiments. This pitfall is often avoidable: simply flag the unexpected data and come back to them later. One can easily confuse the ‘chance-in-a-lifetime’ result with trivial results, failing to follow up on the former or wasting considerable time on the latter. Louis Pasteur [18221895, a] repeatedly said, “In the field of experimentation, chance favors only the prepared mind.” One needs considerable background, to recognize the unexpected result and to evaluate correctly its importance and significance.

Transfer From Other Disciplines

One path to insight that is frequently successful, yet underutilized, is the extension of a technique, algorithm, relationship, or equipment from one field to another:

“For every original discovery there are dozens of important advances which are made simply by recognizing that a scheme developed for one field or application can be applied to another.” [Wilson, 1952]

“Making variations on a theme is really the crux of creativity.” [Hofstadter, 1985]

For example, Einstein’s emphasis on reference frames was a key not only to relativity but also, much later, to the paradigm of plate tectonics in geology. The recent physics paradigm of chaos is creating breakthroughs in oceanography, meteorology, biology, and earthquake mechanics. These examples are of theoretical concepts, but even more consistently productive is the application of new techniques and instruments to empirical research.

Many discoveries are driven by technology, by the sudden ability to make a new type of measurement or to make a much more accurate measurement than before. The inventors of the electron microscope, laser, and CT scan could never have imagined the realms that these devices would explore. Recognition of the applicability of a new instrument or technique is easiest for the person who knows the problem, not for the person who develops the technique. One does not necessarily even have to match the new technique with a specific problem. Discovery can result just from a ‘fishing expedition’ with a new kind of observation of an old phenomenon.

Because science is increasingly specialized, researchers seldom are aware of technical or conceptual developments in ‘unrelated’ fields. Thus a potential application may go unrecognized for many years. This approach also is underutilized because it is haphazard, almost always stumbled upon rather than deliberately sought out. Yet it can be fostered. Seeking new applications to one’s field is a strong incentive for reading outside of one’s field. Such goal-oriented reading can be extremely productive.

Breakthroughs by Amateurs: the Outsider Perspective

Breakthroughs by ‘amateurs’ are a phenomenon that seems to run counter to the philosophy of acquiring all relevant data to assist in reaching an insight. Actually, the ‘amateur’ usually is not a scientific novice, but an experienced scientist who has just changed fields. The neophyte brings to a new field the established disciplines of scientific method but not the prevailing assumptions or prejudices of the entrenched leadership of that field. The newcomer may also bring a technique or concept to a different field, as mentioned above.

A related phenomenon is breakthroughs by young scientists: most revolutions within each field of science are led by the younger generation (in physics, for example, by those under 30 years old). This generation may have more energy than the older generation, but it also has less efficiency and much less knowledge. The higher discovery rate among relative newcomers to a field stems from their greater flexibility of thought, due to less ingrained assumptions and conclusions.

Published errors -- whether in assumptions, data, or interpretations -- are stumbling blocks to further insights, particularly for researchers who have long accepted them. It does not follow, though I have heard the argument made, that it is better for one to avoid reading intensively in one’s specialty. Oliver [1991] points out that seeking breakthroughs outside one’s specialty brings hazards as well as opportunities. For example, physicist Lord Kelvin calculated the age of the earth and was dogmatic that his result was correct. He ignored geologists’ evidence for a much older age, partly because the evidence was not from his field.

Knowing all the relevant evidence is essential, but equally essential is alertness to the basis for one’s assumptions and interpretations, and critical reevaluation of that basis.

Changing fields is a drastic means of chasing insights. Changing subjects within the same field can unleash creativity [Loehle, 1990]. Another pragmatic alternative is simply to try out different perspectives. The following problem illustrates this approach.

A gnat flies deep into your ear and repeatedly collides with your eardrum. How can you solve the problem?

First, consider the consequences of initial failure on future trials (Chapter 5): squirting water into your ear might wash the gnat out, but a dead gnat may be even harder to extract than a live gnat. Instead, try the problem-solving technique of

choosing the perspective of others -- in this case, the gnat’s perspective. Does the gnat want to be in or out? Is there anything that you can do to influence the gnat’s behavior? Gnats, like most flying insects, are attracted to light. So shine a light in your ear, and help the gnat escape.

For those who are more committed to attaining insights than to persisting with their current research projects, consideration of the following questions may suggest more fruitful research directions [Oliver, 1991]. Is the present discipline becoming more isolated and losing touch with the rest of the field? What is its past and present rate of progress and discoveries, and are they accelerating or decelerating? Are many datasets still not understood, thereby indicating the potential for new insights? Is the field undergoing long-term changes, such as from observational to theoretical or perhaps toward increasing interaction with some other discipline? Do gaps between specialties create opportunities for dual-specialty science, and will their exploitation require intensive background study or selection of co-workers in that specialty? Is there an alternative research thrust to that followed by the majority of researchers in the field? ***

From Puzzle Solving . . .

The magnitude of the creative leap forms a continuum, from minor problem solving to major creative insight to mystical experience. The thrill of major creative insight or mystical experience is quite rare, yet most scientists capture a taste of that thrill every day in the small-scale problem solving that is a characteristic part of science. Indeed, many scientists have puzzle-solving hobbies such as chess, bridge, and reading mysteries -- hobbies that further gratify the craving for insights of any size.

Some classic puzzle-solving techniques also foster both insight and scientific problem-solving:

• redefine the problem by breaking it down into several components, then attack one or more of these pieces individually;

• decide which thread to grasp, to start unraveling the puzzle;

• analyze all assumptions and detect inappropriate, overlooked, or invalid assumptions;

• provisionally assume an answer, then look at its implications for the problem.

Perhaps a hobby of puzzle solving can improve our ability to recognize hidden scientific assumptions. Killeffer [1969], among others, suggests that the practice of puzzle solving improves the ability of the mind to see patterns and associations. This ability, like other acquired skills, can be enhanced by practice.

“It began with little things, certain small clinical changes which I observed. Little things can be important. Even more important is the ability -- call it knack, hunch, providence, good luck, whatever -- to know what you are looking for and to put two and two together. A great scientist once said that genius consists not in making great discoveries but in seeing the connection between small discoveries. . . Small disconnected facts, if you take note of them, have a way of becoming connected.” [Percy, 1987]

“It’s just like doing a jigsaw puzzle: whenever you think that there is no piece that can possibly fill a blank space, you don’t just throw up your hands and insist that

only a miracle will solve the problem. You keep looking, and eventually you find something that links together the parts of the puzzle.” [Calvin, 1986]

But in science, unlike in puzzle solving, the problem may be impossible to solve. That uncertainty is part of the challenge. ***

. . . to Mystical Experience

Many scientists will be bothered or even offended by my inclusion of mystical experiences on the continuum of insight intensity. Mystical experiences are automatically lumped with emotions and other non-rational and therefore non-scientific subjects. Perhaps psychologist Abraham Maslow’s term ‘peak experience’ or James Joyce’s term ‘epiphany’ is more palatable. I argue that major creative insight is a goal-oriented subset of mystical experience rather than a fundamentally different phenomenon. But I am content if the reader agrees that the two phenomena exhibit some surprisingly strong parallels, as illustrated by the following examples.

St. Augustine [354-430 A.D., b] described a mystical experience as “a moment of supreme exaltation, followed by gradual absorption back into the normal state, but with resulting invigoration and clearer perception.”

“Then followed months of intense thought, in order to find out what all the bewildering chaos of scattered observations meant, until one day, all of a sudden, the whole became clear and comprehensible, as if it were illuminated with a flash of light. . . There are not many joys in human life equal to the joy of the sudden birth of a generalization illuminating the mind after a long period of patient research.” [Kropotkin, 1899]

Naturalist Annie Dillard read accounts of people blind since birth who were suddenly given sight by cataract surgery, and of their remarkably variable and sometimes frightened reactions to this new vision. She was particularly captivated by the experience of one young girl who stared, astonished, at what she finally recognized as a tree. This was not a tree such as you or I have ever seen, but “the tree with the lights in it” [Dillard, 1974]:

“When her doctor took her bandages off and led her into the garden, the girl who was no longer blind saw ‘the tree with the lights in it.’ It was for this tree I searched through the peach orchards of summer, in the forests of fall and down winter and spring for years. Then one day I was walking along Tinker Creek thinking of nothing at all and I saw the tree with the lights in it. I saw the backyard cedar where the mourning doves roost charged and transfigured, each cell buzzing with flame. I stood on the grass with the lights in it, grass that was wholly fire, utterly focused and utterly dreamed. It was less like seeing than like being for the first time seen, knocked breathless by a powerful glance. The flood of fire abated, but I’m still spending the power. Gradually the lights went out in the cedar, the colors died, the cells unflamed and disappeared. I was still ringing. I had been my whole life a bell, and never knew it until at that moment I was lifted and struck. I have since only very rarely seen the tree with the lights in it. The vision comes and goes, mostly goes, but I live for it, for the moment when the mountains open and a new light roars in spate through the crack, and the mountains slam.”

“Thus my mind, wholly rapt, was gazing fixed, motionless, and intent, and ever with gazing grew enkindled, in that Light, . . . for my vision almost wholly departs,

while the sweetness that was born of it yet distills within my heart.” [Dante Alighieri, 1313-1321]

“The flame of conception seems to flare and go out, leaving a man shaken, and at once happy and afraid. . .” [John Steinbeck, 1954; cited by Calvin, 1986]

***

Living science, for me, is punctuated illumination, less blinding but more frequent than the experiences of Dillard and Dante. I too am struck and go on ringing, and I am so addicted that neither the countless minor frustrations nor occasional stagnations can fully damp the memory and obsession with this ringing. Perhaps someday I will pick a problem that defies solution, search for years without finding, and finally claim that I really became a scientist because of motivations other than the thrill of insight. Or perhaps I will see “the tree with the lights in it.” For now, I go on ringing.

“I feel like shouting ‘Eureka!’, awakening the camp. But caution reasserting itself, I satisfy myself with a broad smile instead, and look overhead at the drifting clouds. I must try this out, see just how much of the universe’s known mechanism can be appreciated from this new viewpoint.” [Calvin, 1986]