As I wrote a few days ago, if we agree that the nature of science is along the lines I have described, next we need to ask why it is so. Platt, in his classic 1964 article on strong inference, briefly mentions a number of answers, which he dismisses without discussion, but that I think are actually a large part of the reason "hard" and "soft" sciences appear to be so different. These alternative hypotheses for why a given science may behave “softly” include, as Platt puts it, “the tractability of the subject, or the quality of education of the men [sic] drawn into it, or the size of research contracts.” In other words, particle physics, say, may be more successful than ecology because it is easier (more tractable), or because ecologists tend to be dumber than physicists, or because physicists get a lot more money for their research than ecologists do.

The second option is rather offensive (to the ecologists at least), but more importantly there are no data at all to back it up. And it is difficult to see how one could possibly measure the alleged differential “education” of people attracted to different scientific disciplines. Nearly all professional scientists nowadays have a Ph.D. in their discipline, as well as years of postdoctoral experience at conducting research and publishing papers. It is hard to imagine a reliable quantitative measure of the relative difficulty of their respective academic curricula, and it is next to preposterous to argue that scientists attracted to certain disciplines are smarter than those who find a different area of research more appealing. It would be like attempting to explain the discrepancy between the dynamism of 20th century jazz music and the relative stillness of symphonic (“classical”) music by arguing that jazz musicians are better educated or more talented than classically trained ones. It’s a non-starter. [Aficionados of classical music don't jump on me: it's just an illustrative analogy, and I love classical music too.]

The other two factors identified and readily dismissed by Platt, though, may actually carry significant weight. The obvious one is money: there is no question that, at least since World War II, physics has enjoyed by far the lion’s share of public funding devoted to scientific research, a trend that has seen some setback in recent years (interestingly, after the end of the cold war). It would be foolish to underestimate the importance that money makes to science (or anything else, for that matter): more funds don’t mean simply that physicists can build and maintain ever larger instruments for their research (think of giant telescopes in astronomy, or particle accelerators in sub-nuclear physics), but perhaps equally important that they can attract better paid graduate students and postdoctoral associates, the lifeblood of academic research and scholarship. Then again, of course, money isn’t everything: our society has poured huge amounts of cash, for instance, into finding a cure for cancer (the so-called “war” on cancer), and although we have made progress, we are not even close to having eliminated that scourge -- if it is at all possible.
Part of the differential ability of scientific disciplines to recruit young talent also deals with an imponderable that Platt did not even consider: the “coolness factor.” While being interested in science will hardly make you popular in high school or even in college, among science nerds it is well understood (if little substantiated by the facts) that doing physics, and in particular particle physics, is much more cool than doing geology, ecology or, barely mentionable, any of the social sciences -- the latter a term that some in academia still consider an oxymoron. The coolness factor probably derives from a variety of causes, not the least of which is the very fact just mentioned that there is more money in physics than in other fields of study, and even the large social impact of a few iconic figures, like Einstein (when was the last time you heard someone being praised for being “a Darwin”?).
The third reason mentioned but left unexamined by Platt is the relative complexity of the subject matters of different scientific disciplines. This is a crucial and yet constantly under-appreciated point, even though it seems to me trivially true that particle physics does in fact deal with the simplest objects in the entire universe: atoms and their constituents. At the opposite extreme, biology takes on the most complex things known to humanity: organisms made of billions of cells, and ecosystems whose properties are affected by tens of thousands of variables. In the middle we have a range of sciences dealing with the relatively simple (chemistry) or the slightly more complex (astronomy, geology), roughly on a continuum that parallels the popular perception of the divide between hard and soft disciplines. That is, a reasonable argument can in fact be made that, so to speak, physicists have been successful because they had it easy. This is of course by no means an attempt to downplay the spectacular progress of physics or chemistry, just to put it in a more reasonable perspective: if you are studying simple phenomena, are given loads of money to do it, and you are able to attract the brightest minds because they think what you do is really cool, it would be astounding if you had not made dazzling progress!
Perhaps the most convincing piece of evidence in favor of a relationship between simplicity of the subject matter and success rate is provided by molecular biology, and in particular by its recent transition from a chemistry-like discipline to a more obviously biological one. Platt wrote his piece in 1964, merely twelve years after Watson, Crick and Franklin discovered the double helix structure of DNA. Other discoveries followed at a breath-taking pace, including the demonstration of how, from a chemical perspective, DNA replicates itself; the unraveling of the genetic code; the elucidation of many aspects of the intricate molecular machinery of the cell; and so on. But by the 1990s molecular biology began to move into the new phase of genomics, where high throughput instruments started churning a bewildering amount of data that had to be treated by statistical methods (one of the hallmarks of “soft” science). While early calls for the funding of the human genome project, for instance, made wildly optimistic claims about scientists soon been able to understand how to make a human being, cure cancer, and so on, we are in fact almost comically far from achieving those goals. The realization is beginning to dawn even on molecular biologists that the golden era of fast and sure progress may be over, and that we are now faced with unwieldy mountains of details about the biochemistry and physiology of living organisms that are very difficult to make sense of. In other words, we are witnessing the transformation of a hard science into a soft one!
I have much more to say, of course, about the soft-hard science continuum, but you will need to wait until the book comes out, hopefully by the end of 2009. Till then.
[This post was a second excerpt from the draft of a chapter of my forthcoming book, Nonsense on Stilts: How to Tell the Difference Between Science and Bunk," to be published next year by the University of Chicago Press]