We all know the story of Albert Einstein’s “cosmological constant,” or lambda, which he invented, then retracted in shame ("Then away with the cosmological constant!"), and then in 1998, with the discovery of the accelerating expansion of the universe, it came back with a vengeance as the mathematical representation of 'dark matter'. 

But surprisingly, the story continues to live on through new discoveries.

 

In 1917, a year after his general theory of relativity was published, Einstein decided to try to apply the field equations of gravitation to the entire universe. But his math was better than he wanted to believe! The field equations implied that the universe couldn't stay static.

But astronomers of the time (Einstein was especially friendly with Erwin Finlay Freundlich, who worked at the Berlin Observatory, and gave Einstein a lot of astronomical information) didn’t know about the universal expansion--they believed that everything visible through telescopes was in the Milky Way.

So Einstein chose to ignore what his mathematics was telling him. He forced his equations to yield only solutions that described a non-expanding, non-collapsing universe. He achieved this by inventing the cosmological constant, λ (the Greek letter), and using it to adjust his equations.

In 1929 Edwin Hubble announced his discovery (with Milton Humason and Vesto Slipher) that the universe was expanding. So Einstein regretted his move: he could have theoretically predicted Hubble’s findings--a result worthy of a second Nobel Prize for him (he deserved a few more, anyway).

In disgust, Einstein famously exclaimed: “If there is no quasi-static world, then away with the cosmological term!” and never considered it again. At least this is what was believed until now.
 
And lambda returned, very unexpectedly, in 1998. That year, two groups of astronomers made an announcement that rocked the world of science: Rather than slowing down since the Big Bang, as everyone had expected (because of the pull of gravity), the universe is actually accelerating its expansion. The discovery won the leaders of the two teams, Saul Perlmutter, and Brian Schmidt and Adam Riess, the Nobel Prize in physics in 2011.

It turns out that the best way to explain the accelerating universe is to revive Einstein’s discarded lambda. The cosmological constant is accepted at the mathematical representation of the mysterious “dark energy” seen to permeate all space and push the universe ever outward at an accelerating rate. Unfortunately, Einstein was not there to witness the reversal of his “greatest blunder,” having died in 1955. And it has been widely assumed that he never reconsidered the cosmological constant. Until now.
 
Irish physicist Cormac O’Raifeartaigh was perusing documents at the Einstein Archives of the Hebrew University in Jerusalem a couple of months ago, when he discovered a hand-written manuscript by Einstein, titled “Zum kosmologischen Problem” (“About the Cosmological Problem”), which had been erroneously filed as a draft of another paper, published in 1931.

But in fact it was an independent manuscript that had never been carefully read before. O’Raifeartaigh’s analysis revealed that in it, Einstein was stubbornly attempting to revive the cosmological constant he had apparently vowed never to use again.

In an article just filed on ArXiv, O’Raifeartaigh and colleagues quote from the newly-read manuscript, where Einstein refers to his earlier invention, lambda (the authors’ translation from the German):

“It is well known that the most important fundamental difficulty that comes to light when one enquires how the stellar matter fills up space in very large dimensions is that the laws of gravity are not in general consistent with the hypothesis of a finite mean density of matter. At a time when Newton’s theory of gravity was still generally accepted, Seeliger had for this reason modified the Newtonian law by the introduction of a distance function that, for large distances r, diminished considerably faster than 1/r². However, I have shown that this can be overcome through the introduction of the so-called ‘λ–term’ to the field equations.” (O'Raifeartaigh, et al., 2014, p. 3.)

He then gives his cosmological equations, in tensor form, from 1917:

(R_ik - ½g_ikR) - λg_ik = kT_ik, where the λ adjusts the metric tensor, g_ik. (R_ik is the Ricci tensor, R is scalar curvature, and T_ik is the stress-energy tensor.)

But Einstein was now aware of Hubble’s discovery of the expansion of the universe, writing:

“On the other hand, Hubbel’s [sic] exceedingly important investigations have shown that the extragalactic nebulae have the following two properties…”

And so Einstein proposed a revision of his model, still with a cosmological constant, but now this constant is responsible for the creation of new matter as the universe expanded. What he says here is key to it all:

“The conservation law is preserved in that by setting the λ-term, space itself is not empty of energy.” (O'Raifeartaigh, et al., 2014, p. 7.)

What we see is that Einstein persistently maintains the use of his lambda, now in the new setting of an expanding universe. Its role is to provide the energy needed to create new particles in the expanding universe so as to maintain a constant density of matter, as Einstein believed happens. (Almost two decades later, a similar “steady state” universe would be proposed by Fred Hoyle, Hermann Bondi, and Tommy Gold. These models of the universe are not supported by modern theories.)

It’s interesting that Einstein repeatedly misspelled the name of Hubble (“Hubbel”). Had he not yet met Hubble in person (he traveled to California in the 1930s) and only heard about his discovery by the time this paper was written? We don’t know. The spelling error does hint at the fact that Hubble’s discovery was not yet well-established so that his name would be recognized by all scientists.

As for why Einstein was so intent on maintaining the use of his discarded lambda: the constant is simply too important—it represents the energy of empty space, a powerful notion that the vacuum of space itself is permeated with forces. Einstein didn’t want to give up on it. And today, we know that the vacuum energy is a very real phenomenon. 

In the 1930s, Einstein exploited it to continuously create new particles from the vacuum so that the density of matter will remain constant through time, while originally in 1917 he used it to hold back a universe that wants to inflate. Today we view the same energy of the vacuum, first proposed by Einstein apparently in error, to explain one of the greatest mysteries of the cosmos: the dark energy accelerating the universal expansion.

Presciently, Einstein refused to let this concept die, regardless of his initial reaction on hearing Hubble’s news. Even with Einstein, old habits die hard. But he couldn't know how right he would be proven--not for his 1917 paper or his (unpublished) 1930s manuscript, but for an idea whose time would come more than 60 years later.