When I can’t sleep, I think about what I’m missing. I glance over at my wife and watch her eyelids flutter. I listen to the steady rhythm of her breath. I wonder if she’s dreaming and, if so, what story she’s telling to herself to pass the time. (The mind is like a shark — it can’t ever stop swimming in thought.) And then my eyes return to the ceiling and I wonder what I would be dreaming about, if only I could fall asleep.

Why do we dream? As a chronic insomniac, I like to pretend that our dreams are meaningless narratives, a series of bad B-movies invented by the mind. I find solace in the theory that all those inexplicable plot twists are just random noise from the brain stem, an arbitrary montage of images and characters and anxieties. This suggests that I’m not missing anything when I lie awake at night — there are no insights to be wrung from our R.E.M. reveries.

Unfortunately for me, there’s increasing evidence that our dreams are not neural babble, but are instead layered with significance and substance. The narratives that seem so incomprehensible — why was I running through the airport in my underwear? — are actually careful distillations of experience, a regurgitation of all the new ideas and insights we encounter during the day.

Look, for instance, at the research of Matthew Wilson, a neuroscientist at the Picower Institute at M.I.T. In the early 1990’s, Wilson was recording neuron activity in the brains of rats as they navigated a difficult maze. (The machines translated the firing of brain cells into loud, staccato pops.) One day, he left the rats connected to the recording equipment after they completed the task. (Wilson was preoccupied with some data analysis.) Not surprisingly, the tired animals soon started to doze off, slipping into a well-deserved nap. And that’s when Wilson heard something extremely unexpected: although the rats were sound asleep, the sound produced by their brain activity was almost exactly the same as it was when they were running in the maze. The animals were dreaming of what they’d just done.

Wilson has spent the last few decades following up on this important discovery. In a 2001 paper published in Neuron, Wilson and Kenway Louie described the behavior of rats that had been trained to run on a circular track. As expected, running on the track generated a distinct pattern of neural firing in the rat hippocampus, a brain area essential for the formation of long-term memory. This is learning at its most fundamental: a flurry of electric cells, trying to make sense of a space.

Here’s where things get interesting: as before, Wilson kept the electrodes in place while the rats drifted off to sleep. (The sleep of rats is very human, and consists of distinct stages, including R.E.M.) The scientists examined 45 dreams and found that 20 of the dreams repeated the exact same patterns of brain activity exhibited while running in a circle. In fact, the correlation between the dream and the reality was so close that Wilson could predict the exact position of the rodent on the track while it was asleep. They were decoding the dream as it was being dreamt.

Why does the brain replay experience? Wilson and others argue that the dreaming rats are consolidating their new memories, embedding these fragile traces into the neural network. While we’re fast asleep, the mind is sifting through the helter-skelter of the day, trying to figure out what we need to remember and what we can afford to forget.

So why are dreams so much more than literal playbacks of the day just passed? Why the non-sequiturs, the long forgotten characters and the unexplained state of public undress? Wilson speculates that dreams are also an attempt to search for associations between seemingly unrelated experiences, which is why it’s so important for the controlling conscious self to disappear. What does this maze have to do with that maze? How can we use the lessons of today to get more food pellets tomorrow? This suggests that the strangeness of our nighttime narratives is actually an essential feature, as our memories are remixed and reshuffled, a mash-up tape made by the mind.

But wait: for the sleep- and dream-deprived, the news gets even worse. In recent years, scientists have discovered that R.E.M. sleep isn’t just essential for the formation of long-term memories: it might also be an essential component of creativity.

In a 2004 paper published in Nature, Jan Born, a neuroscientist at the University of Lübeck, described the following experiment: a group of students was given a tedious task that involved transforming a long list of number strings into a new set of number strings. This required the subjects to apply a painstaking set of algorithms. However, Born had designed the task so that there was an elegant shortcut, which could only be uncovered if the subjects saw the subtle links between the different number sets. When left to their own devices, less than 25 percent of people found the shortcut, even when given several hours to mull over the task. However, when Born allowed people to sleep between experimental trials, they suddenly became much more clever: 59 percent of all participants were able to find the shortcut. Born argues that deep sleep and dreaming “set the stage for the emergence of insight” by allowing us to mentally represent old ideas in new ways.

Or look at a recent paper published by Sara Mednick, a neuroscientist at the University of California, San Diego. She gave subjects a variety of remote-associate puzzles, which require subjects to find a word that’s associated with three other seemingly unrelated words. (Here’s a sample question: “broken,” “clear” and “eye.” The answer is “glass.”) Then, she instructed the subjects to take a nap. Interestingly, subjects who lapsed into R.E.M. during their nap solved 40 percent more puzzles than they did in the morning, before their brief sleep. (Subjects who quietly rested without sleeping or took a nap without R.E.M. showed a slight decrease in performance.) According to Mednick, the dramatic improvement in creativity is due to the fact that R.E.M. “primes associative networks,” allowing us to integrate new information into our problem-solving approach.

While Freud would certainly celebrate this research — as he predicted, dreams have “a psychological structure … which may be assigned to a specific place in the psychic activities of the waking state” — it’s worth pointing out that the stories we invent while sleeping are much more practical than he imagined. For the most part, they don’t reflect the unleashed id, full of unfulfilled sexual desires. Instead, we dream about what we think about: the mazes and mysteries of everyday life.

All this knowledge about the important roles dreams play in our waking lives is fascinating. But it doesn’t make me feel better about my insomnia. Obviously, my old consolation — dreams are nothing but useless melodramas — is clearly false. And though I eventually do fall asleep, lapsing into what I imagine is a rushed state of R.E.M., I can’t help but be jealous of my wife’s twitching eyelids at 2 a.m. She is busy remembering, processing, refreshing —and I am merely awake.

Jonah Lehrer is a contributing editor at Wired Magazine. He’s the author of “How We Decide” and “Proust Was A Neuroscientist” and blogs at The Frontal Cortex.