Mice lead complex social lives. During the course of a day, they may come across other members of their own species—male or female—or any number of predators. Each encounter requires a specific behavioral response.
Unlike humans, mice don’t perform these daily discriminations visually. Instead, they depend on chemical cues in the form of scents and other odors, such as pheromones.
Catherine Dulac, an HHMI investigator at Harvard University, studies the molecular logic underlying the coding of odorant- and pheromone-mediated signals. Dulac is uncovering how connections form between sensory neurons and the brain and how these circuits allow an animal to discriminate a potential mate, a harmless acquaintance, or a predator.
In recent work, Dulac advanced understanding of the function and architecture of the vomeronasal organ (VNO), a structure at the base of the nasal cavity. By studying the response of neurons in the VNOs of mice to chemical cues collected from not just their kin but also rats, snakes, birds, and an assortment of other species, her research team discovered that the VNO is more geared toward detecting cues from predators than from other mice. The findings were published September 2011 in the journal Nature.
A rodent’s main olfactory system detects airborne odor molecules, while non-airborne signals—those left on physical objects by animals or humans, such as pheromones—are detected by the accessory olfactory system. In this accessory system, chemicals are sensed in the VNO, a tubular structure in the nose. Most mammals and reptiles have a VNO, but humans and some other primates appear to lack one.
The VNO houses tens of thousands of neurons, each equipped with receptors for a particular chemical cue. When a pheromone or other chemical signal arrives at the VNO, it fits into its matching receptor like a key in a lock, activating the neuron and sending a message to the brain. Dulac described the first VNO receptor while a postdoc in HHMI investigator Richard Axel’s lab some 15 years ago; since then, about 300 types of receptors have been identified. Yet, scientists had not worked out how many different chemical signals these receptors could detect, nor had anyone tried to map out the relationship between specific receptor types and the chemical signals to which they respond.
Dulac’s team set out to identify which particular VNO receptors were activated by a wide range of chemical cues. They devised an approach using the activation of certain genes as an indicator of neuronal activity.
First, mice were exposed to one of 29 chemosensory cues in the form of bedding from the cages of male and female mice and predators such as snakes and foxes. Bedding material absorbs a mix of chemicals excreted by animals, including urine, feces, saliva, and other gland secretions.
The researchers found that exposure to each individual pheromone from the bedding material activated neurons in the VNO sensitive to that particular chemical cue. They determined which receptor was expressed in the activated cells using 200 molecular probes developed for this purpose.
Dulac was surprised to find that most VNO neurons were activated by predator cues and not by mouse cues. “The animal doesn’t need a lot of receptors to identify another mouse, but it needs a lot of receptors to be able to detect many, many types of predators,” she says.
Some receptors were very specialized, activated only by a particular snake, bird of prey, or mammalian predator. Others were more generalized, responding to an entire class of predators (for example, all snakes). Some were even more generalized and detected all predators.
Another surprising finding was the receptors’ extreme specificity. “We rarely saw receptors that were activated by both predator and mouse cues,” says Dulac. Thus, activation of a single receptor is enough to identify the animal giving off the chemosensory signal. Dulac likens it to a switchboard, where the switch you push tells you very precisely what type of animal is being encountered.
This is the most recent experiment in Dulac’s long line of research challenging the assumptions about pheromones and the role of the VNO. After identifying the first family of pheromone receptors and other molecules unique to VNO neurons, in 1995, she began investigating the role of the VNO in mouse social behavior. What she discovered upended much of the conventional wisdom about pheromones and sexual behavior.
She discovered, for instance, that male mice whose VNO function had been genetically knocked out failed to display stereotypical aggression toward other males and attempted to mate with both male and female mice. “This indicated the VNO does indeed detect pheromones, but these pheromones aren’t essential for mating,” Dulac says. “They are necessary for sex discrimination.”
| In wild type CD-1 mice, the soiled bedding of rats elicits defensive behaviors such as avoidance (top video). Mice whose vomeronasal activity is genetically ablated (TrpC2-/-) no longer avoid rat bedding and instead feed on it (bottom video). |
A similar experiment with female mice revealed that those with nonfunctional VNOs behaved more like males in their sexual and courtship behaviors. The results suggested that the brains of female mice are hard-wired with the same neuronal circuits that underlie male-specific behaviors, but the VNO acts to inhibit male behaviors and activate female ones.
Now that Dulac has identified the chemical signals detected by the VNO and the specific receptors involved, she can manipulate the system to learn how the brain processes sensory information and translates it into a behavioral response. This work opens up questions about learning and identification. Are prior encounters with a signal necessary to recognize it, or is there an innate recognition of predators and members of one’s own species? Dulac is excited about the possibilities and hopes these results will add to scientists’ arsenal of tools for understanding how animals detect, identify, and process sensory signals.
“A central question in neuroscience is how animals are able to tell apart different types of sensory signals,” Dulac says, “regardless of whether they come from the nose, ears, or eyes. What we learn from the chemosensory system in mice may tell us about the principles of neural circuits between sensory systems and the brain more generally.”
