A team of academics from the University of Cambridge, in collaboration with the Smithsonian Tropical Research Institute in Panama, observed the courtship rituals and sequenced the DNA from nearly 300 butterflies to find out how much of the genome was responsible for their mating behaviour.
The research, published today in PLOS Biology, is one of the first ever genome studies to look at butterfly behaviour and it unlocks the secrets of evolution to help explain how new species are formed.
Scientists sequenced the DNA from two different species of Heliconius butterflies which live either side of the Andes mountains in Colombia. Heliconians have evolved to produce their own cyanide which makes them highly poisonous and they have distinct and brightly coloured wings which act as a warning to would-be predators.
Professor Chris Jiggins, one of the lead authors on the paper and a Fellow of St John's College, said: "There has previously been lots of research done on finding genes for things like colour patterns on the butterfly wing, but it's been more difficult to locate the genes that underlie changes in behaviour.
"What we found was surprisingly simple - three regions of the genome explain a lot of their behaviours. There's a small region of the genome that has some very big effects."
The male butterflies were introduced to female butterflies of two species and were scored for their levels of sexual interest directed towards each. The scientists rated each session based on the number of minutes of courtship by the male - shown by sustained hovering near or actively chasing the females.
Unlike many butterflies which use scented chemical signals to identify a mate, Heliconians use their long-range vision to locate the females, which is why it's important each species has distinct wing markings.
When a hybrid between the two species was introduced, the male would most commonly show a preference for a mate with similar markings to itself. The research showed the same area of the genome that controlled the coloration of the wings was responsible for defining a sexual preference for those same wing patterns.
Dr Richard Merrill, one of the authors of the paper, based at Ludwig-Maximilians-Universität, Munich, said: "It explains why hybrid butterflies are so rare - there is a strong genetic preference for similar partners which mostly stops inter-species breeding. This genetic structure promotes long-term evolution of new species by reducing intermixing with others."
The paper is one of two published in PLOS Biology today and funded by the European Research Council which brought together ten years of research by Professor Jiggins and his team.
The second study investigated how factors including mate preference act to prevent genetic mixing between the same two species of butterfly. They discovered that despite the rarity of hybrid butterflies - as a result of their reluctance to mate with one another - a surprisingly large amount of DNA has been shared between the species through hybridisation. There has been ten times more sharing between these butterfly species than occurred between Neanderthals and humans.
Dr Simon Martin, one of the authors of the second paper, from the University of Edinburgh, explained: "Over a million years a very small number of hybrids in a generation is enough to significantly reshape the genomes of the these butterflies."
Despite this genetic mixing, the distinct appearance and behaviours of the two species remain intact, and have not become blended. The researchers found that there are many areas of the genome that define each species, and these are maintained by natural selection, which weeds out the foreign genes. In particular, the part of the genome that defines the sex of the butterflies is protected from the effects of inter-species mating.
As with the genetics that control mating behaviour, these genes enable each butterfly type to maintain its distinctiveness and help ensure long-term survival of the species.
But can the findings translate into other species including humans?
Professor Jiggins said: "In terms of behaviour, humans are unique in their capacity for learning and cultural changes but our behaviour is also influenced by our genes. Studies of simpler organisms such as butterflies can shed light on how our own behaviour has evolved. Some of the patterns of gene sharing we see between the butterflies have also been documented in comparisons of the human and Neanderthal genomes, so there is another link to our own evolution."
"Next we would like to know how novel behaviour can arise and what kind of genetic changes you need to alter behaviour. We already know that you can make different wing patterns by editing the genes. These studies suggest that potentially new behaviours could come about by putting different genes together in new combinations."