Banded mongooses target close female relatives when violently ejecting members from their social groups, University of Exeter scientists have found.
Most animals are less aggressive towards family members, but dominant members of banded mongoose groups target relatives.
The reason for this surprising behaviour is that unrelated mongooses are more likely to fight back - making it more difficult to evict them.
Females are the prime targets because the pups of dominant mongooses are less likely to survive if there are too many females breeding in the group.
"Targeting close relatives for eviction like this is the opposite of what we would expect social animals to do," said lead author Dr Faye Thompson, of the Centre for Ecology and Conservation on the University of Exeter's Penryn Campus in Cornwall.
"Our research shows that related females submit more easily because they are more sensitive to the costs they inflict on their relatives by fighting to stay in the group.
"As dominant banded mongooses need to evict rival females to reduce competition for their own offspring, their best strategy is to target close relatives."
The mass evictions - which are highly violent and often lead to injuries, and sometimes death - result in multiple females being expelled.
In about 50% of cases males are evicted with females.
The trend for targeting related females was only seen in evictions of mongooses that were old enough to defend themselves - supporting the conclusion that relatives are preferentially targeted only when they are capable of resisting eviction.
Senior author Professor Rufus Johnstone, of the University of Cambridge, said: "It seems that aggressive animals can anticipate the possibility of resistance and change their behaviour accordingly.
"This appears to have a big effect on the way they treat relatives and non-relatives, and suggests that latent threats might exert an important influence on social behaviour more generally."
The research, funded by the Natural Environment Research Council and the European Research Council, is the latest finding of an 18-year study of banded mongooses in Uganda.
Professor Michael Cant, of the University of Exeter, who leads the long-term study, said: "We've long wondered why some individuals are marked out for violent attack and eviction, whereas others are permitted to stay.
"Our new study shows that a crucial determinant is whether victims can put up a fight, and predicts that closer kinship sometimes goes hand in hand with more intense aggression."
The paper, published in the journal Proceedings of the National Academy of Sciences, is entitled: "Explaining negative kin discrimination in a cooperative mammal society."