Anthony Stocks, chairman and professor of anthropology at Idaho State University, responds:
"The evolution of smiles is opaque and, as with many evolutionary accounts of social behavior, fraught with just-soism. Among human babies, however, the 'tooth-baring' smile is associated less with friendship than with fright--which, one might argue, is related to the tooth-baring threats of baboons. On the other hand, a non-toothy, not-so-broad-but-open-lipped smile is associated with pleasure in human infants. Somehow we seem to have taken the fright-threat sort of smile and extended it to strangers as a presumably friendly smile. Maybe it is not as innocent as it seems.
"All cultures recognize a variety of mouth gestures as indexes of inner emotional states. As in our own culture, however, smiles come in many varieties, not all of them interpreted as friendly."
Frank McAndrew, professor of psychology at Knox College in Galesburg, Ill., has done extensive research on facial expressions. He answers as follows:
"Baring one's teeth is not always a threat. In primates, showing the teeth, especially teeth held together, is almost always a sign of submission. The human smile probably has evolved from that.
"In the primate threat, the lips are curled back and the teeth are apart--you are ready to bite. But if the teeth are pressed together and the lips are relaxed, then clearly you are not prepared to do any damage. These displays are combined with other facial features, such as what you do with your eyes, to express a whole range of feelings. In a lot of human smiling, it is something you do in public, but it does not reflect true 'friendly' feelings--think of politicians smiling for photographers.
"What is especially interesting is that you do not have to learn to do any of this--it is preprogrammed behavior. Kids who are born blind never see anybody smile, but they show the same kinds of smiles under the same situations as sighted people."
McAndrew suggests several books that will be of interest to readers seeking more information on this topic:
'Non-Verbal Communication.' Edited by R. A. Hinde. Cambridge University Press, 1972.
'Emotion: A Psychoevolutionary Synthesis.' Robert Plutchik. Harper and Row, 1980.
'Emotion in the Human Face.' Second edition. Edited by Paul Ekman. Cambridge University Press, 1982
Source: https://www.scientificamerican.com/article/how-did-the-smile-become-a-friendly-gesture-in-humans/