Eugenics is the process of selectively breeding traits into or out of species. This can be achieved by various means, through successive generations by selective breeding or more immediately via genetic engineering.
Eugenics via genetic engineering to sentient life-forms was banned in the United Federation of Planets due to Earth's experience in its Eugenics Wars in the late 20th century, when genetically augmented humans warred over the planet in attempts to gain complete control. (TOS - Khan comic: "Issue 1"; DS9 episode: "Doctor Bashir, I Presume"; TOS episode: "Space Seed"; et al.)
The ancient Vulcans in their early years were known to employ a primitive form of eugenics in order to promote more valuable psionic as well as other talents into one's clan through marriages that were carefully planned by the elders of the House. (TOS novel: Spock's World)
In 2254, when a lost colony of violent and emotional Vulcans was discovered by the USS Enterprise, Spock described the disruptive effect of returning the inhabitants to Vulcan as being comparable to a lost colony of Khan Noonien Singh's Augment warriors arriving at Earth. (EV comic: "Cloak and Dagger")
The Romulans' attempt at eugenics resulted in the "forgotten" children on Hellguard. Years later, the Romulan Star Empire attempted to "correct" their mistakes by having the survivors killed. Their plans were thwarted by Saavik. (TOS novel: Unspoken Truth)
The Hortas of Janus VI employed selective breeding to steadily improve themselves by choosing the best of their generation to be the sole mother of the next generation of Hortas, resulting in their species being remarkably civilized and intelligent. (TOS novel: The Rise and Fall of Khan Noonien Singh, Volume 1)