Evolution or natural selection was a genetic change in a population over time, from which new species arise. Over millions of years, new traits would arise from natural mutations caused by DNA replication errors or radioactive damage to DNA. Those traits that hampered (or killed) the organism would be selected against. Those traits that gave an advantage to survival would be selected for. Complex traits like reason or emotion could arise from this process, as cooperation was a powerful survival factor.
A common misconception was that evolution always led to more intelligent species, however it only led to species better adapted to their environment - if an environment required an adaptation that was more primitive, then it was actually the more advanced trait for that environment.
Although controversial and illegal in the Federation (for anything other than the prevention of disease) due to the Eugenics Wars, genetic engineering could allow a species to artificially select their genes. Julian Bashir was such a genetically engineered lifeform. (DS9 episode: "Doctor Bashir, I Presume")