A penal colony was a type of prison that separated prisoners from the general populace by placing them in a remote location on a planet or facility.
Originally on Earth penal colonies were established on islands or distant territorial possessions of the various nation states. The first colony on Australia was a penal colony of the United Kingdom. Initially located at Botany Bay, the colony later relocated to Sydney. Despite the adversity, the men of that first Australian penal colony helped to tame the newly discovered continent. Khan Noonien Singh named the DY-100 ship he escaped Earth on SS Botany Bay in remembrance of that colony. (TOS episode: "Space Seed"; TOS novel: The Rise and Fall of Khan Noonien Singh, Volume 2)
Penal colonies were used by the United Federation of Planets to house inmates who were deemed too dangerous to house in other prison facilities. Penal colonies were noted for being lacking in amenities when compared to other penal settlements such as the Penal Settlement in New Zealand on Earth. This was used by authorities to entice criminals to cooperate with them - such individuals were often allowed to serve their sentences in New Zealand instead of a remote prison colony. Alexander Simmons and Galif jav Velk were two men who were allowed to serve their sentences in New Zealand instead of a remote prison colony due to their cooperation with Starfleet.
Some colonies shared worlds with civilian colonies, while other penal colonies were the only settlements on the worlds they were built on. While some were established on Class M planets, others were placed within pressure domes on non-Class M worlds, such as the asylum on Elba II. (DSC novel: Drastic Measures; TOS episode: "Whom Gods Destroy"; ST - The Fall novel: Peaceable Kingdoms)
Jaros II was a penal colony specifically used to house members of Starfleet convicted of crimes. Some instructors at Starfleet Academy would regularly remind cadets of the existence of the colony and how few of those sent there as prisoners were ever allowed to serve in Starfleet again. (TNG episode: "Ensign Ro"; ST - The Fall novel: The Poisoned Chalice)
The Klingon Empire also maintained penal colonies, which made even the most remote penal colonies in the Federation look luxurious by comparison. The most infamous Klingon penal colony was the one on the asteroid Rura Penthe, which became known as The Alien's Graveyard. (TOS movie: The Undiscovered Country)