Red matter was a substance that was created through the processing of decalithium. The formulation of red matter was discovered by the Vulcan Science Academy in the 24th century, and took the form of a red liquid substance that collected in the form of a sphere in a null environment. A single drop of this matter was capable of creating a singularity.
History and specifics
By the year 2387, the Vulcan government had developed the ability to produce red matter. At the time, the Hobus star had grown increasingly unstable and was expected to produce a powerful shockwave that would destroy the Romulan homeworld of Romulus. The only way to stop the destruction of the star was by using red matter to collapse it into a singularity. The Romulan Star Empire briefly considered invading Vulcan in order to acquire the technology which the Vulcans refused to share. Ambassador Spock was able to get enough red matter manufactured which was deployed on the Starfleet ship Jellyfish. However, Spock was too late as the supernova from Hobus destroyed Romulus and extinguished the lives of millions of Romulans. Before the nova could progress further, Spock shot a portion of the red matter into the Hobus star collapsing it into a singularity. (ST comics: "Countdown", "Number One", "Number Four")
This incident led to the Narada and the Jellyfish falling into the singularity where they went back in time leading to the creation of an alternate reality.
During this time, Nero attempted to gain revenge against Spock and sought his emergence from the rift in time where he intercepted the Jellyfish and gained access to red matter. With it, he deployed some of the substance against Vulcan leading to the destruction of the planet and near extinction of the Vulcan species. The crew of the USS Enterprise was later successful in destroying the Narada before it could deploy a red matter weapon against Earth. (TOS movie, novelization & comic adaptation: Star Trek)
Following the Narada's destruction, the Vulcan Ruling Council sent a scout ship to explore what was left of Vulcan from its destruction and recover fragments that may have survived. Among the fragments that survived were microscopic red matter particles that survived Vulcan's destruction. A group of Vulcans under Ambassador Sarek posing as Narada survivors were able to steal the scout ship and the red matter. They planned to give the Imperial Senate the red matter and destroy Romulus.
However, the USS Enterprise under Captain James T. Kirk took a shuttle into the Neutral Zone. After Kirk and his party were taken to Romulus and the Senate, Sarek and Varik stopped Tevok from detonating the red matter. The Romulans, in exchange for the red matter and the Narada's schematics, allowed Kirk, his party and Sarek to leave. Admiral Pike did not like the idea of leaving the red matter in the Romulans's hands. (TOS - Vulcan's Vengeance comics: "Part 1", "Part 2")
On Romulus, the red matter started to decay rapidly. Romulan senators Lucian and Vicia knew they could use the red matter to destroy the surviving Vulcans at Ceti Alpha V. The senators asked Narada survivors Arix and Vella to man ship loaded with the red matter on a suicide mission to destroy every last Vulcan. However, Arix decided to give up this plan and detonated the red matter which took his and Vella's life. (TOS - Legacy of Spock comics: "Part 2", "Part 3", "Part 4")
Despite that setback, the Romulans still had red matter left and agreed to use it against the Klingons with Section 31's assistance to destroy their First City's Great Hall. As Romulan troops prepared to detonate the red matter, Section 31 under their Agent, Lt. Yuki Sulu gunned them down. As Sulu prepared to detonate the red matter, she was convinced to stop this by Captain Kirk because two things: making the Klingons angrier and that she was the same as her brother, Hikaru. Section 31 decided to take the red matter as "nuclear deterrent" and use it to make others fear the same fate as Vulcan. The red matter disappeared with the Section 31 ships. (TOS - The Khitomer Conflict comics: "Part 3", "Part 4")