The Upper Town functioned as the citadel and as the acropolis of the city. It is believed that the citadel was built as the first construction in 2000 BC. Some early sources mention the existence of ruins of structures such as hippodrome, tower, bath and palace. The citadel underwent various modifications in the history. The walls of the Upper Fortress are higher than of the other parts. It is fortified in the south with higher and thicker walls and towers. During the Roman period an aqueduct was built to supply the Upper Town with a freshwater source. The Upper Town was accessible through a double gate with the Middle Town. The imperial palace of the Empire of Trebizond was located in the Upper Town. After the surrender of the city to Ottoman forces a small mosque was built next to the palace. Some epigraphs from Ottoman era, which were found between the citadel walls, can be seen in Trabzon Museum.
The Middle Town, which was built by Alexios II of Trebizond (reigned 1297–1330), is the continuation of the upper and lower parts. It has no regular form. Its two gates, Imaret Kapı (literally: Alms Kitchen Gate) and Zağanos Kapı (Zaganos Pasha Gate) are situated in the west. It has two more gates at other sides, Tabakhane Kapısı (Tannery Gate) and Kule Kapı (Tower Gate). Notable buildings found here are Orta Hisar Mosque (Panagia Chrysocephalos Church), Governor’s mansion, Zağanos Bridge, Kule Hamamı (Tower Bath), Çifte Hamma (Twin Bath), Amasya Mosque, Şirin Hatun Mosque and Musa Pasha Mosque.
The Lower Town stretches in the west from Zağanos Tower down to the sea. Also this part of the fortifications were built by Alexios II of Trebizond. However, an inscription with the tughra of Ottoman sultan Mehmed II (reigned 1444–46 and 1451–81) is situated above the gate in the Moloz Tabya (literally: Rubble Bastion). In the east, there are two gates, Pazarkapı (Marketplace Gate) and Mumhane Kapı (Chandlery Gate). Historical buildings around the Lower Fortress are Molla Siyah Mosque (St. Andreas Church), Hoca Halil Mosque, Pazarkapı Mosque, Kundupoğlu and Yarımbıyıkoğlu manions, Sekiz Düzenli Hamam (bath), Tophane Hamamı (bath), Hacı Arif Hamamı (bath) and İskender Pasha fountains.
The most of the city walls are still standing and are among the city’s oldest buildings. In fact, their oldest part can be dated back to 1st century AD during the Roman Empire era. Historical sources provide information about older stages of their construction. Xenophon, who visited the city in 5th century BC also mentioned the existence of city walls.
In 1921, some of the remaining Christians of the city were ordered by the municipal authorities to dismantle the stone of the Comnenos era fortress and use the rubble for road building.
During the 20th century the valleys on both the western and eastern sides of the middle and upper towns were built up with illegal constructions, in some cases obstructing the view of the walls. In the last two decades the municipality of Trabzon has initiated a series of redevelopment projects surrounding the old town. As of 2017 the western Zagnos valley has been turned into a park with unobstructed view of the walls and aqueduct, and the illegal constructions in the eastern Kuzgun (or Tabakhane) valley have been torn down. Kuzgun valley will eventually also be turned into a park