The area was first inhabited in prehistoric times, 30,000 years ago by cave dwellers on Montgó. Subsequent residents have included Stone and Bronze-age peoples,Romans, Greeks,  Phoenicians, Visigoths, Germanic, Carthaginians, and Moors. Roman fishing boats used the port, and there is evidence that dates the Roman occupation of Xàbia to the 2nd century BC, it makes Xàbia the oldest known Roman site on the coast with a commercial port for fish and minerals.

In the 6th century AD. Christian Visigoth monks, came to Xàbia and founded the monastery of Sant Martí, which likely gave its name to the cape in the area named Cap San Martí. Hermenegild, son of the Visigoth king Leovigild of Toledo, sought refuge in the Monastery after angering his father by marrying a Christian girl. When his father's troops arrived to arrest him all but one monk fled to Portichol; Hermenegild and the old monk were killed.Several people with Visigoth names still live in the area.

There is little left of the Moors other than some inscribed gravestones and ceramics, although they were here from about 714 AD until they were expelled in 1609. The hermitage of Popol, dates to the 14th century.