Authorities restricted access to the compound to Muslim men aged 50 and over, with no restrictions for women, after Sunday's clashes in which seven Palestinian protesters were injured and three arrested.
Ten masked Palestinians were arrested on Monday for throwing stones at security forces just outside the Old City, police said.
Thousands of officers deployed in and around the area, focusing on the Al-Aqsa mosque compound which is holy to both Muslims and Jews, and the Western Wall, the main Jewish pilgrimage site also known as the Wailing Wall.
Two Arab neighbourhoods adjoining the Old City were sealed off as helicopters and a surveillance drone flew overhead.
"These measures were taken to avert new incidents on the compound and the Old City and to prevent stones being thrown at the Jewish faithful who come to pray at the Western Wall," police spokesman Micky Rosenfeld said.
Tension flared on Sunday after police closed access to the Al-Aqsa compound -- known to Muslims as Al-Haram Al-Sharif (the Noble Sanctuary) and to Jews as the Temple Mount -- saying calls for Muslims to gather there were inciting violence.
Clashes broke out after more than 150 people gathered to pray outside the compound. After the prayers, worshippers threw stones and security forces responded with stun grenades and a water cannon.
Rumours had earlier swept through the Old City that the Israeli authorities would allow right-wing Jewish settlers to enter the compound during the week-long Jewish festival of Sukkot.
On Monday an estimated 30,000 Jewish worshippers prayed at the Western Wall below Al-Aqsa for the Priestly Blessing ceremony, a highlight of the Sukkot celebrations.
Rosenfeld earlier said "hostile elements are inciting to violence," pointing the finger at the Islamic Movement, an Arab-Israeli group that regularly calls the faithful to rally to the defence of Al-Aqsa.
Sheikh Azam Al-Khatib, who heads the Islamic trust that manages the compound, claimed the current tension was caused by "Jewish extremists who provoke the Muslim faithful and don't hide their ambition to kick the Muslims out to build a temple."
The compound was the site of the Second Temple, Judaism's holiest site, which was destroyed by the Romans in 70 AD.
In Gaza, thousands of people joined a protest organised by the Islamist Hamas rulers of the Palestinian enclave, calling for a popular uprising or intifada to defend the mosque compound.
The Palestinian Authority on Sunday urged the international community to "immediately intervene and bring the question of the Al-Aqsa mosque before the UN Security Council."
Jordan summoned Israel's ambassador in Amman on Sunday to demand a halt to "repeated violations" at the compound.
A week ago several people were wounded in unrest that erupted after a group of non-Muslims entered the mosque compound. Police said they were French tourists, but the Palestinians insisted they were Israeli extremists.
The site of the compound is the holiest in Judaism and third holiest in Islam, and has often been a flashpoint for Israeli-Palestinian violence.
The second Palestinian intifada began there in September 2000 after a visit by Ariel Sharon, the right-wing politician who became Israeli prime minister the following year.