But their cause is gaining support among both mainstream religious Jews and Israel's government. Jewish visits to the politically-sensitive compound are on the rise, and key Israeli lawmakers are lobbying to end a ban on Jewish prayer there.
The matter reached the highest of official levels this month when Israel asked Jordan, which administers Muslim religious affairs at the site, to consider allowing limited Jewish worship there, according to a Jordanian official.
The visits have unnerved Muslim authorities, who fear that Israel is quietly trying to upset a fragile status quo and encroach upon the site. Similar tensions in the past have boiled over into deadly violence.
"If this happens, there will be lot of bloodshed," said Azzam Khatib, director general of the Waqf, Jordan's Islamic authority that manages the Jerusalem holy site, about the possibility of organized Jewish prayers there.
The site, known to Muslims as the Noble Sanctuary and to Jews as the Temple Mount, is ground zero in the territorial and religious conflict between Israel and its Arab neighbors.
Revered as Islam's third holiest spot, the site's iconic gold-topped Dome of the Rock enshrines the rock where Muslims believe the Prophet Mohammad ascended to heaven.
Jews believe the rock may be where the holiest part of the two ancient temples stood about 2,000 years ago -- and where religious Jews pray a third temple will one day be built.
The site is so holy that Jews have traditionally refrained from praying on the hilltop, congregating instead at the adjacent Western Wall. In recent weeks, Israel's chief rabbis, as well as the rabbi of the Western Wall, have issued directives urging people not to ascend the Temple Mount, arguing that the temple's former location on the mount is unclear and that Jews at the site could inadvertently enter the holiest area of the once-standing temple, where it was forbidden to tread.
Attitudes among Orthodox Jews have been evolving, however, as archaeologists have weighed in about the precise location of the ancient temples -- and of places where Jews would be allowed to tread.
According to figures the Israeli newspaper Makor Rishon said it obtained from the police, Jewish visits to the holy site have jumped from about 5,700 in 2009 to some 8,300 in 2011. Last year, the number dropped slightly to about 7,800 and this year rose to nearly 8,000.
The Temple Institute, an organization that has led efforts to allow Jewish prayer at the site, attributes the drop in 2012 and this year to police closures during religious holidays when many Jews usually visit. There was no immediate comment from the police to confirm the statistics.
In one of the strangest security measures in the Holy Land, visitors identified as Jews receive police escorts and are banned from praying.
Rabbi Chaim Richman of the Temple Institute, a group that has for years been advocating for Jewish prayer at the holy site, said police often harass and remove Jews who ascend the mount and recite prayers. He called for Jewish religious freedom at the site.
"I'm asking for the right to move my lips," Richman said.
Israeli officials declined comment on the matter, but an aide to Israel's deputy minister of religious affairs said the religious affairs ministry has drafted a proposal that would allow for limited Jewish prayer at the site.
"We see great importance to allow equality in freedom of religion" at the site, said the Idit Druyan, the aide to Deputy Minister Eli Ben-Dahan. "There is no reason why one religion is allowed and another religion is not."
Muslims at the site have protested in recent weeks over what they call Jewish encroachment, and Muslim clerics warn against allowing separate hours for Jewish and Muslim prayer at the site, an arrangement that exists at a Hebron holy site and that Druyan said the religious affairs ministry has considered.
A Jordanian official said Israel asked Jordan this year to consider allowing limited amount of Jews to pray in a small area at the site, according to a Jordanian official. The Israeli request was rebuffed, said the official, who spoke on condition of anonymity because he was discussing a classified diplomatic matter.
The Jordanian official said Jordanian King Abdullah II has asked Israeli Prime Minister Benjamin Netanyahu several times, most recently two weeks ago, to prevent Jewish worship at the compound, warning that such a move would provoke Muslims and rekindle anti-Israel sentiment around the Muslim world.
Israeli officials declined to comment on those talks, citing the extreme sensitivity of the matter.
Israel captured the site, located in the walled Old City, and the rest of east Jerusalem from Jordan in the 1967 Mideast war. An Israeli flag was enthusiastically raised on the peak of the Dome of the Rock and a military commander announced what later became an iconic catchphrase: "The Temple Mount is in our hands."
But the flag was quickly taken down, and Israel conceded administrative control of the compound to the Waqf, the Islamic religious body in Jerusalem, to keep the peace. The Western Wall, a remnant of the retaining wall of the ancient second Jewish temple just below the compound, was cemented as Israel's central Jewish pilgrimage site.
When hawkish former Prime Minister Ariel Sharon ascended the mount in 2000 in a demonstration of Israeli sovereignty over the site, it helped trigger the second Intifada, or Palestinian uprising. More than 3,000 Palestinians and 1,000 Israelis were killed in several years of fighting.
The Palestinians claim east Jerusalem, and the Old City, as the capital of their future state. The status of the site remains perhaps the most sensitive and explosive issue in U.S.-brokered peace talks.
Despite the tensions, the Jewish visits continue. Early one morning last week, during limited hours open for tourist visits, eight Orthodox Jews ascended the compound in groups of one and two.
One Israeli, an immigrant from St. Louis, Missouri, walked barefoot despite the morning winter chill, as Jewish pilgrims did millennia ago.
An Israeli police officer and a Waqf representative shadowed the Jewish visitors to ensure they did not break the ban on Jewish prayer. When their minders were distracted, the visitors murmured psalms.
The Jewish visitors said they often resort to tricks to circumvent the police-enforced ban on prayer. Some pretend to talk on the phone but recite prayers instead. Others secretly prostate themselves in prayer while bending down to observe shrubbery, pick up dropped keys or take a photo low to the ground.
Pinchas Rosenfelder, a 44-year-old Toronto native who moved to Israel, said his monthly visits to the site were to uphold Israeli sovereignty of the site.
"If you're not in a place, you lose it," Rosenfelder said. "The lack of a Jewish presence here is not a good thing."
Rabbi David Rosen, an Israeli interfaith activist, said Jews should be allowed to pray at the site in the framework of an Israeli-Palestinian peace deal, but allowing Jewish prayer before peace was reached could torpedo current negotiations.
"The fact something is a right doesn't mean you have to exercise it," Rosen said about Jewish prayer at the site. "It doesn't take much to light this tinderbox."
Associated Press writer Jamal Halaby contributed to this story from Amman, Jordan.
Follow Daniel Estrin at twitter.com/danielestrin