The Chinese ships were sighted Tuesday about 49 kilometers (30 miles) from the island of Yonaguni, in Japan's Okinawa prefecture (state), according to Japan's Defense Ministry. They were about 200 kilometers (125 miles) from a chain of small islands that have sparked a heated dispute between Japan and China.
The ships were believed to be returning to China after training in the Pacific.
Defense Minister Satoshi Morimoto said Japan is monitoring the ships' movement. Japan considers the area part of its contiguous waters, but it is not illegal for foreign vessels to transit them.
It is not unusual for the Chinese navy to transit waters around Okinawa en route to the Pacific, but they usually go through wider straits. The ships included frigates, a guided missile destroyer, a supply ship and two submarine rescue vessels.
Defense Ministry officials said the ships might have been trying to avoid an approaching typhoon.
China's Defense Ministry said the ships were on a scheduled cruising exercise and were acting in a manner that was "appropriate and legal."
Underscoring China's sharper stance, it also protested the scrambling of a Japanese military plane in the direction of the disputed islands, calling that a "gross violation" of Chinese sovereign rights.
"The Chinese military is closely following the actions of the Japanese side and demands Japan halt all actions complicating or escalating the situation," the ministry said in a short statement on its website.
Japan angered China last month by nationalizing part of the chain of uninhabited East China Sea islands called Senkaku in Japanese and Diaoyu in Chinese. The move sparked violent protests in China.
Chief Cabinet Secretary Osamu Fujimura said Tokyo has urged Beijing to "avoid any actions that would go counter to the mutual benefit."
Nearby Taiwan also claims the islands, which are uninhabited but surrounded by rich fishing grounds and possibly lucrative undersea energy deposits.
China and Japan have recently stepped up naval activities in the area around Okinawa because of the dispute, but there have been no clashes between their warships, which have generally stayed away from the disputed islands themselves.
Wary of missteps that could lead to a sudden escalation of tensions, the countries have instead sent less threatening coast guard ships. Over the past week, however, both have made a point of showing off their naval prowess.
Chinese websites were abuzz Monday with photographs of navy pilots practicing touch-and-go landing exercises on China's first aircraft carrier. It wasn't clear when the pictures were taken, and they did not appear on the Defense Ministry's website or in official media.
The carrier was launched last month without aircraft or an accompanying battle group, and actual flight operations could be years away. But it is widely seen as a symbol of China's ambitions to be a leading Asian naval power, especially as it faces sharpening territorial conflicts with Japan and other countries.
Japan's navy, meanwhile, marked its 60th anniversary with a major exercise on Sunday. Japan also plans to hold a joint exercise with the U.S. military later this year, reportedly using a scenario of taking a remote island back from a foreign intruder.
Asked how China sees the reported scenario, Chinese Foreign Ministry spokesman Hong Lei said, "To maintain the peace and stability of Asia-Pacific is beneficial to all sides." He added: "Increasing tension is against the bigger trends of regional security, peace and the buildup of political and security trust. We reserve the right to take further action."
Defense Minister Morimoto declined to confirm the scenario or give other details.
In Sunday's exercise, about 40 ships—including state-of-the-art destroyers, hovercraft able to launch assaults on rough coastlines and new conventionally powered submarines—took part in Fleet Review 2012, the maritime equivalent of a military parade.
About 30 naval aircraft, mostly helicopters, also participated. For the first time, Japan's navy was joined by warships from the United States, Singapore and Australia. Representatives from more than 20 countries, including China, attended the event staged in waters south of Tokyo.
Associated Press writers Ian Mader in Beijing and Mari Yamaguchi in Tokyo contributed to this report.