The Belt and Road project has come under fire for saddling poor nations with debt through big projects that are not economically viable. China rejects the criticism.
Japanese participation could help the initiative's image and allay fears of debtor nations, officials say.
But Japanese defence officials are wary of its military implications, and Tokyo is pushing its Free and Open Pacific Strategy to promote trade and infrastructure in Asia, Africa and the Middle East.
Japan also wants to ensure any joint projects with China are transparent, open and fiscally sound, officials said.
"We're ready to discuss tangible cooperation in third countries, but ... we think we do not need to label this cooperation with some 'initiative'," a Japanese foreign ministry official said.
In a symbol of China's economic rise, Japanese Foreign Minister Taro Kono said this week Japan was ending its development assistance to China, after halting the bulk of aid more than a decade ago. Instead, they will seek ways to help others.
Despite the thaw, mistrust persists.
War-time history still rankles, with China often complaining that Japan has not fully atoned for its occupation of parts of China before and during World War Two.
Japan is wary of China's military spending and its dominance of the South China Sea, through which much of Japan's trade flows.
A recent survey showed 86 percent of Japanese had a “bad image” of China.
"Abe will try to develop relations," said Akio Takahara, a China specialist at the University of Tokyo.
"But at the end of the day, if strategic targets are different, we won't be able to establish a stable relationship."
