The recommendation on the fate of the Welfenschatz, or Guelph Treasure, by a decade-old commission created to help resolve restitution claims isn't binding but carries strong moral weight.
The collection includes silver and gold crucifixes, altars, intricate silverwork and other relics. Some experts have estimated its current value at between 180 and 200 million euros ($248 and $276 million.
The heirs maintained that their ancestors had no choice but to sell the Christian artifacts in 1935 to the Nazi government for less than their value.
The foundation that oversees Berlin's museums said the collectors weren't forced to sell the treasures, arguing among other things that the collection was not even in Germany at the time of its sale.
In its recommendation, the commission wrote that, after thoroughly investigating the sale process, it came to the conclusion that it was not a "forced sale due to persecution." It said it can "not recommend the return of the Welfenschatz to the heirs of the four art dealers and other possible former co-owners."
The president of the museum foundation, Herrmann Parzinger, welcomed the panel's conclusion and praised it as a "thorough recommendation ... that considers all the facts." Representatives of the heirs weren't immediately available for comment.
Germany's culture minister, Monika Gruetters, said she hopes the Jewish heirs will accept the recommendation.
She said it "does not change ... the fact that the German government will continue to do everything to shed light on to the Nazis' art thefts and, when in doubt, will press for restitution."
The collection, which has been on display in Berlin since the early 1960s and is currently at the city's Bode Museum, is considered the largest collection of German church treasure in public hands.