Walker's strong stance is another blow to Kramer as he faces losing his leadership position Tuesday and is under growing pressure to resign the seat he's held since 2007.
"I definitely think the Assembly is right in asking him to step down from his position as a part of their leadership team," Walker said when asked about the situation while in Green Bay.
Walker said he didn't know any details of the case beyond what he's seen in media reports.
"If it's true, I don't think there's any place for someone in a position of public trust to be in office if they've done those things," Walker said.
Kramer, who was elected to the second-most powerful position in the Assembly in September, hasn't commented on the allegations, which surfaced Friday. On Saturday, his staff said Kramer had checked himself into a treatment facility for an unspecified reason.
On Tuesday, Republicans plan to vote on removing Kramer as majority leader and discuss privately whether to fill the position or leave it vacant for the remainder of the year, Assembly caucus chairwoman Rep. Joan Ballweg said.
Kramer's chief of staff, Cameron Sholty, said Monday that Kramer won't attend Tuesday's vote because he's receiving treatment, and again declined to comment about the allegations.
Ballweg, a high-ranking Republican from Markesan, also said Monday she supports removing Kramer from his leadership post.
"Personally, I feel it is wholly inappropriate to have unwanted touching, which is part of the allegations," Ballweg said.
Republicans with direct knowledge of the situation said one of the women involved was a legislative staff member and the other was a lobbyist. The Republicans spoke on the condition of anonymity because they were not authorized by their attorneys to comment publicly about the alleged victims.
Neither the lobbyist nor the staffer in question responded to requests for comment left Monday by The Associated Press.
The vote on Kramer's position is being taken because it's "vitally important" to have someone in that role given that it's the majority leader's job to keep track of where bills stand in the busy final days of the session, said Republican Rep. Dean Kaufert, of Neenah.
"It's a very hectic time for the Legislature, very stressful," Kaufert said Monday. "And it's easy to lose track of a bill or two."
Many high-profile proposals remain for the Legislature to address, including bills that would give lawmakers the power to write academic standards, impose sanctions on failing public and private schools that take voucher students and limit local control on sand mine operations.
But the question over what to do about the majority leader position goes beyond the end of the legislation session. That person also typically plays an important role in helping Republican candidates during an election year by raising money, campaigning and helping with other behind the scenes logistics, Kaufert said.
Republicans currently hold a 60-39 majority in the Assembly. All 99 seats, including Kramer's, are up for election this fall.
Kramer was elected as majority leader to take over for Scott Suder, who left the position early to take a job with Walker's administration.
Kramer is known for his sometimes flamboyant and confrontational style. He has admitted to carrying a concealed handgun on the floor of the Assembly to protect himself and in his previous position presiding over the Assembly was a stickler for the rules, especially those covering spectators in the galleries.
When he was elected majority leader, Rep. Chris Kapenga warned that Kramer had acted inappropriately in the past.
"He was aware that we felt he would be our representative and we expected him to portray himself in a manner that would not embarrass the caucus," Ballweg said Monday, in explaining the effort to remove him as majority leader.
Kramer defeated Rep. Dean Knudson, of Hudson, for the majority leader position.
Knudson issued a statement saying he was not interested in the position and would not run for it now or next year if he wins re-election.
Follow Scott Bauer on Twitter at https://twitter.com/sbauerAP