Jack McInerney, Freddy Adu and Gabriel Gomez also scored for the Union, who led 3-0 at halftime.
The Union move on to host Sporting Kansas City on July 11.
Brian Ombiji and Morgan Langley scored for the City Islanders, who upset the New York Red Bulls in the fourth round. Harrisburg also defeated the New England Revolution on its way to the quarterfinals, marking the fifth time in six seasons it ousted an MLS team in this tournament.
Harrisburg plays in the US Pro Division.
Antoine Hoppenot was taken down in the penalty area in the 68th minute and Pajoy converted the penalty kick to seal the victory for the Union.
Langley's goal in the 54th minute had cut the Philadelphia advantage to 3-2.
The Union played without captain Carlos Valdes. The defender was given a red card in their fourth-round victory against D.C. United.