The A police official said on Tuesday that Saffron-robed Buddhist monks led a mob that broke down gates and entered the walled multi-storied compound near Sri Lanka’s capital, Colombo.
"We have pushed back the mob and the refugees will be relocated in a safer place," AFP quoted the unnamed official as saying.
The police official said the refugees were taken into "protective custody" and had been brought back to their safe house when the mob returned and started throwing stones.
The official added that they were authorized to remain in Sri Lanka pending processing of their papers. The Rohingya were eventually to be resettled in a third country.
Meanwhile, a monk who stormed into the building was filmed by his radical Sinhalese National Force as he urged others to join him and smash the premises.
"These are Rohingya terrorists who killed Buddhist monks in Myanmar," the monk said in a message posted on Facebook, pointing to Rohingya mothers with small children in their arms.
In May, over 30 Rohingya refugees were rescued by the Sri Lankan navy after they were found drifting in a boat off the island's northern waters.
Sri Lanka's extremist Buddhist monks have close links with their ultra-nationalist counterparts in Myanmar. Both have been accused of orchestrating violence against minority Muslims in the two countries.