Public broadcaster NHK showed rescue workers suspended by ropes from police helicopters, pulling victims from the rubble. Others gingerly climbed into windows as they searched for survivors in crushed homes.
Hillsides caved in or were swept down into residential areas in at least five valleys in the suburbs of the western Japanese city after heavy rains left slopes unstable.
Hiroshima prefectural police said 36 people were confirmed dead and at least seven others were missing as of Wednesday night. The Fire and Disaster Management Agency said 15 people were injured, two seriously.
"A few people were washed away and it is hard to know exactly how many are unaccounted for," said local government official Nakatoshi Okamoto, noting that conditions in the disaster area were hindering rescuers.
Authorities issued warnings that additional rain could trigger more landslides and flooding.
The land collapsed so quickly that evacuation advisories came an hour after the first mudslide, officials acknowledged.
"It's so regrettable," Kyodo News service quoted Hiroshima Mayor Kazumi Matsui as saying. "We'll find out what went wrong and take the necessary measures."
Landslides are a constant risk in mountainous, crowded Japan, where many homes are built on or near steep slopes. Torrential rains in the early morning apparently caused slopes to collapse in an area where many of the buildings were newly constructed.
Damage from land and mudslides has increased over the last few decades due to more frequent heavy rains, despite extensive work on stabilizing slopes. In the last decade there have been nearly 1,200 landslides a year, according to the land ministry, up from an average of about 770 a year in the previous decade.