Fire officials say they expect to have the Colby fire fully contained by Wednesday after spending days battling the 1,906-acre blaze in the hills above Glendora and Azusa.
The fire was at least 78% contained by Sunday night as the last mandatory evacuation orders were lifted over the weekend, according to the U.S. Forest Service.
The fire, which broke out Thursday morning, has destroyed at least five homes and 11 other structures. At least six other structures have been damaged.
Authorities believe the fire was sparked by an illegal campfire near Colby Trail and Glendora Mountain Road.
Glendora Police Chief Tim Staab said three men -- identified as Clifford Eugene Henry, 22, of Glendora; Jonathan Carl Jarrell, 23, of Irwindale; and Steven Robert Aguirre, 21, who police say is a transient from Los Angeles -- were cold and tossed paper into an illegal campfire when a gust of wind "just blew embers all over the place."
The men, who were in custody, would face federal charges, Staab said.
Steep terrain, dry weather and, at times, fierce winds made it difficult for firefighters to contain the blaze, which grew quickly.
Over the weekend, Forest Service spokeswoman L'Tagna Watson said the steep terrain is the biggest challenge moving forward as crews try to cut lines to keep the blaze within the containment boundaries.