COSTA MESA — Bryan Rice began as a ball boy for the Costa Mesa High boys' basketball program.

He went on to play for the team. Now he's coaching the Mustangs, guiding Costa Mesa to unprecedented feats.

The Mustangs unleashed a relentless defense Wednesday night to set the school record for most wins in a season with 19. It proved to be a special night for Rice and the Mustangs, as they beat St. Monica Catholic, 46-22, in the first round of the CIF Southern Section Division 4A playoffs.

A home game is next for Costa Mesa, as it will play host to top-seeded Pacific Hills of West Hollywood Friday at 7 p.m. Pacific Hills, the Alpha League champion, beat Nipomo, 82-29, and had lost a coin flip for a home game before Wednesday night.

Costa Mesa (19-9) finished third in the Orange Coast League, but it's been a banner-type year for the Mustangs. In addition to the most wins in a season at the school, they won their first playoff game in seven seasons. They also won the Bell, the trophy given to the winner of the rivalry with Estancia, for the first time in five years.

"It's fun," Rice said of getting to 19 wins this season. "I couldn't do it without my coaches. They're all alumni staff. Having a record is great but being a part of this group is more fun than I've had in a long time. It's a credit to their hard work and it's a credit to the players' hard work. Coming from a program where we didn't win a whole lot of games to now and winning the most games in a season in school history is huge for me and the program."

The Mustangs made some noise with their defense against St. Monica (13-15), holding the Mariners of Santa Monica to three points in the second quarter and two points in the third period. The Mariners played without Troy Whiteto (knee), who was one of St. Monica's top players. Rice didn't think he would've made a difference.

Rice continually pushed his players to hustle on defense. With 38 seconds left, he shouted at them, "Hey get a stop." They did.

Before the game began, he reminded them about becoming the team with the most wins in a season at the school. They responded with perhaps their best effort of the season.

Rice sensed this could be a special year for the Mustangs. It is his fourth season with the Mustangs, and he's seen seniors like Jordan Walden improve throughout the years. Rice took over a team that had won one game the season before and a program that had gone through three coaches in as many years.

There have been ups and downs during Rice's four years. The Mustangs won just three games in his second season. But they've improved to achieve a record-breaking year.

"We knew about the record," said Walden, who led Costa Mesa with 10 points. "That was one of our goals this year as a team to break that record. It's a big accomplishment. We are really proud."

Walden provided a scoring punch. Senior post men Jake Comer and Saaeed El Assadi delivered with a strong effort in the rebounding department. Comer pulled down nine boards to go with his six points and El Assadi had seven rebounds and four points.

Senior point guard Nick Dawson also had four points and guided a solid offense. Sophomore Oronde Crenshaw came off the bench to score eight points.

Everyone seemed happy for Costa Mesa. They laughed. A fan for St. Monica provided some comic relief. He danced with a weird outfit on. He wore bright red gym shorts with a T-shirt and a red cape. On each hand was an oven mitt. On his head was a raccoon hat. He also wore advertisement for some apartment rental website.

For the Mustangs fans, he was a joke. But no one was laughing at Costa Mesa. The Mustangs gave St. Monica no chance to come back.

They scored six points in the final 44 seconds of the third quarter to deliver the knockout. Comer scored with 44 seconds left. Crenshaw later scored on a layup with eight seconds left. Then Marcus Hayes stole the ball and beat the buzzer with a layup for the final points of the third and a 35-10 lead

Costa Mesa hopes to bring the same type of effort against Pacific Hills Friday night.

"We're going to be the underdogs, but we like it that," Walden said. "We are going to work hard in practice and hopefully we'll play as good as we did tonight."

steve.virgen@latimes.com