But the kid pays little attention to the negativity or the posse ready to run him out of town. He believes he has to be rock-solid confident as a leader, everyone in the McKay Center feeding off the head coach.
"Pete Carroll taught me that," Kiffin says. "I have to learn from the negative but focus on the positive.
"I remember when we lost to Texas in the Rose Bowl [in 2006]. I probably shouldn't say this because my wife will get mad, but it was like I didn't want to live.
"Believe me, I understand the pain, and what a loss means to the USC family. It looks like I don't care, and I get that. But inside I'm suffering just like everyone else.
"When I first got here I met Marv Goux, and I heard about the passion of Marv and Dave Levy. And I've seen Ed Orgeron cry after a loss. But as head coach, I don't believe it's in the best interests of the team if I go in the tank."
He moves on now without his dad, Monte, as defensive overseer and Matt Barkley as quarterback. And though it appears sometimes he's detached emotionally, he talks differently.
"Look at my dad; he's here today, he was here Saturday and Sunday and he doesn't have a job," Kiffin says. "Now you know where I get it.
"But looking at the time we had together, even though it didn't work out, Pops got to spend four years with his grandkids and that doesn't always happen in this profession. No one can take the memories away from my kids.
"We were over to my parents' house Sunday and I watched Dad playing games with my kids to get them eating. It was just so cool. That's what I'll miss."
As for Barkley, he says, "It's hard to care about someone more than I do about him. I think so much of his parents as well. I've bounced parenting things off Les because he and Bev have done such a wonderful job raising kids.
"But I spent so much time with Matt and our receivers, I'm concerned how it affected other players. I could see why they might feel neglected; I need to do better."
During a break in the game he steps out of his office onto the Pat Haden Terrace overlooking the practice field. There is a plaque on the wall dedicated to Haden's accomplishments.
"As you can see, he's always over my shoulder," Kiffin jokes. "I'm so fortunate; it's not your usual coach-athletic director relationship. Look at my phone, we text each other constantly with great give and take. I feel I have tremendous support from both the school's president and Pat, but now it's on me."
He says his job is to concentrate on the big picture and reduce distractions such as the jersey-switch controversy.
"All I want to know is if it's legal, yes it is, so I go back to third-down plays," he says. "I should have said, 'Whoa, it's legal, but what about the Kiffin factor?' It's not worth it.
"I know it's my fault there's a Kiffin factor because I've made mistakes. I wear a hoodie in the biting cold, which has nothing to do with completing a second-down pass, but I get criticized. I get it. Bill Belichick wears one the following Sunday and it's no big deal.
"It's all about winning. That's why I have to figure out how we can go 17-3 over the last two seasons but then lose five of the last six.
"What could I have done better when we were 6-2 and suddenly we can't play for the championship? I don't have the answer, but I will."
With the pizza gone and Saban's legacy certified, Kiffin is asked where he thinks USC will be ranked next season?
He grins. "I know I won't be voting.
"But I also know I owe the USC family something better. And we can do that."
twitter.com/LATimesTJSimers
