But for several years, it was Jefferson and his son. Football was inevitable.
"He saw a sign in the park for football sign-ups one day when he was 8," Jefferson said. "He was just this little bitty guy out there."
Jefferson grew into a quality defensive back at Los Angeles Cathedral High. In a 2009 game, he was a match for Gardena Serra receivers Marqise Lee and Robert Woods, both future USC stars. Cathedral lost in overtime.
Oregon wanted him and Jefferson committed. Then Clyde Jefferson was diagnosed with cancer.
"Anthony and his grandfather are very close," said Anthony Jefferson Sr., whose father is now a cancer survivor. "He told me, 'I've got to stay with Granddad so he can see me play.' "
It was a coup for the Bruins.
Jefferson was receiving significant playing time as a freshman in 2010. The opening kickoff against California in the sixth game altered his season.
"I came down and hit a guy and my foot got stuck in the ground," Jefferson said. "The bone just snapped."
Jefferson had surgery and was out for six months. On his first day back, he felt a twinge in his back while lifting weights. It grew worse as the days passed.
"I started feeling it down my leg," Jefferson said. "I found out I had a herniated disk. I needed more surgery."
The injuries lingered in his head after he returned in 2012.
"There were a few times, felt like 'I can go, I can go,'" Jefferson said. "In the back of my mind I knew this wasn't how I used to play. There was never a point where I disregarded my abilities, but it took a toll on my mental outlook."
His father suffered as well.
"I think he lacked confidence," the elder Jefferson said. "He was healed, but he wasn't 100% healed."
That came last spring.
Jefferson again matched steps with receivers. The swagger returned in summer, when he was moved from cornerback to safety. It was a hard-edged process at times. He and Evans scrapped during a practice in August.
"He gets a little hot under the collar sometimes," defensive backs coach Demetrice Martin said. "You like that in a defensive back. But it's a fine line. You have to control it. I've seen maturity and confidence this season."
Opposing teams have seen it too.
Jefferson already had coverage skills. What he has shown is the ability to be a hammer against the run. He was second on the team with nine tackles against Nebraska. He was second with 14 against Stanford.
"I see a different player than I saw when I first got here," Mora said.
That's because Jefferson is again the player he once was.
"I'm blessed," he said. "I came in with all the guys who are now seniors. To get to share the field with them one last season is huge for me. Things are working out."