Kelly, who spoke to reporters about the ongoing investigation for the first time Saturday, recounted his initial reaction when Notre Dame Athletic Director Jack Swarbrick told him about the probe on Thursday.
"I care for those four guys deeply," Kelly said. "They're a part of our program, but I have a job to do and I have another hundred players I have to be concerned with. My focus and attention is on continuing to develop our players on a day-to-day basis."
Kelly said he hadn't spoken to the four players but planned to do so at some point this weekend. He also said there was a chance the players could leave Notre Dame before the university completes its investigation.
Notre Dame announced Friday that wide receiver DaVaris Daniels, linebacker Kendall Moore, defensive end Ishaq Williams and cornerback KeiVarae Russell were being held out of practices and games until the university completes its investigation.
Notre Dame has not suspended the players, and they still have access to athletic facilities, Kelly said.
Kelly, who has yet to be interviewed as part of the investigation, brushed aside potential criticism regarding the academic issues that have affected the football program during his tenure at Notre Dame.
"This isn't the time to have a debate on what my leadership or lack thereof is," Kelly said. "My focus is on my football team and getting this team ready. There's a time and place for that. People have their opinions and certainly they're entitled to them. I'm going to focus on this football team."
If Notre Dame determines the misconduct would have made the students ineligible to play in previous seasons, the school said it planned to vacate wins from those seasons.
The Fighting Irish are scheduled to open their 2014 season at home against Rice on Aug. 30.