The NBA filed formal allegations Monday and set a June 3 hearing in its attempt to oust Sterling from the team he has owned for 33 years.
The league said Sterling's conduct "damaged and continues to damage the NBA and its teams."
That started the process to terminate Sterling's ownership of the team.
Sterling has until Tuesday to respond to the charges.
"The time is laid out in the NBA constitution," Silver said during a news conference Tuesday from New York. "We're following it to the letter, in terms of numbers of days that Mr. Sterling has to respond and when the hearing will be held. And as I said, I know we're doing the right thing here. This is an unprecedented proceeding. Will there be bumps in the road? Presumably, yes."
Silver also said the NBA has informed Sterling's lawyer, Maxwell Blecher, that Sterling will not be given three months to respond to the charges.
Silver said Sterling's wife, Shelly, who owns 50% of the Clippers in a family trust and has said that she will fight to keep her share of the team, wouldn't be allowed to keep the team as well.
Silver said that when the proceedings take place to strip Donald of the team, if the required three-fourths of the NBA owners vote to terminate him, "all ownership interests are terminated as part of that proceeding."
Silver said that despite Shelly's lawyers saying she can't be terminated, that "it just doesn't make sense."
"Our position is under the constitution, based on Mr. Sterling's conduct, if the owners ultimately decide that it is appropriate to terminate his franchise, the interest of all owners is terminated," Silver said.
Sterling's downfall began April 25 when website TMZ released an audio recording in which Sterling told a female friend he didn't want her to associate with African Americans.
In that audio, Sterling made disparaging remarks about blacks, including
The NBA then banned Sterling for life, fined him $2.5 million and has been seeking to take his team away so it can be sold.
"Mr. Sterling still owns the Los Angeles Clippers," Silver said. "Mrs. Sterling, as I understand it through a trust, owns 50% of the team as well.