But on "The Late Show" on Monday, Colbert paid heartfelt tribute to Scalia, who was appointed by President Reagan in 1986 and became the court's most outspoken conservative voice, known for his fiery rhetoric and scathing dissents.
Colbert praised Scalia as an "intellectual giant" with a "great sense of humor."
"He told more jokes and got more laughs than any of the other justices," Colbert said, suggesting that Scalia's wit partly explained his friendships with liberal peers such as Ruth Bader Ginsburg and Elena Kagan.
To illustrate the point, Colbert recalled a brief but memorable encounter with Scalia at the White House Correspondents Dinner in 2006 — the year Colbert mercilessly roasted President George W. Bush in a now-infamous performance.
"Not a lot of people laughed in the front row," said Colbert, describing the awkward mood in the room after his remarks. "No one was even making eye contact with me."
The one exception was Scalia. The justice had recently made headlines for flicking his chin at photographers, inspiring Colbert to make some rude gestures from the podium — much to Scalia's delight.
"After it was over, he came up to me — again, no one's talking to me in the whole damn room — and Antonin Scalia comes up to me and says, 'It's great, it's great. Did you give me one of these?'" Colbert said, flicking his teeth with his thumb.
Assured it would be legal to make such a gesture, Colbert said, "Yeah, I gave you one of these."
"He goes, 'Great stuff, great stuff. Goodnight.' And then he left. And I watched him go and I thought, 'Don't you make me love you, old man,' " Colbert remembered. "So I will forever be grateful for that moment of human contact that he gave me. So I would just like to say, one last time: Justice Scalia, I salute you."
And with that, he flicked his teeth.