"A little less conversation and a little more action," he quipped during Thursday's annual Southeastern Conference football media days.
If there is a preverbal college football hot seat, Dooley's butt has been squarely fixed upon it.
In fact, when he took their job, there were some who were already willing to pull the chair out for him.
It's been very rocky on top of Rocky Top.
Whether fair or not, that's the life of an SEC coach. It's a life that Dooley has been familiar with.
His father Vince spent 34 seasons coaching in the conference. The first seven at Auburn and the last 24 at Georgia where he won six SEC championships and a national title.
So Derek Dooley knows the pressures of being compared to a living legend. He also knows how quickly things can change at a school.
Few programs have the rich history and vibrant tradition of Tennessee and few schools have as passionate of fans as Volunteer fans are.
It's those fans who have been put to the test over the past three-plus seasons.
First there was the firing of future Hall of Fame coach Phil Fulmer – a polarizing figure in Knoxville – and then there was the whole Lane Kiffin fiasco.
In essence, Dooley has been behind the eight-ball since he started in 2010.
During his brief time in Knoxville, he's had to deal with depth issues and injuries to several key players including quarterback Tyler Bray who missed five games last season with a fractured thumb.
Then there's the whole 4-12 record against conference opponents including a loss to Kentucky last season that snapped a decades-old winning streak.
If the seat wasn't hot then, it's blazing now.
"It's been a tough four years in Tennessee," Dooley said. "I know, of course, the SEC has enjoyed taking advantage of our tough times. But there's a nice mood on our team right now that you're not going to have Tennessee to kick around anymore."
"We haven't had the seasons that Tennessee has had in the past but we are going to try and change that," Vols quarterback Tyler Bray added.
And it's not just the coach and players who think that.
"I think Derek Dooley has done a really good job," Alabama coach Nick Saban said earlier in the day. "I think he inherited a very difficult situation."
Dooley is one of several coaches to fall from the coaching branches of the Saban tree. He spent time with him at LSU and with the Miami Dolphins before getting his first head coaching job at Louisiana Tech.
"I think Derek has done a lot of things to create a lot of program stability there," Saban added. "I think that Derek is going to be just fine and do a wonderful job in the future at Tennessee."
By now, the hot seat question has been debated over and over again but in the players' minds, it's not the coach who should be feeling the heat.
"I don't think he's on the hot seat," Bray said. "I think the team is on the hot seat.
"I mean, us as players, we haven't gone out and executed the coaches' game plans how we should have and that's why we are losing."
Teammate Herman Lathers agrees with Bray's sentiment adding it's the player's who haven't been holding up their end of the bargain.
"Right now we want to just play ball and put Tennessee where it should be."