The company also said it expected a key revenue measure to increase from 3% to 5% in the current quarter, which would mark the second straight quarter of gains in more than two years.
Investors were encouraged by the results, pushing shares up nearly 5%, or 29 cents, to $6.25 in aftermarket trading. Shares had risen 33 cents to $5.96 before the release of earnings data.
The results, which cover the crucial holiday shopping season, show signs that Penney is making strides in recovering from a botched transformation plan by its former chief executive.
Ullman is trying to win back shoppers by restoring the sales events and basic merchandise that the company ditched under Johnson in a bid to attract younger, wealthier consumers. Earlier last month, Penney announced thousands of job cuts and the closure of 33 stores as it attempts to get back on the path of profitability.
"J.C. Penney achieved what it set out to do on a number of important fronts in 2013," Ullman said in a statement. "We stabilized our business, both financially and operationally."
But he acknowledged there was more work to do.
Shares are down 86% from the $43 reached in early February 2012 when investor enthusiasm was high amid hopes for Johnson's plan. The stock is down 32% so far this year.
Penney said Wednesday that it earned $35 million, or 11 cents per share, in the three-month period that ended Feb. 1. That compares with a massive loss of $552 million, or $2.51 per share, a year earlier.
Revenue slipped 2.6% to $3.78 billion.
Excluding a tax benefit and other items, Penney had a loss of $206 million, or 68 cents per share. Analysts had expected a loss of 81 cents on revenue of $3.84 billion.
The company said revenue at stores open at least a year, a key measure of retail health, was up 2% in the fourth quarter. That's compared with a 31.7% plunge for the holiday quarter a year earlier.
In the latest year, Penney recorded a loss of $1.39 billion, or $5.57 per share, while revenue dropped 8.7% to $11.86 billion as Johnson's legacy continues to cast a shadow on its results.
Penney recorded a nearly $1-billion loss as revenue dropped 25% to $12.9 billion for the year that ended Feb. 2, 2013, the company's first year of Johnson's failed transformation plan.