The ruling could help put to rest lingering animosities between the Balkan neighbors.
Fighting in Croatia from 1991-95 left around 10,000 people dead and forced millions from their homes.
Tuesday's decision was not unexpected, as the U.N.'s Yugoslav war crimes tribunal, a separate court also based in The Hague, has never charged any Serbs or Croats with genocide in one another's territory.
Croatia brought the case to the world court in 1999, asking judges to order Belgrade to pay compensation. Serbia later filed a counterclaim, alleging genocide by Croat forces during the 1995 "Operation Storm" military campaign.
Rejecting both cases, court President Peter Tomka stressed that many crimes happened during fighting between Serbia and Croatia and urged Belgrade and Zagreb to work together toward a lasting reconciliation.
"The court encourages the parties to continue their cooperation with a view to offering appropriate reparation to the victims of such violations," Tomka told a packed Great Hall of Justice at the court's Hague headquarters, the Peace Palace.
Decisions by the International Court of Justice are final and legally binding.
Tomka said crimes including killings and mass expulsions by both sides constituted elements of the crime of genocide, but the judges ruled that neither Serbia nor Croatia carried out the crimes with the "specific intent" to destroy targeted populations.
Serbian Justice Minister Nikola Selkovic said the decision would usher in "a much better page in our bilateral relations," but his Croatian counterpart, Orsat Miljenic, was not so sure and urged Belgrade to do more to prosecute suspected war criminals.
"I don't see it as any changing or breakthrough moment," Miljenic told reporters at the court. "We are neighbors. We have to cooperate in as many areas as possible."
The case brought by Croatia was not the first time Serbia had faced allegations of genocide at the world court.