On the Internet, traffic is easily tracked. Google and Facebook have algorithms that know what users are searching for. Online retailers can monitor what shoppers are buying. Newspapers can see in real-time how many readers are viewing an article.
The same hasn't been true for traffic in the physical world, where data-gathering has been decidedly more low-tech.
Want to know how many people are getting on at a certain bus stop? Hire someone to stand at the stop with a clicker. Tracking how many passengers are on a train? Do an on-board survey. Figuring out traffic congestion? Get a view from a chopper. Or, as Angelenos do every day, just sit in it.
And if that wasn't cumbersome enough, when companies do finally collect that data, they're often overwhelmed by it. Big data, as it's called, gets its name from the sheer quantity of information that pours in when companies start tracking something. Most companies hit a technical ceiling when it comes time to crunch the numbers.
Urban Engines, a Los Altos, Calif., start-up, wants to change that. The 5-year-old company helps transit agencies, delivery companies and taxi services make sense of all that data. The company employs about 50 people with engineering and data science backgrounds to develop an analytics tool that helps its clients process and understand the data they collect.
"There's a lot of low-cost sensors coming on board, and collecting movement data is now a big trend," said Shiva Shivakumar, co-founder and chief executive of Urban Engines. "Our insight is: How do we take some of the ideas we learned from the digital world and apply it to the real world?"
Many large technology companies, such as Uber, Google and Apple, have already figured out ways to cobble together the vast amounts of movement data their drivers, map makers and devices collect, and make sense of it.
For instance, Google and its subsidiary Waze, a mapping app, use big data to tell drivers in real-time which roads are congested and where accidents have occurred. Uber and its competitor Lyft use big data to identify hot spots where there's high demand for rides.
But smaller companies that don't have those technical resources are turning to Urban Engines for help. The company's Space Time Engine is a platform that lets companies and transit agencies plug in whatever data they have and see what they can do with it.
Instead of having information such as trip data and delivery times sitting in a spreadsheet, the Space Time Engine creates real-time visualizations so agencies can see the movement of people, vehicles and packages on a map. It can also make predictions based on past data.
The company counts among its customers the Bay Area Rapid Transit rail service, the Washington Metropolitan Area Transit Authority and Delhivery, a logistics firm in India. The company, which is backed by big-name venture capital firms such as Google Ventures, Andreessen Horowitz and Greylock Partners, declined to reveal how many clients it has or how much money it has raised.
The Washington Metropolitan Area Transit Authority, for example, had for years analyzed tap-in/tap-out data to find out where passengers were getting on and off, with the hope of improving the commuter experience. But the amount of data it collected overwhelmed its old technology systems. The agency had custom tools but Shyam Kannan, its director of planning, said they were "costly and time-consuming."
Urban Engines streamlined the data that the transit agency collected, enabling it to visualize customer travel patterns, quickly identify delays and congestion, and better understand platform crowding and wait times.
"We are now able to get a granular view into our transit system across all stations in 15-minute increments," Kannan said.
In India, Delhivery has a ground staff of more than 12,000 people who process more than 250,000 packages a day across 375 cities.
Although the company collects its own data about each package, it hit a wall when it came to figuring out where packages were being held up.
"Every shipment gets scanned 30 times before it gets delivered, and the scans tell you how the network is doing," said Kapil Bharati, Delhivery's chief technical officer. "By putting this data into Urban Engines, we could see across the country where parcels were getting delayed and stuck, and whether goods were moving on time."
Now, when packages have fallen behind schedule, the company's central team receives alerts in its system and is able to contact ground staff directly.
Twenty years ago, when companies first started dabbling with big data, computer processors could only handle basic information to answer simple questions. Companies such as Wal-Mart Stores Inc. mined transactional data to figure out what people were buying (FYI: beer and diapers were often bought together), but the machines were prohibitively expensive, and even those who could afford them couldn't do a lot with them.
The turning point came three years ago when the cost of GPS sensors fell dramatically because of the proliferation of smartphones that have GPS, gyrometers, accelerometers and even magnetometers (which measure magnetic fields). The cost of transmitting and storing data also went down as cloud computing became the norm.
Fast-forward to today, and most cellphones, buses and trains are equipped with GPS. Everyday objects such as cars and watches are increasingly connected to the Internet. Big data is everywhere.
Which is why breakthroughs in self-driving cars, app-based ride-hailing and on-demand delivery all happened so recently, said Karl Brauer, senior analyst at Kelley Blue Book.
It's not just that this data is now available, companies can now do more with it, whether it's gleaning insights into where and how people travel, anticipating demand and congestion, or even teaching an autonomous vehicle how to drive.
"Big data is going to tell you all kinds of stories, and companies will get an understanding of the subtle and sometimes drastic differences in transportation options," Brauer said.
Transportation experts say this is just the start. As more things become connected to the Internet and the tools to understand big data become more advanced, there could be a day when bus and train systems become nimble and can quickly react to surges in demand, or even predict commuter behavior.