The small corps of mariners who guide ships in and out of the Port of Los Angeles are by far the best-paid city employees, averaging nearly $450,000 in salary and bonuses last year.
But unlike pilots in other major West Coast ports, the pilots in Los Angeles have a union contract that boosts their incomes in unusual ways, including a clause that guarantees them at least four hours of overtime pay if they are called back to work after their shift ends.
Such generous provisions are meant to help managers deal with the unexpected staffing shortages, but for L.A.'s port pilots, callback pay has become a routine, and significant, bonus.
From July 2015 through June 2016, L.A.'s 11 full-time pilots were called back after 81% of their regular shifts, a Times analysis of city payroll data found.
Pilot Ed Royles was called back after 90% of his shifts, boosting his pay by more than $88,000, the data show. Pilot Brett McDaniel was called back an incredible 150% of the time – he was called back on more occasions than he worked at regular pay -- boosting his compensation by $125,000.
Port piloting consists of boating out to ships waiting to enter the harbor, climbing a rope ladder to the deck and then standing on the bridge to provide local knowledge, including advice on currents and navigational hazards, as the vessel is steered through the congested waterway to the dock. The process is reversed when ships leave the harbor.
Either way, it typically takes about two hours in the Port of Los Angeles, from the time the pilot leaves the station, on a pier at the end Signal Street in San Pedro, to the time he returns, port records show.
The work schedule for pilots in Los Angeles contrasts sharply with those at other West Coast ports.
Pilots in San Francisco, Seattle and the neighboring port of Long Beach are not government employees and they don't work on an hourly basis. Instead, they're on call for a week or two at a time, supervising ships in rotating order. The routine provides continuity for the ports, according to industry officials, and the predictability of large ship movements ensures there's time for pilots to eat, sleep and relax between assignments.
Los Angeles, on the other hand, has pilots work 12-hour shifts that change during the peak shipping traffic hours, virtually guaranteeing that some pilots will be required to work overtime.
The schedule has been in the pilots' union contract for at least three decades, said Port of Los Angeles spokesman Phillip Sanfield, so port officials don't know who asked for it, or why. But it works well, Sanfield said, because it means there are plenty of pilots at the port, on and off duty, to handle traffic during the peak hours.
The callback provision has been around just as long, Sanfield said, and its origin is also a mystery.
The effect, however, is that Los Angeles pilots have received more than $5.3 million for callbacks in the last decade, according to city payroll data.
Callback pay is designed to give employers a way to handle unusual, unanticipated staff shortages brought on by emergencies, such as essential equipment breaking down or a key employee calling in sick, said Al Latham, an attorney who teaches labor law at USC.
It's meant to compensate the employee for the inconvenience of abandoning plans he had for his time off in order to race to work and help with the unexpected. Setting a minimum number of overtime hours for such occasions guarantees the employee that his effort will be worthwhile, even if the problem passes quickly.
But widespread, routine use of callback pay indicates "something is seriously wrong, there's very poor planning," Latham said. "Any normal employer would try to contain those costs" by expanding the workforce or scheduling more people to work during the busiest periods, Latham said.
Port administrators have done neither of those things, allowing most pilots to collect hundreds of thousands of dollars in callback pay in the last decade, according to city payroll records.
In fact, the city was slow to hire replacements after several pilots retired in 2014 and 2015, creating a spike in the number of callbacks. Trainees have been hired, but since they are not allowed to guide ships on their own for 18 to 24 months, it will take time for the number of callbacks to subside, Sanfield said.
"The city has not properly anticipated its needs and only recently has come to grips with a real manpower shortage," said Royles, the pilots union president, who described the extra hours as "mentally exhausting."
But even when the port had its usual complement of 15 pilots in 2010, they were called back after more than a third of their regular shifts, payroll records show.
The high rate of callbacks among L.A.'s pilots also surprises maritime experts because the movement of large ships is so carefully orchestrated, leaving little to chance. Vessel owners have to inform the port of their arrival date and time at least four days in advance, under U.S. Coast Guard rules.
On the day ships arrive, vessel owners want to get them to the dock as early as possible so the longshoremen can begin work. Likewise, vessel owners are eager to shove off for the next port as soon as the dock hands are finished.
"There's a morning rush in and an evening rush out, it's pronounced and very predictable," said Kip Louttit, executive director of the Marine Exchange of Southern California, a nonprofit that monitors ship traffic at the port.
Instead of having more pilots working at regular pay during those peak hours, and fewer in the middle of the night when traffic is light, the L.A. port pilots' contract calls for the same number of pilots to be on duty at all times, and for the shifts to change at 5:30 a.m. and 5:30 p.m., in the middle of the rush hours.
At the Port of Long Beach, pilots are employed by Jacobsen Pilot Services, a family-owned company the city has contracted with for nearly a century.
The firm's president, Tom Jacobsen, did not respond to requests for comment for this story. But a source familiar with the company's operation said Jacobsen pilots are on call for a week at a time, followed by a week off. When the shifts change, it's at 2 a.m., to avoid disruption during the peak traffic hours.
Ports in San Francisco and Seattle, where pilots are independent contractors, follow similar schedules. They offer time off with pay to pilots who work unscheduled shifts, but they do not pay overtime, said Mike Jacob, an attorney for the Pacific Merchant Shipping Assn. Pilots in those ports split the fees paid by shipping companies for their services, much like partners in a law firm divide payments from clients. Last year, pilots in San Francisco made about $430,000 each, and pilots in Seattle made about $360,000, according to industry officials.
In the Port of L.A., pilots are treated like clock-punching hourly employees, despite their high pay. In a typical two-week pay period, they work four 12-hour days, followed by four days off, and then three days on followed by three days off.
Each full-time pilot received base pay of $227,000 last year and a $104,000 bonus based on the amount of cargo that moved through the port.
In addition, the callback provision in their contract requires the city to pay them four hours of overtime — which is 1.5 times their hourly pay — if they agree to begin a ship move, or any other assignment, "after" their regular shift is complete.
In practice, "after" a shift can mean any time: an hour later, a day later, an hour before the pilots' next regular shift begins, Sanfield said.
The Times' analysis of city payroll data shows pilots frequently receive callbacks on days when they didn't work any regular hours -- that's how Brett McDaniel, the pilot who was called back after 150% of his shifts, managed to work more callbacks than regular shifts last year. Three other pilots also worked more callbacks than regular shifts in 2015-16, records show.
None responded to requests for comment.
Because a typical ship move takes about two hours, pilots on callback collect overtime for roughly twice as many hours as they actually work, resulting in hundreds of thousands of dollars paid to them each year for time they weren't on the job, according to a Times analysis of payroll records, some of which the port keeps on a handwritten ledger.
City Administrative Officer Miguel Santana, who negotiated and signed the city's contract with the pilots, said it is up to Harbor Department administrators to decide when and how to use callback pay. He said the city will study whether it's cheaper, overall, to hire more pilots or to continue to pay the current pilots generous overtime.
City records also show that some pilots are paid for multiple four-hour callbacks in a single day. Since July 2010, that has happened more than 500 times, records show.
On May 8, pilot John Betz worked three callbacks. The ship moves took him less than six hours, records show, but he was paid for 12 hours of overtime: $1,962.72.
That practice appears to contradict the contract, which says if a pilot is required to remain on callback duty beyond four hours to complete a ship move or another assignment, he will be paid overtime on an hour-for-hour basis.
Sanfield explained that the port's policy is to treat each ship move as a separate callback, guaranteeing the pilot a full four hours of overtime for each trip.
"I didn't know that was happening," said Santana, the city official who signed the contract. "We'll ask the port why that's happening."
The contract also says the two chief port pilots, Bent Christiansen and Michael Rubino, are supposed to get called back only if no other pilots are available. That happens frequently, the data show.
Rubino was called back 137 times — or more than twice a week -- from July 2015 through June 2016, boosting his pay by nearly $90,000. Christiansen was called back 105 times, boosting his pay more than $68,000.
Money for the callbacks counts toward the pilots' pension, which is based on the best 12 months of pay during their career, according to city retirement officials.
In the last fiscal year, Christiansen's total pay was more than $602,000, records show. Rubino's pay was just short of $500,000. Neither responded to requests for comment.
Last fall, Christiansen played a pivotal role in hiring Rubino's 33-year-old son for the first Los Angeles port pilot job opening in more than a decade. The younger Rubino was chosen over more than 50 other applicants, including ship captains and tugboat masters with years of experience.
The city fired the younger Rubino about a month later after determining he had overstated his experience on his application.
Christiansen and the other port employees involved in hiring Rubino's son have not been disciplined because they did not know about "the qualification issue until after the fact", Sanfield said.