New Metrics for a New TTC

With the changing of the guard in the Mayor’s Office and a shift in the political balance of the TTC Board, it is time to blow the dust off of the metrics in the TTC CEO’s Report and elsewhere. I have written about aspects of this before, and will not belabour earlier arguments. However, in an era of recovery, we need to show what this is actually happening, and that we are getting good use out of the transit infrastructure, notably a large vehicle fleet, that we already own.

The areas of particular interest are:

• Ridership, demand and crowding on routes
• Service quantity and reliability
• Fleet availability, usage and reliability

The CEO’s Report is replete with “Key Performance Indicators” (KPIs), a favourite tool of lazy managers to give the impression a complex organization and process can be reduced to a handful of simple numbers. Either “up” or “down” is considered “good”, and as long as the lines move in the correct direction, gold stars are handed out like confetti. Rarely, if ever, is the underlying process, the product, or the real meaning of the KPIs discussed.

A subtle, pervasive issue for TTC KPIs is the focus on top line numbers for ridership and revenue. This is akin to a restaurateur who counts the receipts and the number of meals sold without asking what brings diners to the door, or even worse, whether they will come back. The goal is to sell more meals, preferably at a low cost. Advertising, not word of mouth, generates new, if not lasting, trade.

Ridership is a rough measure of system use and a point of comparison for post-pandemic recovery, but it does not tell the whole story. Already we know that the bus network which is mainly based in the suburbs has recovered much of its pre-pandemic demand, although this is not distributed the same way with shifts in peak periods and in travel patterns. Off peak recovery is stronger than peak, in part because “work from home” affects less than half of the total demand, and non-work trips still occur.

Even “growth” can be misleading. In pre-pandemic times, the TTC routinely celebrated year-over-year riding growth even while the rate of growth slowed and eventually stalled. A problem flagged at the time was that growth occurred disproportionately in the off-peak where there was surplus capacity. That capacity filled up, but thanks to budget constraints service did not expand to match.

This shows the danger of looking at a single, simple number without understanding the detailed system behaviour, or even worse, of using the simple metric to hide a growing problem. Trimming capacity to demand can be a vicious cycle that prevents growth.

The phrase “subject to budget availability” is a standard caveat on any goals, and it has haunted TTC planning for years, well before the pandemic. That might be a basic part of corporate management, but over many years it has become the foundation of TTC reality. Aim low because aiming higher will cost too much.

This speaks to the split nature of TTC goals. It is supposed to provide transportation, and the motto “Service, Courtesy, Safety” is emblazoned on the TTC’s coat of arms. However, the TTC Board sees its primary role as serving its political masters at Council and especially the Mayor.

I wrote about TTC culture and that motto back in 2010. For context, this was before Andy Byford became CEO, let alone Rick Leary.

The common problem with many KPIs the TTC publishes is that they are one dimensional and report only average values of major variables. They do not necessarily reflect what riders see nor give a sense of the shortfall between what the system achieves and what could be possible.

I have said this before: riders do not experience “average” trips any more than diners in a restaurant experience an “average” meal. A four-star restaurant might outdo itself with a plateful of magic from the kitchen, but an off day could bring overcooked, lukewarm food and indifferent treatment by the wait staff. Getting it right most of the time doesn’t warrant four stars. Getting it right only some of the time doesn’t warrant any. The diners are paying for all four.

On occasion, I am asked how I would change the TTC’s KPIs to better show what is happening. My first response is that many aspects of a transit system cannot be reduced to one-dimensional metrics that compress all of the vital details into simplistic averages.

TTC needs to focus its performance metrics on service-related factors, direct measures of what riders experience. Average values will not do, and the Board needs to understand what these numbers mean. Providing tolerable service on most routes a good deal of the time is not an advertisement for “the better way”. Provide attractive, reliable service and riders will follow.

What Should be Changed

This section summarizes observations from the article.

• Change the crowding standards back to the pre-2023 version to relieve off-peak crowding.
• In budget presentations, clearly show the shortfall between what the budget provides and what the standards say should operate. A list of routes and time periods requiring more service should appear in the CEO’s Report.
• Show network-wide performance indicators at the route and time level with colour-coded maps. Provide these maps through an online portal with regular, automatic updates and access to archival data.
• Tighten the headway standards so that irregular service is not counted as “on time”, and so that the target for performance is well above the current 60% level.
• Create a metric to explicitly report bunching, gaps and missing vehicles.
• Create a metric for excess wait time.
• Measure actual service quality at key points along routes, not just at terminals.
• Report on staffing levels, recruiting, and the ability to operate service.
• Report on the cause of major delays and the actions taken to work around them.
• Report on fleet size by mode including peak scheduled service, spares and the headroom for service growth. Where multiple vehicles types exist, report usage by type.
• Report on fleet reliability by mode and vehicle type without capping reported values.

Note that this article focuses on operations and service for the “conventional” system. A full list of KPIs (or whatever one wants to call them) is beyond my scope here, but the same principles would apply. Avoid oversimplification and consolidation of values that can hide the real system behaviour. Focus on how any aspect of the transit system affects riders and their travels.

Wheel-Trans has its own collection of problems, and I leave them to the TTC’s Advisory Committee on Accessible Transit (ACAT). The TTC does not have an advisory committee to hold management’s feet to the fire for the regular system, and riders must depend on advocates who have no official standing at Board meetings.

The capital budget is a completely separate matter dependent on available financing, priorities, and political interference. The City’s review of its capital budget process contains an prioritization methodology which I believe is badly flawed, but that is a topic for a separate article.

Service Standards

It is impossible to talk about metrics without discussing some of the fundamentals the TTC is supposed to measure, and several of these are found in the TTC Service Standards. The official standards published on the TTC’s website were adopted as part of a much larger report in May 2017 which sets out the results of an internal review and of rider surveys. The report is still worth reading, if only to avoid covering the same ground as if nobody ever thought of this before. My own review of the report is linked below.

A key paragraph appears in the report’s summary:

This update to the TTC service standards took a no cost approach. The updated service standards reflect existing conditions with the goal of continuous improvement over time. It is the TTC’s intention to maintain and update this document on a regular basis. Enhancements to the service standards, such as improvements to the vehicle crowding standard or minimum service levels, will be presented to the Board in a separate report at a later date.

Update to Service Standards, May 18, 2017, Page 1

Two key points are that the review “took a no cost approach” and that “improvements … will be presented to the Board”.

A no cost approach is not aspirational, but seeks only to codify what is already done without adding to the budget. It is a recipe for shuffling resources to for incremental change, or even just to absorb new circumstances with minimal visible effect.

Presentation to the Board is another matter, and in the 2023 Budget Cycle, the crowding standards were altered by management without prior Board or public consultation. This led to additional service cuts which the new Mayor seeks to reverse. Specifically, the off-peak crowding standard was raised well above a seated load (the basis for the previous standard), although this change has not been reflected in the published standards.

The realigned service proposes to:

1. Resume pre-COVID vehicle crowding standards in peak periods, which were temporarily suspended during the pandemic to provide more physical distancing. (50 customers per bus, 130 customers per streetcar, 1000/1100 customers per train on average during the busiest hour)
2. Increase the pre-COVID vehicle crowding standard at off-peak periods with capacity for each route and time period planned based on the busiest hour for 45 customers per bus, 90 customers per streetcar, and 600-650 customers per train on average.

TTC Operating Budget Report 2023, January 9, 2023, Page 26

Note that these standards are “on average”, but there is no metric nor report indicating how widely the actual conditions vary from the standard. More riders will experience crowded buses than empty ones, and so the average experience will be worse than the reported average. This allows management to achieve targets while riders complain of crowded buses.

The CEO’s Report includes a chart showing the percentage of bus trips that hit various crowding levels with bands at 30%, 70%, 100% and 120% occupancy. The values are not subdivided by route or time-of-day. Although this may seem like a distant memory, before 2020 there was a major problem with capacity across the system. Only rarely was the shortfall between capacity and demand reported in detail to the Board, let along potential demand that went unserved.

With bus network weekday ridership back to 84% of the pre-covid level overall, demand pushing against the capacity provided will again be an issue.

Links to reports:

Service Reliablity

Parts of the Service Standards deal with acceptable levels of reliability. Some of these standards have never been reported on, and all of them are quite generous in defining achievement of their goals. The TTC regularly does not hit their targets even on an averaged basis across all routes and time periods. Some routes might be very good, but some routes will be truly atrocious. This distinction is lost in reporting summary values.

There is no measure of scheduled service that does not operate, nor of the effects of gaps and bunching on service quality and wait times.

Unless the Service Standards are revised to reflect Council’s goals, we will have no way to know how far we must travel to reach them. This is not to say the goals should be unattainable, but Council and Toronto’s transit riders should know what will be required. Interim steps might be needed, but at least we can measure progress through them rather than reporting “success” in reaching a low bar.

On-Time & Headway Standards

The +1/-5 “on time” standard gives the appearance of setting a six minute wide window, but for actual service quality, this allows wider gaps. If two buses on a 6 minute scheduled headway arrive one minute early and five minutes late, the gap between them is 12 minutes. Indeed, buses could be running in pairs under this standard and be “on time”

Similarly, the headway standard allows substantial variation. A route might be part of the “Ten Minute Network”, but a scheduled headway can vary in practice from 5 to 15 minutes. Even then, only 60% of the service must meet this “target”.

These variations translate to uneven crowding, rider frustration and the sense that, at least for short trips, TTC service is not worth waiting for.

With the move to the larger Flexity streetcars and the service cuts during the pandemic, many lines operate at a ten minute scheduled headway, but the standards allow for very wide fluctuation, and it is not clear whether this is measured anywhere but the terminals. Many bus routes also have headways in that 5-10 minute band.

Although the TTC does have a standard for headways, what is always reported is “on time” performance at terminals measured against the +1/-5 standard. Headway adherence is not reported.

The TTC does not have a standard for bunching, although pairs, trios, and worse will all fall below the 5 minute mark in the standards. Similarly the gap between bunched vehicles on a 10 minute service will easily exceed the 15 minute upper target. However, 40% could be operating well outside of the target range but the “standard” would be achieved. This is nonsense.

An important factor about bunching is that most riders experience the long wait (and a crowded bus) even though “on average” all of the advertised service is there. An excess wait time metric is needed to convert actual operating data to a measure of rider experience. Moreover, there should be a standard that the excess wait time should be kept to a minimum, and this should be measured along routes, not just at terminals.

I wrote about excess wait time in previous articles and will not duplicate that discussion here. These articles present charts of actual headway distributions on routes to illustrate the wide divergence from the average values advertised by the TTC, and the resulting effective wait times riders encounter.

Without getting too deeply in the mathematical weeds, the important thing about excess wait time is that it is proportional to the square of the headway. This means that as headways widen, excess wait times go up even faster. When there is a gap, more riders accumulate and they wait longer. The short intervals between bunched buses benefit few riders and do not dilute the typical wait experience.

Tracking all of this to produce individual figures would be tedious, but presentation on a map to show overall system performance would be much simpler.

Run As Directed Vehicles

In response to complaints about service, a standard TTC response has been to say that “Run as Directed” or RAD buses are available to fill gaps and selectively provide extra service at key points. These vehicles have several problems including:

• RAD buses are not visible on trip prediction apps, and riders will not know when one is coming.
• Until recently, RAD buses were not visible in historical tracking data and it was impossible to show how they were used to improve service.
• Recent, last-minute schedule changes have triggered the deployment of RAD buses for services such as 903 Scarborough Express and 501L Queen to Long Branch. This reduces the pool of buses available for ad hoc service where there are problems.

TTC has a bad habit of citing the total number of RAD buses on an all day-basis rather than the number in service in each time period. This gives the impression that more vehicles are available than actually exist.

Subway Reliability

Subway lines have frequent service, and they are measured against scheduled headway, not against scheduled times. The allowed variation is 100% of the scheduled value and so for trains intended to arrive every 5 minutes, a range of 0 to 10 minutes is considered “on time”. The target is to achieve this for 95% of trips all day. This has two bizarre effects:

• Half of the service could be missing, but the line would be “on time” because the gaps would be no more than 100% above the scheduled headway.
• A wide gap followed by many closely-spaced trains would count for only one train (the gap train) not being on time. Following trains in the parade would be close together, but with a lower limit of zero minutes, they would be counted as “on time”. (There is also a physical lower bound on headways imposed by the signal system.) This underplays the real effect on riders.

A separate metric looks at the capacity delivered, and the target is to provide 90% of the number of trains scheduled during the peak hour.

Presentation of Route Performance Data

The CEO’s Report includes several charts showing the performance of the system as a whole and of specific parts including the subway lines, bus and streetcar networks. In most cases, these consolidate information for an entire month giving no indication of breakdown by route or time of day. With the current state of the streetcar network and its many diversions, the average stats must then be qualified saying that “routes a, b and c” were really bad, and others were much better. This breakdown should be evident without interpretive text.

In some planning reports, the TTC has displayed the condition of routes on a colour coded map that both separates the data geographically while giving an overview of the entire network. If such a map existed in multiple versions (e.g. AM peak, midday, PM peak, evening), problem areas and times would stand out.

Whether the full set of potential charts should appear in the CEO’s Report, or more appropriately on a page within the TTC site is another matter. Such charts could be generated automatically and posted, including archival versions, for anyone who wanted to review them.

A Look Elsewhere

Dashboards with detailed information are common on major transit systems. Here are links to some in North America:

Route by route breakdowns are found elsewhere, but the TTC reports only limited data at that level. With the automation of much of the number crunching, detailed reporting in close to real time (recent days, not months ago) should be possible.

As a starting point, the TTC and those responsible for directing it should look at other systems for inspiration.

Staffing

No service can operate without drivers on buses and streetcars, and maintenance workers keeping vehicles and infrastructure in shape.

The CEO’s Report does not track hiring or the shortfall, if any, between actual and target staffing levels. In an environment where keeping costs down is the primary goal, short-staffing is a common response. In an office environment, “gapping” means empty desks and work spread among those who remain. On a transit system, “gapping” translates directly to lack of service, and quite literal gaps where scheduled buses and streetcars might have been. Service designs are dictated by available staffing, not necessarily by demand.

Staffing levels are approved in the annual operating budget, and these numbers can only be exceeded with Council approval. Any plan to restore and improve service must include adequate headcount for the people who will operate and maintain it.

Fleet Availability, Usage and Reliability

The Mayor’s goal is to restore service to pre-pandemic levels, but this is only a start, and a few caveats must be noted:

• If the former service level is defined as vehicle or crew hours, the actual service provided will be less than it was because vehicles run more slowly in 2023 due to traffic conditions and increased recovery times.
• Changes in demand patterns will mean that some routes and periods are better off, while others less so even if total service operated stays the same.
• Network restructuring for major changes such as opening of Lines 5 Crosstown and 6 Finch will make direct comparison of services in some areas more difficult.
• Before the pandemic, the TTC was already crowded and the system was under-performing because both budget headroom and fleets were less than needed.
  • The TTC ordered 60 new streetcars to address the shortfall in capacity.
  • The TTC has many elderly buses that do not run very often, if at all, on their books, but these will be replaced with new vehicles in the next few years.

The CEO’s Report includes:

• Bus Mean Distance Before Failure (MDBF) by mode (eBus, hybrid and clean diesel) and month
• Subway MDBF by vehicle type and month
• Streetcar MDBF by month
• Bus and streetcar Road Calls and Change Offs (RCCO) by month
• Percentage of scheduled service operated by mode, weekly average

The MDBF values suffer from various problems:

• Reported values are capped in many cases so that monthly variations and the actual level achieved are not visible.
• Bus MDBF values are segmented by propulsion type (clean diesel, hybrid, battery electric), but some differences are hidden by capped reporting. MDBF values are not subdivided for major bus groups (age, manufacturer) within the fleet.

The percentage of scheduled service operated tells us how many vehicles were fielded, but not how much reserve capacity exists in each fleet, nor the proportion of the fleet routinely out of service for maintenance. The scheduled level of service reflects budget constraints and is not necessarily the level that could be provided or is justified by the Service Standards.

Any plans to improve service by running more vehicles must be informed by knowing how many are reliably available.

See also:

Appendix:

Ridership Metrics

The TTC reports various ridership-related metrics in the monthly CEO’s Report:

• Total ridership by week
• Total boardings by week and by mode
• Ratio of current riding to pre-pandemic levels by mode

Additional ridership data is available on the TTC’s “Planning” page:

• Average weekday and annual total ridership by mode for 2019-2022
• Weekday boardings, vehicle mileage and hours, peak vehicles by route for 2019-2021
• Annual streetcar boardings by route for 2019-2022
• Subway ridership by line and station

For broad system-level tracking, riding and boarding counts show the general direction of system use, but they reveal nothing about quality. Moreover, in an era when riding patterns are changing with work-from-home and a stronger recovery of non-work demand, the shortcomings of a network focused on downtown commuting.

Stats for past years do not tell us what is happening today. In an era when ridership numbers come not from manual counts with checkers riding vehicles, but from automatic passenger counters and fare machine data, there is no excuse for presenting stale data. Various TTC reports do show that current data are available internally for planning and management, but they should be publicly visible to inform debate about the city’s transit present and future.

The City of Toronto Open Data Portal contains various TTC data, although some of these have been replaced by the City’s dashboard (see below), and many are years out of date. Of historical interest is:

• Annual ridership by fare type, mode and period (weekday/weekend) from 1985 to 2019

A City of Toronto dashboard (not linked from TTC site) replaced spreadsheets formerly on the City’s Open Data site. It contains:

• Average Weekday Ridership by Month from 2007 to 2023
• Peak Period Annual Ridership from 2003 to 2022
• Off Peak Annual Ridership from 2003 to 2022
• Monthly Ridership from 2008 to 2023
• Monthly Fare Revenue from 2007 to 2023

This dashboard has system ridership and revenue, but not service (measured as vehicle hours or kilometres) nor a measure of service versus ridership, revenue or costs.

Historical ridership numbers show where the system has been, and in the current situation provide a reference of what the TTC could achieve. However, they do not show the rides not taken because trips are poorly served by the network. There is no breakdown of the transit modal split in various parts of the city nor its evolution as population and destinations have grown outward to and beyond the Toronto (416) boundary.

Service Quality and Delay Reporting

The TTC produces a Daily Customer Service Report, but it has only very broad stats about service quality. There is no archive of past data, nor is there any ability to drill down to see information at the granular level riders could compare to their experience.

The delay data for subway, streetcar and bus network is refreshed fairly regularly on the City of Toronto Open Data Portal. The challenge in analysis of these data lies in the vast number of delay codes for the subway and RT, and by contrast in the very small number of codes for surface modes.

There is no consolidated tracking of incidents or trend analysis for incident types. The only way to access details is to browse through the TTC’s online alerts on Twitter/X, or subscribe to every route for email alerts. Even then, the descriptions can be vague, and at times inaccurate. For example, “operational problems” covers a variety of issues without any hint of how service might be affected.