The concept of continuous development in sports is not limited to athletes and tactics only. It’s essential for men and women who run games too.
The mechanics of refereeing—where to stand, how to move, and who to make the calls—wasn’t uniformly taught in the NBA until the early 1980s, after more than 30 years of the league’s existence. The use of video recorders and video tapes to train referees will not begin in earnest until after a few years. The National Basketball Association didn’t even begin using three full-time officials for its games until 1988.
As the world around him updated, so did the NBA. The division in charge of the league today relies more than ever on the well-known sports buzzword: analytics.
Every call made by NBA referees – and many that are not – are rated by neutral observers and then entered into an extensive database including every league official. This data is used for the ongoing training and development of referees, and it helps determine reference promotions and playoff assignments. Teams are given partial access to selected calls and allowed to make certain inquiries.
This is not a mom-and-pop setting. It’s an entire operation involving more than two dozen full-time employees, from former NBA officials to outside consultants and quantitative analysts. It’s also an area in which the public hasn’t had much of a window, even as analytics and technology have become a larger and larger part of the division in charge of the league in recent years.
FiveThirtyEight has spoken to more than 25 people across the NBA, from league employees responsible for collecting and using data to front-office executives, to understand exactly how this referee ratings and analytics program works.
Many NBA fans recognize Secaucus, NJ, as the home of the league’s Replay Center, but that’s just a small part of the league and the refereeing processes that take place there. Secaucus is also the center for the NBA’s referee review and grading program, which employs more than 25 people in various roles.
The main character here is Stephen Angel, one of the longest serving staffers in the NBA for nearly 14 years. Angel was part of a consulting firm that helped redesign league rulers data in the early 2000s when David Stern was commissioner; He was eventually hired full-time as part of the responsible department, and now holds the role of Senior Vice President of Game Analytics and Strategy.
When Angel began working with the league, the evolution of referee reviews and grading systems matched the limited technology that was available. At the turn of the millennium, the National Basketball Association began assigning observers to attend games in person in their geographic area, then go home and break the same game again via DVR later that night. Angel, with his consulting background, noted potential training problems and ring biases, and the entire system was introduced internally in 2013.
When the NBA brass decided to start sharing referee game scores and reports with individual franchises in 2015, they quickly realized that many teams were focusing on different types of contact and play than the league’s referee advisors — understandable given that teams have their own interests at heart. While consultants focus on the performance of referees at the league level. Standardizing these definitions across the board has become a major part of Angel’s section, a part that survives to this day under the rubric of Clarification Project.
Participants work directly with the NBA Competition Committee—which includes a group of owners, general managers, coaches, referees, and players—as well as representatives from all 30 teams and the referee association. Angel sums up the goal of this project simply: “What actually constitutes a bug? The game is long without a call, except when the whistle goes off,” Angel told FiveThirtyEight. “Can we agree on what counts as a bug?”
The Angel team also deals with the world of integrity. He looks for any and all possible indications of bias, whether conscious or subconscious. The name “Donaghy” is rarely uttered in league circles today, but there is a clear desire to guard against even the suggestion of wrongdoing – especially given the NBA’s stated interest in the sports gambling arena.
“We’re keeping an eye on the Las Vegas gambling lines,” Angel said. “We are looking to see if there is any indication of manipulation or bias. We hope this is a huge and significant waste of time, but we are still diligent in this effort.”
However, the biggest element that Angel oversees is the NBA team of dedicated referee reviewers.
While some roles in the league’s administrative division are filled by former referees, game reviewers are not – again, the goal here is to reduce any potential for bias. Instead, Angel and his crew are looking for “basketball-centric individuals” who have trained or played at a certain level and who have a strong knowledge base but do not have specific ties to the NBA franchises.
In the early years of this program, game reviewer job applicants took tests of their basketball acumen. Despite this, the association quickly realized that those exams were asking the wrong questions.
“We think we can teach what [reviewers] The angel said. “What we need to find in them is the ability to sit still and focus for long periods of time.”
Reviewing NBA referees is a tedious and painstaking task. Getting the job done right might require watching the same three-second playback clip more than a dozen times at multiple angles. A single game review takes between six and eight hours, per multiple members of the department. Dedication and focus are just as important as actual knowledge of basketball.
Today’s interview process focuses more on these types of skills. These are prestigious roles: the NBA has a staff of 15 game reviewers at most, including 10 standard reviewers, three senior reviewers, and two specialist roles (such as one reviewer dedicated solely to gambling streaks and related integrity concepts).
Game reviewers are trained on three job specific components:
- How to classify plays: what constitutes an infraction vs. what is not? What types of plays should be included in game reports?
- What is the responsible reference: The reviewers assess whether there was a foul, but also the referee on the ground was responsible for making (or not making) that call. This includes detailed knowledge of the mechanics of the NBA governors for each of the three positions the officiant holds on the ground (lead, hole, and track).
- Use of game review system technology.
The Game Review System, or GRS, is the technological basis for the modern NBA referee rankings and is an in-house designed system that has been updated several times since the mid-2000s. The goal is to provide reviewers with a standard dashboard that they can access via the computer, with processes that can be applied as equally as possible to each NBA game.
Again, there is a huge focus here on any Plays are included. The league could require reviewers to break each individual dribble in each game, describing “no offense” each time a player successfully moves without traveling; While that would allow the NBA to claim 99.99 percent accuracy on all types of calls, it wouldn’t be a possible use of reviewer time — and the resulting data would be nearly useless.
Instead, the reviewers get great help here from state-of-the-art technology: Spectrum II camera tracking, found on all 29 NBA yards, is integrated into the GRS. This data is used to “pre-tag” many common events in basketball, such as shots, passes, drives, screens and dribbling starts. Instead of manually combing through every second of game action, controlling which games should be covered and which should be left alone, reviewers have a unified shortcut they can rely on.
In general, reviewers can rate – or “tag” – a play in one of four ways:
- Infraction: It is clear that there was a violation of the rules of the NBA occurred in the play in question.
- no offense: It is clear that the violation of the NBA rules did not occur.
- Possible offense: The call was not clear or conclusive. Two subcategories, “slant violation” and “slant violation”, are included in these “judgment calls” to make the final datasets more robust.
- Enhanced review: A final decision on the call can only be reached using an enhanced video review and the judge cannot reasonably see it in real time. For example, the league would not penalize a referee for missing a flight in a play in which a slow-motion video revealed that a player raised his pivot foot fractions of a second before the ball left his hand to dribble.
Reviewers have access to no fewer than nine broadcast angles for each play, as well as many more views often — and a whole host of video enhancement options available to them. In addition to entering one of these four marks for each play, the reviewers also use the video to determine which referee should be in charge on the field. (Tracking technology often plays a role in this task.)
Collaborative spirit is encouraged. The reviewers are in the same physical location in Secaucus and often meet with each other or senior staff in challenging plays. (COVID-19 protocols have disrupted parts of this personal coordination over the past two seasons.)
After six to eight hours, the reviewer will have analyzed one game of the NBA and all potential offenses within it. This process is often repeated by the chief staff auditor; Hundreds of NBA games each season that each get 12-16 hours of review.
Each call and non-call will be automatically assigned to the referees who worked in that game and compared against the reviewers’ scores. (It is important to note, again, that reviewers do the rating infractionsNot the rulers themselves. Their job is to focus only on the players on the field and whether the rules were broken during play; To avoid bias as much as possible, the resulting scores are assigned to officials later and by other league staff.) This allows the corresponding accuracy to be determined. From here, this data will become part of each administrator’s current record. At the league level, this data set is huge: One season will include about 500,000 refereed data points, according to Evan Wash, the NBA’s executive vice president of basketball strategy and analytics.
Wach oversees Angel’s department, as well as a separate group of data scientists whose role involves combing through this data in excruciating detail. They look for trends across individual referees and staff, between players and teams, and even within betting patterns. The Wash team will then work with other key NBA officials — including Monty McCutchen, senior vice president of referee development and training, and Mark Wonderlich, vice president of referee operations — on what they find.
This data analysis often has a direct impact on the ongoing training of referees. For example, as the game has gone through the spacing revolution in the past decade, responsible analysts are beginning to notice relevant positioning issues.
“[As] The game became more ocean-oriented, and we found we were seeing errors in various places on Earth,” Wash told FiveThirtyEight.” Which in turn prompted us to work with Monty, Mark Wonderlich and his crew to rethink the mechanics of refereeing to make sure the referees were in the right place to capture plays. “
Once again, cooperation is key. Wash and his staff are regularly involved in meetings and training sessions with the Wunderlich Group. If the data can identify or enhance an important training area for a single reference, a group of referees, or even every reference in the league, all the better.
They even check their own reviewers! data is kept on the number of times senior audit staff are forced to pass an inappropriate mark from the initial audit when they pass for the second time; Angel and his team address any error trends that appear here. If problems persist, the reviewer may have to find a new job.
Coming Wednesday: How the reference ranking data is used by the league and its teams.