RPI Should RIP

by BruinsRule on Mar 15, 2009 8:42 PM PDT Comment 14 comments

Bumped. GO BRUINS. -N

I started this post on Friday afternoon but didn't have time to finish it. In the intervening 50 hours, the point has become more personal for Bruins fans than it was previously. I think this offseason is the right time for the NCAA selection committee to finally eliminate RPI in favor of a formula that considers margin of victory. By ignoring victory margin, the committee is handicapping itself in its quest to identify the 34 best at-large teams.

This year in particular Consider how the incessant "bubble" discussion would change for Pac-10 teams if a measure like Sagarin's Predictor (which uses margin of victory), Sagarin's Rating (which synthesizes his Predictor with a model that does not consider margin of victory) or Ken Pomeroy's Pythagorean formula were used instead of RPI (rankings through last Thursday's games--I compiled this on Friday; it's close enough to how these ended up for you to get the point):

Team    RPI    Sag Prdctr    Sag Rtg        KenPom
UW         11        14 (-3)        17 (-6)    14 (-3)
UCLA    27        7 (+20)        14 (+13)        6 (+21)
ASU       30       12 (+18)        21 (+9)        13 (+17)
Cal         40       28 (+12)        27 (+12)        29 (+11)
USC       49       30 (+19)        39 (+10)        33 (+16)
Ariz         58       40 (+18)        44 (+14)        38 (+20)
WSU      90       36 (+54)        52 (+38)        31 (+59)
Stan      104      49 (+55)        64 (+40)        50 (+54)
OSU      156    120 (+36)    114 (+42)    121 (+35)
Oregon 177    142 (+35)    140 (+37)    152 (+25)

AVG 74.2 47.8 53.2 48.7
MED: 53.5 33 41.5 32.0

As you can see, in every case but Washington, the numbers in the other rating systems rate each Pac-10 team significantly higher than the RPI system does. According to RPI, USC's rating was low enough that it was not even on the bubble when we played them. Had Kenpom been used, USC may have already been a lock to make the tournament. Of course, they weren't and we lost to them, a loss that cost us another 6 spots in the final RPI rankings.

Why is it the case that RPI is so different from these other two respected rating systems? Simply put, RPI's ignorance of the score of games creates a lot of incongruities that should lead its users to question its effectiveness for its purpose. Here is a sample scenario, if BYU were to beat UCLA by 1 point in Provo, it would have the exact same effect on UCLA's RPI as BYU beating Mississippi Valley St. by 51 in Provo would have on MVSU's. In fact, move the UCLA-BYU game to Pauley, and the game would have a more negative effect on UCLA's RPI than MVSU's because RPI gives 60% weight to road losses and home wins. I'll say it another way: MVSU loses to BYU by 51 in Provo, and we lose to BYU by 1 at Pauley, and RPI would rate MVSU as better than UCLA.

The committee shows much evidence of its recognition that RPI is not an effective measure of a team's strength. For instance, Arizona got a tourney bid despite a #63 RPI, while San Diego State was left out of the tournament with a #34 RPI. UCLA, one spot higher in RPI, picked up a 6 seed, but the Aztecs are NIT-bound.

Of course, had the system used by the committee viewed margin of victory, UCLA's wins would look more impressive. After all, despite finishing 2nd in the Pac-10, we led the conference in point differential by a significant margin, and we also pummeled all of the crummy competition we scheduled in the preseason. Use these other measures, and UCLA is a protected seed.

Alas, we have the RPI this year, and Coach Howland was unable to game the system as well as some other coaches did:

-Siena RPI #18 despite 7 losses and only 12 games against top 100 (compared to UCLA's 18 games vs top 100)

-Utah RPI #10, despite a worse record than UCLA (by 1 less win), no road wins over an RPI top 100 team, a loss to a #195 RPI team Idaho State, a home loss to Cal, and fewer games and wins over top 100 competition. And by the way, their only wins of note were against Gonzaga when the Zags were reeling, SDSU and BYU.

And that's the system that sends us to Philly to play a tough 1st round game and if we're fortunate enough, Villanova in its secondary home.

Addendum:

I thought of this while noting for myself that before the USC game, we were something like 5-5 against RPI top 50 teams. USC was 49th in RPI. Had we beaten USC, they would have fallen out of the top 50 (despite being the same team they were all along), and our record against the top 50 would have fallen to 3-5, which would have been a black mark in the selection process. It struck me as odd that no matter what happened against USC, our record against the top 50 was doomed to get worse.

Of course, we did lose, and our record against the top 50 did also get worse. And a few minutes ago I reflected on the fact that we dropped RPI 6 positions after a neutral court loss to a team that was in the top 50. In the rating systems that consider scoring margin, we didn't drop nearly as far. The irony thickens.

This is a FanPost and does not necessarily reflect the views of BruinsNation's (BN) editors. It does reflect the views of this particular fan though, which is as important as the views of BN's editors.

3 recs | Comment 14 comments | Share

Story-email Email Printer Print

Comments

Couldn't have said it better myself..

I’ve been anti-RPI for a while in this forum but haven’t expressed it in quite that way. It certainly helps that we now having something concrete (our ridiculous seed out East) to hang our hat on and further prove our points.

Look, we got a crappy seed in a crappy environment (I apologize for the lack of eloquence)… According to most of these more accurate rating systems (which I of course whole-heartingly endorse) we are much better than our seed would indicate. Thus we should have a chance against a team that’s supposedly worse than their seed (Villanova).

I sincerely think we’ll beat VCU, maybe by a lot, and then have a legitimately good chance to knock off a great Nova team at home. That switch everyone keeps talking about better be on though.

by zinlinez on Mar 15, 2009 8:58 PM PDT reply actions 0 recs

Each Game is tough

We could win and we could lose anytime, beginning on Thursday. I do have confidence in our team to be able to handle the rough situation and to come out with fire. DC, JS and PAA especially, with their careers on the line, are going to play with the intensity of 6000 suns. I like our chances for the weekend, even though the odds are against us winning both.

by BruinsRule on Mar 15, 2009 9:50 PM PDT reply actions 0 recs

Don't forget bad losses

You know a system is flawed when its #1 team lost a game by 27 points to Clemson. But, that is the RPI (Duke being their champ). Instead of a team being punished for bad losses (or conversely rewarded for being close in its losses as we were in all of ours) the RPI could care less. I’m with you Rule, dump the RPI and use a better system.

by bruinponcho on Mar 15, 2009 10:26 PM PDT reply actions 0 recs

this is very helpful analysis

there’s obviously a lot of subjectivity and politicking behind the scenes in these things…personally though I struggle with the concept of overweighting the Margin of Victory…the KenPom type numbers I’ve seen IMHO overestimate our strength, because of these blowouts…an offense can be a phenomenal offense against bad or mediocre defense, but be less phenomenal against very solid defense, like ours, and “game” these types of statistics.

What makes the Tourney so nice is that, unlike football, you have 64 teams, and the five teams that get left out don’t have a real argument that they could have won the NC. The RPI may be flawed, but the subjectivity, although it will always leave room for bias and stupidity (like AZ’s inclusion this year), is minimized reasonably well in college basketball…

by glassbruin on Mar 16, 2009 12:01 AM PDT reply actions 0 recs

Ken Pomeroy

I have to say that I have slowly been won over by his system but it does have some flaws. Blowout wins have too much of an affect on your ranking. There needs to be some cap for garbage time. A team that is up by 25 at the half is actually punished for putting in reserves and winning by 18. This rewards coaches with bad sportsmanship who run that lead up to 50.

The other problem I have with it is that it fails to account for close wins. Teams that win close games are often more dangerous than those that only win when they can amass a large enough lead, yet there is nothing to account for the ability to win when it is close. I have no idea how that would be done, but I think it would help make the ratings even more accurate.

On the whole, I like Pomeroy’s work. I hate the RPI (any system that ranks the big 10 so highly must be flawed, my eyes aren’t that bad) and I think it should be replaced

by Rhapsode on Mar 16, 2009 1:39 AM PDT up reply actions 0 recs

Since we're on the subject...

…Pomeroy has us beating VCU 75-66 with a total chance of winning of 81%. I’ll take that anyday – but his stats are once again assuming we bring our A game.

by zinlinez on Mar 16, 2009 5:14 AM PDT reply actions 0 recs

I agree, every system has its flaws.

So your premise that complete reliance on the RPI is stupid is something I agree with.

However, the other rankings you point to make UCLA seem like a top # 10 team—which in my opinion would be a worse crime to college basketball than UCLA as # 33 in RPI. Maybe scoring margin should be factored in a bit—I really don’t see why it can’t or shouldn’t be taken into consideration. But to me, it is apparent that it was those meaningless blowouts against mediocre to bad teams that inflated our offensive stats to meaninglessness.

We heard all about how great an offensive team UCLA is this year, and heard it again as soon as CBS announced UCLA as a tournament team and # 6 seed. Pomeroy used it to put us at #9 in his national rankings. But those offensive numbers are in my opinion grossly misleading. Sure our defense is not as good this year, but after watching all but one of UCLA’s games, I can’t say we have a terrific offense either. Not even close. For me, this season has been an indication that pure numbers don’t tell our story this year.

We have been good, and we have drubbed some bad teams. But we haven’t been able to score well against good teams in big games when it really matters. Our offense stops scoring when the pressure mounts. It has been brutal. Aside from Collison at Cal, I can’t think of an important game where we dug down and scored when it was necessary. I’m sure there have been such moments, but I’m struggling to remember them.

I guess my point is, if you look at all the rankings posted above, RPI, Ken Pom, Sag, etc. — in my opinion only the RPI captures a true reflection of how good UCLA was this year on a national scale. The other numbers based systems just don’t reflect something about UCLA this year.

Maybe the other systems work better than the RPI for the majority of NCAA teams this year—that is entirely possible. For instance, I sort of agree with Ken Pom’s top 10 based on the college basketball season (with the exception of UCLA). So, in the aggregate, maybe changing the RPI or using other ratings systems is a good idea. But, for better or worse, I think UCLA’s season, if anything, highlights the flaws of those other systems this year—rather than prove their value.

#7, #14, #6 – the non-RPI rating systems tout UCLA as an exceptional team. But in my opinion, the only thing exception about UCLA’s season is that it tricked a lot of smart numbers based ratings systems into calling us really good.

by rfirpo on Mar 16, 2009 8:18 AM PDT reply actions 0 recs

The flaw in your argument

Is that you’re choosing the conclusion supported by less data over the one supported by more. You admit that you agree with 9 out of the top 10, but you’ll let the one you don’t agree with swing your opinion to another system. We’re talking about the usefulness and accuracy of systems as a whole, not how accurately you think they portray UCLA. So while you talk a lot about how you think these systems have overrated UCLA, you’ve actually done little to show that the RPI is better or even close.

by Tydides on Mar 16, 2009 8:48 AM PDT up reply actions 0 recs

Actually I partially retract this until further notice

I tried to do a runthrough of a comparison between the actual results and the predictions but I was stopped by the fact that it’s more difficult to obtain RPI results from before the tournament starts. Based on the RPI results after the tournament vs. Kenpom before (I know, apples to oranges comparison), the results are mixed when doing block comparisons by round (16, 8, 4) over the past three years. I started doing a deviation comparison and the initial results are slightly to moderately in favor of kenpom, but I almost got caught doing it at work so I’ll probably put this project on hold for a while.

by Tydides on Mar 16, 2009 9:50 AM PDT up reply actions 0 recs

Your point is good and valid.

I recognized that too. Still, I think its fair to challenge any system by saying “while x is likely a better system than y, x really is misleading as to team z.” That’s where I am coming from.

While I wish UCLA had a higher seed, the suggestion that a non-RPI ratings system would have more validly seeded UCLA annoys me a tad, because for whatever reason, this year UCLA was ranked way too high in those other systems in my opinion. In fact, the other numbers are so far off in my mind that they don’t offer any validation or comfort to me as a fan of UCLA. And if I don’t think a system works for the team I know the most about, it leaves me wondering if it works for other people’s teams (in other words, just because I think the rest of KenPom’s top 10 comports with what I know, maybe fans of those schools disagree). But anyway, your point is totally valid and fair.

By the way, I’m almost caught doing all of this at work all the time!

by rfirpo on Mar 16, 2009 10:58 AM PDT up reply actions 0 recs

Haha, I hear you

I actually wish the RPI wasn’t used at all along with any other synthesized numbers, so I don’t advocate the committee using KenPom either. Weighting by arbitrary numbers just seems like a recipe for trouble, and is a big reason why people distrust the BCS so much. I believe there is more than enough information in the raw data: wins/losses, conference standings, last X games, injury status, etc. to determine which are the most worthy teams.

While I believe we earned a 5 seed, taking into consideration that we don’t know the status of DC, I can see a plausible reason for moving us down 1 spot. If he’s not close to 100%, then we’re pretty much a 6 seed or lower. I don’t like it, but injuries are part of the game.

by Tydides on Mar 16, 2009 11:15 AM PDT up reply actions 0 recs

Illinois

is without their point guard due to injury. They should’ve definitely been moved down.

by lil eg not cs on Mar 16, 2009 12:09 PM PDT up reply actions 0 recs

Agree Ty.

I don’t mind the 6, except its a 6 in Philly near Nova. Had we been a 4 or 5, we are probably in boise. Even a six is fine, but I hate that its in Philly. I would take any other 6 seed over ours. The crazy thing is that, according to the S curve, we were the 2nd worst # 6 seed—meaning we were almost a # 7!

And the NCAA committee didn’t give a rats you know what to keeping teams close this year in my opinion.

by rfirpo on Mar 16, 2009 3:34 PM PDT up reply actions 0 recs

Even the Committee thought RPI undervalued UCLA

As a result of its conclusion, the committee gave us a 6 seed instead of the 9 that the RPI would have warranted. I’m not saying that we’re the 9th best team in the country, and I agree that Pomeroy ratings tend to shoot up when you beat a bad team by a lot rather than when you beat a good team by a little. All computer ranking systems have deficiencies. To me, RPI is more deficient than the others because it ignores one of the components that allows one to assess a teams’ relative strength, and that is the score of the game. Although victory margin in certain circumstances can be misleading (take, for example, the UCLA at ASU game where an even ball game became a 7-point loss due to one bad call and a few free throws by the leading team, or the occasional 10-point overtime victory) , it is my view that these outliers tend to even out over an entire season. In fact, statisticians will tell you that it is in fact true that these things even out. They will also tell you that winning close games (being clutch) is not a particular skill that can be measured. There is a regression to the mean. This is why, for example, in baseball, teams with a great winning percentage in 1-run games in one year tend to be poor in 1-run games in the next year.

You say that other than Cal, we haven’t scored when necessary. How easy it is to forget the wins when the losses hurt so much. So, here goes a list of close games where I remember that we scored down the stretch to win: Miami (OH), Washington, @ Stanford, @ USC, @Cal. In our loss against WSU, we scored a ton down the stretch. Basically, in close games, we scored enough to win in 5 of them, and not enough to win in 6 of them (the other 2 losses—USC and Arizona—were not particularly close). As luck would have it, that’s right about 50-50.

by BruinsRule on Mar 16, 2009 3:32 PM PDT up reply actions 0 recs