Sorting by Highest Rating: order sometimes looks wrong

[shauny]

Sometimes when sorting stories by “Highest Rated” you’ll notice this oddity - the ratings are not in the correct order. But this is not a bug, it’s a feature!

See here for example. This screenshot was taken June 2024, so it may be out of date when you read this, but it is used for an example.

It shows that the highest rated Eleventh Doctor and Valarie story is Broken Hearts at 4.6 / 5, then Sins of the Flesh gets 4.5 / 5, The Last Stand… gets 4.5 / 5, The End gets 4.4 / 5 and then… Victory of the Doctor gets 4.5 / 5 - quick, someone email me there is a bug!

All is revealed if you hover over the ratings and you see this little tooltip:

This story (The Last Stand) had 78 votes, and achieved 4.5. The other story further down (Victory of the Doctor) also gets 4.5 but with fewer votes - only 57.

The Wilson Score Algorithm

There is an algorithm on the site called The Wilson Score which makes sure that stories with a low number of votes but a high rating cannot appear higher than stories with many more votes.

Here’s the algorithm

For example say there was a story that got 100% score! Perfect, amazing! But only 10 fans voted it, so they are probably biased.

Whereas another story got 99%, incredible! And voted by 100 people.

This algorithm ensures that a story cannot jump straight to the top of the ranks simply by having a high score from a low number of users.

So why not show a lower rating?

I toyed with the idea of showing a lower rating instead, so there was no confusion and everything looked in order. But the problem is, a story may be really good, but it gets penalised for only having a few ratings. Showing a story getting something like 2/5 just because it only has a few ratings would be very misleading.

So we’re in a lose-lose situation!

I get asked about this a lot, so I have created this post to link to the next time someone asks!

[Tian]

I like this a lot. Now I need to google this sorting score algorithm

[shauny]

[Tian]

This is an awesome solution to this problem

[deltaandthebannermen]

God…Doctor Who fans are such nerds…

[BillFiler]

We prefer Nerdvians

[Jae]

That’s some fun math

[deltaandthebannermen]

Cheeky bastard!

[Shayleen]

I was wondering about that when I sorted by highest rating, and did wonder if it was something like this. Makes sense!

[PlumPudding]

I think the way you’ve done it works well (although I worry how often you’re going to have to explain it). If, for example, a niche story with a small audience gets a few positive reviews, it makes sense to have it fall below something with wider appeal. But, that score may still encourage others to experience that story for themselves.
It also hopefully discourages a certain type of review bombing where people give low scores to stories they haven’t seen so that their favourite stories end up at the top (though that isn’t a problem here currently, and hopefully won’t become one.)

[shauny]

I actually just had a dream about review bombing. (Totally normal thing to have a nightmare about lol).

It’s something bigger sites have to worry about, I know that IMDB has a formula that reduces the impact of 1 and 10 star ratings to try and prevent people doing it.

You get people rating a story in a way that they think will affect the overall score, rather than rating how they honestly feel.

I hope the way this site works, that your profile shows your ratings, means people are more encouraged to rate things honestly.

[15thDoctor]

The above formula is perfect, have wondered a lot about this (yes, I’m also that guy) so great to see it’s in hand.

A way to avoid review bombing is to not to give a reviewer’s scores any weight unless they meet certain requirements, this could be having been on the site for a certain amount of time, or having used scores other than just 5* and 1/2*s. This would stop someone blanket giving half stars to their least favourite era, logging off and never coming back. Given the size of the community right now it’s probably not necessary to take these steps, but when it grows it’s something you could consider?

[shauny]

Yes that’s something I have considered.

Not an issue at the moment fortunately.

[dema1020]

As somebody who has studied stats on a grad school level, this makes me so happy. I was wondering about the unusual order some rankings took so it is cool this isn’t just some bug or something. Instead this is one of the smartest approaches to ranking content I’ve ever seen. You’re my hero, @shauny