Ranking system flawed

11/1/2014 11:13:02 AM
Gravatar
Total Posts 516

Ranking system flawed

I don't even know why I make that post (chances for updates are at roughly 0%), but here we go:

As you might know, the Free For All room uses the Elo rating system to determine the ranks of players. Behind your rank lies a score (points). How many points you win or lose against another player only depends on the score difference. If a 200 points player wins against a 250 points player, s/he should gain as many points as a 400 points player against a 450 points. Now have a look of the score distribution in Cultris II:

  • rank 1: 1882 points
  • rank 2: 1389 points
  • rank 3: 1257 points
  • rank 4: 1012 points
  • rank 5:  471 points
  • ...
  • rank 20: 361 points
  • rank 50: 263 points
  • rank 100 222 points
  • rank 200:190 points

You just have to look at the top 5 to realize that the Elo implementation in Cultris II is flawed. Somehow, the number of lost/gained points per round is limited. The score I gain for beating a top 3 player is peanuts. I assume, it makes no difference if I beat a rank 1 or a rank 20 player (a score difference of 1000 is treated as a score difference of 100). From my observations, a rank 50 player will rank up when playing against a rank 20 player, but lose ranks against a rank 1 or rank 100 player. A rank 20 player will rank up when playing against a rank 1 or rank 200 player, but lose ranks against a rank 50 player.

This shouldn't happen with a proper implementation of the Elo system. As a side effect, some people tend to avoid the Free For All room when certain other players are around (because they would lose ranks otherwise).

11/2/2014 1:41:32 PM
Gravatar
Total Posts 52

Re: Ranking system flawed

Basically it will just be ELO with a k-factor of around 35 and an average score of 1000. The actual number for k is based on parameter optimization and we're going to use the one that gives the best accuracy over all players. Currently I have it at 72 divided by the number of players in the round. This means it's better to win against a good player in 1vs1 than in a group with 4 other people.

What Acriaos said in http://gewaltig.net/Forums/Thread.aspx?pageid=1&t=366~1 does seem to support what you're saying. If the number of players in the room were not taken into account for the score calculation, score changes would be more varied per match. Slickalei once mentioned that he only got 1 point per win in a room with many players.

But perhaps the problem is that the ELO system wasn't designed for more than two competitors. In a room with more than two players, better players will place higher most of the time. This could explain why a good player's score continually rises.

11/2/2014 4:41:55 PM
Jes
Gravatar
Total Posts 210

Re: Ranking system flawed

What i have thought for a while is pretty consistent with what you say, misstake. I know that in many ELO-systems (for example the one used in chess), there is implemented a max factor of what you can win/lose against one player. This basically means that when someone is very good and their rating goes beyond a certain threshold, they will pretty much gain and lose the same amount almost nomatter who they play, that meaning anyone outside top 4. This can pretty much make rating go rampant for anyone who are capable of going beyond somewhere that currently seems to be around 500. 

I dont know if this is the case, but it is very clear that something is very off.

11/3/2014 12:12:03 AM
Gravatar
Total Posts 516

Re: Ranking system flawed

But perhaps the problem is that the ELO system wasn't designed for more than two competitors. In a room with more than two players, better players will place higher most of the time. This could explain why a good player's score continually rises.

Elo also works with more than two players. Let's say you play in a 5 player room and get 3rd, then this is rated as a win against the 4th and 5th and a loss against the 1st and 2nd.

The problem with the implementation in Cultris is that as long as you can maintain a certain win ratio (placing better than your opponent in roughly 7 of 8 games), you will rank higher and higher. It doesn't matter how many points your opponents are behind. Theoretically this shouldn't happen with Elo. I don't know the constants used in Cultris, but generally it's like: if you need a win ratio of 10:1 games to be 100 points better than your opponent, then you need a ratio of 100:1 for 200 points, and a ratio of 1000:1 for 300 points.

11/3/2014 10:34:33 AM
Gravatar
Total Posts 52

Re: Ranking system flawed

Maybe the problem is that the scoring system doesn't account for the FFA mechanics, where the better players' advantages are compounded, depending on the number of players, so that they are able to win more often than the system predicts.

The lines received are an average of all the lines sent. The more players there are in the room, the less lines the better players will receive relatively since they send the most. Tbe less lines received, the more lines they block as well as send. Their speeds are generally faster as well, which helps all these aspects. All of these factors only increase their chances of winning.

11/3/2014 7:32:08 PM
Gravatar
Total Posts 516

Re: Ranking system flawed

You pretty much described what happens, if a top player is in a room with many mediocre players. The other extreme is when a mediocre player is in a room with many top players. This situation is in favor of the mediocre player because the top players will top out each other - with lucky blocking and downstacking the mediocre player will stay alive longer than some top players. Free For All ranking can never be completely accurate. Very agressive players (high BPM, making missdrops, concentrating more on comboing than blocking) will of course perform better in small rooms. These players would be ranked much higher in a 1-on-1 ranking. Another issue is disconnecting.

But this thread is about a different aspect. There's something wrong with the Elo implementation for sure. To achieve a 1000 points lead you have to compensate a loss with thousands of wins. This is definitely not the case. If player A has a win ratio of 10:1 over player B, and player B has a win ratio of 10:1 over player C, then Elo expects player A to have a win ratio of 100:1 over player C. This roughly matches the reality. On the other hand Cultris hands out points as if player A has a win ratio of 10:1 over player C.

As a result the top players can increase their lead arbitrarily because they can surpass that critical win ratio against nearly everyone outside of the top 50. Some of their points are transferred to other players in the top 20. So we have the following situation:

  • Cultris ignores the score difference, if it becomes too big
  • Currently, the score difference is bigger than it should be in theory

A player will benefit from rooms with:

  • very worse players (because Cultris behaves as if the score difference was only 100 points)
  • slightly better players (because score difference is currently bigger than it should be)

If you are in such a room you will rank up, even if playing bad. On the other hand, everything is fine if the room is balanced.