At 'How long is' we are constantly collecting all the reviews of all the games on steam and use this data to get a good idea of how much time people spend playing their games. Some basic statistical principles show us that the sample playtimes we get from the reviews are often a reliable indicator. The two most important questions are:
The first question is, how big is our sample actually? I was not able to find a reliable source on what percentage of players post steam reviews but estimates are that this is around 1-2% on average. This assumption is also confirmed when we look at the recently launched STAR WARS Jedi: Fallen Order. This game has 25.806 reviews at the time of writing, if this is 2% of the playerbase, it results in an estimated 1.3 million players which is in line with the 1-2 million estimate at Steamspy. We also see this confirmed with different games of different genres: Sally Face (2016, Indie-Adventure), Monster Hunter World (2018, Action) and Path of Exile (2013, RPG).
Now that we know that our sample size is between 1% and 2%, we can calculate how many reviews a game needs before we can rely on the playtime statistics. If we aim for a 95% confidence level with a margin of error between 4% and 8%, we need a minimum of 150 reviews. Currently 9784 games on steam have 150+ reviews. You can still use our website for games with less than 150 reviews but be aware that this data may be less representative.
If reviews would solely be written by people who love the game, the hours played of reviewers would be far above the average and would not be reliable. If only people that dislike the game would write reviews, the opposite would most likely be true. Fortunately it seems like neither of both are true as people have a lot of reasons to write steam reviews.
According to some interviews of Rick in the linked post, people write reviews to share with friends, to think critically about them, to keep track of games, provide feedback to developers, to be funny, use it to upvote/downvote and spread political messages. The wide variety of motives people have for writing reviews on steam, makes us believe that this sample can be random enough to be representative for the entire playerbase of a game.
When we have more than 150 reviews on a game, we should be able to rely on the playtimes of these reviews to be representative of all the players of this game. The more reviews a game has, the more reliable the playtimes will be.