"More thoughts on the recent "Pokéstop and Eggs" research by TSR research team"

#PokemonGO: Two days ago, an analysis of "Pokéstops and Eggs" research from TSR research team has been posted. The idea of this project was to show if egg distance is in any way related to a Pokéstop the egg is obtained from. Only 2 km and 5 km egg distances were considered. The conclusion was "yes, they are related", with a p-value = 0.0111.The next day, user /u/LDinthehouse posted an interesting follow-up to this research in which he showed that, even when we simulate the results with controlled ratio of 40% 2 km and 60% 5 km, the results visually look similar to those from TSR researchers, suggesting that TSR research was inconclusive. However, he didn't offer any concrete reason why TSR research was inconclusive nor pointed to any concrete mistake that was made in the process.But now, I would like to suggest some possible mistakes in the TSR research. Many thanks to /u/erlendig who pointed me in this direction with his thoughts and observations, even though I failed to see the importance of sample sizes at first.The test used in this project is Chi-squared test of independence. This is the data that was used in this test:Observed frequencies26 researchers collected total of 1715 eggs with 2 km and 5 km distances from different Pokéstops. Each researcher collected somewhere between 15 and 169 eggs with those distances. Columns are sorted in a decreasing order of total eggs collected.In the table with expected frequencies, I highlighted the cells with the smallest expected frequencies:Expected frequenciesI will quote couple of facts for Chi-squared test, for which I was able to find a source.(...) caution when there are small expected counts. (...) cells that have expected frequencies less than five. Some statisticians hesitate to use the chi-square test if more than 20% of the cells have expected frequencies below five, especially if the p-value is small and these cells give a large contribution to the total chi-square value.SourceThis is very important because it is exactly what happened in TSR research. The p-value was indeed small (0.0111) and the cells with the smallest expected frequencies (the ones associated with 4 trainers who collected less than 40 eggs) contributed huge ~44% of the total chi-square value, the result obtained by /u/tasteface.The chi-square test is also sensitive to small expected frequencies in one or more of the cells in the table.SourceThe chi-square statistic is only approximated by the chi-square distribution, and that approximation worsens with small expected frequencies.Cochran (1952), who is generally considered the source of this rule, acknowledged that the number ”5” seems to be chosen arbitrarily.SourceThe rule which says "all expected frequencies should be 5 or greater" can be found in many places. However, this quote suggests that it is just a rule of thumb and that we have to take with caution any conclusion derived from a table which contains expected frequencies close to 5. And in TSR research there are couple of expected frequencies like that.For all of these reasons, the original p-value of 0.0111 derived in the TSR research should be taken with caution.To solve the problem with highlighted cells in the Expected frequencies table, we can join last two columns of the table together. If null hypothesis is true, then this should make no difference, so we are allowed to do it. This means that, if we are able to obtain p-value less than 0.05 even after joining columns, the conclusion that egg distance is related to a Pokéstop will be stronger.However, this test gives p-value = 0.0693, which is above 0.05.If we join last three columns together, to lose the next smallest expected frequency as well, then p-value = 0.1383.TL;DRThere still might be a connection between egg distance and a Pokéstop. Several trainers have reported that they noticed certain Pokéstops giving more 10k eggs than others etc. But TSR research, IMO, doesn't prove it. We might be better off not jumping to conclusions just yet. A bigger sample is needed. via /r/TheSilphRoad http://ift.tt/2gZwTtI