"[Research] Statistical analysis on capture rates"

#PokemonGO: After reading this very interesting post by RhyniD who looked at the effect of the colored ring size and "Nice" throws on the capture rates of Pokemon, I decided to have a second look at the data that RhyniD meticulously collected.There are two hypotheses of interest that I thought could be tested more robustly using Bayesian statistics:(1) Does the formula "1-(1-TCR)f", with f decreasing from 2 to 1 with increasing radius, describe well the effect of the ring size on capture rate ? (as posited at the end of RhyniD's post, TCR means theoretical capture rate - based on the circle color. Note that a larger f means a better capture rate).(2) Is there any statistically significant improvement on capture rate when throwing "Nice" (or "Great" or "Excellent") balls ?In order to answer these questions, one has to calculate the likelihood of an observed outcome given the "true calculations" which are performed on Niantic's servers. Since we do not know the actual formula, I decided to use the one described above, where X could take any value from 0 to 3. The question then becomes: "How does X depend on the circle's radius" ? If Hypothesis (1) is true, we should have something like f = (2-R), where R is the size of the radius (which goes from zero to 0.99).This is a bit analogous to the problem of predicting the outcome of N coin flips. If you throw 3 coins for example, the "outcome" that you have just observed is a sequence of all the results. For example, [TAILS,HEADS,HEADS] would be an outcome. You could translate this to maths with "TAILS=0" and "HEADS=1" and then the outcome would be [0,1,1] - an array of ones and zeroes. We can do exactly the same with catching Pokemon (0 = fail, 1 = success).Let's call the outcome array "d_j" in what follows. "j" is simply the tag number for each value in the array (e.g., d_1 is the first element of the outcome array). If the probability of getting heads is "q" (you would expect q=0.5 for a normal coin), then the likelihood (probability) of observing the outcome p_j is given by:L = PROD[ (1-q)1-d_j*qd_j ],where "PROD" is a product of all elements j. This likelihood is also called the "binomial Likelihood".Now, this Likelihood actually gives us the reverse of what we're seeking - if you know the value of "q", then you can calculate L given your measured outcome d_j. What we actually want is an estimate on "q" given the observed outcome. Without going into too much detail, this can be done with Bayesian statistics. For those who know what I'm talking about, I'll assume a flat prior on the value of "q", for simplicity.In the present case, what we actually want to estimate is a little different than "q", because "q" is given by:q = 1-(1-TCR)fAnd we actually want to estimate f.I went ahead and coded this, using RhyniD's data that consists of a sample of 200 throws and their outcomes. Since 200 is not a huge number, it is not possible to choose a small range in circle radii and repeat the problem for many radii and see how the estimate for "f" varies. For this reason, what I did instead is that I tried many values of "maximal circle radius" - this is common practice in statistics. The idea is that you estimate "f" for the complete sample, and then repeat it for only the throws that had a radius smaller than 0.9, and then for all throws that had a radius smaller than 0.8, etc. This way you can actually see whether "f" goes up with smaller circle radii, which is expected here.What you get is this figure. The blue curve is the best estimate for "f" as a function of the maximum circle radius, and the shaded pale blue region around it is the region that is also acceptable within the error bars (for the stats geeks, it's the 1-sigma range). This blue region represents uncertainties which are entirely due to small number statistics (if we get more than 200 throws, this region will shrink). As expected, these uncertainties are larger for smaller maximum radii, because we're constrained to a smaller sample size. I have stopped the x axis at max_R > 0.4 because the error bars get crazy large below that.If it is true that f = (2-R), we should expect our best estimates for "f" to vary from "2.0" (at the smallest radii) to ~1.4 (at the largest radii), with a smooth decrease in between. This may seem counter-intuitive at first given the formula f = (2-R), but the reason for "f" converging to ~1.4 is that the X axis here is the maximum radius, not the radius itself. Hence with a maximum radius of 0.99, we're using the whole sample, and we're getting a diluted result of all radii. This exact value "1.4" comes from a simulated data set that I have generated using f = (2-R).Here are several interesting observations that we can make by looking at the figure:(1) The hypothesis f = (2-R) is not bad at all; we see a "smooth" decrease that is statistically consistent with f values that go from ~2.5 to ~1.6 (instead of 2.0 to 1.4).(2) There seems to be a plateau between radii of 0.7 and 1.0. In other words, there is no basis in this data set to say that the circle size affects capture rate above R = 0.7. I strongly suspect that this may only be due to the current lack of data at R > 0.7 (see this histogram of the radii values in RhyniD's sample)(3) However, at R < 0.7 there is relatively good evidence in the data set that a smaller ring yields a better capture rate. This is not a very strong result statistically speaking (the shaded regions are almost as large as the actual variation), but it is likely to be true given Niantic's statement. Getting way more than 200 throws would really help here.Now onto the next question; do "Nice" throws improve capture rate ? I have divided the data set into two categories to answer this question; (a) all throws with no "Nice" (or better) bonus, and (b) all throws with a "Nice" (or better) bonus. I have then performed the exact same analysis on the two data sets. Here are the results. You can observe that:(1) Both data sets show this trend of better capture rates with smaller radii.(2) The plateau seems slightly better for "Nice" throws in terms of capture rate. Roughly, "Nice" throws converge to f = 1.8 ± 0.3, and regular throws converge to f = 1.6 ± 0.2 if you include all data. This is not a statistically significant result yet, but rather a hint that "Nice" throws may actually help, and that more data is needed to answer the question.(3) At maximum radii between 0.5 and 0.7, we can't see any difference in this data set between "Nice" and regular throws.(4) At small radii (R < 0.5), there seems to be some differences in the current data set (i.e., "Nice" throws would be better for capture), but again it's not statistically significant because the blue and red shaded regions overlap.Unfortunately, most of the answers to the questions above are "we need more data", but there seem to be hints of (1) a slight improvement in the capture rate of "Nice" throws, and (2) a stronger improvement in capture rate with smaller radii.Thanks again to RhyniD for his post that inspired this analysis, and don't hesitate to ping we if you would like me to re-run my codes on a larger data set ! All I need is a set of throws with the following 4 informations for each throw: "Fail or Success (0/1)", "Circle Radii", "Theoretical Capture rate (TCR) based on the circle color", and "Bonus Type (No bonus/Nice/Great or Excellent)". via /r/TheSilphRoad http://ift.tt/2cHWr1v