"Our EX data collection is flawed. Let’s talk about how to Stardardize and Collaborate better."

#PokemonGO: Bottom Line on top: There are gaping holes in our data and some methodology flaws in our studies. If we can standardize data collection and make some assumptions we can come up with something meaningfulLonger version: I’m an Actuary who manages a team that spends 100% of its time working with and analyzing data. I love this stuff as much as I love Pokémon (62m xp, 127k+ catches), and I have become obsessed with solving the EX raid system. The work has been successful (my community is swimming in passes, but that’s a different story for a different day); however, there are still some missing pieces to the puzzle.While the work in this forum has been incredible, nearly every study regarding distribution rates hits at least one of these flaws:1) Selection Bias. Casual players who randomly hit an ex eligible gym and don’t get a pass are extremely unlikely to report anything. By missing this cohort, all of our numbers about low level players and/or bronze gym holders are incredibly misleading. We need to start building in assumptions for this.2) Each gym is different. One local gym by me spit out passes to 16 out of 18 players 2 weeks ago and a similar % on Monday. A Starbucks in Manhattan is clearly not giving passes to 89%. When a study aggregates results from multiple gyms, we need to start classifying them differently based on volume instead of placing in one bucket.3) Players often hit more than one EX eligible location in a week. How do we keep track of their data? If they get a pass from 1, do we throw their trials from the others out? Or do they still have any value? If one player hits 8 spots and fails at each, does their failure count 8 times when we aggregate? I’ve yet to hear an argument that convinced me what the best way to do this is.4) Every pass distribution period is different and Niantic may be tweaking the formula each time. Different periods should be treated differentlyThis is only the tip of the iceberg... it’s only what i have observed so far. I think in a perfect world we’d have a centralized place to report, but with some local “leadership” able to place gyms into buckets base in volume and make other adjustments. I still have no idea how to get the non-reporters involved, but coming up with assumptions regarding the size of that bucket is a good place to start.Thanks for reading. Let’s figure this out in the comments below. via /r/TheSilphRoad http://ift.tt/2BGxB9a