"Simulation accuracy : Analysis"

#PokemonGO: I hesitated a looooong time before posting this analysis, but....Well, here we are !So today, I wanna talk about Simulator accuracy for Pokémon GO and how they're not... well, accurate.To demonstrate this, I took the liberty of downloading and analyzing this raid video by KaitoNolan. I took specifically this video as it contains most of the things I wanna talk about (Mostly Deaths and Quick/Charged attacks.), It's also a clean run as it contains no rubberbending of any kind and no transfered charged attack upon death.Quite the barbaric term I used here. Basically, if one of your pokemon starts a charged attack and dies before the move hits the opponent, then the rest of that move's timer is carried on the next pokemon, which won't be able to act until that timer is finished So let's get on with the analysis.Analysis method :The video was analyzed using a video encoder software called "Yua". It displays a timestamp aswell as a Frame counter (which will come in handy later in this post).The starting point of the Raid I choose is when the Run away and the switch button appears (You can start attacking). The ending point is when those buttons disappear (Attacks are no longer accepted by the game). This results in a 3816 frames long battle (or 127.327 seconds RTA). Note that the video runs at 29.97 FPSAttacks were counted by checking if the move caused the Boss's HP bar to flash. I also double checked if the energy bar was increasing accordingly. Time starts on the first frame the energy is incremented and ends X seconds later according to the GAME_MASTER data.Damage Data comes from https://ift.tt/2Ld5uE3 and moves data comes from the GAME_MASTER file.The team :5 Pokémons were involved in the raid fight, the 6th went un-used.Let's introduce our fighters :Gengar, Hex/Shadow Ball, 15/15/15 IVs, Lv. 40Alakazam, Psycho Cut/Shadow Ball, 15/15/15 IVs, Lv. 40Bannete, Shadow Claw/Shadow Ball, 15/15/15 IVs, Lv. 40Absol, Snarl/Dark Pulse, 15/13/15 IVs, Lv.40Cacturne, Sucker Punch/Dark Pulse, 15/15/15 IVs, Lv. 40Basic data :This first data will count the number of successful attacks during the raids, aswell as the dodges. The resulting total damage is also displayed.https://ift.tt/2L43NvP splitted the data into "waves". Each wave starts with a Fast attack and end with either a death or a Charged attack.The damage seems to checks out with our knowledge of Tier 3 bosses having 3k HP, however you might notice that I highlighted 2 numbers :Alakazam Wave 4 (Fast Attack)Cacturne Wave 4 (Fast Attack)What happened there is that in the 4th Psycho Cut of Alakazam's Wave 4 hit Gengar at the same time it's Focus Blast landed on Alakazam, killing him. I'm not sure if the hit actually counts in that case but I couldn't notice any rubberbending, so I left it for transparency.For Catcturne's Wave, the 4th Sucker Punch landed on the same frame as the fight ended. As it is the server who decides when a fight ends, I'd be surprised if this attack was actually accepted by the server and sent back to the client in less than a frame as there is always some latency. So again, not sure if that hit counted, but I left it for transparency.However, this does not change a thing. One of those hits are valid for sure. (I'd personally say the Psycho Cut is the valid one.)If the Psycho Cut is the one invalid, the damage count would fall to 3008. If the Sucker Punch is the one invalid, the damage count falls to 3007. Either way, it's still above 3k.Now that we have those datas, it's time for the part that interests me : The timing.Applying Pokébattler's timing data :So on my side, the fight lasts for 127.327 seconds *in real time*, so I'm gonna take Pokébattler's timing data which includes Quick/Charged delay and Death time, apply it to every wave and see if it checks out.But before I do, to keep things fair, I'm gonna track the wasted time first. That means the time where Kaitonolan was inactive or the time wasted/gained by dying during a move.Those timings are as follow :Gengar wasted time : Frames 801-822 (0.701 seconds).Alakazam wasted time : Frames 1102-1118 (0.533 seconds); supposedly dies 0.23 seconds early. We'll go with 7 frames (0.233 seconds), so 0.3 seconds wasted.Banette wasted time : Dies 6 frames early (-0.2 seconds)Absol wasted time : Frames 3033-3056 (0.767 seconds)Cacturne wasted time : Frames 3599-3621 (0.734 seconds); Recovers theoritically 0.4 at the end (Fight ends at the beginning of the quick attack damage windows, which is 0.3s out of the 0.7s cooldown). Total of 0.334 seconds lost.Alright, now we have some fair data to work with, they will be added in the form of "Wasted time" in the spreadsheet. Regarding the game data, we are working with :Hex = 1.2 secondsPsycho Cut = 0.6 secondsShadow Claw = 0.7 secondsSnarl = 1.1 secondsSucker Punch = 0.7 secondsShadow Ball = 3 secondsDark Pulse = 3 secondsDodge = 0.5 secondAnd now we apply Pokébattler's data :Death time = 1.9 secondsQuick attack lag = 0.028 secondCharged attack lag = 0.070 second.We run the results :https://ift.tt/2Ld5vI7 as you can clearly see, there is a massive discrepency of around 6.5 seconds in those calculations. It doesn't seem like much here because the fight was cleared by a 49 seconds margin. But it makes quite the difference when simulating tighter fights.So why such a discrepency ?Let's run 2 Hypothesis.Hypothesis N°1 : Lag is not accurate :Pokébattler's calculation of lag is merely an average of the expected lag within a fight. What if KaitoNolan had way less lag than what you would normally expect ?With that in mind, we're gonna run the same timing simulations but with more optimism, as in no lag whatsoever.https://ift.tt/2L2egYx seconds off. We're closer, but this is still a big discrepency.Hypothesis N°2 : The clock is lying (and the game runs slightly faster in the video than intended) :So maybe our values are correct but the in-game timer is wrong.To check that, let's rewind the video back to Frame 0. The clock starts at 177 seconds. However, it only stays as such for 6 frames (0.2 seconds) before ticking down to 176, which would imply the clock starts around 177.2 seconds.Going forward in time, the raid stops 5 frames after the clocked displayed 49 seconds, which would implyed there is still around 49.833 seconds on the clock when the raid stops.Making the substraction, 177.200-49.833 = 127.367. Merely 0.04 slower than the RTA, so I'd say there is no issues with the clock.Second AnalysisSo, why are we getting such a difference between Pokébattler's time and the in-game time ?To find out, let's go back to the beginning of the video.However this time, not only will we count the number of attacks, but we will also note down their timestamp, or the time in the video where the move landed.https://ift.tt/2Ld5xjd how was lag calculated here ?For the first hit of the raid, I simply substracted the first window damage of Hex according to the GAME_MASTER.For every consecutive Quick attacks, I substracted the difference in timing aswell as the cooldown for the move. If there was any dodge or wasted time during the attack, I also substracted that aswell, since they're not part of the lag.For every charged attack, I substracted the remaining cooldown of the previous Quick Attack move, the first damage window for the charged move, and again, any potential wasted time.For a Fast attack following a Charged attack, I substracted the remaining cooldown of the charged attack and the first damage window of the quick attack.The average lag obtained is around 24ms. This is close from what Pokébattler uses for their simulations.However, you might have noticed something : I left out the lag between a death of a pokemon and the next pokemon's Fast Attack. If you are observant, you might also notice that they are far faster than the supposed 1.9 second death time delay that Pokébattler has calculated for their data.You may have also noticed that I made no distinction between Fast attacks lag and Charged attacks lag. That is not important but I thought I'd mention it.So what if we stand on the fact that the Death Time is 1.9 second and that the game is just glitchy ?If we do that, we encounter 2 issues :There would be 2 invalid Psycho Cuts, 2 invalids Shadow Claws, 1 invalid Snarl and 2 invalids Sucker Punch.This makes a discrepency in the energy count, since there shouldn't be enough Quick attacks in every Wave 1 (Except Gengar) to throw their respective charged move.But for the sake of argument, let's assume that it's another glitch from the game that gives us free energy. We still have another issue.Mainly, the damage counter is down to 2941 ! 59 HP short from defeating the boss. This wouldn't be such a big issue in this case, because there is still almost 50 seconds on the timer, but imagine on tighter fights, this would be a big deal !So it looks like there is a massive issue on our hands regarding death time. This is one of the two main inaccuracy of simulators that I wanted to point out. So let's set the record straight and review the mechanics of the battle.Battle Mechanics :Death time.To be fair, I'm not sure I understood how Pokébattler made their calculations. If I got this correctly, they calculated between the moment of death and when the combat log showed "super effective" for the first time.However, Combat log only applies when the pokémon is physically present on the field (When it's name and CP appear). But while the combat log doesn't lie, the game does.There is always a discrepency between what's shown on screen and what's happening in the game. It's mostly noticeable on Android where you can click buttons before they actually appear on screen.GoBattleSim has taken the approach of setting 1 second as their death time in accordance to the GAME_MASTER "SWAP_DURATION_MS" value. I cannot blame them for that, but that value is actually a trap.So how does it work then ?First, it's important to note that the game treats swapping and deaths like 2 distincts methods of switching.When a pokémon is manually swapped out, the game applies the "SWAP_DURATION_MS" delay of 1 second. Here is the tricky part : When the pokémon arrives on the field, it does not have a proper target assigned. This is where another value of the GAME_MASTER comes into play : "RETARGET_SECONDS". This value applies an extra delay of 0.5 second, after which the pokemon gets it's target assigned and input is permitted for the player.However, a death is not treated as a swap by the game. Instead, upon a pokémon's death, the next one is immediately out on the field. But yet again, it has not target assigned, so, the game calls upon the "RETARGET_SECONDS" value and add an extra 0.5 second delay, after which the pokemon gets it's target assigned and input is permitted for the player.So, Death time would appear to be around 0.5 seconds, but this is not exactly true. It could be lower or higher depending the phone that you have.Since most of the time, you're gonna die from a Charged attack, there is gonna be lots of particles displayed on the field, combined with the death animation, this can cause a lot of lags on devices. So the value could be lower if the game compensate for lag too much, or higher if it does not compensate enough.I demonstrate this behaviour in the following video : https://www.youtube.com/watch?v=dEWqa_F93i0But then, why the delay ?Even if the Pokémon's HP reaches 0 on the client side, the death animation and the swap won't happen until the client receives an acknowledgment that the pokémon is dead from the server.I believe this 0.5 seconds is to account for most latency on the client-server communication and prevent an unfairness during battles for those who have higher latency. This way, I personally suggest the server sends out a packet containing the timestamp of the Pokémon's death, and then the client would swap the pokémon and allow input at the timestamp + 0.5 seconds (and not from when the swap actually happen), this would account for network lag, but not device lag however (In this scenario, the client could allow firing an attack earlier or later than intended if previous lags has offset the in-game battle timer)Attacks lags." Lag is defined as unintentional delay caused by processing time, programmer bugs, internet delays and other side effects of being a multi-player game. Quick Attack Lag is the extra delay between the end of a quick attack and the start of the next attack. Quick attacks are queued on the server side, and generally have very short delays."" Special Attack Lag is very similar to Quick Attack Lag. It is defined as the delay between the end of a special attack and the beginning of the next quick attack. For unknown reasons, Special Attack Lag is higher than Quick Attack Lag. "Source : https://ift.tt/2L4ueS8 definition of a lag is correct, but "Quick attacks are queued on the server side" seems to be a bit confusing to me. As this seems open for interpretation on how the whole system works, here is my hypothesis :To prevent delays, the game can "store" attacks. For example, if you double tap your screen rapidly, 2 quick attacks will be stored in the game, and they will be both executed consecutively on screen without further input from the trainer.The game then handles locally the cooldown of attacks, aswell as the damage to the defender and your energy.The defender however, works in a "predictive mode". That means the server sends the signal to the client to make the defender make either a Quick or charged attack. However, if a brief communication error were to occur, the client would try to predict up to 1 attack from the defender until receiving correction from the server.So how does the server fits in all of this ?My personal hypothesis would be that the client sent to the server a data in the form of "I damaged the defender at X seconds remaining on the clock", one after the other, and the server would only check for the validity of the move, and would then update the battle on the server side.But if the server receives an invalid move from the client, then the server would resend every battle data that it has to the client, and the client will updates accordingly.Lastly, if the server happens to lose communication for a certain period of time, then the server would stop the defender from acting any further while awaiting reconnection. The timer however would continue to count down to 0.This hypothesis was made by running 2 experiments.The first experiment I made was to go on a battle and intentionally cut the communication with the server, after which I would start attacking over and over with only quick attacks to see if the game would allow me to do so.Now you're probably wondering how do I prevent the game from communicating with the server without triggering the "No Internet connection" error ?Well, the way the game checks for connection is by simply looking if there is an active connection or not rather than wether or not it has access to Internet. So by setting up a hotspot on a second phone that has no SIM card nor Wi-Fi and connecting to it, we can effectively trick the game into thinking that there is a connection while there isn't.The expected results to confirm the attack was queued server side is if :The game didn't allow inputs from the player in this situation.The game didn't have any delay coming from the server.However, as seen in the video below, I was able to attack the defender "just fine" on the client side.As for the delay, it is still present : https://ift.tt/2Ld5xQf second experiment was to go on a battle, rapidly tap on the screen to store 2 quick attacks in memory and quickly cut the connection before the second attack gets a chance to display on screen.This was made by switching the screen's phone off by pressing the power button after the 2 taps on the screen.The expected results to confirm the attacks were queued server side is if :The second attack damage wasn't rolled back upon reconnecting to the server.The energy from the second attack wasn't rolled back upon reconnection to the server.As seen in the video below, neither of those results were achieved. I tried many times with different timings, but I could not get it to not roll back the second attack.You can find my experiments on those videos :https://youtu.be/tBppAKNr-WYhttps://youtu.be/g27PGmyxJfIRemember that those experiments merely constitute my evidence for the case, not undisputable proof, so don't take it as absolute truth. My understanding/logic of the whole thing CAN be flawed. Unless someone goes ahead and checks the full communication method between the client and the server in a fight, how a battle work will be left to debate.So I suggest the lag is entirely client-sided, but now you might be wondering why ?Remember at the beginning of this post, I told you that KaitoNolan's video was running at 29.97 FPS. In addition, my 2 experiments videos were encoded at a full 30 FPS, but it's a bit to leave it at that for the anlysis.I can't speak for KaitoNolan, but as for myself, the RAW footage that I got to work with was not constant. It was oscillating between 25 and 29.12 FPS at best in a normal gym battle.For reference, my first exepriment originally runned at an average of 28.082191 FPS according to VLC, even though AZ Recorder was set to record at 30 FPS.So if the recording doesn't reach a constant 30 FPS, I highly doubt the game does. And even if it did, the framerate between the original and the encoded footage would still be different, causing inconsistencies when analyzing it.So now, you might be wondering why I didn't analyze the RAW footage then ?I couldn't. Vegas wouldn't open the file and Yua was distorting the video, extending it and creating lag where there was any. Furthermore, the footage from KaitoNolan's video was directly taken from YouTube, which means it was already NOT the original footage anymore, since it was already edited.So now, does it mean that the lag is merely a result of the framerate difference and does not actually exist in-game ?Well, no. As we saw in the results, there are negative and positive lag, so it's proof that the game constantly tries to correct itself. However, there is still an amount of around 2.6 seconds of lag, I'm almost certain such a diffrence can not be caused by the framerate unless you were to completely slow down the video.But how far does it makes my analysis off ?. Well let's check out !Side note : It does make sort of a difference that the lag is client based instead of server based, becausetas lag isn't equal for every device, it would be pointless to simulate battles with a preset lag that isn't accurate to your device. Just wanna throw that out there.New data, new calculations :So with everything we've learned so far, it's time to make another calculation on KaitoNolan's raid.But before that, we need to fill in the last of the lags with our new death delay of 0.5 seconds. So I'm gonna substract that aswell as the first timing for the damage window of the Quick Attack.Quick rundown :Alakazam first attack : 0.264 seconds of lagBanette first attack : -0.050 seconds of lagAbsol first attack : -0.148 seconds of lagCacturne first attack : -0.099 seconds of lagWith that, our new total lag amounts to 2.570 seconds.So, in addition of our lagless time and new death time, 2.570 seconds will be added to the total result.https://ift.tt/2L52L2M 0.055 second discrepency. That's not perfect, but it's extremely closer than our 2 previous results !Now where could those last 55ms of inaccuracy ?Well, I have 2 hypothesis :I could have missed 1 or 2 wasted frames.I'm only human after. Interpreting what is a wasted frame based upon where I think a Fast attack ended is pretty subjective.But it wouldn't explain everything. After all, if I missed 1 wasted frame, the inaccuracy would go down to 0.022 and if I missed 2, the inaccuracy would go down to -0.012. Either way, it's not 0.The lag calculation is inaccurate.Again, this is as far as video analysis can bring us.You see, when I calculated lag for a charged attack for example, I'd take the remaining cooldown of the current Fast attack and add to it the window of the Charged attack and then substract the whole on the timing difference.Let's take for example Banette. When I calculate his lag on Shadow Ball, I take the remaining cooldown of the previous Shadow claw, which is theoritically 0.45 seconds, add to that the 2.4 seconds necessary for Shadow ball to hit, and substract the whole timing to the difference between timestamps.However, this sum adds up to 2.85 seconds, which is impossible to get on a 30 FPS video.The closest we can get is either 85 frames (2.833 seconds) or 86 frames (2.866 seconds), so there is bound to be some lag on my spreadsheet. In this case, it happens to be a 0.019 second lag. The same thing can also be noted when calculating Absol's lag.As I'm not exactly sure how I could correct this inaccuracy, I prefer to leave it as-is.Conclusion :So, does that mean that simulators are not to be trusted ?Well, in the case of GoBattleSim, for example, the only inaccuracy are those regarding death time (and potentially lag for some of you), so the only consequence would be that the Time To Win displayed would be higher than what it's supposed to display.Futhermore, if you happen to use dodging during your runs, then the simulators can never be accurate to your playstyle, since the most popular simulators on the market either dodge nothing or dodge all charged/all attacks.The real issue is that a higher death time mess with the rankings of Glass cannons, which are obviously prone to dying a lot.So as a general rule of thumb, always take a simulator result with a grain of salt.If a simulator deems a fight as do-able but you can't do it, maybe it doesn't simulate enough lag compared to what your device is outputting and vice versa.Unfortunately, this doesn't hold true for every simulator...Big (?) issue with Pokébattler :As I said in the post, Death delay and attacks lags were the 2 things that bothered me regarding simulation accuracy.For the purpose of this post, I analyzed a few random battle logs from Tier 4/5 bosses and came across another inaccuracy that seems to be specific to Pokébattler (a.k.a I didn't notice this on GoBattleSim)Normally, when a Raid Boss fires a 1-bar charged attack. There is a "buffer" where the boss must then throw at least 2 Fast attacks minimum before being allowed to throw another charged move. I guess on that part, Niantic decided that 1 bar moves were obligatory way more powerful than multi-bar charged attacks and so they put that in place to prevent the raid boss from decimating teams in 30 seconds.However, it seems like Pokébattler doesn't take that into account. Analyzing 3 battle logs (It was really difficult, for some reason, my browser kept crashing when loading the full log. Those logs were from Ho-oh battles with Extrasensory/Steel Wing and Solar Beam aswell as Steel Wing/Fire Blast), I found out that the raid boss was allowed in the simulation to fire off charged move with as little as 1 quick attack between each charged. Sometimes, they were even fired back to back !This is a big issue as it heavily penalize once again Glass cannon, but it also favors more tanky Pokémons to get higher into the leaderboard (More damage = More energy, so more charged attacks/damage)This issue doesn't occur when simulating as solo attempts since the boss doesn't get the chance to reach 100 energy having firing off 2 or 3 quick attacks (In general ? There might be some cases where you can pull that off.) So tier 3 and below in general are spared from this inaccuracy (as well as solo attempts in Tier 4 and above, but they are only accurate as solo attempt, not multiplayer attempts).Personal conclusion :At the state where things currently stand, I personally drew off 2 conclusions through this whole analysis :GoBattleSim looks like the best simulator out there at the moment.It has what you expect from a simulator. You can customize from players to the content of your parties. It feels very user-friendly and doesn't even require signing up, so it's accessible to everyone from the get go.It has it's share of inaccuracies as stated above, BUT, and this is why I love this simulator, you can even change the battle parameters used for the simulations ! So that means for example that you can change your lag to whatever value you feel is right from your gameplay experience and get a much more accurate simulation than with the average preset. To edit those settings, Click on "Customise" and in the sub-menu, click "Battle Settings", you can change them from there. The only downside is that this window is not exactly user-friendly, although it's not that hard to read. It feels like a better presentation of the thing would be much welcome.I highly advise to avoid Pokébattler for simulating Tier 4 and higher.I'm not sure how high of an impact the second issue mentioned has on Pokébattler's simulator, It may be quite a lot, it may not, but it definitely have an impact. As such, I can't regard Pokébattler's leaderboard of counters as accurate (Although it will get most of the order right a lot of time, because Pokémon GO is such an unbalanced game.).Again, this only stands as my personal opinionAnd with that, I conclude my analysis regarding simulation accuracy.Feel free to debate my points in the comments and correct me if you see anything wrong with my analysis.And lastly, thank you for reading this far (if you didn't skip here). This was really a big project, and being alone, an exhausting amount of work. via /r/TheSilphRoad https://ift.tt/2KZqd1c