Bug: We Keep Seeing Dead People!

[klizilii]

On 12/14/2023 at 3:09 PM, JackieKo said:

• For groups who are seeing dead players during their Ops run, what happens when the "dead" players is set as a focus target?
• Do they show up as alive again or does nothing happen? 

 

 

I can add them as the focus target so that I have an alternate way to select them since I can't select dead players using the ops frames.  I can use the focus target to make them my main target to heal them.  I do recall them having HP the majority of the time when my focus target, but iirc there may have been the odd time where they don't show the correct HP in the focus target (but I can't confirm if this is true or just a bad memory on my part).

So, they are "alive" in the focus target but it changes nothing with the operations frames.

 

The workarounds I know of to fix them appearing dead in operations frames include:

• Can set them as focus target to be able to select them and heal/do things to them (still dead in ops frames)
• Quit the group, leave phase and rejoin group (simply leaving the phase and re-entering is not enough).
• Have the player kill themselves and rez them (regular rez works)
• Release from death last as the healer tends to avoid this bug happening at all.

Based on that, I would say the problem has something to do with the player seeing the bug dies, and not so much when someone else dies.  Since the bug can be avoided by being the last one to release/rez and being the one to quit group & leave phase fixes it.  If the "dead" player leaves they come back and will still appear dead to me in ops frames.  It is random enough that I haven't seen a pattern other than I also have to die for the bug to happen where people appear dead in the ops frame.

[xxSHOONYxx]

On 12/15/2023 at 7:07 PM, Jerba said:

You're correct; I must have confused the dates. But we have definitely seen the bug on live 7.0 in TFB, on PTS 7.1 in R-4, and on live 7.1 in R-4. And when we encountered it in TFB, it was definitely the same bug as in R-4 and nothing that could be fixed with /follow (that's a different bug, related to players appearing in the wrong position after a teleport, and has nothing to do with this bug).

You may be getting your dates confused with tfb too. At release of 7.0 some nim ops had actual dps checks and terror was one of those fights. Many people wiped constantly for hours and did not see that bug on 7.0, at least no one that i know got it. And with how many people wiped on that boss if it happened on 7.0 it would have been reported

Edited December 18, 2023 by xxSHOONYxx

[JenHarkness]

Thanks everyone for the information on the bug. Particular thanks to klizilii who gave me the most thorough answer to the focus target question Jackie forwarded on our behalf. I also appreciate the videos from Jerba, although unfortunately you didn't have your target or focus target in the videos, so they didn’t provide the information we needed.

I wanted to take a minute to respond to a couple of the comments with some additional information so you folks can get a better feel for why this bug is tricky and why I particularly asked about focus target.

First a bit of background information - there are many servers that make up the SWTOR backend, and they communicate with each other through a fairly complicated system of messages and replication of data. The simplest of those is that your character data is on the Area Server for whatever world you're in, say Hutta, but there are also copies of that on your computer and partial copies of it on the computers of your group members.

Now where things get complicated. We use messages from our group server to let you see information like group member hit points when you’re in a different area or a certain distance away from them. 

A lot of you have noticed that the bug is particularly happening when there is a long distance between the death and the respawn points, and that is likely related because that's one of the times we switch from "area" information to "group" information. We are unable to reproduce this because this only happens when one of the "area" or "group" updates is slow due  to one of those servers being overloaded. This Is almost impossible to simulate in our development environment.

I asked about focus target because if your client was stuck trusting out of date "group" information, then setting the not-dead person as your focus target would force the client to get a new copy of "area" information. So if you use focus target and the focus target has hitpoints, that tells me that the version of the data on the server is correct, and it's just the UI communication that's out of sync.

So what are we doing now?  First, we turned on some more powerful error logging last night (which is a bit laggy, so we don't have it on often) and we're going to see if we see something in those logs that points us in the right direction.

Additionally, I've been using the various information in the thread to look through the code to see if I can spot anything wrong. Because SWTOR is a distributed system with messages that can arrive in various orders, code inspection like this is not efficient and often doesn't find the problem, but we're definitely not giving up on it.
 

[Nee-Elder]

2 minutes ago, JenHarkness said:

Thanks everyone for the information on the bug. Particular thanks to klizilii who gave me the most thorough answer to the focus target question Jackie forwarded on our behalf. I also appreciate the videos from Jerba, although unfortunately you didn't have your target or focus target in the videos, so they didn’t provide the information we needed.

I wanted to take a minute to respond to a couple of the comments with some additional information so you folks can get a better feel for why this bug is tricky and why I particularly asked about focus target.

First a bit of background information - there are many servers that make up the SWTOR backend, and they communicate with each other through a fairly complicated system of messages and replication of data. The simplest of those is that your character data is on the Area Server for whatever world you're in, say Hutta, but there are also copies of that on your computer and partial copies of it on the computers of your group members.

Now where things get complicated. We use messages from our group server to let you see information like group member hit points when you’re in a different area or a certain distance away from them. 

A lot of you have noticed that the bug is particularly happening when there is a long distance between the death and the respawn points, and that is likely related because that's one of the times we switch from "area" information to "group" information. We are unable to reproduce this because this only happens when one of the "area" or "group" updates is slow due  to one of those servers being overloaded. This Is almost impossible to simulate in our development environment.

I asked about focus target because if your client was stuck trusting out of date "group" information, then setting the not-dead person as your focus target would force the client to get a new copy of "area" information. So if you use focus target and the focus target has hitpoints, that tells me that the version of the data on the server is correct, and it's just the UI communication that's out of sync.

So what are we doing now?  First, we turned on some more powerful error logging last night (which is a bit laggy, so we don't have it on often) and we're going to see if we see something in those logs that points us in the right direction.

Additionally, I've been using the various information in the thread to look through the code to see if I can spot anything wrong. Because SWTOR is a distributed system with messages that can arrive in various orders, code inspection like this is not efficient and often doesn't find the problem, but we're definitely not giving up on it.
 

Christmas  comin' early this year from 'Broadsword'  ^  SO GREAT  to hear from another actual  DEV today  @JenHarkness with your  behind-the-code  insight.   Words cannot express how much  loyal $ubscribers like me ( who've been here since 2011 beta ) appreciate  posts like the one you posted here today.

Best of luck with the bug-hunting  and i know our guild(s)  will help with screenshots & videos as best we can.

 ./salute!

[Jerba]

Thanks for your detailed reply @JenHarkness; I totally understand why this is so difficult to debug.

We were very confused why you asked about the focus target because the hitpoints have always been correctly replicated; only the brain state (dead/lucid/stunned) is out of sync.
When I put a "dead" player in my target or focus target, it shows them at full HP but that is no indication it forced a replication of the character; the hitpoints were correctly replicated the whole time.
The reason why the player shows zero HP in the ops group frame is because that UI element is coded to hide the HP bar if the player is in the dead brain state. However, the target & focus target frame always show the HP bar, regardless of the brain state.

You can see it in this video: https://www.youtube.com/watch?v=bXy_pBqH9KU

• At 0:11 I hover over the player in the ops frame; note how the tooltip correctly shows them at full HP:
• At 0:19 I put them in my focus target, and yes it shows them at full HP in the focus target frame but the bug is still present - the player remains in the "dead" brain state.
• At 0:47 it still shows them dead in the ops frame (despite tooltip, target frame and focus target frame showing them at full HP) and the cursor has the revive icon:

Here's another video: https://www.youtube.com/watch?v=m08MMS7VRpo Sadly, the player did not use focus target but you can see at 0:26 that the hitpoints are correctly replicated in ops frame, character frame, target frame and target of target frame; only the HP bar below the nameplate is bugged. (But that might be a different issue and not related to the "dead" player bug; not sure.)

I do not think there is more information we can provide but if there is any way we can help further, please let us know!

And also, please talk to the designers about adding a workaround, e.g. adding an item to stun oneself, or a command similar to /stuck for either stunning oneself, or killing oneself out of combat. (Obviously, care must be taken to not trigger any effect procs or adding Resolve, so it is not abused in PvP etc.) That way, we can quickly work around this bug without wasting minutes on it every time it occurs.

Edited December 22, 2023 by Jerba

[Jerba]

To clarify @JenHarkness, since I fear there is still a misunderstanding: Your post reads like you consider a character dead if they are at 0 HP, but that is not the case!

Whether or not a player is dead has nothing to do with HP, in fact:

• A dead player can be at 10% HP or even full HP if they are being healed after the killing blow. This is caused by race conditions in the effect system, where the heal action is queued up while the player is still alive, and executed after the killing blow.
• An alive player can be at 0 HP! Let's assume they start with 100/100 HP and receive 100 damage; they are now at 0/100 HP but they are not dead! The game client will still show them alive and they can run around and cast abilities. To reiterate, the minimum amount of hitpoints of an alive player is not 1/100 but 0/100, and they'd have to receive 101 damage to die.

Unfortunately, many game systems rely on checking whether the player has 0 HP, instead of checking whether the brain state is set to dead, for example:

• If an alive player is at 0 HP, then jumps (by pressing spacebar) and lands on the ground, the client will play the faint animation (assuming they died), blocking abilities for a second until the animation to get back up has finished playing. Here, the animation system clearly checks the HP instead of checking the brain state.
• If a boss uses an AoE to target nearby players, alive players at 0 HP will not be considered, and will not receive cleave damage. The targeting system ignores players at 0 HP, instead of checking the brain state.

These edge cases are obviously super rare, but anyone who's been around long enough has experienced them.

Please forgive me if I'm wrong but your reply sounds like you are making the same fallacy, thinking that if we're seeing dead players, the hitpoints must be wrongly replicated when in fact, the brain state is wrongly replicated. I don't mean to be rude, I honestly want this bug to be fixed, so I hope I could explain it correctly. Please let me know if anything is still unclear.

[Crystal_Mind]

On 12/18/2023 at 6:28 PM, xxSHOONYxx said:

You may be getting your dates confused with tfb too. At release of 7.0 some nim ops had actual dps checks and terror was one of those fights. Many people wiped constantly for hours and did not see that bug on 7.0, at least no one that i know got it. And with how many people wiped on that boss if it happened on 7.0 it would have been reported

If you do a search for "dead in frame" in theorycrafters you can see discussion of this bug as early as April 27, 2022 (patch 7.0.2). I think the first time I saw it was in July of that year. Even now I don't run into it too often, but once I see it I'll keep seeing it all night until I relog or the game hangs in the loading screen.

I think the funniest thing to me is that while I can't heal the "dead" target directly, I can still throw snowballs and sparkle powder at them.

[Jerba]

Since it hasn't been mentioned yet, I want to thank the devs for seemingly stealth-fixing this bug in 7.4.1. At least, we have not encountered either the "dead players" nor the "stuck in loading screen" bug since, and have not heard from any other groups seeing the bug. It might be that due to lower population numbers, the servers are not as busy but here's hoping it is finally gone.

Also thank you very much for fixing the tooltips in 7.5 so the duration is back to being refreshed every second. Raiding is finally fun again!