Monsters keep track of all the damage they receive from players and they also keep track of any heals that are made to these players. This information is kept in the monster's threat list.
When a player first attacks a monster, they are added to the threat list along with any other players in their party ready for the monster to keep track of their damage and healing.
PlayerA does 1,000 damage to a monster, the player is added to the threat list with 1,000 threat.
PlayerA then does another 1,000 damage to the same monster, their threat will increase to 2,000.
If PlayerB, a healer, heals PlayerA for 1,000 HP, PlayerB's threat will increase to 500 threat.
Why only 500 and not 1,000? Because damage and heals have different multipliers when converted to threat.
Damage 100%. So 1,000 damage is 1,000 threat.
Heals (including heals from potions) 50%. So a 1,000 heal is 500 threat.
Note: Overhealing (heals that are not needed) do not cause any threat. If a player is healed for 1,000 HP but only needed 250 HP, only 250 of the heal will cause threat.
If PlayerA's pet then does 1,000 damage to the monster, the pet is added to the threat list with 1,000 threat AND 200 threat is added to it's owner, PlayerA, because 20% of a pet's damage is added as threat to it's owner.
This results in:
PlayerA: 2,200 threat
PlayerA's pet: 1,000 threat
PlayerB: 500 threat
Monsters will usually* target the player with the most threat - so in this case, it will target and attempt to attack PlayerA.
*See the Monster Skill target type section.
TANK CLASS SKILLS
Due to the tank classes having lower damage output compared to the damage classes - how do they keep their threat higher than the damage classes?
Most tank damage skills cause additional bonus threat. The amount is displayed in the tooltip of skills with bonus threat.
THREAT SECONDARY STAT
Threat can be increased or decreased by utilising the Threat+ and Threat- stats that can be found on consumable items, passive skills, active skill buffs, and armor and equipment.
Equipping an item with 10% Threat will increase all threat from your damage by 10%. So 1,000 damage will cause 1,100 threat. It will also increase the bonus threat of tank skills, so if a skill has 5,000 bonus threat, it will be increased to 5,500 threat.
The same goes for reducing threat. Equipping items or learning passive skills that reduce threat by -10% Threat will reduce all threat from your damage. So 1,000 damage will cause 900 threat.
Negative Threat is capped at -90% but there is no cap on positive Threat.
Players are removed from monster threat lists when they die, use a hide skill, or go far out of range for a while, or when a threat reset is used - like the Threat Reset Potion.
Pets are removed from monster threat lists if they die or you dismiss them, but the additional threat they provided to their owner remains.
Tanks have several Provoke skills that when used successfully on a monster, force it to target and attempt to attack the tank, ignoring their threat list for however long the Provoke debuff lasts. When the Provoke debuff runs out or is removed somehow, the monster will start using it's threat list again and target the player with the most threat.
Do note that Provoke is temporary while it lasts, and only 1 Provoke is effective so don't waste multiple Provoke skills at the same time. It's a skill to temporarily force a monster to target you while you use other skills to build your threat and hopefully have the most threat by the time it runs out so it keeps you targeted. It does not permanently increase your threat in any way unless the skill itself also does some damage.
MONSTER THREAT RESET
Some boss monsters will reset their threat list during their encounter. You can tell when this happens because "Boredom" will be displayed in the middle of the screen.
When this happens, the priority of the tank should be to build as much threat as possible as quickly as possible - so use a Provoke to keep the boss on you, the tank, and use your high damage skills to build threat. The damage classes should be very careful not to build more threat than the tank and it's a good idea to stop damaging the boss for a moment to give the tank some time to build some threat, otherwise they risk the boss targeting them instead.
Bosses with threat reset include Laioha's Knight and Special, Decomus' Sharlach and Frecky, Wood Garden's Radon, Von Stein's Mansion's Baron Von Stein, World Bosses Sakun, Tundro, Lost Mysterion, Namazu, and Sraclone.
MONSTER TARGET COUNT
Non-boss-type monsters check how many other monsters have a player targeted before deciding to attack them. The amount varies throughout the game, but for dungeons it is usually set to 50, meaning upto 50 monsters can attack a player at once. If that player gets near to, or attacks an additional monster, it will ignore them and attack somebody else in the party.
There are a few exceptions to this rule. Some dungeon monsters have a lower target limit. For example - the Vipers in Golden Temple have a limit of 10. If their target already has 10 monsters attacking (not just other Vipers, but any monsters) they will switch to another member in the party.
MONSTER FOLLOW RANGE/TIME
Monsters have a max follow range and follow time that is reset each time they successfully damage their target or their target damages them.
Lets say a monster has a max follow range of 50 meters and a max follow time of 30 seconds.
The monster will target and chase a player for upto 50 meters or for 30 seconds, whichever comes first, and then it will try to find a new target. If there is no new target, it will reset back to it's original spawn location.
If the monster catches the player and successfully damages them (ie not a miss), OR the player successfully damages the monster, the follow range will reset to 50 meters from the monster's current location and follow time will reset to 0 seconds - meaning it will follow for an additional 50s.
Most dungeon bosses, but not all, have a relatively short follow range to make it more difficult to move them from their spawn location but also to make it possible to reset them if needed.
Max follow time around 20s
Max follow range around 50m
Dungeon trash monsters vary greatly depending on the type of monster and what it is intended to do, but generally they have longer follow times and follow ranges so they cannot be ignored or bypassed easily by just running past them. They will give up eventually if you can get far enough away from their original spawn point without them hitting you or anyone else in the party, but depending on the layout of the dungeon it may not be possible to bypass them.
Max follow time 30-60s
Max follow range 150-400m but sometimes much more
So it's not just a case of having the most threat. You need to be aware of how many monsters are attacking you, for how long, and for how far you have lured them without hitting them.
MONSTER SKILL TARGET TYPES
Monsters will usually target and try to use their skills on the player or pet at the top of their threat list, but there are some skills, usually only used by bosses, that are setup to use a different method of choosing a target. Most bosses have several of these types of skills but they usually have a cooldown so they can't be used frequently.
In these cases, there is nothing the tank can do to prevent the monster targeting somebody else - these skills are setup specifically to target something based on certain conditions to add flavour so it's not just a basic "tank and spank" encounter.
Random: Target a random player or pet.
This is used commonly by a lot of boss skills but it has been toned down in recent times as it was a bit unfair to the melee damage classes that would nearly always be chosen due to their proximity to the boss. It's still commonly used but the damage of these skills has been reduced considerably or they have been given a casting time and red warning circle to give a chance to avoid them.
It is also used by some dungeon trash monsters on their actual regular attacks to make them target and attack players very randomly - for example, the Vipers and Tomb Sand Worm-Worms in Golden Temple. In these cases there is no warning circle as those are generally only used by bosses.
Furthest Away: Target the player that is furthest away from the monster's current position.
This is commonly used for bosses with ranged skills and is often used to incapacitate healers and ranged classes.
2nd Highest Threat: Target the player or pet that is 2nd on the threat list.
This one isn't as commonly used as Random and Furthest Away but it is used by a few world and dungeon bosses.
There are other types like highest HP, lowest HP, lowest threat, specific class (Sorcerer, Prophet, Crusader etc.), but they are not used for any encounter currently.
So as you can see, there are many reasons why a monster attacks somebody other than the tank that isn't always related to threat. Some of them can be countered with some skill and knowledge of the monster's abilities and it's encounter, but some of them not - they are just part of the game to make things less predictable.