Ok, this is the formula, simplified a bit to make it easier to read, kind of lol.

valueA = attackersAtkPwr (a random amount between attackers min/max atk but to simplify let's just use the max atk number)
valueB = targetsDef (targets defense slightly randomised by a few hundred points here and there to give a different number each time - don't even ask me why it works like this I have no idea lol, but lets just simplify and use the actual defense number)
valueC = 1 - ( ( 2 * valueB ) / ( 2 * ( pow( valueB, 1 ) ) + 500 + ( 150 * attackersLv ) - ( 20 * targetsLv ) ) )
valueD = skillAttackPlus (skill additional damage, so if a skill says it does 1000 bonus damage, it's 1000 here).
valueE = 100 + skillAttackPer ) / 100 (skill additional % damage, so if a skill says it does 50% bonus damage, it's 50 here).
valueF = 1 (this will always be 1 when the attacker and target are the same level, but it's gets much more complicated if there is a level difference between attacker and target)
valueG = ( ( ( valueA * valueE ) + valueD ) * valueC ) * valueF
valueH = addPerTotalDamage (+boss damage, +pvp damage etc.)
valueI = addTotalDamage (%boss damage, %pvp damage etc.)
valueJ = valueG * valueH + valueI

Example 1
Players is level 85 and Monster is level 85
Attacker has 16846 Max Physical Atk
Target has 6450 Physical Defense
Skill does not have any bonus damage, ie it's an auto-attack.

valueA = 16846
valueB = 6450
valueC = 1 - ( ( 2 * 6450 ) / ( 2 * ( pow( 6450, 1 ) ) + 500 + ( 12750 ) - ( 1700 ) ) ) = 0.44
valueD = 0
valueE = ( 100 + 0 ) / 100 = 1
valueF = 1
valueG = ( ( ( valueA * valueE ) + valueD ) * valueC ) * valueF = 7412.24
valueH = 1
valueI = 0
valueJ = valueG * valueH + valueI = 7412.24

So yeh, hitting that monster with a basic autoattack, that doesn't crit, should be around 7.4k damage. If your autoattack is a multihit like on dagger or dual sword users, that number will be spread across the two hits, so like 3k and 4.4k hits.


Example 2 - further simplified
Players is level 85 and Monster is level 85
Attacker has 16846 Max Physical Atk
Target has 30000 Physical Defense (Training Dummy E200 D30000)
Skill does not have any bonus damage, ie it's an auto-attack.

valueA = 16846
valueB = 30000
valueC = 1 - ( 60000 / ( 60000 + 500 + 12750 - 1700 ) ) = 0.17
valueD = 0
valueE = ( 100 + 0 ) / 100 = 1
valueF = 1
valueG = ( ( ( valueA * valueE ) + valueD ) * valueC ) * valueF = 2863.82
valueH = 1
valueI = 0
valueJ = valueG * valueH + valueI = 2863.82


Example 3 - further simplified
Players is level 87 and Monster is level 87
Attacker has 16846 Max Physical Atk
Target has 50000 Physical Defense (Training Dummy E300 D50000)
Skill does not have any bonus damage, ie it's an auto-attack.

valueA = 16846
valueB = 50000
valueC = 1 - ( 100000 / ( 100000 + 500 + 13050 - 1740 ) ) = 0.11
valueD = 0
valueE = ( 100 + 0 ) / 100 = 1
valueF = 1
valueG = ( ( ( valueA * valueE ) + valueD ) * valueC ) * valueF = 1853.06
valueH = 1
valueI = 0
valueJ = valueG * valueH + valueI = 1853.06


So to answer your original question, yes there is a diminishing return on defense stat, the more you get, the less effective it becomes - it's the same with most games to stop defense blocking out all incoming damage and being completely overpowered. Other things are also coming into play though, like skills with ignore defense on them. A lot of the higher level monsters in dungeons have a certain amount of ignore defense built into most of their skills.

If you do the tests yourself on those two training dummies, take into consideration that they are classed as Bosses so if you have any Boss Damage on your items, set effects etc, you need to add them into the calculations. Also the second one is level 87 which also makes a big difference if you are level 85, so you can't really test on that one at the moment.