Possible AMD+Vulkan (or RDR2 implementation) tessellation issue

Discussion in 'Videocards - AMD Radeon Drivers Section' started by wirapuru, Dec 10, 2019.

Tags:
  1. wirapuru

    wirapuru New Member

    Messages:
    8
    Likes Received:
    5
    GPU:
    5700 XT Nitro+
    (First of all sorry if this is not in the proper forum, I couldn't really tell. Please warn me and/or move/delete it as it should be done, if not).

    Since RDR2 on PC launch we have seen a lot of issues but this one either is too specific to my system - although some other people had it as well - or too buried under so many others.

    I'll start with the pictures, which should tell enough about it:
    [​IMG]
    [​IMG]
    [​IMG]

    Also, I don't have them at hand but deep snow also has some similar glitches about it (not exactly like that egg carton effect, more like a post-earthquake effect that goes all through it).

    I've tried a lot of ON/OFF settings there to see if anything affects it. The only thing so far was switching the game engine from Vulkan to DX12, but sadly another issue: with DX12 it'll randomly turn into a "black screen state" where I can only hear some game sounds (pretty much like some device connect/disconnect and flicker issues I've seen around here). I can't do anything besides hard-resetting the PC.

    Anyone has any idea what to test, try and observe in here? I've been waiting between AMD drivers updates and Rockstar updates, but no changes so far (it happens since day 1 of both game and 19.11.1 drivers). I'd really appreciate any directions or tips.
     
    connos likes this.
  2. JonasBeckman

    JonasBeckman Ancient Guru

    Messages:
    15,970
    Likes Received:
    1,885
    GPU:
    AMD S. 5700XT Pulse
    Yeah saw it a while ago on the AMD Reddit as something from the November driver that enhanced performance for RDR2 but a problem occurs when running the game with Vulkan and snow deformation looks weird but I suppose the dynamic tessellation effect is overall kinda broken then going by the above screens. (Incorrect shader or something during optimization if they did something here in the driver?)

    Guessing it just needs a driver update to fix it but who knows when that will be.
     
    connos and wirapuru like this.
  3. wirapuru

    wirapuru New Member

    Messages:
    8
    Likes Received:
    5
    GPU:
    5700 XT Nitro+
    Thanks @JonasBeckman that's already more than the silence I've been seeing everywhere I posted this so far. Although not much hopeful since it has been happening since 19.11.1 (officially the RDR2-ready one, and the first one I've used since installing this card).

    Do you have by any chance any links to these discussions? I'd like to follow them.

    (Also, any news about the DX12 black screens? Ever heard about that anywhere?)
     
  4. JonasBeckman

    JonasBeckman Ancient Guru

    Messages:
    15,970
    Likes Received:
    1,885
    GPU:
    AMD S. 5700XT Pulse
    Trying to find it but nothing so far, might have been one of the hundreds of posts in the 19.11.x driver release topics asking about it and maybe that's where I saw it mentioned although I can see earlier drivers had broken deformation for the snow effect as a topic but from the Red Dead Redemption 2 Reddit though it looks to be the same problem.
    https://www.reddit.com/r/RDR2/comments/ds7lzj/pc_deformed_snow_glitch/

    Guessing it's something for the 5700 / 5700 XT and whatever AMD changed.
    Don't know much about the black screen issue though, issue itself has been a thing with Navi but not API specific or game specific so could be something Rockstar needs to address with the D3D12 implementation if it's something that the game is doing with the API.

    Going by issues with stuttering and crashes from NVIDIA users it sounds like D3D12 support overall just isn't the best even with the current patches the game has received but I haven't picked up the game myself so I can only go by what I am reading and comparing with patch notes and experiences from users after these updates are released and how that affected things both good and bad.


    EDIT: Ah so there is a workaround at least, guessing it's shader related then.

    Change the setting for that and the shader likely gets reloaded and then it works for a while and either the driver or something in the game is changed and it breaks again if I'm reading it correctly possibly said shader getting re-initialized again on loading another area.


    EDIT: Just a guess but say the water displacement effect uses tessellation and AMD in a effort to optimize the game for a what was it up to a 8% performance increase simplified the shader so a replacement is used instead or a driver "hack" takes effect but now that same shader is also responsible for terrain deformation and since these persist the effect instead looks wrong although how that would be missed during testing is something I can't quite understand.
    (The game begins in a snowy area it kinda should have been discovered pretty quickly and it's a bit too broken to just accept and roll with it as a side effect of the adjustment the driver is doing for the water displacement.)
     
    Last edited: Dec 10, 2019
    G13Homi and wirapuru like this.

  5. Spider-Vice

    Spider-Vice Member

    Messages:
    22
    Likes Received:
    2
    GPU:
    Radeon RX 5700 8G
    Interesting insight! AMD's shaders for this game are different than NVIDIA's, as the shader cache present in the Documents folder for the game has a different format from that generated on NVIDIA (even has AMD Shader Compiler headers), so this could very well be something Rockstar did that the AMDGPU/OpenGPU stuff they are using doesn't like. Then again it could be a driver bug too and hopefully it's addressed in Adrenalin 2020.

    Strange that the water quality option affects it, I wonder if lowering Tessellation settings helps too.
     
  6. wirapuru

    wirapuru New Member

    Messages:
    8
    Likes Received:
    5
    GPU:
    5700 XT Nitro+
    Wow, I've been reading a lot of your posts and always impressed with the details - but I can't thank you enough for bringing those up.

    About the water details: tbh it makes a lot of sense, although I probably overlooked them (idk if you saw it anywhere, but they have a "casual" high-level low/med/high setting for Water in general, then if you enable Advanced Settings you can change Water Refraction, Reflection and Physics). I'll test all combinations I can in here to see if any pattern follows through and report back.

    Sadly that maybe isn't enough for an accurate analysis of the issue, which comes together with what you just said above:
    Yeah, it's another of those issues sometimes happen, sometimes not... and when they do, not always at the same level. Ugh. That's why I think it maybe passed through the QA, or just was a low priority one since so specific and uncommon to happen..? (Idk how those work inside R* and AMD so just speculation here.)

    But in my case they almost always happen, so I guess I'll just get a spot on swamp areas, try to do like a 5 reloads to see how many times that happen. Then I'll test each of the Water settings until I get some acceptable stats.

    Anyways, thank you a lot @JonasBeckman I have finally something to test in here. I should get back soon with some results.
     
    Last edited: Dec 10, 2019
    connos likes this.
  7. Astyanax

    Astyanax Ancient Guru

    Messages:
    4,413
    Likes Received:
    1,279
    GPU:
    GTX 1080ti
    vulkan pipeline caches are hardware specific.
     
    Spider-Vice likes this.
  8. wirapuru

    wirapuru New Member

    Messages:
    8
    Likes Received:
    5
    GPU:
    5700 XT Nitro+
    After playing around 2 hours with all combinations I could test:

    Water Quality (a higher-level setting, which seems to be a "profile" that controls the more advanced ones): High/Medium/Low

    (The following are the settings that probably are set as High/Med/Low according to the above one - but you can individually set them if you activate Advanced Settings)

    Water Refraction: High/Medium/Low
    Water Reflection: High/Medium/Low
    Water Physics: 0-1-2-3-4 of 4

    I've tried all combinations there. I could save at one spot on Story mode where I could walk around an area for 30secs until that glitch would show up. The good thing is when I've tried those you could actually see the changes without having to restart the game.

    This was very explicit and actually confirmed what Jonas said, that the glitch seems to be related to the Water settings after all: whenever I changed any of the factors of the combinations, the glitch was gone, like if that reloaded something in the engine (and it only happened when changing Water settings - Tesselation i.e. it didn't). But the glitch was back if I just ran around the same swamp area for around 30secs.

    After trying all of the combinations and having almost always the same results both on quantity and quality for those glitches I went to try a weird thing: changing the `deepsurfaceQuality` directly on the settings.xml file. That setting does not exist on the ingame settings - for some reason - but it was found out to be responsible for the distance where some effects - like i.e. wagon trails between others - fade out. Even setting everything Ultra people still had those trails fading out too near. So the only way to set them to Ultra was by editing that file. I did it again - because I had done it before but updates reset it - and suddenly the glitches that happened 90% of the times were now happening in max 10-20% of those, and in a much smaller scale than before.

    Anyways, I wont make this great wall of Chine of text bigger - for anyone with this issue, until they actually fix it - try to do this:

    1. Edit your `\Documents\Rockstar Games\Red Dead Redemption 2\Settings\system.xml` file
    2. Find the `deepsurfaceQuality` setting, it should say something like `<deepsurfaceQuality>kSettingLevel_High</deepsurfaceQuality>`.
    3. Change it to `<deepsurfaceQuality>kSettingLevel_Ultra</deepsurfaceQuality>` and save it

    It should mitigate the issue for now. I'll update in here if I find anything else. Thanks again Jonas for your directions and finds.
     
    Krteq and connos like this.
  9. Astyanax

    Astyanax Ancient Guru

    Messages:
    4,413
    Likes Received:
    1,279
    GPU:
    GTX 1080ti
    can't you fix this by turning off radeon optimized tesselation in radeon settings?
     
  10. JonasBeckman

    JonasBeckman Ancient Guru

    Messages:
    15,970
    Likes Received:
    1,885
    GPU:
    AMD S. 5700XT Pulse
    Hmm that sounds like it could be a possibility if AMD has a profile override in place and that's what they are doing here and not something with the shaders, some games even show glitches for otherwise fairly alright overrides like 32x that would be giving enough steps to not cause any glitches such as when hitting 16x or even lower. :)

    And then a difference in how this gets processed by Vulkan compared to DirectX 12 perhaps since it mainly seems to be breaking when using Vulkan. Wonder if clearing the cache could help but maybe not.
    (Older pipeline files where the game is storing these and other settings and AMD's own Vulkan shader cache perhaps.)
     

  11. wirapuru

    wirapuru New Member

    Messages:
    8
    Likes Received:
    5
    GPU:
    5700 XT Nitro+
    Sorry, I didn't mention that before because posts were already such walls of text.. but yeah I did try to turn on/off "AMD Optimized" tessellation - and also tried manual > 64x one (but none of the other ones) before, no difference here. I will try lower ones later and report back.
     
    connos likes this.
  12. wirapuru

    wirapuru New Member

    Messages:
    8
    Likes Received:
    5
    GPU:
    5700 XT Nitro+
    I forgot to say this too: my overall impression is that I actually have so much less stutter than before when those glitches were happening - I wouldn't know for sure but yeah, it seems more smooth overall. I can imagine how bugged calculations would cause stutters, after all it is an unexpected side effect - so maybe it's just speculation. But although I've been trying to ignore all those technical issues (for immersion, I've really been waiting for this game before anything else), I cannot avoid noticing every single stutter..

    But I could only play for 1-2 hours now at max. As I play more I'll report anything I find. Thanks for the suggestions though - the real goal here is try to find the issue so maybe this could be actually fixed where it should by either AMD or Rockstar.
     
  13. connos

    connos Maha Guru

    Messages:
    1,213
    Likes Received:
    27
    GPU:
    AMD 480 4GB
    Thank you man for this post. I will try you recommendations. I experiment a lot with this problem but never found a fix. Again Thank you.
     
    wirapuru likes this.
  14. Backstabak

    Backstabak Master Guru

    Messages:
    498
    Likes Received:
    190
    GPU:
    Gigabyte Rx 5700xt
    It has happened to me too since the release day. I have tried a few options and it is true that reloading the settings fixes things for a while, but under Vulkan they always appear anyway. Luckily for me though, DX12 works without any issues. I think this bug is related to the Rockstar's implementation of Vulkan, as this haven't happened anywhere else, so a fix must come from them.

    The "black screen" state you describe that happens with DX12 sounds like driver crash, which shouldn't be happening either. I'd probably just try to reinstall with DDU and try it in DX12.
     
  15. Spider-Vice

    Spider-Vice Member

    Messages:
    22
    Likes Received:
    2
    GPU:
    Radeon RX 5700 8G
    wirapuru likes this.

  16. Astyanax

    Astyanax Ancient Guru

    Messages:
    4,413
    Likes Received:
    1,279
    GPU:
    GTX 1080ti
    no, its definitely a Radeon driver math bug, of which AMD's vulkan support has a number of graphical issue causing problems at the current time.
     
  17. wirapuru

    wirapuru New Member

    Messages:
    8
    Likes Received:
    5
    GPU:
    5700 XT Nitro+
    I'll try that again - tbh since I had some weird stutters trying some drivers to try to fix that tessellation weirdness I've seen that AMD drivers seems to be much more affected with leftovers here or there than Nvidia, at least in my experience. But I'm only 1 month or such with this gpu, and really didn't have much time to mess with it. But yeah, the first time I finally did a double safe-mode-reboot with DDU it was the only time those stutters stopped (although later I found out the actual culprit was probably Realtek drivers, ugh..)

    Anyways, I'm not risking drivers leftovers anymore - I'll try doing that later if I can and report back. Thanks for the suggestion
     

Share This Page