We need to talk about UE4 Shader compilation issues

Dunno what you mean that UE4 under dx11 "is fine", it isn't, all UE4 games I've tried, regardless of the API fall into the category of unplayable to me or borderline unplayable due to the insane stutter all games on this engine have; SW: Fallen Order, Outer Worlds, Remnant: From The Ashes and a lot more, this engine is ill on PC and it is ill because it gets no treatment, on consoles it's fine because it's used properly. Games on other engines run without these issues. I managed to get Days Gone and FF7: Remake thanks to dxvk libraries which "skips" shader compilation and does an initinal shader compilation with massive stutters for 3 to 5 minutes once you start the game, but when it's finished, the game would only stutter due to your hardware issues or limitations, not because the engine is struggling. It makes UE4 games behave miles better, which means the key to solve the issues this engine has is the Vulkan API, but devs are moronic ignorants that don't care about feedback and important information as this. Watch these "experts" keep using dx12 and making even more UE4 games on PC stutter to no end.

Also, Horizon Zero Dawn is a great port now and does not stutter so no idea what you are talking about. It does compile shaders every so often when you are playing, but if you have a CPU with 8 threads or more, the game takes advantage of that; it loads your CPU but the game itself does not have any stutter close to what UE4 have. it may stutter, but having 1 or 2 frames dropped in separate intervals is not an issue, having MASSIVE framerate drops every 10-20 seconds like you get them in UE4, that's a major issue.

Also I wouldn't care to sit through a 10 min shader compilation screen if that means the game will not have stuttering when entering new areas, when a new effect killing some enemy appears, a new NPC shows up, you know, like the freaking UE4 that loads shaders on "the fly" and never compiles anything. Dxvk "skips" this and once it loads its own shaders, UE4 can be free of stuttering. What inspired to search for a fix for FF7 Remake is that I wanted to play it so badly, but knowing it's on UE4, I knew it would stutter to no end, so I searched and found a good solution. This should be done on dev side, but eh, whatever, this will do for now. Here are a few video examples:

 

And games on PC under UE4 that to this date stutter a lot just will stay that way and those who bought them but can't enjoy them properly just have to "suck it up" and just "wait" for UE5 games? Lmao. First they should try fixing most UE4 games on PC and then move on to UE5. If they are that lazy, just add a shader compilation screen at the start of the damn game and be done with it, that would greatly reduce the hitching on these games.

 

I have to say I generally question the value of these new low level API's. I know the deep engine code guys at AAA's say they want the flexibility, but in practice what does it actually give you? A lot of engineering overhead and room for error compared to a higher level API. The very best implementations like say id's Tech 5 engine, offer at most, at most, a few fps difference with Vulkan compared to OpenGL. And the bad implementations, which is virtually every other engine, offer stuttering, lower performance, crashes, and a lack of interoperability with software tools and driver settings for enthusiast users. It's so much more effort for a developer to create something that is worse than DX11 or GL4 runtimes. And features like raytracing and others in the future will be locked behind these API's. What is the point of raytracing in Battlefield 5 if you have to enable DX12 to get it working? And by enabling it lose fps, frame pacing, stuttering, crashing, etc. There is no point.

And for indie developers, that same engineering effort goes from bad to crippling. They say there will be libraries that replace the higher level functionality. They always say that when they remove high level functions from the API's. But it never happens. It's been 5 years and where are these libraries? They don't exist and never will. This drives indie developers towards locked platforms like Unreal and Unity, creating another pointless duopoly with top down control where there didn't need to be one.

 

There is nothing for "creative people" to optimize in Epic's engine in this particular case. Go read Epic's own documentation on the matter.

 

link?

 

This? https://docs.unrealengine.com/4.27/en-US/

 

https://docs.unrealengine.com/4.26/en-US/SharingAndReleasing/PSOCaching/
https://docs.unrealengine.com/5.0/en-US/optimizing-rendering-with-pso-caches-in-unreal-engine/

 

Oh, I thought you meant you had something new I had not seen before - nothing in those links is related to people not optimizing assets in a level/map/area/scene correctly, but simply confirms that people are not optimizing.

It supports my point.

You are saying that creative people (I hate to point this out to you) have nothing to optimize, then you tell me to read something in the documentation, you give me a link (thank you) but that only confirms that people have to optimize.

What am I missing?

 

You're missing the point. The process of PSO collection for caching and (pre)compilation is completely out of "creative people" hands and should be performed by programmers, devops and QA. Nothing there in any way concern "creative people".

What you're suggesting though as an "optimization" is a conscious simplification of graphics - which could (not necessarily btw) lead to less PSOs being generated with simpler shaders in them thus possibly leading to less and/or smaller hitches at them being compiled at runtime. This isn't "optimization" to begin with and it won't solve the issue since for that you'd still ideally want to have all PSOs collected and preprocessed - meaning that we're back to step 1, which is out of "creative people" hands.

And yes, the way it is done presently in UE4/5 is far from optimal as the process is pretty badly designed even for those developers who have the manpower to handle all this. And with the engine being "free" and "indie friendly" a lot of dev houses using it simply don't have such manpower - especially as said process is needed only for PC versions of games they are making. This is something which should be "fixed" by Epic at this point.

 

Ah.

So - when they are testing the environments for performance and can clearly see there are issues - they do not think it has anything to do with them? Gotcha.

 

What kind of PCs are you running? Is UE5 faster with shaders?

 

wow what about kena and ff xv? does it work for that also?

 

You would have to test it, I guess it could. Just throw this where the game .exe is: https://www.mediafire.com/file/aykrdbj6vx48sel/dxvk.rar/file

Keep in mind the game could stutter badly and then it settles once the vulkan cache is fully build.

 

That does not matter, the engine should pre-compile all shaders before playing a game, even if it's faster than UE4, if UE5 behaves the same way, the same story will be repeated.

 

They don't see any issues because they are running cached shaders.

 

When testing performance?

 

I think your missing a link. Creatives are the people who are creating assets, level design/building, prop/clutter placement, lighting, textures etc. The only "optimizing" they might do is just to hit targets. It's not optimization for the general game per say. That part is handled like dr_us said. They may send stuff back to the creative side and go "That lighting isn't performant enough. Can you cut down to X". But something like shader compilation... that is not something any of the creatives on a game would really have to know or interact with at all.

 

Fully aware, and yes you are right, yet - there is a problem in passing the buck, kicking the can down the road, and lo and f***ing behold, there are performance issues when they are testing the levels/maps out...that is their responsibility, yet the shaders ARE the responsibility of the people creating the shaders, as they are only created by creatives when they are building assets to be placed into levels/maps by them.

In UE4/5, you can easily see what the shader compilation would be for ANY material you are applying to a mesh - these are NOT hidden and esoteric pieces of data - right-click on ANY asset in the content browser and it will give you the option of viewing the 'size map' in pretty color-coded formatting just how much resources that asset takes up.

Yet, this is not their problem? They created it! Just as, they created the level and the map, and have tested it, and can see the draw calls, and can feel the performance whilst testing it.

Hey! This level is not running at a zippy rate when the player stands in X location - let's run the tests OMFG I have got 50 trillion shaders completely unoptimised, oh well, I'll leave that for someone else to resolve...

Please...

 

They usually run the game with shader cache being populated simply because they do it so often (it's their work) that whatever users will see when running it for the first time ever - they won't.
Also as I've said already even if they will see hitches related to PSO compilation they can't do much about them. "Optimization" in the form of removal of all materials with shaders from your graphics by the artist is hardly a solution.

 

I tried the latest official release of DXVK with an old Frostbite3 game I still like to test things with (Dragon Age Inquisition) and it was a complete stutterfestival.
Your previous post seemed to suggest to me that DXVK does some kind of pre-compilation and pre-caching of all shaders in the game during the first few minutes of gameplay. But it's nothing like that, shaders seams to be compiled on-the-fly as needed while moving around on the map (just like with native DX 11.1) and the stutter is horrible with DXVK (at least for me and in this particular game).