1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

36 cores / 72 HT

Discussion in 'Die-hard Overclocking & Case Modifications' started by venturi, Nov 18, 2015.

  1. venturi

    venturi Master Guru

    Messages:
    239
    Likes Received:
    82
    GPU:
    4x Titan V 32gb CEO
    PART 1 of 4

    Et Ali

    I rebuilt the prior box and this is its new incarnation
    Yes, dropped the water cooling



    I tried to make an ultimate pc. My personal rules is that it had to be low maintenance, stable, very powerful and do anything I threw at it.

    I run at 4096x2160 true 4k and I like my games to be full eye candy and I like to run a lot of VM for the various things I do.

    While at first glance one would not notice, but there where a lot of details worked out in this build.


    Well here are my specs and some pictures, please enjoy

    thx

    V




    Asus Z10PE-D8 WS dual mobo (version 1.04, bios 1202)
    (2) E5-2699 v3 (36cores/72HT) (O.C. 2.6/4.0 Turbo) (sequential sr numbers)
    256GB LR Samsung ram DDR4 ecc reg
    (QUAD SLI) 4 Titan X SC
    Corsair Aix1500 DIGITAL PSU
    1 Samsung NVMe 950Pro pcie 3.0 (os drive)
    8 Samsung 850 Pro 1TB SSD RAID 0 (apps drive)
    9 noctua fans (almost silent)
    Corsair Aluminum Red LED Keyboard, red level 10 mouse
    LG 31MU97z 4096x2160 true 4K at 60hz dp
    CaseLabs Mercury
    2 modified Scythe ninja 3 rev b heatsinks
    Thermal Grizzly Kryonaut TIM
    Windows server 2012 R2, Ubuntu 15, LinuxMint
    1lb. of flesh and a pint of blood

    OC rock stable up to 107mhz using bios profiles, etc etc
    I've had it at 109, 111, and 112.5, but didn't feel comfortable at that level.
    I run it at 105mhz by default


    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]
     
    Last edited: Nov 29, 2015
  2. venturi

    venturi Master Guru

    Messages:
    239
    Likes Received:
    82
    GPU:
    4x Titan V 32gb CEO
    Part 2 of 4

    medium to high load temps when room is 75F

    [​IMG]


    My prior über workstation was the Z9PED8 WS and unfortunately it can't do NVMe or m.2 with 4x lanes,


    for compatibility and speed the xp941 is very fast and as is the sm951

    in order of speed (high to higher) it's xp941, sm951 ahci, sm951 nvme, 950pro nvme.

    in order of heat generated (warm to warmer) it's xp941, 950pro nvme, sm951 name, sm951 ahci

    I describe the heat from the drives as warm. Under load all chips are less than 49c on all models.
    Having 4 titan x sc right above them may increase the environment heat, but its still with normal parameters.

    If you look closely at the picture you'll see that I have heatsinks on many motherboard components as well.

    There was a company this summer that did a review of the z10 and took a thermal image of the board, so when I saw the review I used the picture as guide to get rid of hot spots

    in addition the 950pro only has chips on the top surface which aids in cooling and allows for all those little aluminum heatsinks.


    The hottest thing on the whole board now is the chipset controller the c612, and that runs about 104F to about 109F and that sits directly under 2.5 out of 4 video cards.

    I can't imagine the drives on motherboard being a heat issue, maybe, but I don't know it might be a heat issue in a tight enclosed environment like a surface or tablet laptop


    All temps are so farrrr below spec I'm not concerned ;)

    [​IMG]

    [​IMG]


    By the wat here are the more relevant bios settings for stability. The bios has many screens but I'm posting some of the more important settings


    Make sure you have the latest bios 1202

    [​IMG]

    Pic the default auto unless you want overclocking profiles, but first make sure your system is stable

    [​IMG]

    Main index
    [​IMG]

    Enable Above 4G decoding or you WILL NOT realize the benefit of the 4th card and you will otherwise have stability issues and STUTTERING
    [​IMG]

    These are your best cpu parameters for speed and stability
    [​IMG]

    if your cpu configuration supports it pick these specific setting in the qpi
    if possible select "cluster on die" in snoop
    [​IMG]

    Enable numa
    Enable acpi
    Enable default mem unless you are overclocking
    [​IMG]

    In CSM enable UEFI when possible, here is a mixed config just as an example
    [​IMG]

    In boot make sure you enable UEFI and Legacy
    [​IMG]

    Also if you use UEFI, those pesky separate raid config splashes are bypassed in the initial boot sequence, saving time.

    I hope that helps
     
    Last edited: Nov 18, 2015
  3. venturi

    venturi Master Guru

    Messages:
    239
    Likes Received:
    82
    GPU:
    4x Titan V 32gb CEO
    Part 3 of 4

    In comparison to the Asus Z10PED8 WS, there have been significant changes in motherboards, but not many if you consider the amount of time that has passed since I first got into this around the mid 90's.

    I've been building my own personal workstation since 1998. In 1999 and 2000 was water-cooling a dual cpu rig with raid. I even water-cooled the video card back then. I had to use dual power supplies at that time. I used to run an obsidian x-24 card then moved up to 3d wildcat dual GPU/CPU video card etc. I did one of the first sli rigs on a test mule than mobo. Then I moved to an 8 core dual 2 motherboard rig from tyan. Then more tyan boards and the first quad sli build that I know of. (Lots of different builds to end up to the referenced one today)

    This just happens to be the 7th incarnation of the original build. Great for games- and when I have to do work from home.


    Anyhow, here are some of my first PCs from 15+ years ago

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]
     
  4. venturi

    venturi Master Guru

    Messages:
    239
    Likes Received:
    82
    GPU:
    4x Titan V 32gb CEO
    Part 4 of 4

    My "Blue Period" and one of the first quad SLI ever

    (NVidia had not even made the quad Sli sound yet, but beta driver support was there in developer drivers)
    and always multi cpu and raid ;)

    [​IMG]

    [​IMG]


    [​IMG]

    "Cable Zen" was a pain then too

    [​IMG]

    [​IMG]


    Earlier this year water-cooling

    [​IMG]

    [​IMG]

    The last one was my water-cooling attempts on the Z10PED8 WS build. While it worked well, the results were not any better than air cooling and not with the maintenance and complexity of the liquid cooling. So not the box is completely air cooled now.

    Bringing us back to "today”:

    [​IMG]

    [​IMG]

    [​IMG]




    V
     
    Last edited: Dec 1, 2015

  5. lmimmfn

    lmimmfn Ancient Guru

    Messages:
    10,349
    Likes Received:
    51
    GPU:
    AorusXtreme 1080Ti
    Nice setup, i thought the WS mobos wouldnt overclock?

    Sounds like a great system to play around with, did you try running 4 VM's with each of the titans 1:1 per VM? i would be curious as to the game performance, i thought it was cool on Linus Tech tips when they did it for 2 VM's.
     
  6. vbetts

    vbetts Don Vincenzo Staff Member

    Messages:
    14,525
    Likes Received:
    1,073
    GPU:
    RTX 2070FE
    BLK overclock and overvolt! :D

    But my god, not enough cores!!!!!!!!!!!!!!!! Love it!
     
  7. venturi

    venturi Master Guru

    Messages:
    239
    Likes Received:
    82
    GPU:
    4x Titan V 32gb CEO
    I have not tried that, but its on my todo list.
    I've run the VMs the normal way so far only (allocate resources, parameters etc)

    as far as the mobo, it has a lot of overclocking potential and tremendous details at every parameter
     
  8. vbetts

    vbetts Don Vincenzo Staff Member

    Messages:
    14,525
    Likes Received:
    1,073
    GPU:
    RTX 2070FE
    Curious question!

    Have you tried to make it into a lan party rig? :D

    I'll have to find the article again, but a small computer shop built a server with 8 video cards dedicated to 8 different virtual servers, and allocated I believe 2 cores and 4gb of ram per instance for some low powered lan gaming.
     
  9. venturi

    venturi Master Guru

    Messages:
    239
    Likes Received:
    82
    GPU:
    4x Titan V 32gb CEO
    ....no sir, THAT had not occurred to me, although my family uses it to serve up multiplier games, just not in the venue you described ;)
     
  10. nhlkoho

    nhlkoho Ancient Guru

    Messages:
    7,678
    Likes Received:
    314
    GPU:
    RTX 2080ti FE
    Holy ****. The cheapest I see those CPU's for is $2,300 a piece. This PC costs more than most used cars.

    Step 1....have money.
     

  11. maur0

    maur0 Master Guru

    Messages:
    751
    Likes Received:
    34
    GPU:
    point of view gtx 570 1gb
    one doubt this motherboard Asus Z10PE-D8
    with dual cpu memory run at 4 channel or 8 channel?
     
  12. holystarlight

    holystarlight Master Guru

    Messages:
    782
    Likes Received:
    55
    GPU:
    Nvidia 2080 TI SLI
    Oh damn! Id love to send you my 13tb worth of bluray rips and get them converted to a more appropriate size! wouldn't take much time with 36 cores/cuda cores, of pure Powaaah!!

    Jokes aside. That's pretty badass system!!
     
  13. BLEH!

    BLEH! Ancient Guru

    Messages:
    5,866
    Likes Received:
    63
    GPU:
    Sapphire Fury
  14. venturi

    venturi Master Guru

    Messages:
    239
    Likes Received:
    82
    GPU:
    4x Titan V 32gb CEO
    Thank you ;)
     
  15. CPC_RedDawn

    CPC_RedDawn Ancient Guru

    Messages:
    7,647
    Likes Received:
    183
    GPU:
    Zotac GTX1080Ti AMP
    Please god post your cinebench scores!!! And also a pic of it using 72 threads!!!
     

  16. Spets

    Spets Ancient Guru

    Messages:
    2,979
    Likes Received:
    41
    GPU:
    RTX 2080Ti XC Ultra
    Please do! :D
     
  17. dwiewolverine

    dwiewolverine Member Guru

    Messages:
    163
    Likes Received:
    0
    GPU:
    Strix GTX 980 Ti SLI
    same boat..

    36 cores / 72 HT Reply to Thread...:banana:
     
  18. venturi

    venturi Master Guru

    Messages:
    239
    Likes Received:
    82
    GPU:
    4x Titan V 32gb CEO
    Cinebench and benchmarking

    short answer: 4700


    details with higher scores:
    I originally just dropped in a shot of the score, then I thought about all the details that go into it. So forgive what should have been a one sentence answer, the devil is in the details:

    CINEBENCH
    I've run a few benchmarks that make more sense to me but I'll run some of the more generic ones so folks can have relevant comparisons, which I guess is the basis for a benchmark ;)

    An example here is Cinebench 15, however(1) it should be noted that cinebench may see all the cores but only utilize them 1-12% of their capacity so its not a relevant example other than a somewhat synthetic benchmark. (2) cinebench does not invoke the turbo mode out of the intel xeons, so the score is running at the base 2.3ghz. It would be interesting to see what would happen if Cinebench could force turbo mode :)


    Cinebench cpu scores on windows 764 were:

    normal (100): 4696 and 4721

    mild overclock (103): 4803 and 4792

    so it also lacks consistency, but here is a pick if the last one from the prior 764 build:


    [​IMG]


    On win srv 2012 r2 and the scores are actually higher, so I don't really see the synthetic benches as accurate or consistent

    Win2012 r2 cinebench 15

    Server 2012 R2
    [​IMG]


    mild overclock (103)

    4861
    4804
    4799
    4830

    Hence I don't have 100% faith in Cinebench 15. It is however handy for some quick checks and it is relevant to other folks' measurements.

    temps stayed low during the cinebench benchmark

    [​IMG]

    I did spend considerable time designing how to air-cool that much hardware in a small case:


    [​IMG]

    [​IMG]

    I'm currently benchmarking the system using AIDA64 and I'll post those scores as I wrap up.

    ***Very Important:
    If you are like me and made an SLI profile for the openGL part of cinebench, the score is about 150pts lower because when you're running the cpu part of cinebench, the NVidia driver doesn't know the difference so its running the SLI profile full bore while you're doing your cpu bench. Turn off SLI and THEN do your cpu bench :) (..damn details)




    I'll run some gaming benchmarks in srv 2012 r2, however those also are not as relevant. "Heaven" for example runs a high score, median score, low low high. Every time, its a reproducible pattern, so I'm not sure why the software mimics that pattern but it does, and its not load or temperature dependent.

    The problem with video benchmarks is that not everyone benchmarks the same parameters for an apples to apples comparisons (exception is locked parameters such as cinebench)

    Example, I run at real 4096x2160 with max eye candy, occlusion, parallax, ssao, AF16. FSAA etc. The gaming benchmarks will be skewed at the very least by the fact its at real 4K, but I'll run those shortly. I also edit and modify games to show all it can and I modify the cfg and ini etc to use more threads in both physics and cpu. I also script it to run cpu affinity for real cores (not HT) and all cores (real cores and the HT threads)

    everyone says they have 4k, but what they are referring to is 3840x2160, my monitor and set up is 4096x2160.

    The benchmarks I like are the built in Metro Last Light, the Painkiller Black edition built in bench, the doom 3 built in bench, the FEAR built in bench etc


    SLI profiles are custom with max visuals and tweaked, in most cases I run AFR2 4GPU SLI mod profiles with the multi threaded optimization

    I had ti use NVidia inspector to force 4gpu. Right now I'm using 353.62, there are too many compromises in NVidia's drivers 354.xx to 369.xx.


    Game BENCHMARKS, all at 4096x2160 maxed visuals, maxed threads, maxed physics (again, parameters are well beyond the game settings menu)

    Doom 3, modified, cfg maxed, using full suite of higher textures, higher lighting, rag doll, liquids, ssao, occlusion, adv physics mods: 97FPS

    [​IMG]



    Painkiller Black cfg maxed 424FPS

    Cryostasis, full physics, threading, full eye candy, textures, etc 291FPS

    Wolfenstein 2009, cfg maxed, havok physics threading, game threading, max eye candy, veil fully rendered: 147FPS

    SOMA 151 FPS -- this is very experimental as I had to start from scratch on that profile. Also I'm still discovering adding visual enhancements. Also SOMA was abear to get 4GPU to actually be used, not just ON.

    FEAR cfg maxed, ssao, physics, occlusion, true 4k, liquids, lighting, destructive physics, particle motion etc etc Min 146FPS, Max 281FPS, Av 188FPS

    [​IMG]


    I'm currently tweaking "Outlast", and I'm tweaking the Unreal 4 engine "Kholat" game. The new engine can really take advantage of the hardware if configured correctly.


    At the end of the benchmarking the video card temp high was 69-71C (158F) the cpus highest was 41C (105F) ram temp was 44C (111F)

    there are other types of benchmarks, that are more important to me

    DATA read write Throughput -->

    example, using MACRIUM Reflect Server version, I can compress and back up 1.9TB full system and data in 6 min (6:18) to a full self restoring image.

    [​IMG]

    and I can restore full windows, apps, and mail data blank blank to fully loaded in about 1 1/2 minutes (1:23)

    [​IMG]

    The Asus Z10PE-D8 WS has a jumper for LSI raid or for INTEL raid. I benchmarked both and looked at their features. The LSI raid requires install drivers as well as the fact that it 1/2 the throughput of the INTEL raid. Also, the intel raid supports TRIM down to the individual sad drives but the LSI raid DOES NOT. So beware of that issue.

    And of course one has to make sure the m.2 pcie 4x drive and the 8 ssd raid are setup correctly
    [​IMG]

    [​IMG]

    I hope that helps on initial benches and perception, and forgive my long winded explanation, the devil is in the details.


    Ciao


    V
     
    Last edited: Dec 1, 2015
  19. airbud7

    airbud7 Ancient Guru

    Messages:
    7,135
    Likes Received:
    3,658
    GPU:
    pny gtx 1060 xlr8

    hahaha...agree :banana:
     
  20. Spets

    Spets Ancient Guru

    Messages:
    2,979
    Likes Received:
    41
    GPU:
    RTX 2080Ti XC Ultra
    Thanks Venturi, that really is the ultimate PC :D
     

Share This Page