Updates From Guru3D Folders

Discussion in 'Folding@Home - Join Team Guru3D !' started by Svein_Skogen, Aug 14, 2010.

  1. J_J_B

    J_J_B Member Guru

    Messages:
    159
    Likes Received:
    0
    GPU:
    EVGA: 2xSLI 580HC, 480HC
    Well... lots of testing here but never did find the cause of the general slowdown in the SMP2 client. I'm currently running the 263.14 drivers on the GPUs but there is no change.

    Process Lasso has improved things a quite a bit (nice find Iancook!). I have set the default CPU affinity for "fahcore_a3.exe" (the SMP2 client) to use "cores" 0,1,2,3,4,5 and the affinity for "fahcore_15.exe" (the GPU3 client) to use cores 6 and 7. This makes things nice and neat - instantly saw cores 0 through 5 go to 100% usage. The GPU3 clients tend to eat about 60% of core 6 and 75% of core 7. I also run a virtual machine and found that I had to reconfigure the SMP2 client to only use 95% of requested CPU resources in order maintain responsiveness in the VM. This is with the SMP2 priority set to "low".

    I've narrowed in on a maximum set of overclocks for my GTX-580s. I have the cores at 925 MHz and the memory at 2150 MHz. Could probably get the memory higher but I'm not going to bother. The second of the two cards is a little less stable than the other. To achive stablility the voltage has been increased from EVGA's standard for the 580 HC of 1.088 to 1.125. The allowable max that EVGA has set is 1.150 volts. I found that 1.150 volts was not enough to make the second card stable at a core speed of 950 Mhz so pulled both cores back to 925 and I'll keep the extra voltage in reserve in case I start erroring out work units when Stanford gives us some high atom-count WUs again.

    I noticed that the PPD of the 580s was not increasing as it should as I overclocked. It turns out that I need to run GPU-Z with the "/GTX580OCP" flag in order to turn off nVidia's power management functions, else the PDD did not go up in a sensible fashion... I actually saw a very slight decrease during some tests.

    So here is the issue with the 580 overclocks... each 580 is pulling about 330 watts at the above clocks. When we do get issued one of those high atom-count work units from Stanford I am predicting that the wattage pulled by each card will go up to as high as 450 watts. Those are furmark-like extreme burning numbers and I don't want to kill these cards so I'm going to pull the cards back to 900 MHz on the cores and play with that for a while. It is conceivable that I'll need to return the cards to default values (850 MHz on the cores) when such a WU comes down though.

    1000 watts are currently being pulled by the main PSU. Only about 80 watts of that total is going to the GTX-480, which is on its own auxillary PSU. I need to pick up a second power meter from the home improvement store so I can monitor power draw by both PSUs.

    It seems that the GPU cooling loop has adequate ability in that the temperatures of the 3 GPUs are maxing out at 51 C under their current Stanford work-unit loads. I designed the loop upon the recommendation of other forum members of one 120mm type radiator per GPU, and then added a fourth for good measure since that rule was valid back in the GTX-295 days and I figured Fermi would draw more power. In this case there are two 120 type rads and one 240 rad for a combined surface area of 480mm. The problem now is a lot of the GPU heat passes through the case, which decreases the cooling efficiency of the CPU/MB cooling radiator, which exhausts air out through the top of the case. I'm currently running with the side cover off, which nets me an extra 10 degrees of cooling on the CPU and probably 5 degrees on the GPUs. I'm certain now that I'm going to cut holes for two 120-mm fans in the top-right area of the side panel's plex-glass. I have an unused channel on my fan controller that can be used to power those additional fans. If two fans don't help, I may need to add a 3rd. The Obsidian 800D is a great case for modders, but it's stock airflow is quite poor.
     
    Last edited: Jan 5, 2011
  2. iancook221188

    iancook221188 Ancient Guru

    Messages:
    1,725
    Likes Received:
    11
    GPU:
    GTX 670 SLI / GTX 460 SLI
    sound like your getting to the bottem of the problems anyway and the credit need to go to panther he suggested that to me

    did not think of that, other folder have been finding that with the 580 overcurrent porection GTX580OCP kicking in, wow 330w for one 580 and could go up to 450w :eek:
     
  3. J_J_B

    J_J_B Member Guru

    Messages:
    159
    Likes Received:
    0
    GPU:
    EVGA: 2xSLI 580HC, 480HC
    Yup - folding really stresses our cards and with all the processing capability that a 580 has, a higher binned GPU capable of remaining stable at high overclocks could damage itself with overcurrent protection disabled! I think my PSU would shut down before the cards were run like that for any significant length of time but I'd rather not find out.

    The 450 watts is an estimate but I think it is a reasonable one. I know I can hit that right now with a Furmark Extreme-Burning test, which is why I quickly stopped Furmark'ing a few weeks ago when I realized just how extreme a test it really is.

    I know from Stanford's last high atom-count WU that my GTX-480 pulls 350 watts while folding those monsters.
     
  4. J_J_B

    J_J_B Member Guru

    Messages:
    159
    Likes Received:
    0
    GPU:
    EVGA: 2xSLI 580HC, 480HC
    Just noticed that my SMP2 TPF was down to 3min 51sec while folding a P6076 work unit. This is with -SMP 6, requested CPU usage of 95%, and priority of "low". This is with the three GPU3 clients folding and the virtual machine running.

    Perhaps the low TPF values I've been experiencing on the SMP2 client were simply related to the particular work units that were being pulled down??

    In any case, the SMP2 PPD is reading 12385. That seems more on-par with what PantherX was reporting.
     

  5. iancook221188

    iancook221188 Ancient Guru

    Messages:
    1,725
    Likes Received:
    11
    GPU:
    GTX 670 SLI / GTX 460 SLI
    yer look out for porject 6701/2 they are very slow and low ppd on the smp the bigadv 2684 is also a slow one all the other should get about the same
     
  6. Ghost15

    Ghost15 Active Member

    Messages:
    77
    Likes Received:
    0
    GPU:
    EVGA 465 SC 1GB
    Finally got the OC'ing done, was stable on IntelBurnTest, and to my shock tonight, i got an unstable_machine 48% done with a bigadv! DOH!

    [22:19:59] mdrun returned 255
    [22:19:59] Going to send back what have done -- stepsTotalG=250000
    [22:19:59] Work fraction=0.4114 steps=250000.
    [22:20:03] logfile size=98709 infoLength=98709 edr=0 trr=25
    [22:20:03] logfile size: 98709 info=98709 bed=0 hdr=25
    [22:20:03] - Writing 99247 bytes of core data to disk...
    [22:20:04] ... Done.
    [22:20:06]
    [22:20:06] Folding@home Core Shutdown: UNSTABLE_MACHINE
    [22:20:10] CoreStatus = 7A (122)
    [22:20:10] Sending work to server

    I upped the vCore one notch from the below settings just now to see if it helps.

    Current Times: P6900 (BigAdv) TPF: 23:42 , PPD: 53261.2 Credit: 87659


    145x29 = 4.20GHz
    Memory: 1012.5Mhz (DDR 2025) 2:14 ratio, 9-9-9-30 Timings
    vCore 1.375
    vDimm 1.65
    QPI PLL 1.3
    IOH vCore 1.125
    rest stock.

    Max Temp under IntelBurnTest is 58c/47c (Core 4 always runs alot cooler than the rest for some reason) no load temp is 24c/18c
     
    Last edited: Jan 7, 2011
  7. Mr.Bigtime

    Mr.Bigtime Ancient Guru

    Messages:
    20,794
    Likes Received:
    8
    GPU:
    ASUS 1080TI
    after formating my PC and having issues, i will start to FOLD again in sunday i guess.
     
  8. J_J_B

    J_J_B Member Guru

    Messages:
    159
    Likes Received:
    0
    GPU:
    EVGA: 2xSLI 580HC, 480HC
    HQ IT is migrating our company's infrastucture to new firewalls this weekend so I am lugging my rig home tonight to primarily do some much needed gaming on it for the next 3 days. It will do a little folding while there but not a whole lot. Won't be folding in earnest again until Tuesday morning.

    Afraid I calculated how much it would cost me to use this rig to fold 24/7 at home... an additional $143 per month on top of my normal $85 electric bill!! Are the rest of you high-rate folders eating such costs or are you also folding offsite at an office location? I don't feel that bad folding in the office because we have about 120 employees at this location and many use space heaters at their desks. Then there's the fact that many workstations, servers, lights stay on 24/7...etc. In the end, my rig is a 1300-watt space heater that I'm certain is pulling only a very small portion of the building's total load.
     
  9. iancook221188

    iancook221188 Ancient Guru

    Messages:
    1,725
    Likes Received:
    11
    GPU:
    GTX 670 SLI / GTX 460 SLI
    well i calculated that i should not be pulling more than 750 from the wall for all 3 rigs so it not to bad. jj it sound like you have very high elec bill, i just contribute to the elec in my house
     
  10. Ghost15

    Ghost15 Active Member

    Messages:
    77
    Likes Received:
    0
    GPU:
    EVGA 465 SC 1GB
    I used a Kill-A-Watt to measure actuall usage, I have 80+ Eff PSU's in both my main PC and the Server.

    These were the numbers with my 280, not sure if it changed much with the 465 folding on it.

    Main PC (980x 4.15 OC, 280 OC)

    Watt Usage:
    240-250w On-Idle
    447-485w Full Load CPU & GPU

    Cost per 24 Hours:
    $1.05-$1.10 On-Idle
    $1.96-$2.13 Full Load CPU & GPU

    Cost Per 31 Days:
    $32.70-$34.06 On-Idle
    $60.90-$66.08 Full Load CPU & GPU

    Total Per kWh: 0.18313 USD

    Server (965 OC 3.8GHz)

    Watt Usage:
    240-250w On-Idle
    410-415w Full Load CPU

    Cost per 24 Hours:
    $1.05-$1.10 On-Idle
    $1.81-$1.82 Full Load CPU

    Cost Per 31 Days:
    $32.70-$34.06 On-Idle
    $55.86-$56.54 Full Load CPU


    Yes, they both use about the same wattage Idle.

    Total Cost to run both Folding (100% Load) for 1 month:

    $116.76-$122.62 (USD) Plus about $50-$60 for my other electic usage

    Yes I 'eat' this cost, as I run this all at my home. It does offset my heating (Gas) bill, as they do a good job heating my condo.
     

  11. J_J_B

    J_J_B Member Guru

    Messages:
    159
    Likes Received:
    0
    GPU:
    EVGA: 2xSLI 580HC, 480HC
    That is true about the heating... though it would not offset my oil heating costs enough to justify it in my wife's mind (there's my true problem :)).

    What I'm planning to do long term is upgrade the components in my current system (memory, CPU, etc.) and hold onto the old ones so I can gradually build a second X58-based system that will take the place of this rig at the office. That way I'll be able to keep something folding 24/7, though the folding rate will certainly be much less.
     
  12. J_J_B

    J_J_B Member Guru

    Messages:
    159
    Likes Received:
    0
    GPU:
    EVGA: 2xSLI 580HC, 480HC
    Sigh.....

    Was having odd problems getting my monitor at home to work today. It worked only on one of the two DVI headers of the video card in the secondary PCIe slot today. This was a nuisance since the system post info shown when the system first powers on is only displayed on the 1st DVI header of the 1st video card. Windows was displaying fine with the monitor connected to the second card once the logon screen appeared. So, I investigated this evening...

    Keep in mind that I was moving the monitor connector between cards while trying to figure this out... and that made things worse because it turns out there was a bent pin in the monitor's DVI plug. This has caused the thin plastic wall between two of the pins in FIVE of the DVI connectors on the cards (all four connectors on the 580 Hydro Coppers and one on the 480 Hydro Copper) to break. The small pins in each little divider (there are two) were pushed around, resulted in poor contact or even a short in one case. I wasn't clued into the reason until I wiggled the cable a little bit during system startup and saw the system post screens using card 1, albeit with one color missing. Contrary to what you are thinking, it didn't take much force to insert the DVI plug and cause all this damage - no more force than that which is sometimes needed when the shield around a DVI plug has been deflected just a tiny bit.

    I have used a sewing needle to gentle remove broken pieces of the plastic between the two affected pins in each socket and reposited the pins such that the video connections are once again working.

    It just really bugs me (sorry to vent here but I had to tell the story)... this is $2000 worth of new hardware damaged in just a couple minutes of troubleshooting and these cards now have no resale value as-is. I guess I could send the cards into EVGA to have new DVI sockets soldered on but that would be at my expense and would not be cheap.
     
  13. iancook221188

    iancook221188 Ancient Guru

    Messages:
    1,725
    Likes Received:
    11
    GPU:
    GTX 670 SLI / GTX 460 SLI
    that does not sound like fun :cry:

    how much will it cost to get it repaired and does that mean you would keep your warranty im guessing that the damage could affect the rma if needed

    could you make the 580 the second card how many montiors are you needing to power
     
  14. J_J_B

    J_J_B Member Guru

    Messages:
    159
    Likes Received:
    0
    GPU:
    EVGA: 2xSLI 580HC, 480HC
    I only use one monitor, and all the connectors are working again because I got into each one with a sewing needle to remove a few bits of broken plastic and reposition the two pins that were affected in each connector.

    I sent a tech support message into EVGA explaining what happened and requesting an estimate on the cost to solder four new DVI connectors onto the two 580 cards and one on the 480. I expect the cost to be several hundred per card so I'll probably just be carefull and live with this. There's no chance of ever selling these now, unless the problem is repaired - guess I'll be folding with them for years to come until they fry! If the connectors were to get iffy then I could use a proper solder removal gun along with some copper braid to remove the existing connectors before soldering new ones on myself.

    You're probably right about the RMA. I don't know how EVGA is but I wouldn't put it past most companies to blame something as simple as this on a larger problem. "Umm, yeah the main GPU chip fried because you had a bent pin in the DVI connector."

    I can't help but feel stupid about this but then I think of the 55 widescreen monitors I just distributed and set up in the office... it took more force to push their DVI plugs onto our laptop docking station DVI connectors then what I used here.

    Lesson learned... always check for bent pins on a monitor plug. I had no idea the material the connectors are contructed from could be so easily damaged. Come to think of it, I don't think the black material making up the socket walls is LCP (liquid crystal polymer) as used to be the case. That material is quite rigid and probably would have resisted normal forces.
     
  15. Mr.Bigtime

    Mr.Bigtime Ancient Guru

    Messages:
    20,794
    Likes Received:
    8
    GPU:
    ASUS 1080TI
    {Edited} iam so happy and glad about my current points..dont know what happened magically..updated some windows patches etc...just that.. :D look man look..11k points!

    1 M here i come!!!!

    Post edited and moved to appropriate Thread {F@HTL}
     
    Last edited by a moderator: Jan 19, 2011

  16. Athlonite

    Athlonite Maha Guru

    Messages:
    1,322
    Likes Received:
    39
    GPU:
    Pulse RX5700 8GB
    normally it shouldn't get damaged that easy you must of been pushing on the plug extremely hard to break them ...... albeit you've done a temp job to fix them if you've got a steady hand and patience you could just find a few dead older cards from a PC shop anything they would throw out that has an DVI connection on it de-solder however many you want from the dead cards and de-solder / re-solder the old/new one in place on your 580 and 480 :nerd: I've done it a couple of times for people takes about 30~35mins for each DVI connection
     
  17. Athlonite

    Athlonite Maha Guru

    Messages:
    1,322
    Likes Received:
    39
    GPU:
    Pulse RX5700 8GB
    yay just moved into spot 198 (my bad misread)
     
    Last edited: Jan 22, 2011
  18. iancook221188

    iancook221188 Ancient Guru

    Messages:
    1,725
    Likes Received:
    11
    GPU:
    GTX 670 SLI / GTX 460 SLI
  19. Athlonite

    Athlonite Maha Guru

    Messages:
    1,322
    Likes Received:
    39
    GPU:
    Pulse RX5700 8GB
    ah fudge pucket everytime I take my watchful eye of my PC the SMP A3 WU frigs up it'll be folding along nice as pie then as soon as im not watching it blam EUE at 88% for the 1st one and prolly the second one aswell temps are all fine the CPU doesn't get over 50c and both GPU WU's seem to be working fine aswell grrrrr just can't seem to win
     
  20. PantherX

    PantherX Folding@Home Team Leader Staff Member

    Messages:
    1,380
    Likes Received:
    0
    GPU:
    Nvidia GTX 1080 Ti
    In that case, maybe your RAM Voltage needs to be tweaked a little to get it fully stable.

    BTW, when you get the WU to EUE at 88% is it the same WU as before or not? If it is the same WU, it might be a bad WU so you have to report it here.
     

Share This Page