Performance Counter monitoring plugin

Discussion in 'Rivatuner Statistics Server (RTSS) Forum' started by stangowner, Sep 14, 2009.

  1. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    PerfCount

    Current Version: PerfCount 0.9
    OS: XP/Vista/7 (32 & 64 bit)
    PreReq: VC++ 2008 Redist

    This plugin allows available Performance Counters to be added to the RivaTuner Hardware Monitoring module. Standard performance counters include detailed information about logical/physical disks, processor, processes, network, ram/paging, and much, much more. Windows includes hundreds of values that can be queried out of the box. If you're new to performance counters, you can open the performance monitor in Windows (Start->Run->perfmon<enter>). Click the '+' button to add a counter and browse the list of available categories and counters. Some software/drivers also make counters available, so you may have additional data. For instance, you'll see categories for .NET and MSSQL if you have either of those installed.

    Here is an example of some of the data that can be extracted. These examples are using the ModType and ModFactor values in PerfCount.cfg for unit translation. Please read that file for additional information.

    [​IMG]

    Here is a brief description of the files included in the download.
    PerfCount.dll - the actual RivaTuner hardware monitoring plugin
    Perfcount.cfg - the configuration for the plugin, which instructs which counters to query, any necessary formatting to perform, and other misc RivaTuner configuration information
    PerfCount.rth - the RivaTuner help file for this plugin
    PerfCountCLI.exe - a command line application to retreive the performance counter syntax and test reading it prior to incorporating it into the PerfCount.cfg - the proper syntax for the counters are "\\Computer\PerfObject(ParentInstance/ObjectInstance#InstanceIndex)\Counter" (\\Computer can be left out for the local machine)

    Installation:
    1 - Copy the PerfCount.dll and PerfCount.cfg files into the "%RivaTunerInstall%\PlugIns\Monitoring" folder
    2 - Copy the PerfCount.rth file into the "%RivaTunerInstall%\Help\Plugins\Monitoring" folder
    3 - Enable the plugin via the hardware monitoring setup->plugins screen

    PerfCountCLI.exe can be run from any directory and is RivaTuner independent. Running the application with no arguments will prompt you to select a counter. Running it with one argument will attempt to query the counter provided as an argument. I'll incorporate human readable error codes in the future, but for now you can reference the error codes here.

    Here is an example:
    Code:
    C:\temp>PerfCounterCLI.exe
     
    Counter selected by user: \\MyPCName\Processor(_Total)\% Idle Time
    Press any key to stop collection
     
    "09/14/2009 09:16:07.954","94.167376908148768"
    "09/14/2009 09:16:08.954","100"
    "09/14/2009 09:16:09.954","98.174499329772118"
    "09/14/2009 09:16:10.953","98.174499329772118"
     
    C:\temp> PerfCounterCLI.exe "\Processor(_Total)\% Idle Time"
     
    Counter provided via argument: \Processor(_Total)\% Idle Time
    Press any key to stop collection
     
    "09/14/2009 09:16:55.378","99.176279935177959"
    "09/14/2009 09:16:56.378","95.169157513554609"
    "09/14/2009 09:16:57.377","99.176279935177959"
     
    C:\temp> PerfCounterCLI.exe "\LogicalDisk(C:)\Free Megabytes"
     
    Counter provided via argument: \LogicalDisk(C:)\Free Megabytes
    Press any key to stop collection
     
    "09/14/2009 09:18:24.028","8338"
    "09/14/2009 09:18:25.028","8338"
    "09/14/2009 09:18:26.027","8338"
     
    C:\temp>
    
    NVPerfCount

    Current Version: NVPerfCount 0.9.1
    OS: XP/Vista/7 (32 bit OS only)
    PreReq: NVIDIA PerfKit, VC++ 2008 Redist

    This plugin allows GPU Performance Counters from NVIDIA's PerfKit to be added to the RivaTuner Hardware Monitoring module. Counters include detailed information about the driver, ROP, Shader, Textures, Memory and much, much more.

    NVPerfCount64

    Current Version: NVPerfCount64 0.9.3
    OS: XP/Vista/7 (64 bit OS only)
    PreReq: NVIDIA PerfKit, VC++ 2008 Redist

    Same as NVPerfCount, but for 64 bit systems.
    ** you must use the version of PerfKit designed for 32 bit apps on 64 bit systems

    - Nick

    EDIT:
    - Removed NVIDIA PDH information as this only works on 32 bit XP
    - Removed ATI PerfStudio Information
    - Added NVPerfCount and revised layout to make first post more friendly for the latest info concerning these plugins.
     
    Last edited: May 3, 2010
  2. Unwinder

    Unwinder Moderator Staff Member

    Messages:
    12,960
    Likes Received:
    45
    Awesome! Good job, Nick. Was thinking about adding a plugin for accessing OS performance counters myself a few times ;)

    P.S. Making the thread sticky
     
  3. burebista

    burebista Ancient Guru

    Messages:
    1,693
    Likes Received:
    0
    GPU:
    MSI GTX 1060 GAMING X 6GB
    Yep, I've played last night a couple of hours with it and I like it. I can monitor whatever Windows counters I want but I'm stuck at nVidia ones. Probably it has something to do with my Vista64 and GTX 260. :help:
    Anyway, good job stangowner. As always. :thumbup:
     
  4. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    Hi Unwinder. Thanks for the extremely well documented SDK examples. I started with the Everest one, so you did 80% of the work ;)

    I really just need to do some work on the export option for logging. Once I do that, if you want the source so you can proof & include in the base install, just let me know :)

    Hi Burebista. I have not played with PerfKit too much yet....just been trying to get the plugin done. But I have also had similar results in the few minutes I've played with it. On Win7 x64 with a GTX280, I have not been able to get the counters to show. On Vista x86 with a QuadroFX 4000, it creates the categories in perfmon, but no valid counters are available. Even when trying to query them manually, PerfCounterCLI.exe returns 0x800007D5 (PDH_NO_DATA) No data to return. So I reverted to my wife's old machine with XP and a GeForce 8800GTS just to test and prove it out. I did not spend more then a few minutes playing with it yet. When I get to some further testing, I'll post anything that may be of value.

    What do you show in perfmon after enabling the driver instrumentation and adding counters via nvdevcpl.cpl? BTW - one quirk I did notice on the XP machine is even hitting "Apply" in the dev panel does not enable the counter. You actually have to click OK.
     

  5. burebista

    burebista Ancient Guru

    Messages:
    1,693
    Likes Received:
    0
    GPU:
    MSI GTX 1060 GAMING X 6GB
    Man I was so sad that I wasn't able to monitor nVidia so I've uninstalled PerfKit but I'll install again today and if you want to help I'll be all ears. :D
    All I can remember is that after adding counters via nvdevcpl.cpl in permon appear a new category (Video something, i forgot) and %GPU Busy was an entry (I'll post a screenshot in a hour or so) but I cannot make it move in your plugin. :(
     
  6. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    Yeah, I know the ATI boys have GPU Usage available....and was hoping this could make it available for us without needing to wait for the 300 series cards :) And my Win7x64/GTX280 machine is the one I want to monitor it on :(

    If it displays in the perfmon window, that is a promising sign. Did you actually add it in perfmon and does it work (I used FurMark to get that 100% usage in the pic above)? If it works in perfmon, then we can test with PerfCounterCLI.exe to ensure the proper syntax. If we can get it working there, then we can just add it to PerfCount.cfg and you should be good to go.

    Let me know when you have more info.
     
  7. burebista

    burebista Ancient Guru

    Messages:
    1,693
    Likes Received:
    0
    GPU:
    MSI GTX 1060 GAMING X 6GB
    OK, installed again. Pick from nvdevcpl.cpl only gpu_idle and hit OK.

    [​IMG]

    Then in perfmon Video Scheduler appears but with %GPU Busy as counter. :3eyes:

    [​IMG]

    Unfortunately despite idle or Furmark load output is the same: 100 in perfmon and 0 in your PerfCounterCLI.

    Now what? :bang:
     
  8. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    That looks good.

    hmmm. Are you sure Video Scheduler is not there even without PerfKit? PerfKit should add the counters under NVIDIA GPU Performance. See page 4 (pdf page 9/32) in the User Guide. This is where I found them on XP - and I used the GPU0/% gpu_idle counter that is pictured.

    Not sure why PerfCounterCLI is outputting 0. However, it is not 100 in perfmon. The last reading was 7.4 million. Its just off the chart.

    Do you show any counter categories under NVIDIA...?
     
  9. burebista

    burebista Ancient Guru

    Messages:
    1,693
    Likes Received:
    0
    GPU:
    MSI GTX 1060 GAMING X 6GB
    I've uninstalled PerfKit, reboot and Video Scheduler is still there. :(
    So it looks like I cannot add nVidia counters.
     
  10. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    Yeah, I'm not sure what is creating that, but I suspected this was the issue. So nothing under N listed? This is what I saw on my machine as well.

    Let me give it a descent go on my machine tonight to see if I can do anything. If not, perhaps I'll have to create another plugin to use the NVIDIA APIs directly, and not reply on the PDH counter implementation which appears to not have 64 bit support, not just on XP but on Vista/7 as well.
     

  11. burebista

    burebista Ancient Guru

    Messages:
    1,693
    Likes Received:
    0
    GPU:
    MSI GTX 1060 GAMING X 6GB
    Unfortunately no nVidia entry in perfmon, only that Video Scheduler. :(
    Anyway, thanks for your time wasted with me. :D
     
  12. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    My pleasure :cheers:
     
  13. DSK

    DSK Banned

    Messages:
    17,925
    Likes Received:
    0
    GPU:
    HD5770/BenQ G2220HD
    Wow im really likeing theses niffty plug-ins :D

    Great work :thumbup:
     
  14. chinobino

    chinobino Master Guru

    Messages:
    957
    Likes Received:
    1
    GPU:
    MSI GTX980Ti Lightning
    This is a great start strangowner!

    I noticed that NVIDIA System Tools v6.05 has GPU usage and GPU FB (Frame Buffer) usage stated as a percentage.

    If you would like to have a look at the "NVIDIA General Purpose Ring0 driver" you can download it here.

    Maybe this could be helpful...

    [edit: I tested on GTX280 SLI and 8800GTX Tri-SLI and it gave valid data for both using Windows 7 64-bit]
     
    Last edited: Sep 15, 2009
  15. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    Guess that was a bad idea. I tried 3 OSes, 2 video cards and several drivers. Couldn't even get it to work on 32 bit Vista.

    Can't get the values to list at all in perfmon. I creates the category but no counters available. And PerfCounterCLI.exe can't return a value. You can see that it errors adding the counter when I have not enabled it nvdevcpl.cpl. Once I enable it, it successfully adds the counter, but returns crap when querying it.

    Code:
    C:\temp\PerfCount_v0.9>PerfCounterCLI.exe "\NVIDIA GPU Performance(GPU0/% gpu_idle)\GPU Counter Value"
    
    Counter provided via argument: \NVIDIA GPU Performance(GPU0/% gpu_idle)\GPU Counter Value
    
    [COLOR="Red"][B]PdhAddCounter[/B] failed with status 0xc0000bb8.
    The specified object is not found on the system.[/COLOR]
    
    C:\temp\PerfCount_v0.9>PerfCounterCLI.exe "\NVIDIA GPU Performance(GPU0/% gpu_idle)\GPU Counter Value"
    
    Counter provided via argument: \NVIDIA GPU Performance(GPU0/% gpu_idle)\GPU Counter Value
    
    [COLOR="Red"][B]PdhCollectQueryData[/B] failed with 0x800007d5.
    No data to return.[/COLOR]
    You'll have to give me a little time to get something going. I have a few ideas to try in the next few days.

    In the meantime.....at least you get hundreds of standard counters :)

    - Nick
     

  16. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    I started to test NVIDIA's PerfAPI directly, and not rely on the PDH implementation. Initial results look promising.

    Here is the "gpu_idle" performance counter on a Vista 32 system with a Quadro card (one of the systems where phd does not work). I ran FurMark for a few seconds to see the percentage change. The value and cycle are in picoseconds, and the percentage is idle/elapsed*100.

    C:\Nick\Visual Studio 2005\Projects\RivaTuner\NVPerfCountCLI\debug>NVPerfCountCLI.exe

    09/16/2009 08:48:48.246, value:212545331 ps, cycle:212645764 ps, percent:99.952770
    09/16/2009 08:48:49.246, value:212545331 ps, cycle:212645764 ps, percent:99.952770
    09/16/2009 08:48:50.246, value:421141690 ps, cycle:425275315 ps, percent:99.028012
    09/16/2009 08:48:51.246, value:421141690 ps, cycle:425275315 ps, percent:99.028012
    09/16/2009 08:48:52.247, value:306797972 ps, cycle:425274848 ps, percent:72.141104
    09/16/2009 08:48:53.247, value:306797972 ps, cycle:425274848 ps, percent:72.141104
    09/16/2009 08:48:54.248, value:4243062 ps, cycle:425494491 ps, percent:0.997207
    09/16/2009 08:48:55.248, value:4243062 ps, cycle:425494491 ps, percent:0.997207
    09/16/2009 08:48:56.248, value:1750923 ps, cycle:425277655 ps, percent:0.411713
    09/16/2009 08:48:57.248, value:1750923 ps, cycle:425277655 ps, percent:0.411713
    09/16/2009 08:48:58.248, value:1505583 ps, cycle:425279303 ps, percent:0.354022
    09/16/2009 08:48:59.248, value:1505583 ps, cycle:425279303 ps, percent:0.354022
    09/16/2009 08:49:00.248, value:2982386 ps, cycle:425281036 ps, percent:0.701274
    09/16/2009 08:49:01.248, value:2982386 ps, cycle:425281036 ps, percent:0.701274
    09/16/2009 08:49:02.249, value:1315419 ps, cycle:425277468 ps, percent:0.309308
    09/16/2009 08:49:03.249, value:1315419 ps, cycle:425277468 ps, percent:0.309308
    09/16/2009 08:49:04.249, value:1932394 ps, cycle:425418449 ps, percent:0.454234
    09/16/2009 08:49:05.249, value:1932394 ps, cycle:425418449 ps, percent:0.454234
    09/16/2009 08:49:06.249, value:2884216 ps, cycle:425144307 ps, percent:0.678409
    09/16/2009 08:49:07.249, value:2884216 ps, cycle:425144307 ps, percent:0.678409
    09/16/2009 08:49:08.249, value:212406383 ps, cycle:425274995 ps, percent:49.945655
    09/16/2009 08:49:09.249, value:212406383 ps, cycle:425274995 ps, percent:49.945655
    09/16/2009 08:49:10.249, value:424057771 ps, cycle:425279494 ps, percent:99.712725
    09/16/2009 08:49:11.249, value:424057771 ps, cycle:425279494 ps, percent:99.712725
    09/16/2009 08:49:12.250, value:425102249 ps, cycle:425271589 ps, percent:99.960181
    09/16/2009 08:49:13.250, value:425102249 ps, cycle:425271589 ps, percent:99.960181

    I'll test several platforms and other counters to see how that goes.
     
  17. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    I did a little more testing, and not getting anything good on my Win7 (x86 & x64) GTX280 machine. I have a support request into NVIDIA, but have not yet received a reply. And their developer's forum indicates many others are not having any luck neither. It seems it was never fully developed past Win XP x86, even though there is some success on Vista 32 bit. Its a shame, because I can enumerate over 300 performance counters on my GTX280 and over 200 on a Quadro :(

    I've been out of a computer for 2 weeks due to a PSU that I am getting replaced. Once I get up and running again, I'll continue to take a look at this and other options. But if I had to decide on the limited testing so far, I would not develop a plugin for the NVPerfAPI. Not until NVIDIA finishes/updates it.

    Of course I'll continue to finish the existing PerfCount plugin once I get my machine up and running.
     
  18. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    I received a response from NVIDIA and they are telling me that NVPerfAPI should work perfectly fine on 32 and 64 bit version of Linux, Windows XP, Vista and 7.

    I'll be taking another look at this to see if I can get it working properly.
     
  19. stangowner

    stangowner Master Guru

    Messages:
    586
    Likes Received:
    0
    GPU:
    2xMSI N550GTX-Ti Cy II OC
    Burebista, feel like testing a few things with me again? Or anyone else with Vista/7 x64 and a nvidia card (this build will not work on 32 bit systems, and Windows 7 is the only OS I have tested so far - it should be fine for Vista x64, but not sure yet about XP x64).

    This is a new test application using the NVPerfAPI directly. Works basically the same as before, but does not involve the Windows Performance Counters at all. This is the command line application only - it is not yet a plugin for RTHM. You can get it here.

    USE: No argument dumps the name and description (if available) for all avalable counters. Adding a counter name as an argument causes it to be queried every second until stopped.

    I would recommend running NVPerfCountCLI.exe from a command prompt window, not just double clicking it. This way, when querying all counters you can pipe it to 'more' so you can see one screen at a time ("NVPerfCountCLI.exe | more"). Its also more convenient for providing arguments.

    Here is an example on my Windows 7 Ultimate RTM box with a single GTX280 and 191.07 drivers. I installed PerfKit 6.5 Vista 64-bit from here. You must install both components (HUD and SDK) as the SDK alone does not include one required dll. You do not need gDEBugger.

    Code:
    C:\temp>NVPerfCountCLI.exe
    NVPerfAPI initialization successful
    Number of available counters: 385
    
    No argument provided - querying all available counters
    Counter #: Counter Name (Counter Description)
    
    Counter 0: OGL frame time
    Counter 1: OGL driver time waiting
    Counter 2: OGL driver waits for GPU
    Counter 3: OGL driver waits for kernel
    Counter 4: OGL driver waits for lock
    Counter 5: OGL driver waits for render
    Counter 6: OGL driver waits for swap
    .
    .
    .
    Counter 383: GPU Bottleneck_gpu_min
    Counter 384: GPU Bottleneck_gpu_max
    Press any key to exit
    
    And here are a few snippets for common counters. In these test, I started collection, started FurMark, closed FurMark, stopped collection. FurMark is very heavy on the shaders, and these numbers are indicating that.

    Code:
    C:\temp>NVPerfCountCLI.exe gpu_idle
    NVPerfAPI initialization successful
    Number of available counters: 385
    
    Counter provided via argument: [B][COLOR="Red"]gpu_idle[/COLOR][/B]
    Successfully added counter: gpu_idle
    
    Press any key to stop collection
    
    value: 4136  cycle: 4137  percent: 99.975830
    value: 4136  cycle: 4137  percent: 99.975830
    value: 30137184  cycle: 266766735  percent: 11.297204
    value: 30137184  cycle: 266766735  percent: 11.297204
    value: 7761654  cycle: 650849067  percent: 1.192543
    value: 7761654  cycle: 650849067  percent: 1.192543
    value: 7329362  cycle: 651204435  percent: 1.125509
    value: 7329362  cycle: 651204435  percent: 1.125509
    value: 36969084  cycle: 150974870  percent: 24.486914
    value: 36969084  cycle: 150974870  percent: 24.486914
    value: 41051631  cycle: 42751626  percent: 96.023560
    value: 41051631  cycle: 42751626  percent: 96.023560
    
    Code:
    C:\temp>NVPerfCountCLI.exe rop_busy_gpu0
    NVPerfAPI initialization successful
    Number of available counters: 385
    
    Counter provided via argument: [B][COLOR="Red"]rop_busy_gpu0[/COLOR][/B]
    Successfully added counter: rop_busy_gpu0
    
    Press any key to stop collection
    
    value: 0  cycle: 4706  percent: 0.000000
    value: 0  cycle: 4706  percent: 0.000000
    value: 412859  cycle: 54516447  percent: 0.757311
    value: 412859  cycle: 54516447  percent: 0.757311
    value: 104591243  cycle: 633106666  percent: 16.520319
    value: 104591243  cycle: 633106666  percent: 16.520319
    value: 113053608  cycle: 652005706  percent: 17.339359
    value: 113053608  cycle: 652005706  percent: 17.339359
    value: 116179096  cycle: 650789617  percent: 17.852020
    value: 116179096  cycle: 650789617  percent: 17.852020
    value: 29284917  cycle: 200350541  percent: 14.616838
    value: 29284917  cycle: 200350541  percent: 14.616838
    value: 1668  cycle: 43095425  percent: 0.003870
    value: 1668  cycle: 43095425  percent: 0.003870
    
    Code:
    C:\temp>NVPerfCountCLI.exe shader_busy_gpu0
    NVPerfAPI initialization successful
    Number of available counters: 385
    
    Counter provided via argument: [B][COLOR="Red"]shader_busy_gpu0[/COLOR][/B]
    Successfully added counter: shader_busy_gpu0
    
    Press any key to stop collection
    
    value: 0  cycle: 76466  percent: 0.000000
    value: 0  cycle: 76466  percent: 0.000000
    value: 25498  cycle: 205279050  percent: 0.012421
    value: 25498  cycle: 205279050  percent: 0.012421
    value: 571821217  cycle: 707958581  percent: 80.770439
    value: 571821217  cycle: 707958581  percent: 80.770439
    value: 582335121  cycle: 708226401  percent: 82.224426
    value: 582335121  cycle: 708226401  percent: 82.224426
    value: 577432481  cycle: 708336752  percent: 81.519493
    value: 577432481  cycle: 708336752  percent: 81.519493
    value: 115974056  cycle: 186829853  percent: 62.074692
    value: 115974056  cycle: 186829853  percent: 62.074692
    value: 22282  cycle: 46690262  percent: 0.047723
    value: 22282  cycle: 46690262  percent: 0.047723
    
    Code:
    C:\temp>NVPerfCountCLI.exe geom_busy_gpu0
    NVPerfAPI initialization successful
    Number of available counters: 385
    
    Counter provided via argument: [B][COLOR="Red"]geom_busy_gpu0[/COLOR][/B]
    Successfully added counter: geom_busy_gpu0
    
    Press any key to stop collection
    
    value: 47  cycle: 76655  percent: 0.061314
    value: 47  cycle: 76655  percent: 0.061314
    value: 4274  cycle: 136002783  percent: 0.003143
    value: 4274  cycle: 136002783  percent: 0.003143
    value: 3990  cycle: 707903106  percent: 0.000564
    value: 3990  cycle: 707903106  percent: 0.000564
    value: 2733492  cycle: 708216087  percent: 0.385969
    value: 2733492  cycle: 708216087  percent: 0.385969
    value: 2693796  cycle: 708398173  percent: 0.380266
    value: 2693796  cycle: 708398173  percent: 0.380266
    value: 2660478  cycle: 708288155  percent: 0.375621
    value: 2660478  cycle: 708288155  percent: 0.375621
    value: 1177189  cycle: 350371057  percent: 0.335984
    value: 1177189  cycle: 350371057  percent: 0.335984
    value: 412  cycle: 46767178  percent: 0.000881
    
    Let me know how you make out or if you have any questions.

    - Nick
     
  20. burebista

    burebista Ancient Guru

    Messages:
    1,693
    Likes Received:
    0
    GPU:
    MSI GTX 1060 GAMING X 6GB
    Sure man, anytime. :)
    I did only two tests, gpu_idle
    [​IMG]

    and busy.
    [​IMG]

    I've tried GPU Bottleneck too but it's outputs 0 regardless GPU in idle/Furmark or my CPU at 2 GHz or 3.5GHz. :D

    [​IMG]

    Frankly I dunno what counters are relevant from all 89 but I hope you'll shed some light. :)

    I'm all ears now. :D
     

Share This Page