Windows: Line-Based vs. Message Signaled-Based Interrupts ...

Discussion in 'Operating Systems' started by mbk1969, May 7, 2013.

  1. mbk1969

    mbk1969 Ancient Guru

    Messages:
    3,535
    Likes Received:
    0
    Graphics Card::
    GeForce GTX 970
    ... or another attempt to improve latencies

    Little bit of theory:
    From "Windows Internals" by Mark Russinovich, David A. Solomon, Alex Ionescu
    Line-Based vs. Message Signaled-Based Interrupts
    Shared interrupts are often the cause of high interrupt latency and can also cause stability issues. They are typically undesirable and a side effect of the limited number of physical interrupt lines on a computer. For example, in the previous example of the 7-in-1 media card reader, a much better solution is for each device to have its own interrupt and for one driver to manage the different interrupts knowing which device they came from. However, consuming four IRQ lines for a single device quickly leads to IRQ line exhaustion. Additionally, PCI devices are each connected to only one IRQ line anyway, so the media card reader cannot use more than one IRQ in the first place.

    Other problems with generating interrupts through an IRQ line is that incorrect management of the IRQ signal can lead to interrupt storms or other kinds of deadlocks on the machine, because the signal is driven “high” or “low” until the ISR acknowledges it. (Furthermore, the interrupt controller must typically receive an EOI signal as well.) If either of these does not happen due to a bug, the system can end up in an interrupt state forever, further interrupts could be masked away, or both. Finally, line-based interrupts provide poor scalability in multiprocessor environments. In many cases, the hardware has the final decision as to which processor will be interrupted out of the possible set that the Plug and Play manager selected for this interrupt, and there is little device drivers can do.

    A solution to all these problems is a new interrupt mechanism first introduced in the PCI 2.2 standard called message-signaled interrupts (MSI). Although it remains an optional component of the standard that is seldom found in client machines, an increasing number of servers and workstations implement MSI support, which is fully supported by the all recent versions of Windows. In the MSI model, a device delivers a message to its driver by writing to a specific memory address. This action causes an interrupt, and Windows then calls the ISR with the message content (value) and the address where the message was delivered. A device can also deliver multiple messages (up to 32) to the memory address, delivering different payloads based on the event.

    Because communication is based across a memory value, and because the content is delivered with the interrupt, the need for IRQ lines is removed (making the total system limit of MSIs equal to the number of interrupt vectors, not IRQ lines), as is the need for a driver ISR to query the device for data related to the interrupt, decreasing latency. Due to the large number of device interrupts available through this model, this effectively nullifies any benefit of sharing interrupts, decreasing latency further by directly delivering the interrupt data to the concerned ISR.

    Finally, MSI-X, an extension to the MSI model, which is introduced in PCI 3.0, adds support for 32-bit messages (instead of 16-bit), a maximum of 2048 different messages (instead of just 32), and more importantly, the ability to use a different address (which can be dynamically determined) for each of the MSI payloads. Using a different address allows the MSI payload to be written to a different physical address range that belongs to a different processor, or a different set of target processors, effectively enabling nonuniform memory access (NUMA)-aware interrupt delivery by sending the interrupt to the processor that initiated the related device request. This improves latency and scalability by monitoring both load and closest NUMA node during interrupt completion.


    Now practice.
    Checking for PCI devices working in MSI-mode.
    Go to Device Manager. Click in menu "View -> Resources by type". Expand "Interrupt request (IRQ)" node of the tree. Scroll down to "(PCI) 0x... (...) device name" device nodes. Devices with positive number for IRQ (like "(PCI) 0x00000011 (17) ...") are in Line-based interrupts-mode. Devices with negative number for IRQ (like "(PCI) 0xFFFFFFFA (-6) ...") are in Message Signaled-based Interrupts-mode.

    Trying to switch device to MSI-mode.
    You must locate device`s registry key. Invoke device properties dialog. Switch to "Details" tab. Select "Device Instance Path" in "Property" combo-box. Write down "Value" (for example "PCI\VEN_1002&DEV_4397&SUBSYS_1609103C&REV_00\3&11583659&0&B0"). This is relative registry path under the key "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\".

    Go to that device`s registry key ("HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1002&DEV_4397&SUBSYS_1609103C&REV_00\3&11583659&0&B0") and locate down the subkey "Device Parameters\Interrupt Management". For devices working in MSI-mode there will be subkey "Device Parameters\Interrupt Management\MessageSignaledInterruptProperties" and in that subkey there will be DWORD value "MSISupported" equals to "0x00000001". To switch device from legacy- to MSI-mode just add these subkey and value.

    Before adding these key and value (or changing "MSISupported" to "0x00000001" in case subkey and value already exist) you have to perform safety steps like doing backup (creating system restore point at least).

    Do tweak one device -> reboot and check (1) if it is displayed in Device Manager as correctly working device; (2) if its IRQ became negative -> if no (1) and no (2) then either remove subkey "MessageSignaledInterruptProperties" (if you added it) or change "MSISupported" to "0x00000000" and reboot.

    Theoretically if device driver (and platform = chipset) unable to perform in MSI-mode it should ignore mentioned subkey and value. But who knows...

    Afaik most intensive/active IRQ devices are network controller, SATA controller and USB controller.

    I have found that network controller is already working in MSI-mode (both at work and at home). I have successfully switched SATA controllers, video- and sound-cards into MSI-mode (both at work and at home). USB 2 controllers ignored "MSISupported" (both at ...), but USB 3 controllers at parents` PC switched to MSI-mode. And Intel MEI device at home rig became incorrect after adding "MSISupported", so I reverted it. And most important is that at work rig (which is old and weak) LatencyMon showed stable level of latencies (at heavy load) which is almost twice lower then before these tweaks, and usual stutters at heavy loads (VisualStudio + VMWare + SmartSVN + Skype + mp3-player) are gone.

    Edit: If you have old rig and there are no devices working in MSI-mode, then may be you should not experiment with this tweak.
    As this tweak is risky then you probably should not tweak devices which can prevent Windows to boot (by their failure) - SATA controllers, videocards. Especially if they do not share their IRQ with another devices.

    Edit: Troubleshooting topic with info from "Windows Internals":
    Accepting the Boot and Last Known Good
    Besides starting services, the system charges the Sevice Control Manager (SCM) with determining when the system’s registry configuration, HKLM\SYSTEM\CurrentControlSet, should be saved as the last known good control set. The CurrentControlSet key contains the Services key as a subkey, so CurrentControlSet includes the registry representation of the SCM database. It also contains the Control key, which stores many kernel-mode and user-mode subsystem configuration settings. By default, a successful boot consists of a successful startup of auto-start services and a successful user logon. A boot fails if the system halts because a device driver crashes the system during the boot or if an auto-start service with an ErrorControl value of SERVICE_ERROR_SEVERE or SERVICE_ERROR_CRITICAL reports a startup error.

    The SCM obviously knows when it has completed a successful startup of the auto-start services, but Winlogon (%SystemRoot%\System32\Winlogon.exe) must notify it when there is a successful logon. Winlogon invokes the NotifyBootConfigStatus function when a user logs on, and NotifyBootConfigStatus sends a message to the SCM. Following the successful start of the auto-start services or the receipt of the message from NotifyBootConfigStatus (whichever comes last), the SCM calls the system function NtInitializeRegistry to save the current registry startup configuration.

    Windows maintains several copies of CurrentControlSet, and CurrentControlSet is really a symbolic registry link that points to one of the copies. The control sets have names in the form HKLM\SYSTEM\ControlSetnnn, where nnn is a number such as 001 or 002. The HKLM\SYSTEM\Select key contains values that identify the role of each control set. For example, if CurrentControlSet points to ControlSet001, the Current value under Select has a value of 1. The LastKnownGood value under Select contains the number of the last known good control set, which is the control set last used to boot successfully. Another value that might be on your system under the Select key is Failed, which points to the last control set for which the boot was deemed unsuccessful and aborted in favor of an attempt at booting with the last known good control set. Figure 4-15 displays a system’s control sets and Select values.

    NtInitializeRegistry takes the contents of the last known good control set and synchronizes it with that of the CurrentControlSet key’s tree. If this was the system’s first successful boot, the last known good won’t exist and the system will create a new control set for it. If the last known good tree exists, the system simply updates it with differences between it and CurrentControlSet. Last known good is helpful in situations in which a change to CurrentControlSet, such as the modification of a system performance-tuning value under HKLM\SYSTEM\Control or the addition of a service or device driver, causes the subsequent boot to fail. Users can press F8 early in the boot process to bring up a menu that lets them direct the boot to use the last known good control set, rolling the system’s registry configuration back to the way it was the last time the system booted successfully.
    So if you choose to try and switch PCI device into the message-mode, you better do add "MSISupproted" registry value to one device and reboot. If Windows failed to boot you restart and press F8 and choose boot option "last known good configuration" (which must be the configuration without "MSISupported").

    Edit: Improper Propagation of SATA Message Signaled Interrupts on AMD SB950
    Description
    The SATA Message Signaled Interrupt (MSI) from higher port numbers may not properly propagate to port 0 before being sent to the driver.

    Potential Effect on System
    On a platform with SATA MSI enabled and activity occurring on SATA devices connected to three or more ports, if the failing condition occurs, the driver will not receive the interrupt and the system will hang.

    Suggested Workaround
    Do not expose the MSI capability in the SATA controller. This change is implemented in CIMx version 1.1.0.2.

    Fix Planned
    No

    Edit: Here is Intel`s paper (thanks to n1hilist):
    http://www.intel.co.za/content/dam/...hite-papers/msg-signaled-interrupts-paper.pdf

    Edit: PCI Express overview

    Edit: Here is utility for switching MSI-mode
    http://www.mediafire.com/download/hrxw4rsb9xnw1ei/MSI_util.zip

    Edit: MSI utility v2.
    http://www.mediafire.com/file/2kkkvko7e75opce/MSI_util_v2.zip
    Change log:
    - New devices grid layout with 4 columns - name, irq, msi, limit, where msi is checkbox column (for MsiSupported value), limit is textbox column (for MessageNumberLimit value) which accepts either empty string or integer value in range of 0..2048 (where 0 has the same meaning as empty string - no limit).

    Edit: Here is the script for switching MSI-mode
    http://www.mediafire.com/download/rcgtxc73fp00x26/MSI_mode_utils.zip

    Open it in text editor to read the help or execute in PowerShell:
    get-help .\MSI_mode_utils.ps1 -full

    Remind you that to be able to execute PowerShell scripts you should specify execution policy by:
    Set-ExecutionPolicy -ExecutionPolicy Unrestricted
    or
    Set-ExecutionPolicy -ExecutionPolicy RemoteSigned

    And I strongly recommend to use script with argument 'reg' first to make the backup reg-file before turning MSI-mode on.
     
    Last edited: Jul 31, 2017
  2. thatguy91

    thatguy91 Ancient Guru

    Messages:
    6,452
    Likes Received:
    0
    Graphics Card::
    XFX RX 480 RS 4 GB
    So this has no effect on the IRQ's for the Numeric Data Processor and System CMOS/Real time clock? Both of these use IRQ's.
     
  3. mbk1969

    mbk1969 Ancient Guru

    Messages:
    3,535
    Likes Received:
    0
    Graphics Card::
    GeForce GTX 970
    Are these the PCI devices in your rig? At mine they are ISA ones.
     
  4. Nurse13

    Nurse13 New Member

    Messages:
    6
    Likes Received:
    0
    Graphics Card::
    Gskill 8gig
    Great write up, although i found it somewhat difficult to follow and re-read it many times as i was confused to what the main "wanted" goal was and what was to be created.

    This worked for my video card but not asus xonar essence stx audio device. I have the latest xonar software with no addons. I have always believed some of my problems (latency, DPC, etc.) has been related to his.. or maybe im just that stupid to think so i dont know. However, i'd thought i'd try. I had initially messed up by not creating the "MessageSignaledInterruptProperties" subkey first before adding "MSIsupported". I went back and re-read the post and seen the mistake which then after a reboot showed that this worked for my video card.

    IS there something that prevents the change to MSIsupported mode or a software conflict/reason this wont work as like the USB 2 controllers you tried?
     

  5. mbk1969

    mbk1969 Ancient Guru

    Messages:
    3,535
    Likes Received:
    0
    Graphics Card::
    GeForce GTX 970
    It`s my clumsy english. Sorry.

    As for MSI-mode, device itself has to use (implement) MSI capability, and its driver too.

    And what about network card and SATA controller in your rig? I suspect that video card is not the source of mass interrupt bursts...
     
    Last edited: May 14, 2013
  6. Nurse13

    Nurse13 New Member

    Messages:
    6
    Likes Received:
    0
    Graphics Card::
    Gskill 8gig
    Network adapter is Intel 82579V. It is already at a -2 IRQ. On a side note if audio wasn't the problem then either this is or maybe usb. I notice that if i mess with the performance settings or interruption/filter settings my net has extreme amount of drops. Youtube videos stop loading, my games freeze periodically as if i lose my connection and regain it. I have read so much into it and have not found the cause still after 6 months. My net is good, my modem is fine as well (just got firmware update i think yesterday even).

    TBH i havent even looked at my SATA controller. Its in Line-based mode.
    I will look at it tonight when i get time to mess with things again.
     
  7. mbk1969

    mbk1969 Ancient Guru

    Messages:
    3,535
    Likes Received:
    0
    Graphics Card::
    GeForce GTX 970
    Have you tried:
    - tweaking NetworkThrottlingIndex;
    - tweaking TCP/IP (there is recommendation to enable TCP RSS when NIC is working in MSI-mode);
    - disabling CPU core parking;
    - this thread
    ???
     
  8. Nurse13

    Nurse13 New Member

    Messages:
    6
    Likes Received:
    0
    Graphics Card::
    Gskill 8gig
    I have slowly got into Von's page on performance tweaks. Many i've done before and or found else where so i found myself either repeating or skipping.


    I monkeyed with tonight with the IRQ and upon reboot i got a blue screened before windows could log in. I ended up just reinstalling windows because i did not have system restore on that partition as i do with this one; why i dont remember.

    And with a new fresh install i can hopefully nail what is causing my problems by introducing 1 thing at a time.

    I did core parking and the network throttling i put to 0.

    This maybe unheard off but by god i believe its keyboard or mouse related but there is no way to really test.No way mean i dont have replacements or software that i know of to find problems. I have g100 keyboard and g500 mouse.
    It seems that my disconnects only happen during me actually using my hardware. Example me spectating in call of duty or bf3 or tf2i have yet to see this same phenomenon happen. My disconnects and by this i mean my game freezes and connection drops and returns are made worse by excessive input from me. So putting the peices together makes me rule out internet issues however it acts just like an internet issue. Hell it reminds of the House episode where they believe its cancer but every test shows that is not cancer. Strange.

    I have no idea gentlemen. I am ready to slap this on ebay, all of it, and say the hell with it. I'll build from the ground up when i have time.

    In vons thread he goes into device manager to disable stuff, so i did to. Looking under device manager i see x4 HID complaint keyboard and same for mouse. I see "virtual keyboard" and "virtual mouse". Which makes me wonder, but that about it.

    I dont know. I would seriously pay someone to sit at my computer and do all that is necessary to find out why i can't use it.
     
    Last edited: May 15, 2013
  9. mbk1969

    mbk1969 Ancient Guru

    Messages:
    3,535
    Likes Received:
    0
    Graphics Card::
    GeForce GTX 970
    Definitely you need network troubleshooting guru. May be from your internet provider.
     
  10. Nurse13

    Nurse13 New Member

    Messages:
    6
    Likes Received:
    0
    Graphics Card::
    Gskill 8gig
    Yea but im not sure if they would find anything. I'm at the point now i dont care anymore-i've given up.

    To make my day even better my keyboard decided not to work today in its current usb slot, so i had to change usb slots t type this messsage. Why?...
     

  11. mbk1969

    mbk1969 Ancient Guru

    Messages:
    3,535
    Likes Received:
    0
    Graphics Card::
    GeForce GTX 970
    Generic recommendation for not working USB ports is to shutdown PC, remove power cable for 5-10 minutes. Or try powered USB hub.

    As for network, imo troubles are 20% of internal hardware-software and 80% of external network protocol ones. Especially when xDSL modem involved.
    Btw, you can try discrete NIC, if you suspect your (most probably integrated) one is culprit.
     
  12. Nurse13

    Nurse13 New Member

    Messages:
    6
    Likes Received:
    0
    Graphics Card::
    Gskill 8gig
    I believe that i might have fixed the issue, fingers are crossed. If it isn't then i truly will just take a 12 gauge and light this whole thing up.

    Two issues: 1) turned modem into a bridge via NAPT mode disabled; known issue with my modem (surfboard from motorola).
    2) Keep getting usb3 power surge message. Switched 1 input to FP usb and the other to the back. So far working.

    The strangest thing happened to me as well. I went through the bios to see if maybe it was power related, turned everything to stock voltages as i had been using part of an OC profile but had de-clocked my cpu long ago just never returned the voltages. Log into windows to find that my mouse is super laggy, my cpu usage is 80 percent, my ram is almost maxed at 8gigs. Logitech gaming software was using over 2 gigs of ram... all hell broke loose. I went back and re did an overlock on my system said why the hell not after i couldn't fix the damn thing lets fry it.
    Well it works now.

    Uhm sorry i have derailed this thread. I believe i noticed some* maybe some improvement with these tweaks. I will re-do them on my fresh install of windows.But the SATA controller is off limits.
    Thanks
     
  13. mbk1969

    mbk1969 Ancient Guru

    Messages:
    3,535
    Likes Received:
    0
    Graphics Card::
    GeForce GTX 970
    Yeah, that was USB troubleshoot # 1 - use back panel ports instead front panel ones.
     
  14. Cyberdyne

    Cyberdyne Ancient Guru

    Messages:
    2,927
    Likes Received:
    0
    Graphics Card::
    GTX1080 Arctic Hybrid III
  15. mbk1969

    mbk1969 Ancient Guru

    Messages:
    3,535
    Likes Received:
    0
    Graphics Card::
    GeForce GTX 970
    Yeah, seems correctly. Audio-card is not the source interrupts bursts, I assume. SATA controllers, USB-controllers, NICs are...
     

  16. pipes

    pipes Member Guru

    Messages:
    156
    Likes Received:
    0
    Graphics Card::
    2X R9 290x
    Hi man, I have do all you say in this guide, only 3:
    Intel(R) 7500/5520/5500/X58 I/O Hub PCI Express Root Port 1 - 3408
    Intel(R) 7500/5520/5500/X58 I/O Hub PCI Express Root Port 7 - 340E - 340e
    Intel(R) 7500/5520/5500/X58 I/O Hub PCI Express Root Port 3 - 340A - 340a


    I can't make in msi-mode, Why You know?
     
  17. Cyberdyne

    Cyberdyne Ancient Guru

    Messages:
    2,927
    Likes Received:
    0
    Graphics Card::
    GTX1080 Arctic Hybrid III
    Probably because of this,
     
  18. mbk1969

    mbk1969 Ancient Guru

    Messages:
    3,535
    Likes Received:
    0
    Graphics Card::
    GeForce GTX 970
    Only those 3 were switched into MSI-mode?
    You see, for MSI-mode must participate: chipset, device and device drivers.
     
  19. pipes

    pipes Member Guru

    Messages:
    156
    Likes Received:
    0
    Graphics Card::
    2X R9 290x
  20. pipes

    pipes Member Guru

    Messages:
    156
    Likes Received:
    0
    Graphics Card::
    2X R9 290x
    is not in msi-mode and I cant change into msi-mode :bang:
     

Share This Page