ProcessPluginFramework in the form of windows system service

Discussion in 'Operating Systems' started by mbk1969, May 19, 2013.

  1. tappingbowl

    tappingbowl Guest

    Messages:
    209
    Likes Received:
    1
    GPU:
    Nvidia
    Okay, removed it.

    Well, tried it. Everything looks fine. Except one thing, 'MaximumTimerResolutionUntilExit' - does not like to work in 'pipeline'. Disables other plugins, going after it, completely.
    Probably, because it does not complete, until process exited.
    Except this, no errors so far. Though, i did experience 'Stopping' state of the service pretty much, while setting a new syntax. I have no idea why it does not want to stop, doing it randomly or looks so. I even did a more precise configuration for processes.
    Configurated plugins for games:
    Code:
    maximumtimerresolutionuntilexit
    affinityset()|priorityhigh|disableprocesspriorityboost|iopriority(3)|sleep()|replacethreadspriority(0,15)|replacethreadspriority(2,15)|replacethreadspriority(1,15)|replacethreadspriority(-15,-2)
    Now looking forward for 'recurrent' and 'thread affinity' plugins. Something like this:
    [application]
    Loop(60000)|Recurrent(0,15|1,15|2,15|-15,-2)

    Multiple replacement. Possible?

    Edit: A little malfunction found. 'TraceProcess' - count lines in [application], so if 'pipeline' is used - it counts it as one line. Therefore, not a real number of plugins is shown.
     
    Last edited: Mar 8, 2014
  2. mbk1969

    mbk1969 Ancient Guru

    Messages:
    15,541
    Likes Received:
    13,561
    GPU:
    GF RTX 4070
    Exactly. All 'UntilExit' plugins do wait for a process stop.

    Yes, but not with '|' symbol. It needs inner symbol for a pair, and outer symbol for delimiting the pairs. Recurrent(0 15,1 15,2 15,-15 -2) ? Or Recurrent(0 15/1 15/2 15/-15 -2) ?

    I don`t want to add saving the number of plugins specified for a process, because that info is used only in TraceProcess, and the info is of low interest.
     
    Last edited: Mar 8, 2014
  3. tappingbowl

    tappingbowl Guest

    Messages:
    209
    Likes Received:
    1
    GPU:
    Nvidia
    Yeah, it was an example. I guess, you better know which one will not conflict with the internal structure of service. Though i like the second more ('/').

    Oh, okay. Though it did misled me a bit, before i figured it out. Since you said it is for troubleshooting purposes.
     
  4. mbk1969

    mbk1969 Ancient Guru

    Messages:
    15,541
    Likes Received:
    13,561
    GPU:
    GF RTX 4070
    Source file was updated. It can be downloaded here - http://www.mediafire.com/download/rabc9ckn8qi72q8/PPF_source_v3.zip - and service binary can be rebuild (by renaming or removing and starting the Install-PPFService.ps1).

    Change log:

    - plugin ReplaceThreadsPriority was rewritten to support multiple priorities pairs per call. Syntax:

    ReplaceThreadsPriority(cur1 new1/cur2 new2/...)

    where 'curN newN' - pair of current and replacement priority thread levels.

    - added new plugin ReplaceThreadsPriorityUntilExit, which applies specified thread priority replacements at its start and then applies specified thread priority replacements at each creation of thread.
    Syntax:
    ReplaceThreadsPriorityUntilExit(cur1 new1/cur2 new2/...)

    where 'curN newN' - pair of current and replacement thread priority levels.

    - seems that hanging service stopping was fixed.
     
    Last edited: Mar 10, 2014

  5. tappingbowl

    tappingbowl Guest

    Messages:
    209
    Likes Received:
    1
    GPU:
    Nvidia
    Works well. No errors registered. Looks like PPF service stopping is fixed.

    A few questions:
    - Segments in 'ReplaceThreadsPriority' plugins '(cur1 new1/cur2 new2/...)' - are they processed in this order or simultaneously? First 'cur1 new1' then 'cur2 new2' etc.
    - If i use 'ReplaceThreadsPriorityUntilExit' - there is no need in using 'ReplaceThreadsPriority'?
    - If 'ReplaceThreadsPriorityUntilExit' is placed in the end of 'pipeline' - will it work without any problems? Or it needs a dedicated line?

    I did try to apply 'ReplaceThreadsPriorityUntilExit' - to some running applications. Minor CPU usage have been observed. To lessen it even more, i would like to specify a 'looping' delay. Like you offered: 'Loop|ReplaceThreadsPriorityUntilExit'. Or in a another way, if you have a some better idea already. Because as i understand it will not be possible to set the delay in a such way for 'UntilExit' plugin. Since it is running constantly. So, the only option is configuring "checks" internally. Like, 'ReplaceThreadsPriorityUntilExit(500\cur1 new1/cur2 new2/...)'. Plugin checks for non-zero CPU time threads of a specified application, every 500 milliseconds.

    Thank you for your hard work!
    PS. Try to open PPF service downloading link in my quote of your message. It is also like that, here: http://forums.guru3d.com/showpost.php?p=4778846&postcount=118

    Edit: One more question. If i will use 'ReplaceThreadsPriorityUntilExit' on PPF service, will it cause a loop? Like, it will never will end. PPF is monitoring self threads - thread created - thread created to process a thread - thread applied - thread created - etc. So, i guess it is not recommended to use this plugin on PPF service?
     
    Last edited: Mar 10, 2014
  6. mbk1969

    mbk1969 Ancient Guru

    Messages:
    15,541
    Likes Received:
    13,561
    GPU:
    GF RTX 4070
    They processed simultaneously. Internally the dictionary is built from specified segements, and when threads are enumerated replacement took place by dictionary.

    No need, because 'UntilExit' version calls replacement once at start.

    It will. No decicated line is needed, only the last place in pipeline.

    I am working on injecting the timer into current logic...

    Edit: But in context of thread priority level there is no much need in looping checks, because it is rare that app sets priority level repeatedly. I mean it is usual action at thread`s creation - to set its priority level, but checking it constantly is strange. And imo it is better to leave alone such threads. Creation of thread most probably is the sign of some step in app`s life, you see.
    But for CPU affinity plugin timer logic is needed.

    Fixed.

    It may be if PPF is under heavy load. But by design plugins are started in thread pool which means that if thread pool manager has idle threads at hands it will not create new ones.
     
    Last edited: Mar 10, 2014
  7. tappingbowl

    tappingbowl Guest

    Messages:
    209
    Likes Received:
    1
    GPU:
    Nvidia
    That is not really convenient, i guess. For example, the goal is to replace threads: 0 15/-15 0. What if '-15' threads will become '0' threads faster, than '0' threads will turn it to '15' threads?

    Okay.

    Understood.

    Just thought, if there are threads that have locked 'Base Priority', not sure though. My current usage of 'ReplaceThreadsPriorityUntilExit' - a one instance at a time. So, if i run a game - only this game use this plugin. I tried to configure this plugin for other applications, running in background - result is i have seen a lot of running threads in PPF service with minor CPU usage. I will try to experiment with these applications. Using less amount of thread replaces.

    Alright, i will try with this a few more times.

    Edit: Just tried 'ReplaceThreadsPriorityUntilExit' with 'notepad'. One constantly cycling thread is running within PPF service. Also, 'unsecapp.exe' is created.

    PS. PPF service experienced "Stopping" state a couple of times.
     
    Last edited: Mar 11, 2014
  8. mbk1969

    mbk1969 Ancient Guru

    Messages:
    15,541
    Likes Received:
    13,561
    GPU:
    GF RTX 4070
    I failed to phrase description. Threads are enumerated by one pass, one loop, so any thread will not be affected twice. And only one pair of transition will be applied.

    And? Each 'ReplaceThreadsPriorityUntilExit' instance works in single thread. It needs to un-subscribe from WMI events after process is finished. I don`t know what will happen with WMI if we skip that.

    Edit: You overestimate importance of that plugin. It needs dedicated thread. We don`t know to which threads we apply replacement and for what. Plugins that set process priority class are much more interesting and much more lightweight.
     
    Last edited: Mar 11, 2014
  9. tappingbowl

    tappingbowl Guest

    Messages:
    209
    Likes Received:
    1
    GPU:
    Nvidia
    So threads with '-15' changed to '0' will not become '15'? I meant their origin 'Base Priority' level.

    Oh, that is subscription. Thought it will work in a way as 'Win32_ProcessStartTrace'. I mean i did not see such activity with monitoring processes start. Yeah, ignoring unsubscribe could probably do something to WMI, a slowdown or a corruption.

    Probably you are right. Still i would like to use that one. Because it change priorities of threads in a process. Kind of a same thing do plugins that set priority classes. If it was possible to set a process priority lower than '4' without tweaking threads...

    You mean 'UntilExit' version needs a dedicated thread? Well, non-'UntilExit' version works different. If it could just launch itself, say every 5-10 minutes... Or 'UntilExit' version could have some kind of a delay-pause. Just realized that it is getting list of threads once and then watching for events (new threads) - if you pause it, it will just stop watching. After resume it will continue watching and thus will skip threads that were created during "sleep time". Does it mean there is no point in such delay?

    And downside of non-'UntilExit' version (with recurrent timer) - threads replacements must be made more strict. For example, script started -> threads gathered -> threads levels are changed '0' to '15', '-15' to '0'. Plugin sleeps. Plugin activated -> threads gathered -> threads levels are changed '0' to '15', '-15' to '0'. That means, threads, that were originally '-15' became '15'. Well, it is possible to avoid by setting '-15' to '-2'. Or some complicated algorithms and buffers that are going to keep "threads history" or something (compare thread time creation with sleep time). Though i do not like the latter.

    Well, the one that suits better is first version, but looped. Though it has some restrictions. If you really do not want to make that plugin then do not.

    Maybe try thread affinity one? That one seems more promising.


    Edit: Got a couple of questions. Will PPF service unsubscribe from WMI events if it is stopped from services, but applications with 'ReplaceThreadsPriorityUntilExit' are still in a running state? And will it do so if it is killed from the task manager?

    Tried to update WEI (Windows Experience Index) - failed to do so. Got this error:
    Code:
    Error: Cannot Measure Time Stamp Counter (TSC) Frequency.
    Figured out the culprit. It is affinity setting for 'explorer.exe'. When 'winsat' is launched PPF service is not that fast in applying all cores for 'winsat'. Thus 'winsat' is unable to measure TSC. It needs access to all cores. Solution is to either set affinity of 'explorer.exe' to all cores or launch 'cmd.exe' and set affinity of 'cmd' to all cores and use command line:
    Code:
    winsat formal -restart clean
    That behavior made me wonder... is inability of TSC measuring somehow make applications (games, other system processes) not stable / faulty?
     
    Last edited: Mar 12, 2014
  10. mbk1969

    mbk1969 Ancient Guru

    Messages:
    15,541
    Likes Received:
    13,561
    GPU:
    GF RTX 4070
    No.


    No. This needs refactoring. But nothing will work when process is killed.

    It is the design of WEI that prevents it to start with no full affinity.

    Edit: Download PPF_source_v3.zip. Check closing service.
     
    Last edited: Mar 12, 2014

  11. tappingbowl

    tappingbowl Guest

    Messages:
    209
    Likes Received:
    1
    GPU:
    Nvidia
    In other words makes sure it is somewhat objective.

    Compiled. Monitoring started. It takes some time to trigger it. Happens randomly.
     
  12. drbaltazar

    drbaltazar Guest

    Messages:
    416
    Likes Received:
    0
    GPU:
    amd 7950 3gb
    Is message signal interrupt involved?if so set MSI max to 2 so if one interrupt of the same device is already in use you have another avail for same device?
     
  13. mbk1969

    mbk1969 Ancient Guru

    Messages:
    15,541
    Likes Received:
    13,561
    GPU:
    GF RTX 4070
    Source file was updated. It can be downloaded here - http://www.mediafire.com/download/rabc9ckn8qi72q8/PPF_source_v3.zip - and service binary can be rebuild (by renaming or removing and starting the Install-PPFService.ps1).

    Change log:

    - plugin PauseMonitoringUntilExit is no longer supported, so you should avoid it.

    - reactions to process stopping and to service stopping were extended to avoid possible resource leek.
     
  14. tappingbowl

    tappingbowl Guest

    Messages:
    209
    Likes Received:
    1
    GPU:
    Nvidia
    Deleted 'PauseMonitoringUntilExit' from plugins folder, i guess not needed anymore?
    No success in triggering "stopping" state with previous version. Will monitor this one.
     
  15. mbk1969

    mbk1969 Ancient Guru

    Messages:
    15,541
    Likes Received:
    13,561
    GPU:
    GF RTX 4070
    Source file was updated. It can be downloaded here - http://www.mediafire.com/download/rabc9ckn8qi72q8/PPF_source_v3.zip - and service binary can be rebuild (by renaming or removing and starting the Install-PPFService.ps1).

    Change log:

    - support for plugins 'PriorityLowerProcessesUntilExit' and 'AffinityAwayProcessesUntilExit' is fixed (there was a bug in service code).

    - further steps to improve disposal of ProcessAPIHelper instances to decrease either PPF` memory load and risk of resource leek.
     

  16. tappingbowl

    tappingbowl Guest

    Messages:
    209
    Likes Received:
    1
    GPU:
    Nvidia
    Updated to latest revision. No success in triggering "Stopping" state of the service with previous revision as well. Probably fixed. Will monitor though. No errors in logs so far.
     
  17. mbk1969

    mbk1969 Ancient Guru

    Messages:
    15,541
    Likes Received:
    13,561
    GPU:
    GF RTX 4070
    tappingbowl, I need ideas (strategies) for recurrent thread affinity plugin. Till now I have:
    - spread most busy threads among cores and leave others not affinitized - just for avoiding the core switch for the most busy threads;
    - spread most busy threads among cores and then spread remaining least busy threads;
    - split cores to pool for most busy threads and pool for least busy threads.


    Edit:

    Source file was updated. It can be downloaded here - http://www.mediafire.com/download/rabc9ckn8qi72q8/PPF_source_v3.zip - and service binary can be rebuild (by renaming or removing and starting the Install-PPFService.ps1).

    Change log:

    - support of plugin 'TraceProcess' reworked so that plugin continues to work until process is finished.

    - further steps to decrease PPF` memory load.
     
    Last edited: Mar 15, 2014
  18. tappingbowl

    tappingbowl Guest

    Messages:
    209
    Likes Received:
    1
    GPU:
    Nvidia
    Hm.. My guess is: all threads need to be separated. Though a little more intelligent way could be implemented, depending on a number of available cores for a process.

    1 - Service checks number of available cores.
    2 - Count all threads, determine usage (list them in usage by descending order).
    3 - 2 Cores? Go to 5.
    4 - 3 or more? Go to 6.
    5 - Separate threads starting from a busiest one - 1st core, a second busiest - 2nd core, third busiest - 1st core etc.
    6 - Separate threads starting from a busiest one - 1st core, a second busiest - 2nd core (if more cores, third busiest - 3rd core, fourth busiest - 1st core), third busiest - 1st core etc. And last available core - lowest busiest threads (need to decide % of busyness, for example take CPU time of a most busy thread and take a % from that - it will be a threshold for lowest busy threads).

    Maybe you could improve it or have a better idea.

    Good job! Memory usage dropped by 10 MB. One question, is that required to include '[PollingInterval=WMI]' in .ini now? So far service stops correctly.
     
    Last edited: Mar 15, 2014
  19. mbk1969

    mbk1969 Ancient Guru

    Messages:
    15,541
    Likes Received:
    13,561
    GPU:
    GF RTX 4070
    Problem is quantity of threads in process usually much greater than quantity of CPU cores.

    No. You better remove that section.

    I did test today - started 500 instances of Notepad from script (i.e. almost instantly). And PPF - with single plugin 'PriorityBelowNormal' in ini-file - managed to cope with load by same 11 threads which were there in idle state of service. I was really impressed. I understand (1) that PowerShell script is not quick enough to say that all instances were started at the same time, and (2) that WMI events have huge latencies, and (3) that ProcessExplorer might was not quick enough to show me actual threads in service. But nevertheless...

    I managed to download and install MS VisualStudio 2013 Express and now I am wondering - why the heck I didn`t do that year ago.
     
    Last edited: Mar 15, 2014
  20. tappingbowl

    tappingbowl Guest

    Messages:
    209
    Likes Received:
    1
    GPU:
    Nvidia
    I meant to separate threads between two cores in a circular way (threads sorted out by load levels):
    thn = thread high, n - a number that is indexed by PPF service, indicates a load level of a thread. tln = thread low, same with a number.
    th1
    th2
    th3
    th4
    th5
    th6
    tl1
    tl2
    tl3
    tl4

    2 cores are available:
    th1 - set affinity core 1
    th2 - set affinity core 2
    th3 - set affinity core 1
    th4 - set affinity core 2
    th5 - set affinity core 1
    th6 - set affinity core 2
    tl1 - set affinity core 1
    tl2 - set affinity core 2
    tl3 - set affinity core 1
    tl4 - set affinity core 2

    If more than 2 cores are available:
    th1 - set affinity core 1
    th2 - set affinity core 2
    th3 - set affinity core 1
    th4 - set affinity core 2
    th5 - set affinity core 1
    th6 - set affinity core 2
    tl1 - set affinity core 3
    tl2 - set affinity core 3
    tl3 - set affinity core 3
    tl4 - set affinity core 3

    If more than 3 cores are available:
    th1 - set affinity core 1
    th2 - set affinity core 2
    th3 - set affinity core 3
    th4 - set affinity core 1
    th5 - set affinity core 2
    th6 - set affinity core 3
    tl1 - set affinity core 4
    tl2 - set affinity core 4
    tl3 - set affinity core 4
    tl4 - set affinity core 4

    Though i would reconsider the above "circular" manage. For example, we have not an even number of high threads:
    th1 - set affinity core 1
    th2 - set affinity core 2
    th3 - set affinity core 3
    th4 - set affinity core 3
    th5 - set affinity core 2
    tl1 - set affinity core 4
    tl2 - set affinity core 4
    tl3 - set affinity core 4
    tl4 - set affinity core 4

    For 2 cores:
    th1 - set affinity core 1
    th2 - set affinity core 2
    th3 - set affinity core 2
    th4 - set affinity core 1
    th5 - set affinity core 2
    th6 - set affinity core 1
    tl1 - set affinity core 2
    tl2 - set affinity core 1
    tl3 - set affinity core 2
    tl4 - set affinity core 1

    After separating 'th1' and 'th2' - "affinity setting order" will invert. Thus 'th3' - will go for 2nd core, not 1st. It should reduce quantity of threads on core 1.

    The main idea to make a load on the cores more even, yet keep them separated as possible. 'th1' - will have highest load, highest cycles etc. No need to interrupt it. On my little experience there is 2 or 3 threads with extreme load per process. Though some could have more, like professional applications. The main and important thing is to separate two highest loaded threads, if there are only two cores available and interrupt highest loaded thread less than others.
    Okay, done. PPF service stops normally, i guess it is a win.

    Uh, oh. So PPF service managed to set priority for 500 notepad instances with only 11 threads? Or ProcessExplorer lied a bit and there were more? Though not really understood about PowerShell script quickness.
    Yeah, i did tell you about what other people were referring to. Some people been told: WMI is slow and have a large CPU usage. That is why i thought implementing kernel mode driver will suit this purpose better.

    Um.. does that mean that something interesting and amazing awaits us? :hopes, hopes: :cheers:
     
    Last edited: Mar 16, 2014

Share This Page