diff --git a/README.md b/README.md index 103ac90..f4007de 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ -`PSParallelPipeline` is a PowerShell module featuring the `Invoke-Parallel` cmdlet, designed to process pipeline input objects in parallel using multithreading. It mirrors the capabilities of [`ForEach-Object -Parallel`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object) from PowerShell 7.0+, bringing this functionality to Windows PowerShell 5.1, surpassing the constraints of [`Start-ThreadJob`](https://learn.microsoft.com/en-us/powershell/module/threadjob/start-threadjob?view=powershell-7.4). +`PSParallelPipeline` is a PowerShell module featuring the `Invoke-Parallel` cmdlet, designed to process pipeline input objects in parallel using multithreading. It mirrors the capabilities of [`ForEach-Object -Parallel`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object) from PowerShell 7.0+, bringing this functionality to Windows PowerShell 5.1, surpassing the constraints of [`Start-ThreadJob`](https://learn.microsoft.com/en-us/powershell/module/threadjob/start-threadjob). # Why Use This Module? @@ -33,7 +33,7 @@ Measure-Command { ## Common Parameters Support -Unlike `ForEach-Object -Parallel` (up to v7.5), `Invoke-Parallel` supports [Common Parameters](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_commonparameters?view=powershell-7.4), enhancing control and debugging. +Unlike `ForEach-Object -Parallel` (up to v7.5), `Invoke-Parallel` supports [Common Parameters](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_commonparameters), enhancing control and debugging. ```powershell # Stops on first error @@ -60,7 +60,7 @@ Get a single, friendly timeout message instead of multiple errors: # Invoke-Parallel: Timeout has been reached. ``` -## [`$using:`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes?view=powershell-7.4#the-using-scope-modifier) Scope Support +## [`$using:`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes#the-using-scope-modifier) Scope Support Easily pass variables into parallel scopes with the `$using:` modifier, just like `ForEach-Object -Parallel`: @@ -70,7 +70,7 @@ $message = 'world!' # hello world! ``` -## `-Variables` and `-Functions` Parameters +## `-Variables`, `-Functions`, `-ModuleNames`, and `-ModulePaths` Parameters - [`-Variables` Parameter](./docs/en-US/Invoke-Parallel.md#-variables): Pass variables directly to parallel runspaces. @@ -87,7 +87,22 @@ $message = 'world!' # hello world! ``` -Both parameters are quality-of-life enhancements, especially `-Functions`, which adds locally defined functions to the runspaces’ [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate)—a feature absent in `ForEach-Object -Parallel`. This is a far better option than passing function definitions into the parallel scope. +- [`-ModuleNames` Parameter](./docs/en-US/Invoke-Parallel.md#-modulenames): Import system-installed modules into parallel runspaces by name, using modules discoverable via `$env:PSModulePath`. + + ```powershell + Import-Csv users.csv | Invoke-Parallel { Get-ADUser $_.UserPrincipalName } -ModuleNames ActiveDirectory + # Imports ActiveDirectory module for Get-ADUser + ``` + +- [`-ModulePaths` Parameter](./docs/en-US/Invoke-Parallel.md#-modulepaths): Import custom modules from specified directory paths into parallel runspaces. + + ```powershell + $moduleDir = Join-Path $PSScriptRoot "CustomModule" + 0..10 | Invoke-Parallel { Get-CustomCmdlet } -ModulePaths $moduleDir + # Imports custom module for Get-CustomCmdlet + ``` + +These parameters are a quality-of-life enhancement, especially `-Functions`, which incorporates locally defined functions to the runspaces’ [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate)—a feature absent in `ForEach-Object -Parallel` and a far better option than passing function definitions into the parallel scope. The new `-ModuleNames` and `-ModulePaths` parameters simplify module integration by automatically loading system-installed and custom modules, respectively, eliminating the need for manual `Import-Module` calls within the script block. # Documentation diff --git a/docs/en-US/Invoke-Parallel.md b/docs/en-US/Invoke-Parallel.md index a8c55ed..c3f9bc5 100644 --- a/docs/en-US/Invoke-Parallel.md +++ b/docs/en-US/Invoke-Parallel.md @@ -15,19 +15,24 @@ Executes parallel processing of pipeline input objects using multithreading. ```powershell Invoke-Parallel - -InputObject [-ScriptBlock] + [-InputObject ] [-ThrottleLimit ] + [-TimeoutSeconds ] [-Variables ] [-Functions ] + [-ModuleNames ] + [-ModulePaths ] [-UseNewRunspace] - [-TimeoutSeconds ] [] ``` ## DESCRIPTION -The `Invoke-Parallel` cmdlet enables parallel processing of input objects in PowerShell, including __Windows PowerShell 5.1__, offering functionality similar to `ForEach-Object -Parallel` introduced in PowerShell 7.0. It processes pipeline input across multiple threads, improving performance for tasks that benefit from parallel execution. +The `Invoke-Parallel` cmdlet enables parallel processing of input objects in PowerShell, including +__Windows PowerShell 5.1__ and PowerShell 7+, offering functionality similar to `ForEach-Object -Parallel` introduced in +PowerShell 7.0. It processes pipeline input across multiple threads, improving performance for tasks that benefit from +parallel execution. ## EXAMPLES @@ -42,7 +47,10 @@ $message = 'Hello world from ' } ``` -This example demonstrates parallel execution of a script block with a 3-second delay, appending a unique runspace ID to a message. The [`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes?view=powershell-7.4#the-using-scope-modifier) is used to pass the local variable `$message` into the parallel scope, a supported method for accessing external variables in `Invoke-Parallel`. +This example demonstrates parallel execution of a script block with a 3-second delay, appending a unique runspace ID to +a message. The [`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes#the-using-scope-modifier) +is used to pass the local variable `$message` into the parallel scope, a supported method for accessing external +variables in `Invoke-Parallel`. ### Example 2: Demonstrates `-Variables` Parameter @@ -55,7 +63,9 @@ $message = 'Hello world from ' } -Variables @{ message = $message } ``` -This example demonstrates the [`-Variables` parameter](#-variables), which passes the local variable `$message` into the parallel scope using a hashtable. The key `message` in the hashtable defines the variable name available within the script block, serving as an alternative to the `$using:` scope modifier. +This example demonstrates the [`-Variables` parameter](#-variables), which passes the local variable `$message` into +the parallel scope using a hashtable. The key `message` in the hashtable defines the variable name available within the +script block, serving as an alternative to the `$using:` scope modifier. ### Example 3: Adding to a thread-safe collection with `$using:` @@ -65,7 +75,8 @@ Get-Process | Invoke-Parallel { ($using:dict)[$_.Id] = $_ } $dict[$PID] ``` -This example uses a thread-safe dictionary to store process objects by ID, leveraging the `$using:` modifier for variable access. +This example uses a thread-safe dictionary to store process objects by ID, leveraging the `$using:` modifier for +variable access. ### Example 4: Adding to a thread-safe collection with `-Variables` @@ -85,7 +96,8 @@ function Greet { param($s) "$s hey there!" } 0..10 | Invoke-Parallel { Greet $_ } -Functions Greet ``` -This example imports a local function `Greet` into the parallel scope using [`-Functions` parameter](#-functions), allowing its use within the script block. +This example imports a local function `Greet` into the parallel scope using [`-Functions` parameter](#-functions), +allowing its use within the script block. ### Example 6: Setting a timeout with `-TimeoutSeconds` @@ -93,7 +105,8 @@ This example imports a local function `Greet` into the parallel scope using [`-F 0..10 | Invoke-Parallel { Start-Sleep 1 } -TimeoutSeconds 3 ``` -This example limits execution to 3 seconds, stopping all running script blocks and ignoring unprocessed input once the timeout is reached. +This example limits execution to 3 seconds, stopping all running script blocks and ignoring unprocessed input once the +timeout is reached. ### Example 7: Creating new runspaces with `-UseNewRunspace` @@ -117,17 +130,45 @@ This example limits execution to 3 seconds, stopping all running script blocks a # 9af7c222-061d-4c89-b073-375ee925e538 ``` -This example contrasts default runspace reuse with the `-UseNewRunspace` switch, showing unique runspace IDs for each invocation in the latter case. +This example contrasts default runspace reuse with the `-UseNewRunspace` switch, showing unique runspace IDs for each +invocation in the latter case. + +### Example 8: Using the `-ModuleNames` parameter + +```powershell +Import-Csv users.csv | Invoke-Parallel { Get-ADUser $_.UserPrincipalName } -ModuleNames ActiveDirectory +``` + +This example imports the `ActiveDirectory` module into the parallel scope using `-ModuleNames`, enabling the +`Get-ADUser` cmdlet within the script block. + +### Example 9: Using the `-ModulePaths` parameter + +```powershell +$moduleDir = Join-Path $PSScriptRoot "CustomModule" +0..10 | Invoke-Parallel { Get-CustomCmdlet } -ModulePaths $moduleDir +``` + +This example imports a custom module from the specified directory using `-ModulePaths`, allowing the `Get-CustomCmdlet` +function to be used in the parallel script block. + +> [!NOTE] +> +> The path must point to a directory containing a valid PowerShell module. ## PARAMETERS ### -Functions -Specifies an array of function names from the local session to include in the runspaces' [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate). This enables their use within the parallel script block. +Specifies an array of function names from the local session to include in the runspaces’ +[Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate). +This enables their use within the parallel script block. > [!TIP] > -> This parameter is the recommended way to make local functions available in the parallel scope. Alternatively, you can retrieve the function definition as a string (e.g., `$def = ${function:Greet}.ToString()`) and use `$using:` to pass it into the script block, defining it there (e.g., `${function:Greet} = $using:def`). +> This parameter is the recommended way to make local functions available in the parallel scope. +Alternatively, you can retrieve the function definition as a string (e.g., `$def = ${function:Greet}.ToString()`) and +use `$using:` to pass it into the script block, defining it there (e.g., `${function:Greet} = $using:def`). ```yaml Type: String[] @@ -175,11 +216,8 @@ Accept wildcard characters: False ### -ThrottleLimit -Sets the maximum number of script blocks executed in parallel across multiple threads. Additional input objects wait until the number of running script blocks falls below this limit. - -> [!NOTE] -> -> The default value is `5`. +Sets the maximum number of script blocks executed in parallel across multiple threads. Additional input objects wait +until the number of running script blocks falls below this limit. The default value is `5`. ```yaml Type: Int32 @@ -195,7 +233,8 @@ Accept wildcard characters: False ### -TimeoutSeconds -Specifies the maximum time (in seconds) to process all input objects. When the timeout is reached, running script blocks are terminated, and remaining input is discarded. +Specifies the maximum time (in seconds) to process all input objects. When the timeout is reached, running script blocks +are terminated, and remaining input is discarded. > [!NOTE] > @@ -231,11 +270,13 @@ Accept wildcard characters: False ### -Variables -Provides a hashtable of variables to make available in the parallel scope. Keys define the variable names within the script block. +Provides a hashtable of variables to make available in the parallel scope. Keys define the variable names within the +script block. > [!TIP] > -> Use this parameter as an alternative to the [`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes?view=powershell-7.4#scope-modifiers). +> Use this parameter as an alternative to the +[`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes#scope-modifiers). ```yaml Type: Hashtable @@ -249,6 +290,54 @@ Accept pipeline input: False Accept wildcard characters: False ``` +### -ModuleNames + +Specifies an array of module names to import into the runspaces’ +[Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate). +This allows the script block to use cmdlets and functions from the specified modules. + +> [!TIP] +> +> Use this parameter to ensure required modules are available in the parallel scope. Module names must be discoverable +via the [`$env:PSModulePath`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_psmodulepath) +environment variable, which lists installed module locations. + +```yaml +Type: String[] +Parameter Sets: (All) +Aliases: mn + +Required: False +Position: Named +Default value: None +Accept pipeline input: False +Accept wildcard characters: False +``` + +### -ModulePaths + +Specifies an array of file paths to directories containing PowerShell modules (e.g., `.psm1` or `.psd1` files) to import +into the runspaces’ [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate). +This enables the script block to use cmdlets and functions from custom or local modules. + +> [!NOTE] +> +> Paths must be absolute or relative to the current working directory and must point to valid directories containing +PowerShell modules. If an invalid path (e.g., a file or non-existent directory) is provided, a terminating error is +thrown. + +```yaml +Type: String[] +Parameter Sets: (All) +Aliases: mp + +Required: False +Position: Named +Default value: None +Accept pipeline input: False +Accept wildcard characters: False +``` + ### CommonParameters This cmdlet supports the common parameters. For more information, see [about_CommonParameters](http://go.microsoft.com/fwlink/?LinkID=113216). @@ -268,14 +357,19 @@ Returns objects produced by the script block. ## NOTES - `Invoke-Parallel` uses multithreading, which may introduce overhead. For small datasets, sequential processing might be faster. -- Ensure variables or collections passed to the parallel scope are thread-safe (e.g., `[System.Collections.Concurrent.ConcurrentDictionary]`), as shown in Examples 3 and 4. -- By default, runspaces are reused from a pool to optimize resource usage. Using `-UseNewRunspace` increases memory and startup time but ensures isolation. +- Ensure variables or collections passed to the parallel scope are thread-safe (e.g., use +[`ConcurrentDictionary`](https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentdictionary-2) or similar), as shown in +[__Example #3__](#example-3-adding-to-a-thread-safe-collection-with-using) and +[__Example #4__](#example-4-adding-to-a-thread-safe-collection-with--variables), +to avoid race conditions. +- By default, runspaces are reused from a pool to optimize resource usage. Using [`-UseNewRunspace`](#-usenewrunspace) increases memory and +startup time but ensures isolation. ## RELATED LINKS [__ForEach-Object -Parallel__](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object) -[__Runspaces Overview__](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.runspace?view=powershellsdk-7.4.0) +[__Runspaces Overview__](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.runspace) [__Managed threading best practices__](https://learn.microsoft.com/en-us/dotnet/standard/threading/managed-threading-best-practices) diff --git a/module/PSParallelPipeline.psd1 b/module/PSParallelPipeline.psd1 index 44a05bd..675434f 100644 --- a/module/PSParallelPipeline.psd1 +++ b/module/PSParallelPipeline.psd1 @@ -11,7 +11,7 @@ RootModule = 'bin/netstandard2.0/PSParallelPipeline.dll' # Version number of this module. - ModuleVersion = '1.2.3' + ModuleVersion = '1.2.4' # Supported PSEditions # CompatiblePSEditions = @() diff --git a/src/PSParallelPipeline/Commands/InvokeParallelCommand.cs b/src/PSParallelPipeline/Commands/InvokeParallelCommand.cs index f4bccc8..9c4a6eb 100644 --- a/src/PSParallelPipeline/Commands/InvokeParallelCommand.cs +++ b/src/PSParallelPipeline/Commands/InvokeParallelCommand.cs @@ -43,6 +43,17 @@ public sealed class InvokeParallelCommand : PSCmdlet, IDisposable [Alias("funcs")] public string[]? Functions { get; set; } + [Parameter] + [ValidateNotNullOrEmpty] + [ArgumentCompleter(typeof(ModuleCompleter))] + [Alias("mn")] + public string[]? ModuleNames { get; set; } + + [Parameter] + [ValidateNotNullOrEmpty] + [Alias("mp")] + public string[]? ModulePaths { get; set; } + [Parameter] [Alias("unr")] public SwitchParameter UseNewRunspace { get; set; } @@ -57,7 +68,9 @@ protected override void BeginProcessing() InitialSessionState iss = InitialSessionState .CreateDefault2() .AddFunctions(Functions, this) - .AddVariables(Variables, this); + .AddVariables(Variables, this) + .ImportModules(ModuleNames) + .ImportModulesFromPath(ModulePaths, this); PoolSettings poolSettings = new( ThrottleLimit, UseNewRunspace, iss); @@ -90,7 +103,7 @@ protected override void ProcessRecord() } catch (OperationCanceledException exception) { - _worker.WaitForCompletion(); + CancelAndWait(); exception.WriteTimeoutError(this); } } @@ -116,7 +129,7 @@ protected override void EndProcessing() } catch (OperationCanceledException exception) { - _worker.WaitForCompletion(); + CancelAndWait(); exception.WriteTimeoutError(this); } } diff --git a/src/PSParallelPipeline/ExceptionHelper.cs b/src/PSParallelPipeline/ExceptionHelper.cs index bd60e9a..182e0ad 100644 --- a/src/PSParallelPipeline/ExceptionHelper.cs +++ b/src/PSParallelPipeline/ExceptionHelper.cs @@ -1,11 +1,13 @@ using System; +using System.IO; using System.Management.Automation; +using Microsoft.PowerShell.Commands; namespace PSParallelPipeline; internal static class ExceptionHelper { - private const string _notsupported = + private const string NotSupported = "Passed-in script block variables are not supported, and can result in undefined behavior."; internal static void WriteTimeoutError(this Exception exception, PSCmdlet cmdlet) => @@ -50,7 +52,7 @@ internal static void ThrowIfVariableIsScriptBlock(this PSCmdlet cmdlet, object? } cmdlet.ThrowTerminatingError(new ErrorRecord( - new PSArgumentException(_notsupported), + new PSArgumentException(NotSupported), "PassedInVariableCannotBeScriptBlock", ErrorCategory.InvalidType, value)); @@ -67,7 +69,7 @@ internal static void ThrowIfInputObjectIsScriptBlock(this object? value, PSCmdle new PSArgumentException( string.Concat( "Piped input object cannot be a script block. ", - _notsupported)), + NotSupported)), "InputObjectCannotBeScriptBlock", ErrorCategory.InvalidType, value)); @@ -84,9 +86,49 @@ internal static void ThrowIfUsingValueIsScriptBlock(this PSCmdlet cmdlet, object new PSArgumentException( string.Concat( "A $using: variable cannot be a script block. ", - _notsupported)), + NotSupported)), "UsingVariableCannotBeScriptBlock", ErrorCategory.InvalidType, value)); } + + internal static void ThrowIfInvalidProvider( + this ProviderInfo provider, + string path, + PSCmdlet cmdlet) + { + if (provider.ImplementingType == typeof(FileSystemProvider)) + { + return; + } + + ErrorRecord error = new( + new NotSupportedException( + $"The resolved path '{path}' is not a FileSystem path but '{provider.Name}'."), + "NotFileSystemPath", + ErrorCategory.InvalidArgument, + path); + + cmdlet.ThrowTerminatingError(error); + } + + internal static void ThrowIfNotDirectory( + this string path, + PSCmdlet cmdlet) + { + if (Directory.Exists(path)) + { + return; + } + + ErrorRecord error = new( + new ArgumentException( + $"The specified path '{path}' does not exist or is not a directory. " + + "The path must be a valid directory containing one or more PowerShell modules."), + "NotDirectoryPath", + ErrorCategory.InvalidArgument, + path); + + cmdlet.ThrowTerminatingError(error); + } } diff --git a/src/PSParallelPipeline/Extensions.cs b/src/PSParallelPipeline/Extensions.cs index c014ea1..2051b10 100644 --- a/src/PSParallelPipeline/Extensions.cs +++ b/src/PSParallelPipeline/Extensions.cs @@ -5,7 +5,9 @@ using System.Management.Automation; using System.Management.Automation.Language; using System.Management.Automation.Runspaces; +using System.Runtime.CompilerServices; using System.Text; +using System.Threading.Tasks; namespace PSParallelPipeline; @@ -62,13 +64,55 @@ internal static InitialSessionState AddVariables( return initialSessionState; } + internal static InitialSessionState ImportModules( + this InitialSessionState initialSessionState, + string[]? modulesToImport) + { + if (modulesToImport is not null) + { + initialSessionState.ImportPSModule(modulesToImport); + } + + return initialSessionState; + } + + internal static InitialSessionState ImportModulesFromPath( + this InitialSessionState initialSessionState, + string[]? modulePaths, + PSCmdlet cmdlet) + { + + if (modulePaths is not null) + { + foreach (string path in modulePaths) + { + string resolved = cmdlet.ResolvePath(path); + initialSessionState.ImportPSModulesFromPath(resolved); + } + } + + return initialSessionState; + } + + private static string ResolvePath(this PSCmdlet cmdlet, string path) + { + string resolved = cmdlet.SessionState.Path.GetUnresolvedProviderPathFromPSPath( + path: path, + provider: out ProviderInfo provider, + drive: out _); + + provider.ThrowIfInvalidProvider(path, cmdlet); + resolved.ThrowIfNotDirectory(cmdlet); + return resolved.TrimEnd('\\', '/'); + } + internal static Dictionary GetUsingParameters( this ScriptBlock script, PSCmdlet cmdlet) { Dictionary usingParams = []; IEnumerable usingExpressionAsts = script.Ast - .FindAll((a) => a is UsingExpressionAst, true) + .FindAll(a => a is UsingExpressionAst, true) .Cast(); foreach (UsingExpressionAst usingStatement in usingExpressionAsts) @@ -147,13 +191,15 @@ internal static InitialSessionState AddVariables( paramBlock: null, statements: new StatementBlockAst( extent: ast.Extent, - statements: [ new PipelineAst( - extent: ast.Extent, - pipelineElements: [ new CommandExpressionAst( + statements: [ + new PipelineAst( extent: ast.Extent, - expression: lookupAst, - redirections: null) - ]) + pipelineElements: [ + new CommandExpressionAst( + extent: ast.Extent, + expression: lookupAst, + redirections: null) + ]) ], traps: null), isFilter: false); @@ -162,4 +208,15 @@ internal static InitialSessionState AddVariables( .GetScriptBlock() .InvokeReturnAsIs(); } + + internal static Task InvokePowerShellAsync( + this PowerShell powerShell, + PSDataCollection output) + => Task.Factory.FromAsync( + powerShell.BeginInvoke(null, output), + powerShell.EndInvoke); + + internal static ConfiguredTaskAwaitable NoContext(this Task task) => task.ConfigureAwait(false); + + internal static ConfiguredTaskAwaitable NoContext(this Task task) => task.ConfigureAwait(false); } diff --git a/src/PSParallelPipeline/ModuleCompleter.cs b/src/PSParallelPipeline/ModuleCompleter.cs new file mode 100644 index 0000000..f69317a --- /dev/null +++ b/src/PSParallelPipeline/ModuleCompleter.cs @@ -0,0 +1,31 @@ +using System; +using System.Collections; +using System.Collections.Generic; +using System.Management.Automation; +using System.Management.Automation.Language; + +namespace PSParallelPipeline; + +public sealed class ModuleCompleter : IArgumentCompleter +{ + public IEnumerable CompleteArgument( + string commandName, + string parameterName, + string wordToComplete, + CommandAst commandAst, + IDictionary fakeBoundParameters) + { + using PowerShell ps = PowerShell + .Create(RunspaceMode.CurrentRunspace) + .AddCommand("Get-Module") + .AddParameter("ListAvailable"); + + foreach (PSModuleInfo module in ps.Invoke()) + { + if (module.Name.StartsWith(wordToComplete, StringComparison.InvariantCultureIgnoreCase)) + { + yield return new CompletionResult(module.Name); + } + } + } +} diff --git a/src/PSParallelPipeline/PSTask.cs b/src/PSParallelPipeline/PSTask.cs index cbd3b43..9942297 100644 --- a/src/PSParallelPipeline/PSTask.cs +++ b/src/PSParallelPipeline/PSTask.cs @@ -15,6 +15,8 @@ internal sealed class PSTask private const string StopParsingOp = "--%"; + private bool _canceled; + private readonly PowerShell _powershell; private readonly PSDataStreams _internalStreams; @@ -36,7 +38,7 @@ private PSTask(RunspacePool pool) _pool = pool; } - static internal PSTask Create( + internal static PSTask Create( object? input, RunspacePool runspacePool, TaskSettings settings) @@ -55,9 +57,9 @@ internal async Task InvokeAsync() try { using CancellationTokenRegistration _ = _token.Register(Cancel); - _runspace = await _pool.GetRunspaceAsync().ConfigureAwait(false); + _runspace = await _pool.GetRunspaceAsync().NoContext(); _powershell.Runspace = _runspace; - await InvokePowerShellAsync(_powershell, _outputStreams.Success).ConfigureAwait(false); + await _powershell.InvokePowerShellAsync(_outputStreams.Success).NoContext(); } catch (Exception exception) { @@ -81,13 +83,6 @@ private static void SetStreams( streams.Warning = outputStreams.Warning; } - private static Task InvokePowerShellAsync( - PowerShell powerShell, - PSDataCollection output) => - Task.Factory.FromAsync( - powerShell.BeginInvoke(null, output), - powerShell.EndInvoke); - private PSTask AddInput(object? inputObject) { if (inputObject is not null) @@ -119,19 +114,22 @@ private PSTask AddUsingStatements(Dictionary usingParams) private void CompleteTask() { - _powershell.Dispose(); - if (!_token.IsCancellationRequested && _runspace is not null) + if (_canceled) { - _pool.PushRunspace(_runspace); + _runspace?.Dispose(); return; } - _runspace?.Dispose(); + _powershell.Dispose(); + if (_runspace is not null) + { + _pool.PushRunspace(_runspace); + } } - private void Cancel() + internal void Cancel() { _powershell.Dispose(); - _runspace?.Dispose(); + _canceled = true; } } diff --git a/src/PSParallelPipeline/RunspacePool.cs b/src/PSParallelPipeline/RunspacePool.cs index cd9bbc7..98cd7d5 100644 --- a/src/PSParallelPipeline/RunspacePool.cs +++ b/src/PSParallelPipeline/RunspacePool.cs @@ -38,7 +38,8 @@ internal void PushRunspace(Runspace runspace) if (UseNewRunspace) { runspace.Dispose(); - runspace = CreateRunspace(); + _semaphore.Release(); + return; } _pool.Enqueue(runspace); @@ -52,18 +53,19 @@ private Runspace CreateRunspace() return rs; } + private Task CreateRunspaceAsync() => + Task.Run(CreateRunspace, cancellationToken: Token); + internal async Task GetRunspaceAsync() { - await _semaphore - .WaitAsync(Token) - .ConfigureAwait(false); + await _semaphore.WaitAsync(Token).NoContext(); if (_pool.TryDequeue(out Runspace runspace)) { return runspace; } - return CreateRunspace(); + return await CreateRunspaceAsync().NoContext(); } public void Dispose() diff --git a/src/PSParallelPipeline/Worker.cs b/src/PSParallelPipeline/Worker.cs index 079d9ea..52f88b8 100644 --- a/src/PSParallelPipeline/Worker.cs +++ b/src/PSParallelPipeline/Worker.cs @@ -57,10 +57,10 @@ private async Task Start() { Task task = await Task .WhenAny(tasks) - .ConfigureAwait(false); + .NoContext(); tasks.Remove(task); - await task.ConfigureAwait(false); + await task.NoContext(); } tasks.Add(PSTask @@ -72,9 +72,10 @@ private async Task Start() { } finally { - await Task - .WhenAll(tasks) - .ConfigureAwait(false); + if (tasks.Count > 0) + { + await Task.WhenAll(tasks).NoContext(); + } _output.CompleteAdding(); } diff --git a/tests/PSParallelPipeline.tests.ps1 b/tests/PSParallelPipeline.tests.ps1 index 417ada7..17e89e7 100644 --- a/tests/PSParallelPipeline.tests.ps1 +++ b/tests/PSParallelPipeline.tests.ps1 @@ -152,16 +152,16 @@ Describe PSParallelPipeline { 1..100 | Invoke-Parallel @invokeParallelSplat } | Should -Throw -ExceptionType ([TimeoutException]) $timer.Stop() - $timer.Elapsed | Should -BeLessOrEqual ([timespan]::FromSeconds(2.2)) + $timer.Elapsed | Should -BeLessOrEqual ([timespan]::FromSeconds(3)) $timer.Restart() { $invokeParallelSplat = @{ ThrottleLimit = 5 TimeOutSeconds = 1 ErrorAction = 'Stop' - ScriptBlock = { Start-Sleep 10 } + ScriptBlock = { $_; Start-Sleep 10 } } - 1..100 | Invoke-Parallel @invokeParallelSplat + 1..1000000 | Invoke-Parallel @invokeParallelSplat } | Should -Throw -ExceptionType ([TimeoutException]) $timer.Stop() $timer.Elapsed | Should -BeLessOrEqual ([timespan]::FromSeconds(1.2)) @@ -183,6 +183,53 @@ Describe PSParallelPipeline { } } + Context 'ModuleCompleter' { + It 'Provides completion based on existing modules' { + Complete 'Invoke-Parallel -ModuleNames ' | + Should -Not -BeNullOrEmpty + } + } + + Context 'ModuleNames Parameter' { + It 'Loads a Module from Name' { + $null | Invoke-Parallel -ModuleNames Microsoft.PowerShell.Utility { + (Get-Module).Name + } | Should -Be Microsoft.PowerShell.Utility + } + } + + Context 'ModulePaths Parameter' { + BeforeAll { + $modulePath = Join-Path $PSScriptRoot .\TestModule\ + $invalidModulePath = Join-Path $modulePath TestModule.psm1 + $modulePath, $invalidModulePath | Out-Null + } + + It 'Loads a Module from Path' { + $null | Invoke-Parallel -ModulePaths $modulePath { + Get-Message + } | Should -Match '^Hello world from' + } + + It 'Should throw if path is not a directory' { + { + $null | Invoke-Parallel -ModulePaths $invalidModulePath { } + } | Should -Throw -ExceptionType ([ArgumentException]) + } + + It 'Should throw if path is not FileSystem Provider' { + { + $null | Invoke-Parallel -ModulePaths function:Complete { } + } | Should -Throw -ExceptionType ([NotSupportedException]) + } + + It 'Should load modules in parallel' { + Measure-Command { + 0..4 | Invoke-Parallel -ModulePaths $modulePath { } + } | Should -BeLessOrEqual ([timespan]::FromSeconds(7)) + } + } + Context '$using: scope modifier' { It 'Allows passed-in variables through the $using: scope modifier' { $message = 'Hello world from {0:D2}' @@ -262,7 +309,7 @@ Describe PSParallelPipeline { Assert-RunspaceCount { Measure-Command { & $testOne | Should -HaveCount 5 } | ForEach-Object TotalSeconds | - Should -BeLessThan 2 + Should -BeLessThan 3 Measure-Command { & $testTwo | Should -HaveCount 10 } | ForEach-Object TotalSeconds | @@ -290,6 +337,7 @@ Describe PSParallelPipeline { $rs = [runspacefactory]::CreateRunspace($Host, $iss) $rs.Open() } + AfterAll { $rs.Dispose() } @@ -313,7 +361,7 @@ Describe PSParallelPipeline { $ps.Stop() while (-not $task.AsyncWaitHandle.WaitOne(200)) { } $timer.Stop() - $timer.Elapsed | Should -BeLessOrEqual ([timespan]::FromSeconds(2)) + $timer.Elapsed | Should -BeLessOrEqual ([timespan]::FromSeconds(4)) if ($ps.HadErrors) { $ps.Streams.Error | Write-Host -ForegroundColor Red @@ -334,34 +382,34 @@ Describe PSParallelPipeline { Assert-RunspaceCount { 0..10 | Invoke-Parallel @invokeParallelSplat | - Select-Object -First 1 - } -TestCount 100 + Select-Object -First 3 + } -TestCount 50 Assert-RunspaceCount { $invokeParallelSplat['UseNewRunspace'] = $true 0..10 | Invoke-Parallel @invokeParallelSplat | - Select-Object -First 10 - } -TestCount 100 + Select-Object -First 7 + } -TestCount 50 } It 'Disposes on OperationCanceledException' { $invokeParallelSplat = @{ - ThrottleLimit = 300 - TimeOutSeconds = 1 + ThrottleLimit = 30 ScriptBlock = { Start-Sleep 1 } } Assert-RunspaceCount { + $invokeParallelSplat['TimeOutSeconds'] = Get-Random -Minimum 1 -Maximum 4 { 0..1000 | Invoke-Parallel @invokeParallelSplat } | Should -Throw -ExceptionType ([TimeoutException]) - } -TestCount 50 # -WaitSeconds 1 + } -TestCount 20 Assert-RunspaceCount { + $invokeParallelSplat['TimeOutSeconds'] = Get-Random -Minimum 1 -Maximum 4 $invokeParallelSplat['UseNewRunspace'] = $true - $invokeParallelSplat['ThrottleLimit'] = 1001 { 0..1000 | Invoke-Parallel @invokeParallelSplat } | Should -Throw -ExceptionType ([TimeoutException]) - } -TestCount 50 # -WaitSeconds 1 + } -TestCount 20 } } } diff --git a/tests/TestModule/TestModule.psm1 b/tests/TestModule/TestModule.psm1 new file mode 100644 index 0000000..20c76dc --- /dev/null +++ b/tests/TestModule/TestModule.psm1 @@ -0,0 +1,2 @@ +Start-Sleep 2 +function Get-Message { "Hello world from $([runspace]::DefaultRunspace.InstanceId)!" } diff --git a/tests/common.psm1 b/tests/common.psm1 index 448faf9..8a26b2b 100644 --- a/tests/common.psm1 +++ b/tests/common.psm1 @@ -48,8 +48,14 @@ function Assert-RunspaceCount { Start-Sleep $WaitSeconds } - Get-Runspace | - Should -HaveCount $count -Because 'Runspaces should be correctly disposed' + $shouldSplat = @{ + HaveCount = $true + Because = 'Runspaces should be correctly disposed' + ErrorAction = 'Stop' + ExpectedValue = $count + } + + Get-Runspace | Should @shouldSplat } } } diff --git a/tools/ProjectBuilder/Module.cs b/tools/ProjectBuilder/Module.cs index a2ff7cf..bb227b4 100644 --- a/tools/ProjectBuilder/Module.cs +++ b/tools/ProjectBuilder/Module.cs @@ -24,11 +24,11 @@ public sealed class Module private string? Release { get => _info.Project.Release; } - private readonly UriBuilder _builder = new(_base); + private readonly UriBuilder _builder = new(Base); - private const string _base = "https://www.powershellgallery.com"; + private const string Base = "https://www.powershellgallery.com"; - private const string _path = "api/v2/package/{0}/{1}"; + private const string Path = "api/v2/package/{0}/{1}"; private readonly ProjectInfo _info; @@ -96,7 +96,7 @@ private string[] DownloadModules(ModuleDownload[] modules) } Console.WriteLine($"Installing build pre-req '{module}'"); - _builder.Path = string.Format(_path, module, version); + _builder.Path = string.Format(Path, module, version); Task task = GetModuleAsync( uri: _builder.Uri.ToString(), destination: destination, @@ -129,10 +129,10 @@ private static async Task WaitTaskAsync( } private string GetDestination(string module) => - Path.Combine(PreRequisitePath, Path.ChangeExtension(module, "zip")); + System.IO.Path.Combine(PreRequisitePath, System.IO.Path.ChangeExtension(module, "zip")); private string GetModulePath(string module) => - Path.Combine(PreRequisitePath, module); + System.IO.Path.Combine(PreRequisitePath, module); private static async Task GetModuleAsync( string uri, @@ -153,7 +153,7 @@ private static async Task GetModuleAsync( private static string InitPrerequisitePath(DirectoryInfo root) { - string path = Path.Combine(root.Parent.FullName, "tools", "Modules"); + string path = System.IO.Path.Combine(root.Parent.FullName, "tools", "Modules"); if (!Directory.Exists(path)) { Directory.CreateDirectory(path);