Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 20 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

</div>

`PSParallelPipeline` is a PowerShell module featuring the `Invoke-Parallel` cmdlet, designed to process pipeline input objects in parallel using multithreading. It mirrors the capabilities of [`ForEach-Object -Parallel`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object) from PowerShell 7.0+, bringing this functionality to Windows PowerShell 5.1, surpassing the constraints of [`Start-ThreadJob`](https://learn.microsoft.com/en-us/powershell/module/threadjob/start-threadjob?view=powershell-7.4).
`PSParallelPipeline` is a PowerShell module featuring the `Invoke-Parallel` cmdlet, designed to process pipeline input objects in parallel using multithreading. It mirrors the capabilities of [`ForEach-Object -Parallel`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object) from PowerShell 7.0+, bringing this functionality to Windows PowerShell 5.1, surpassing the constraints of [`Start-ThreadJob`](https://learn.microsoft.com/en-us/powershell/module/threadjob/start-threadjob).

# Why Use This Module?

Expand All @@ -33,7 +33,7 @@ Measure-Command {

## Common Parameters Support

Unlike `ForEach-Object -Parallel` (up to v7.5), `Invoke-Parallel` supports [Common Parameters](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_commonparameters?view=powershell-7.4), enhancing control and debugging.
Unlike `ForEach-Object -Parallel` (up to v7.5), `Invoke-Parallel` supports [Common Parameters](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_commonparameters), enhancing control and debugging.

```powershell
# Stops on first error
Expand All @@ -60,7 +60,7 @@ Get a single, friendly timeout message instead of multiple errors:
# Invoke-Parallel: Timeout has been reached.
```

## [`$using:`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes?view=powershell-7.4#the-using-scope-modifier) Scope Support
## [`$using:`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes#the-using-scope-modifier) Scope Support

Easily pass variables into parallel scopes with the `$using:` modifier, just like `ForEach-Object -Parallel`:

Expand All @@ -70,7 +70,7 @@ $message = 'world!'
# hello world!
```

## `-Variables` and `-Functions` Parameters
## `-Variables`, `-Functions`, `-ModuleNames`, and `-ModulePaths` Parameters

- [`-Variables` Parameter](./docs/en-US/Invoke-Parallel.md#-variables): Pass variables directly to parallel runspaces.

Expand All @@ -87,7 +87,22 @@ $message = 'world!'
# hello world!
```

Both parameters are quality-of-life enhancements, especially `-Functions`, which adds locally defined functions to the runspaces’ [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate)—a feature absent in `ForEach-Object -Parallel`. This is a far better option than passing function definitions into the parallel scope.
- [`-ModuleNames` Parameter](./docs/en-US/Invoke-Parallel.md#-modulenames): Import system-installed modules into parallel runspaces by name, using modules discoverable via `$env:PSModulePath`.

```powershell
Import-Csv users.csv | Invoke-Parallel { Get-ADUser $_.UserPrincipalName } -ModuleNames ActiveDirectory
# Imports ActiveDirectory module for Get-ADUser
```

- [`-ModulePaths` Parameter](./docs/en-US/Invoke-Parallel.md#-modulepaths): Import custom modules from specified directory paths into parallel runspaces.

```powershell
$moduleDir = Join-Path $PSScriptRoot "CustomModule"
0..10 | Invoke-Parallel { Get-CustomCmdlet } -ModulePaths $moduleDir
# Imports custom module for Get-CustomCmdlet
```

These parameters are a quality-of-life enhancement, especially `-Functions`, which incorporates locally defined functions to the runspaces’ [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate)—a feature absent in `ForEach-Object -Parallel` and a far better option than passing function definitions into the parallel scope. The new `-ModuleNames` and `-ModulePaths` parameters simplify module integration by automatically loading system-installed and custom modules, respectively, eliminating the need for manual `Import-Module` calls within the script block.

# Documentation

Expand Down
138 changes: 116 additions & 22 deletions docs/en-US/Invoke-Parallel.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,24 @@ Executes parallel processing of pipeline input objects using multithreading.

```powershell
Invoke-Parallel
-InputObject <Object>
[-ScriptBlock] <ScriptBlock>
[-InputObject <Object>]
[-ThrottleLimit <Int32>]
[-TimeoutSeconds <Int32>]
[-Variables <Hashtable>]
[-Functions <String[]>]
[-ModuleNames <String[]>]
[-ModulePaths <String[]>]
[-UseNewRunspace]
[-TimeoutSeconds <Int32>]
[<CommonParameters>]
```

## DESCRIPTION

The `Invoke-Parallel` cmdlet enables parallel processing of input objects in PowerShell, including __Windows PowerShell 5.1__, offering functionality similar to `ForEach-Object -Parallel` introduced in PowerShell 7.0. It processes pipeline input across multiple threads, improving performance for tasks that benefit from parallel execution.
The `Invoke-Parallel` cmdlet enables parallel processing of input objects in PowerShell, including
__Windows PowerShell 5.1__ and PowerShell 7+, offering functionality similar to `ForEach-Object -Parallel` introduced in
PowerShell 7.0. It processes pipeline input across multiple threads, improving performance for tasks that benefit from
parallel execution.

## EXAMPLES

Expand All @@ -42,7 +47,10 @@ $message = 'Hello world from '
}
```

This example demonstrates parallel execution of a script block with a 3-second delay, appending a unique runspace ID to a message. The [`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes?view=powershell-7.4#the-using-scope-modifier) is used to pass the local variable `$message` into the parallel scope, a supported method for accessing external variables in `Invoke-Parallel`.
This example demonstrates parallel execution of a script block with a 3-second delay, appending a unique runspace ID to
a message. The [`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes#the-using-scope-modifier)
is used to pass the local variable `$message` into the parallel scope, a supported method for accessing external
variables in `Invoke-Parallel`.

### Example 2: Demonstrates `-Variables` Parameter

Expand All @@ -55,7 +63,9 @@ $message = 'Hello world from '
} -Variables @{ message = $message }
```

This example demonstrates the [`-Variables` parameter](#-variables), which passes the local variable `$message` into the parallel scope using a hashtable. The key `message` in the hashtable defines the variable name available within the script block, serving as an alternative to the `$using:` scope modifier.
This example demonstrates the [`-Variables` parameter](#-variables), which passes the local variable `$message` into
the parallel scope using a hashtable. The key `message` in the hashtable defines the variable name available within the
script block, serving as an alternative to the `$using:` scope modifier.

### Example 3: Adding to a thread-safe collection with `$using:`

Expand All @@ -65,7 +75,8 @@ Get-Process | Invoke-Parallel { ($using:dict)[$_.Id] = $_ }
$dict[$PID]
```

This example uses a thread-safe dictionary to store process objects by ID, leveraging the `$using:` modifier for variable access.
This example uses a thread-safe dictionary to store process objects by ID, leveraging the `$using:` modifier for
variable access.

### Example 4: Adding to a thread-safe collection with `-Variables`

Expand All @@ -85,15 +96,17 @@ function Greet { param($s) "$s hey there!" }
0..10 | Invoke-Parallel { Greet $_ } -Functions Greet
```

This example imports a local function `Greet` into the parallel scope using [`-Functions` parameter](#-functions), allowing its use within the script block.
This example imports a local function `Greet` into the parallel scope using [`-Functions` parameter](#-functions),
allowing its use within the script block.

### Example 6: Setting a timeout with `-TimeoutSeconds`

```powershell
0..10 | Invoke-Parallel { Start-Sleep 1 } -TimeoutSeconds 3
```

This example limits execution to 3 seconds, stopping all running script blocks and ignoring unprocessed input once the timeout is reached.
This example limits execution to 3 seconds, stopping all running script blocks and ignoring unprocessed input once the
timeout is reached.

### Example 7: Creating new runspaces with `-UseNewRunspace`

Expand All @@ -117,17 +130,45 @@ This example limits execution to 3 seconds, stopping all running script blocks a
# 9af7c222-061d-4c89-b073-375ee925e538
```

This example contrasts default runspace reuse with the `-UseNewRunspace` switch, showing unique runspace IDs for each invocation in the latter case.
This example contrasts default runspace reuse with the `-UseNewRunspace` switch, showing unique runspace IDs for each
invocation in the latter case.

### Example 8: Using the `-ModuleNames` parameter

```powershell
Import-Csv users.csv | Invoke-Parallel { Get-ADUser $_.UserPrincipalName } -ModuleNames ActiveDirectory
```

This example imports the `ActiveDirectory` module into the parallel scope using `-ModuleNames`, enabling the
`Get-ADUser` cmdlet within the script block.

### Example 9: Using the `-ModulePaths` parameter

```powershell
$moduleDir = Join-Path $PSScriptRoot "CustomModule"
0..10 | Invoke-Parallel { Get-CustomCmdlet } -ModulePaths $moduleDir
```

This example imports a custom module from the specified directory using `-ModulePaths`, allowing the `Get-CustomCmdlet`
function to be used in the parallel script block.

> [!NOTE]
>
> The path must point to a directory containing a valid PowerShell module.

## PARAMETERS

### -Functions

Specifies an array of function names from the local session to include in the runspaces' [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate). This enables their use within the parallel script block.
Specifies an array of function names from the local session to include in the runspaces’
[Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate).
This enables their use within the parallel script block.

> [!TIP]
>
> This parameter is the recommended way to make local functions available in the parallel scope. Alternatively, you can retrieve the function definition as a string (e.g., `$def = ${function:Greet}.ToString()`) and use `$using:` to pass it into the script block, defining it there (e.g., `${function:Greet} = $using:def`).
> This parameter is the recommended way to make local functions available in the parallel scope.
Alternatively, you can retrieve the function definition as a string (e.g., `$def = ${function:Greet}.ToString()`) and
use `$using:` to pass it into the script block, defining it there (e.g., `${function:Greet} = $using:def`).

```yaml
Type: String[]
Expand Down Expand Up @@ -175,11 +216,8 @@ Accept wildcard characters: False

### -ThrottleLimit

Sets the maximum number of script blocks executed in parallel across multiple threads. Additional input objects wait until the number of running script blocks falls below this limit.

> [!NOTE]
>
> The default value is `5`.
Sets the maximum number of script blocks executed in parallel across multiple threads. Additional input objects wait
until the number of running script blocks falls below this limit. The default value is `5`.

```yaml
Type: Int32
Expand All @@ -195,7 +233,8 @@ Accept wildcard characters: False

### -TimeoutSeconds

Specifies the maximum time (in seconds) to process all input objects. When the timeout is reached, running script blocks are terminated, and remaining input is discarded.
Specifies the maximum time (in seconds) to process all input objects. When the timeout is reached, running script blocks
are terminated, and remaining input is discarded.

> [!NOTE]
>
Expand Down Expand Up @@ -231,11 +270,13 @@ Accept wildcard characters: False

### -Variables

Provides a hashtable of variables to make available in the parallel scope. Keys define the variable names within the script block.
Provides a hashtable of variables to make available in the parallel scope. Keys define the variable names within the
script block.

> [!TIP]
>
> Use this parameter as an alternative to the [`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes?view=powershell-7.4#scope-modifiers).
> Use this parameter as an alternative to the
[`$using:` scope modifier](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_scopes#scope-modifiers).

```yaml
Type: Hashtable
Expand All @@ -249,6 +290,54 @@ Accept pipeline input: False
Accept wildcard characters: False
```

### -ModuleNames

Specifies an array of module names to import into the runspaces’
[Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate).
This allows the script block to use cmdlets and functions from the specified modules.

> [!TIP]
>
> Use this parameter to ensure required modules are available in the parallel scope. Module names must be discoverable
via the [`$env:PSModulePath`](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_psmodulepath)
environment variable, which lists installed module locations.

```yaml
Type: String[]
Parameter Sets: (All)
Aliases: mn

Required: False
Position: Named
Default value: None
Accept pipeline input: False
Accept wildcard characters: False
```

### -ModulePaths

Specifies an array of file paths to directories containing PowerShell modules (e.g., `.psm1` or `.psd1` files) to import
into the runspaces’ [Initial Session State](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.initialsessionstate).
This enables the script block to use cmdlets and functions from custom or local modules.

> [!NOTE]
>
> Paths must be absolute or relative to the current working directory and must point to valid directories containing
PowerShell modules. If an invalid path (e.g., a file or non-existent directory) is provided, a terminating error is
thrown.

```yaml
Type: String[]
Parameter Sets: (All)
Aliases: mp

Required: False
Position: Named
Default value: None
Accept pipeline input: False
Accept wildcard characters: False
```

### CommonParameters

This cmdlet supports the common parameters. For more information, see [about_CommonParameters](http://go.microsoft.com/fwlink/?LinkID=113216).
Expand All @@ -268,14 +357,19 @@ Returns objects produced by the script block.
## NOTES

- `Invoke-Parallel` uses multithreading, which may introduce overhead. For small datasets, sequential processing might be faster.
- Ensure variables or collections passed to the parallel scope are thread-safe (e.g., `[System.Collections.Concurrent.ConcurrentDictionary]`), as shown in Examples 3 and 4.
- By default, runspaces are reused from a pool to optimize resource usage. Using `-UseNewRunspace` increases memory and startup time but ensures isolation.
- Ensure variables or collections passed to the parallel scope are thread-safe (e.g., use
[`ConcurrentDictionary<TKey, TValue>`](https://learn.microsoft.com/en-us/dotnet/api/system.collections.concurrent.concurrentdictionary-2) or similar), as shown in
[__Example #3__](#example-3-adding-to-a-thread-safe-collection-with-using) and
[__Example #4__](#example-4-adding-to-a-thread-safe-collection-with--variables),
to avoid race conditions.
- By default, runspaces are reused from a pool to optimize resource usage. Using [`-UseNewRunspace`](#-usenewrunspace) increases memory and
startup time but ensures isolation.

## RELATED LINKS

[__ForEach-Object -Parallel__](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/foreach-object)

[__Runspaces Overview__](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.runspace?view=powershellsdk-7.4.0)
[__Runspaces Overview__](https://learn.microsoft.com/en-us/dotnet/api/system.management.automation.runspaces.runspace)

[__Managed threading best practices__](https://learn.microsoft.com/en-us/dotnet/standard/threading/managed-threading-best-practices)

Expand Down
2 changes: 1 addition & 1 deletion module/PSParallelPipeline.psd1
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
RootModule = 'bin/netstandard2.0/PSParallelPipeline.dll'

# Version number of this module.
ModuleVersion = '1.2.3'
ModuleVersion = '1.2.4'

# Supported PSEditions
# CompatiblePSEditions = @()
Expand Down
19 changes: 16 additions & 3 deletions src/PSParallelPipeline/Commands/InvokeParallelCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,17 @@ public sealed class InvokeParallelCommand : PSCmdlet, IDisposable
[Alias("funcs")]
public string[]? Functions { get; set; }

[Parameter]
[ValidateNotNullOrEmpty]
[ArgumentCompleter(typeof(ModuleCompleter))]
[Alias("mn")]
public string[]? ModuleNames { get; set; }

[Parameter]
[ValidateNotNullOrEmpty]
[Alias("mp")]
public string[]? ModulePaths { get; set; }

[Parameter]
[Alias("unr")]
public SwitchParameter UseNewRunspace { get; set; }
Expand All @@ -57,7 +68,9 @@ protected override void BeginProcessing()
InitialSessionState iss = InitialSessionState
.CreateDefault2()
.AddFunctions(Functions, this)
.AddVariables(Variables, this);
.AddVariables(Variables, this)
.ImportModules(ModuleNames)
.ImportModulesFromPath(ModulePaths, this);

PoolSettings poolSettings = new(
ThrottleLimit, UseNewRunspace, iss);
Expand Down Expand Up @@ -90,7 +103,7 @@ protected override void ProcessRecord()
}
catch (OperationCanceledException exception)
{
_worker.WaitForCompletion();
CancelAndWait();
exception.WriteTimeoutError(this);
}
}
Expand All @@ -116,7 +129,7 @@ protected override void EndProcessing()
}
catch (OperationCanceledException exception)
{
_worker.WaitForCompletion();
CancelAndWait();
exception.WriteTimeoutError(this);
}
}
Expand Down
Loading