Skip to content

Implementation of Readable iteration helpers that does not rely on Async Iteration#64429

Open
lukiano wants to merge 1 commit into
nodejs:mainfrom
lukiano:chore/faster-map-2
Open

Implementation of Readable iteration helpers that does not rely on Async Iteration#64429
lukiano wants to merge 1 commit into
nodejs:mainfrom
lukiano:chore/faster-map-2

Conversation

@lukiano

@lukiano lukiano commented Jul 11, 2026

Copy link
Copy Markdown

Hi, I'd like to offer an alternative implementation of Readable iteration helpers like find() that don't rely on for await (...) to consume items from the stream. This results in improved performance when the data is already available, such as when creating a Readable from an array, as shown in the attached screenshot (executed on a MacBook M2 Max).

The main con is increased duplication and code complexity. My understanding is that these helpers are still in the experimental phase, which I hope makes a change like this easier to accept.

There are many commits in the branch, but I'll squash them before merging.
Edit: I merged them to fix CI issues

The affected helpers are

  • map()
  • filter()
  • reduce()
  • find()
  • toArray()
  • some()
  • every()
  • drop()

Unaffected helpers:

  • flatmap()
  • take()

There's also a change to the from() method that increases the buffer watermark when the source is an array of data.
At first, I coded the changes manually, but eventually I had assistance from AI to keep backward compatibility. Still, there's a small break as can be seen in the updated test in test/parallel/test-stream-reduce.js.

The file benchmark/streams/operator-throughput.js allows interested parties to run the benchmarks on their computers or modify them to try other scenarios. I can remove it before merging.

stream-operator-throughput-summary

@nodejs-github-bot

Copy link
Copy Markdown
Collaborator

Review requested:

  • @nodejs/performance
  • @nodejs/streams

@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. stream Issues and PRs related to the stream subsystem. labels Jul 11, 2026
@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 88.68552% with 68 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.24%. Comparing base (e6a8d06) to head (fae33f0).
⚠️ Report is 321 commits behind head on main.

Files with missing lines Patch % Lines
lib/internal/streams/operators.js 88.53% 66 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #64429      +/-   ##
==========================================
- Coverage   92.01%   90.24%   -1.78%     
==========================================
  Files         379      741     +362     
  Lines      166972   241811   +74839     
  Branches    25554    45592   +20038     
==========================================
+ Hits       153639   218211   +64572     
- Misses      13041    15147    +2106     
- Partials      292     8453    +8161     
Files with missing lines Coverage Δ
lib/internal/streams/from.js 91.55% <100.00%> (+4.79%) ⬆️
lib/internal/streams/operators.js 90.49% <88.53%> (-5.66%) ⬇️

... and 546 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@lukiano lukiano force-pushed the chore/faster-map-2 branch from fae33f0 to 17e0105 Compare July 12, 2026 18:24
@lukiano

lukiano commented Jul 12, 2026

Copy link
Copy Markdown
Author

I'm seeing #64447, which may change the measured performance of the current implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-ci PRs that need a full CI run. stream Issues and PRs related to the stream subsystem.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants