After adding FLS to a Permission Set during field creation, all Permission Sets are pulled

[FabienTaillon]

Describe the bug
When "Field-Level Security for Permission Sets during Field Creation" is activated (Setup/User Management Settings/Field-Level Security for Permission Sets during Field Creation), adding FLS on one Permission Set ends up in all Permission Sets being pulled.

To Reproduce

• use this repo: https://github.com/FabienTaillon/repro-pull-all-permsets
• create a scratch org
• push everything
• create a new text field on account
• on FLS page, select read on Permission Set 1
• pull everything

Even though only 1 Permission Set is modified, all of them are pulled, even some standard ones.
This also happens when using the "Set Field-Level Security" button on an existing field.
This is painful as we don't want to retrieve unmodified Permission Sets that may be in the org but not part of our project.

Expected behavior
Only the modified Permission Set should be pulled.

Screenshots
All Permission Sets:

Only one which has object permissions is selected:

Everything is pulled:

[FabienTaillon]

Case #46558125 created.