fix(ug_util): prioritize user-data.users over the default user config

[mostafaCamel]

Proposed Commit Message

fix(ug_util): prioritize user-data.users over the default user config

Fixes GH-6703

Additional Context

Test Steps

Unit tests

tox -e py3 -- tests/unittests/distros/test_user_data_normalize.py::TestUGNormalize

Integration tests

• Create a new ~/.config/pycloudlib.toml file and put [lxd] in this file

CLOUD_INIT_OS_IMAGE='noble' CLOUD_INIT_PLATFORM=lxd_container tox -e integration-tests -- tests/integration_tests/modules/test_users_groups.py tests/integration_tests/modules/test_set_password.py

Merge type

• Squash merge using "Proposed Commit Message"
• Rebase and merge unique commits. Requires commit messages per-commit each referencing the pull request number (#<PR_NUM>)

[mostafaCamel]

The first commit I pushed is failing as expected as I only added the unit test

The expected failure

TOTAL                                                          33026   6291  12224   1597    79%
=========================== short test summary info ============================
FAILED tests/unittests/distros/test_user_data_normalize.py::TestUGNormalize::test_users_dict_override_default_attribute - assert True is False
= 1 failed, 5641 passed, 5 skipped, 13 xfailed, 2 xpassed, 84 warnings in 118.45s (0:01:58) =
py3: exit 1 (129.32 seconds) /home/runner/work/cloud-init/cloud-init> .tox/py3/bin/python -m pytest -vv --cov=cloudinit --cov-branch --color=yes pid=2384
  py3: FAIL code 1 (135.99=setup[6.67]+cmd[129.32] seconds)
  evaluation failed :( (136.04 seconds)
Error: Process completed with exit code 1.

[mostafaCamel]

#6703

[mostafaCamel]

After pushing the code fix, all the unit tests now pass

[holmanb]

Please add an integration test.

[mostafaCamel]

Note to myself about the integration tests (will work on it the weekend):

• Declare 2 contansts the python integration: a plain-text password and its hash
  • I first need to run openssl passwd -6 "yourplainpassword" on your terminal to generate the hash and pass it as a constant in the python script
• 2 ubuntu lxc containers A (with cloud-config.users ["default", {"name": "ubuntu", "hashed_passwd": "superpassword"}]) and B (with cloud-config.users ["default", {"name": "ubuntu", "hashed_passwd": "superpassword", "lock_passwd": False}])
• confirm that the 2 contianers are up and running ( I don't know if the test helpers have such a function... can skip it given the following steps)
• grep -E "^ubuntu:!<hashedpassword>: /etc/shadow for container A (and that there is only one occurence) ... notice the exclamation amrk as it indicates the password is locked
• grep -E "^ubuntu:!<hashedpassword>: /etc/shadow for container B (and that there is only one occurence)
  • No need to confirm there is there is only one occurrence in grep as ^ubuntu: is enough as usernames are unique
• Console access to container A should fail
• Console access to container B should succeed

[mostafaCamel]

Pushed a new commit with the integration tests:

• added a new test_hashed_password_without_lock_passwd_override_is_locked in tests/integration_tests/modules/test_set_password.py to confirm the locked_password:True nehabior (excalamtion mark in /etc/shadow)
• added a new test_default_user_settings_override method (NOTE: marked it as a ci test) in tests/integration_tests/modules/test_users_groups.py to confirm the override of defaultsettings actually works (shell and lock_passwd)
• added a new test_default_user_settings (NOTE: did not mark it as a ci test) in tests/integration_tests/modules/test_users_groups.py as "a negative control" to confirm the default settings of the ubuntu user. I don't think it is worth of running in CI hence I did not put the ci pytest mark
• moved the function fetch_and_parse_etc_shadow to tests/integration_tests/util.py as it is now used by multiple test files

The integration test succeeded locally CLOUD_INIT_OS_IMAGE='noble' CLOUD_INIT_PLATFORM=lxd_container tox -e integration-tests -- tests/integration_tests/modules/test_users_groups.py::test_default_user_settings_override
Howeve, the same test fails in CI .... the assertion fails as there is a dicrepancy between the expected (the override /bin/sh) and the actual which seems to still be the default)...
it is as if my code fix is not being ingested in the deb package used for the CI

I ran again the integration test locally qith questing to amtch the CI and the local test still succeeds CLOUD_INIT_OS_IMAGE='questing' tox -e integration-tests -- tests/integration_tests/modules/test_users_groups.py::test_default_user_settings_override

[mostafaCamel]

I confirmed that the debian package generated in the CI task (link in the comment above) contains my change

• went into this step Archive debs as artifacts and I download the zip from the outputted downlaod url
• Unzipped the file which created the .deb file in cloud-init-questing-deb/cloud-init-base_26.1-1~bddeb~25.10.1_all.deb on my local
• Ran dpkg-deb -x cloud-init-questing-deb/cloud-init-base_26.1-1~bddeb~25.10.1_all.deb cloud-init-questing-deb to extract the files
• Confirmed that my code fix is in the file cloud-init-questing-deb/usr/lib/python3/dist-packages/cloudinit/distros/ug_util.py

So normally this should lead to the code being ingested in the CI tests given that the Run integration tests step is running CLOUD_INIT_CLOUD_INIT_SOURCE="$(ls /home/runner/work/_temp/cloud-init-base*.deb)" CLOUD_INIT_OS_IMAGE=questing CLOUD_INIT_LOCAL_LOG_PATH=./cloudinit_logs tox -e integration-tests-ci -- --color=yes tests/integration_tests/

[mostafaCamel]

Pushed a commit with some additional prints. Confirmed that I am running the debain package

/usr/bin/cloud-init 26.1-1~bddeb~25.10.1

Package: cloud-init-base
Status: install ok installed
Priority: optional
Section: admin
Installed-Size: 2994
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: all
Source: cloud-init
Version: 26.1-1~bddeb~25.10.1
Replaces: cloud-init (<< 25.1~), cloud-init-base
Depends: cloud-guest-utils | cloud-utils, dhcpcd-base, iproute2, netcat-openbsd, netplan.io, procps, python3, python3-debconf, python3-requests
Recommends: eatmydata, gdisk, gnupg, python3-apt, software-properties-common
Suggests: openssh-server, ssh-import-id
Breaks: cloud-init (<< 25.1~), cloud-init-base
Conffiles:
 /etc/cloud/cloud.cfg eeaeae68aa0cf2a59134fe341491e769
 /etc/cloud/cloud.cfg.d/05_logging.cfg a7649f3a7332b9e9e0511dfa195d0212
 /etc/cloud/cloud.cfg.d/README f5175bd4df5c37ce781f93d20f59561a
 /etc/cloud/templates/chef_client.rb.tmpl a0844ddc9a42776d41a03d62d10ea139
 /etc/cloud/templates/chrony.conf.ubuntu.tmpl 981b6e81533cbfe2011a72cf83f0f858
 /etc/cloud/templates/hosts.debian.tmpl 941773df489d046d87ae491c3c95d8ec
 /etc/cloud/templates/ntp.conf.ubuntu.tmpl c4917fcac8096871f30136d046d2dade
 /etc/cloud/templates/sources.list.ubuntu.deb822.tmpl 4531340ef6e63ffeda6c35e8cf219bfd
 /etc/cloud/templates/timesyncd.conf.tmpl 9c6b3af8058efc219987863f45bb9198
 /etc/profile.d/Z99-cloud-locale-test.sh 2a8d91f29f0510d142fd03ede447d6b7
 /etc/profile.d/Z99-cloudinit-warnings.sh f45ec83cecfebd00c6071e56895c20bc
 /etc/rsyslog.d/21-cloudinit.conf d4cf2e5d3cb9914cf7e6cdc08d298339
 /etc/logrotate.d/cloud-init-base 1136c49642289d614837572cac18d9f6 obsolete
Description: initialization and customization tool for cloud instances
 Cloud-init with minimal dependencies, refer to cloud-init for more
 information.
Homepage: https://cloud-init.io/
/usr/bin/cloud-init