[pull] devel from ansible:devel#572
Open
pull[bot] wants to merge 2062 commits intophilipsd6:develfrom
Open
Conversation
github-actions
Bot
added
component:api
component:ui
community
component:awx_collection
labels
github-actions
Bot
added
the
dependencies
* migrate settings using the existing authenticator framework * add method to get settings value to gateway client * add transformer functions for settings * Switched back to PUT for settings updates * Started wiring in testing changes * Added settings_* aggregation results. Added skip-github option. Added tests. Assisted-by: Cursor * Added --skip-all-authenticators command line argument. Added GoogleOAuth testing. Added tests for skipping all authenticators. Assisted-by: Cursor * wip: migrate other missing settings * update login_redirect_override in google_oauth2 * impement login redirect for azuread * implement login redirect for github * implement login redirect for saml * set LOGIN_REDIRECT_OVERRIDE even if no authenticator matched * extract logic for login redirect override to base class * use urlparse to compare valid redirect urls * Preserve the original query parameters * Fix flake8 issues * Preserve the query parameter in sso_login_url Gateway sets the sso_login_url to /api/gateway/social/login/aap-saml-keycloak/?idp=IdP The idp needs to be preserved when creating the redirect * Update awx/main/utils/gateway_client.py Co-authored-by: Chris Meyers <chrismeyersfsu@users.noreply.github.com> * Update awx/main/management/commands/import_auth_config_to_gateway.py Co-authored-by: Chris Meyers <chrismeyersfsu@users.noreply.github.com> * list of settings updated * Update awx/main/utils/gateway_client.py Co-authored-by: Chris Meyers <chrismeyersfsu@users.noreply.github.com> * Update awx/sso/utils/base_migrator.py Co-authored-by: Chris Meyers <chrismeyersfsu@users.noreply.github.com> * fix tests --------- Co-authored-by: Andrew Potozniak <potozniak@redhat.com> Co-authored-by: Madhu Kanoor <mkanoor@redhat.com> Co-authored-by: Chris Meyers <chrismeyersfsu@users.noreply.github.com>
* fix: inconsistent return values in github migrator * feat: check setting value before updating and report correct status * fix linter issues
* feat: exit code 1 if any migration fails * update tests * remove unused variables
* Working branch for testing DAB RBAC changes * AAP-48392 Handle DAB RBAC either before or after new type model (for merge) (#16045) * Handle DAB RBAC either before or after new type model * Translate CT to DAB CT * Fix for rearrangement of post_migration methods * Directly include RBAC service URLs * Add a run before remote permission additions * Sync old rbac to remote rbac (#7025) Signed-off-by: Seth Foster <fosterbseth@gmail.com> * Set DAB requirement back to devel --------- Signed-off-by: Seth Foster <fosterbseth@gmail.com> Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
…#7037) * Added better error handling and messaging when the service token authentication is broken. Allowed for GATEWAY_BASE_URL to override the service token's base url if it is set in the environment variables. Co-Authored-By: Cursor (claude-4-sonnet) * Removed GATEWAY_BASE_URL override for service token auth.
…#7038) * Slightly better error handling for non 200 status codes from Gateway. * Apply suggestion from @chrismeyersfsu Co-authored-by: Chris Meyers <chrismeyersfsu@users.noreply.github.com> --------- Co-authored-by: Chris Meyers <chrismeyersfsu@users.noreply.github.com>
* fix: tacacs+ -> TACACSPLUS Gateway doesn't allow `+` to be used in slug. AAP-50774 * Fixed assertion --------- Co-authored-by: Andrew Potozniak <potozniak@redhat.com>
* handle login redirect for the oidc migrator * handle updating login override redirect centrally in the settings migrator * update unit tests
Syncing from new rbac to old rbac locally calls the disable_rbac_sync() context manager. If rbac sync is disabled, we do not need to remote sync, as we can assume the remote syncing already occurred in the viewset. Signed-off-by: Seth Foster <fosterbseth@gmail.com>
* Marked APIs legacy OAuth applications as deprecated * Readded deprecation * Fixed linter * Added more deprecated mark to Oauth2 Api apps * Fixed deprecation errors * Fix tests
* disable authenticators that require updating the redirect URL and add groups claim to AzureAD migrator * update tests
…entication export (#7047) This commit completely refactors how SSO organization and team mappings are processed and exported for Gateway authentication, moving from a group-based approach to a more flexible attribute-based system. Key Changes: - Introduced new process_sso_user_list() function for centralized user processing - Enhanced boolean handling to support both native booleans and string representations - Added email detection and regex pattern support for flexible user matching - Refactored trigger generation from groups-based to attributes-based system Gateway Mapping Enhancements (awx/main/utils/gateway_mapping.py): - Added email regex detection for automatic email vs username classification - Added pattern_to_slash_format() for regex pattern conversion - Enhanced process_sso_user_list() with support for: - Boolean values: True/False and ["true"]/["false"] - String usernames and email addresses with automatic detection - Regex patterns with both username and email matching - Custom email_attr and username_attr parameters - Refactored team_map_to_gateway_format() to use new processing system - Refactored org_map_to_gateway_format() to use new processing system - Changed trigger structure from {"groups": {"has_or": [...]}} to attribute-based triggers - Improved naming convention to include trigger type in mapping names Comprehensive Test Coverage (awx/main/tests/unit/utils/test_auth_migration.py): - Added complete TestProcessSSOUserList class with 8 comprehensive test methods - Enhanced TestOrgMapToGatewayFormat with string boolean and new functionality tests - Enhanced TestTeamMapToGatewayFormat with string boolean and new functionality tests - Added tests for email detection, regex patterns, and custom attributes - Verified backward compatibility and integration functionality - All existing tests updated to work with new attribute-based trigger system Breaking Changes: - Trigger structure changed from group-based to attribute-based - Mapping names now include trigger descriptions for better clarity - Function signatures updated to include email_attr and username_attr parameters Co-Authored with Claude-4 via Cursor Co-authored-by: Peter Braun <pbraun@redhat.com>
https://issues.redhat.com/browse/AAP-51127 Co-authored-by: Peter Braun <pbraun@redhat.com>
* fix: enable workload identity credentials for project updates * Add explanatory comment for credential context handling * Revert build_passwords
…ancestor_artifacts` (#16381) * Add new test for artfact precedence upstream node vs outer workflow * Fix bugs, upstream artifacts come first for precedence * Track nested artifacts path through ancestor_artifacts on root nodes * Fix case where first root node did not get the vars * touchup comment * Prevent conflict with sliced jobs hack
* Reorder URLs so that Django debug toolbar can work * Move comment with URL move
Adds support for testing external credentials that use OIDC workload identity tokens. When FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED is enabled, the /test endpoints return JWT payload details alongside test results. - Add OIDC credential test endpoints with job template selection - Return JWT payload and secret value in test response - Maintain backward compatibility (detail field for errors) - Add comprehensive unit and functional tests - Refactor shared error handling logic Co-authored-by: Daniel Finca <dfinca@redhat.com> Co-authored-by: melissalkelly <melissalkelly1@gmail.com>
* Improved performance of the config endpoint by reducing database queries in GET /api/controller/v2/config/
The cloud credential used by inventory updates was not going through the OIDC workload identity token flow because it lives outside the normal _credentials list. This overrides populate_workload_identity_tokens in RunInventoryUpdate to include the cloud credential as an additional_credentials argument to the base implementation, and patches get_cloud_credential on the instance so the injector picks up the credential with OIDC context intact. Co-authored-by: Alan Rominger <arominge@redhat.com> Co-authored-by: Dave Mulford <dmulford@redhat.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* support bitbucket_dc webhooks * add test * update docs
retrieve_workload_identity_jwt_with_claims is now in a separate utility file, not in jobs.py Signed-off-by: Seth Foster <fosterbseth@gmail.com>
…th exponential backoff. (#16415) controller collection should retry transient HTTP errors with exponential backoff
Fix rrule fast-forward producing wrong occurrences across DST boundaries The UTC round-trip in _fast_forward_rrule shifts the dtstart's local hour when the original and fast-forwarded times are in different DST periods. Since dateutil generates HOURLY occurrences by stepping in local time, the shifted hour changes the set of reachable hours. With BYHOUR constraints this causes a ValueError crash; without BYHOUR, occurrences are silently shifted by 1 hour. Fix by performing all arithmetic in the dtstart's original timezone. Python aware-datetime subtraction already computes absolute elapsed time regardless of timezone, so the UTC conversion was unnecessary for correctness and actively harmful during fall-back ambiguity. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Correctly restrict push actions to ownership repos * Use standard action to see if push actions should run * Run spec job for 2.6 and higher * Be even more restrictve, do not push if on a fork
* Fixed black reformating * Make test simulate 500k hosts in real world scenario
#16425) fix: do not include secret values in the credentials test endpoint and add a guard to make sure credentials are testable
…has unprompted labels (#16426) * AAP-41742: Fix workflow node update failing when JT has unprompted labels PATCH extra_data on a workflow node fails with {"labels":["Field is not configured to prompt on launch."]} when the node has labels associated but the JT has ask_labels_on_launch=False. The serializer was passing all persisted M2M state from prompts_dict() to _accept_or_ignore_job_kwargs() on every PATCH, re-validating unchanged fields. Fix scopes validation to only the fields in the request; full re-validation still occurs when unified_job_template is being changed. * Capture attrs keys before _build_mock_obj mutates them _build_mock_obj() pops pseudo-fields (limit, scm_branch, job_tags, etc.) from attrs. Computing requested_prompt_fields after the pop would miss those fields and skip their ask_on_launch validation. * Include survey_passwords when validating extra_vars prompts prompts_dict() emits survey_passwords alongside extra_vars. _accept_or_ignore_job_kwargs uses it to decrypt encrypted survey values before validation. Without it, encrypted password blobs are validated as-is against the survey spec. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…normalized FK (#16332) * perf: stop eagerly updating Host.last_job_host_summary on every job completion The playbook_on_stats wrapup path bulk-updates last_job_host_summary_id on every host touched by a job. In the Q4CY25 scale lab this query had a median execution time of 75 seconds due to index churn on main_host. Replace all reads of the denormalized FK with a new classmethod JobHostSummary.latest_for_host(host_id) that queries for the most recent summary on demand. This eliminates the write-side bulk_update of last_job_host_summary_id entirely. Changes: - Add JobHostSummary.latest_for_host() classmethod - Serializer: use latest_for_host() instead of obj.last_job_host_summary - Dashboard view: use subquery instead of FK traversal for failed hosts - Inventory.update_computed_fields: use subquery for failed host count - events.py: remove last_job_host_summary_id from bulk_update - signals.py: simplify _update_host_last_jhs to only update last_job - access.py/managers.py: remove select_related/defer through the FK The FK field on Host is left in place for now (removal requires a migration) but is no longer written to. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix .pk AttributeError, add job_template annotations, annotate host sublists - Add 'pk' to AnnotatedSummary dynamic type (fixes AttributeError in get_related) - Add job_template_id and job_template_name to subquery annotations so list views include these fields in summary_fields.last_job (matching detail views) - Traverse job__ FK from JobHostSummary instead of using separate UnifiedJob subquery with OuterRef on another annotation (cleaner SQL, avoids alias issue) - Annotate all host sublist views (InventoryHostsList, GroupHostsList, GroupAllHostsList, InventorySourceHostsList) to prevent N+1 queries Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Update test_events to use JobHostSummary.latest_for_host instead of stale FKs Tests were asserting host.last_job_id and host.last_job_host_summary_id which are no longer updated. Use JobHostSummary.latest_for_host() to derive the same data, matching the new read-time derivation approach. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove stale failures_url from deprecated DashboardView The failures_url linked to ?last_job_host_summary__failed=True which filters on the now-stale FK. The dashboard count itself was already fixed to use a subquery annotation. Since DashboardView is deprecated and has_active_failures is a SerializerMethodField (not filterable), remove the failures_url entirely rather than creating a custom filter. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply black formatting to changed files Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Refactor: replace 10 subquery annotations with bulk prefetch Instead of annotating every host queryset with 10 correlated subqueries (summary + job + job_template fields), annotate only _latest_summary_id and bulk-fetch the full JobHostSummary objects after pagination via select_related('job', 'job__job_template'). This reduces the SQL from 10 correlated subqueries to 1 subquery + 1 IN query, addressing review feedback about annotation overhead on host list views. - _annotate_host_latest_summary: only annotates _latest_summary_id - _prefetch_latest_summaries: bulk-fetches and attaches to host objects - HostSummaryPrefetchMixin: hooks into list() after pagination - Serializer uses real JobHostSummary objects (no more AnnotatedSummary) - to_representation always overwrites stale FK values Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Refactor: move latest summary to QuerySet._fetch_all + Host.latest_summary Per review feedback, replace the view-level HostSummaryPrefetchMixin with a custom QuerySet that bulk-attaches summaries at evaluation time (like prefetch_related), and a Host.latest_summary property as the single access point. - HostLatestSummaryQuerySet: overrides _fetch_all() to bulk-fetch JobHostSummary objects with select_related after queryset evaluation - HostManager now inherits from the custom queryset via from_queryset() - Host.latest_summary property: uses cache if available, falls back to individual query - Remove _annotate_host_latest_summary, _prefetch_latest_summaries, HostSummaryPrefetchMixin from views — no more list() override needed - Remove last_job/last_job_host_summary from SUMMARIZABLE_FK_FIELDS - Serializer uses obj.latest_summary and DEFAULT_SUMMARY_FIELDS loop Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix: scope annotation to views, restore license_error/canceled_on - Remove with_latest_summary_id() from HostManager.get_queryset() to avoid applying the correlated subquery to every Host query globally (count, exists, internal relations) - Apply with_latest_summary_id() in get_queryset() of the 6 host-serving views only - Restore license_error and canceled_on to last_job summary fields to avoid breaking API change Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Guard _fetch_all() to skip bulk-attach on non-annotated querysets Without this guard, _fetch_all() would set _latest_summary_cache=None on every host in non-annotated querysets (e.g. Host.objects.filter()), masking the per-object fallback query in Host.latest_summary. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove name from last_job_host_summary and canceled_on from last_job summary Per reviewer feedback: these fields were not in the original API contract via SUMMARIZABLE_FK_FIELDS and their addition would be an API change. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Add functional tests for HostLatestSummaryQuerySet and Host.latest_summary Tests cover: - with_latest_summary_id() annotation and most-recent selection - _fetch_all() bulk-attach behavior on annotated querysets - _fetch_all() skips non-annotated querysets (preserves fallback) - .count() and .exists() do NOT trigger _fetch_all - Host.latest_summary cache hits (zero queries) and fallback - Host.latest_job property - select_related on bulk-attached summaries (no N+1) - Chaining preserves annotation - Multiple jobs / partial host coverage Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Apply black formatting to test_host_queryset.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ben Thomasson <bthomass@redhat.com> * Fix flake8 F841: remove unused job1/job2 variables in tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Ben Thomasson <bthomass@redhat.com> * Add comment explaining why Prefetch was not used for host latest summary Django Prefetch cannot handle latest per group -- [:1] slicing fetches 1 record globally, not per host (Django ticket #26780). The custom _fetch_all override uses the same 2-query pattern as prefetch_related internally, customized for this use case. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix null handling to keep old behavior --------- Signed-off-by: Ben Thomasson <bthomass@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: AlanCoding <arominge@redhat.com>
…#16432) * [AAP-72722] Use url instead of jwt_aud for workload identity audience The OIDC credential plugin's jwt_aud field is being removed. Use the plugin's url field as the audience when requesting workload identity tokens, since the target service URL is the appropriate audience value. Assisted-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Defer ansible_facts in HostManager to avoid fetching large JSON column in host list queries (AAP-68023) The host list endpoint (GET /api/v2/hosts/) fetches the ansible_facts JSON column unnecessarily, contributing to the 7.8s median query time at scale. This column can be very large and is not used by the list serializer. Changes: - HostManager.get_queryset() now defers ansible_facts - finish_fact_cache call site uses .only(*HOST_FACTS_FIELDS) to eagerly load ansible_facts when actually needed, avoiding N+1 queries - Unit test mocks updated to support .only() queryset chaining - Points DAB dependency at the RBAC query optimization branch for combined testing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> ---------
* git worktree friendly precomit install * worktrees don't have a .git directory. Before, docker-compose would trigger pre-commit install and fail. * make docker-compose work in git worktree * AWX tries to discover the version via info stored in .git/ dir. setuptools-scm is capable of finding the .git/ dir, starting from a worktree, but is unable because only the worktree is mapped into the container, not the .git/ dir itself. Thus, we have to detect and pass the version into the container from outside. That is why this change landed in the Makefile.
Add a fallback that checks for `gateway_sessionid` when no cookie matches `session_cookie_name`, mirroring the existing fallback in `Connection.login()`. The finally block now cleans up whichever cookie name was actually used. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* add converted atf tests * fix bulk settings test
* Change fact processing loop to use file listing * Fix some test * Address coderabbit comments * Handle saving facts in batches to keep memory low * Improve log about mismatch in response to review comment
…_requires (#16444) AAP-74276: Replace setuptools with packaging in awxkit install_requires The Python 3.12 upgrade replaced distutils.version.LooseVersion with packaging.version.Version but did not update awxkit's install_requires. setuptools is no longer needed at runtime since pkg_resources was also replaced with importlib.metadata. This causes ModuleNotFoundError on standalone CLI installs where packaging is not present. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Ryan Williams <3375653+ryankwilliams@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )