From 7558d9485699e049dc723ab7176bf4a09348f1a2 Mon Sep 17 00:00:00 2001 From: Paul Van Eck Date: Tue, 17 Mar 2026 03:09:38 +0000 Subject: [PATCH] [Doc] Consolidate relevant wiki content Signed-off-by: Paul Van Eck --- README.md | 2 +- README.rst | 4 +- doc/README.md | 56 ++++++++- doc/dev/README.md | 12 +- doc/dev/common_issues.md | 176 ++++++++++++++++++++++++++ doc/dev/dataplane_generation.md | 180 +++++++++++++++++++++++++++ doc/dev/mgmt/README.md | 7 +- doc/dev/mgmt/generation.md | 144 +++++++++++++++++++-- doc/dev/mgmt/mgmt_release.md | 2 +- doc/dev/mgmt/tests.md | 7 -- doc/eng_sys_checks.md | 19 +++ doc/python_version_support_policy.md | 19 +++ 12 files changed, 597 insertions(+), 31 deletions(-) create mode 100644 doc/dev/common_issues.md create mode 100644 doc/dev/dataplane_generation.md create mode 100644 doc/python_version_support_policy.md diff --git a/README.md b/README.md index 015f0f32728a..cc99e9d09e83 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ You can find service libraries in the `/sdk` directory. ### Prerequisites -The client libraries are supported on Python 3.9 or later. For more details, please read our page on [Azure SDK for Python version support policy](https://github.com/Azure/azure-sdk-for-python/wiki/Azure-SDKs-Python-version-support-policy). +The client libraries are supported on Python 3.9 or later. For more details, please read our page on [Azure SDK for Python version support policy](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/python_version_support_policy.md). ## Packages available diff --git a/README.rst b/README.rst index 38562bc870e2..7e115bcc9f20 100644 --- a/README.rst +++ b/README.rst @@ -19,7 +19,7 @@ access Management (Virtual Machines, ...) or Runtime (ServiceBus using HTTP, Bat Complete feature list of this repo and where to find Python packages not in this repo can be found on our `Azure SDK for Python documentation `__. -The SDK supports Python 3.7 or later. For more details, please read our page on `Azure SDK for Python version support policy `__. +The SDK supports Python 3.7 or later. For more details, please read our page on `Azure SDK for Python version support policy `__. If you're currently using the ``azure`` package < 1.0 then please read important information in `this issue `__. @@ -60,7 +60,7 @@ For further samples please visit the `Azure Samples website `__. +For detailed documentation about our test framework, please visit this `Azure SDK test tutorial `__. Need Help? ========== diff --git a/doc/README.md b/doc/README.md index 53c9ae8a984c..4c738335d74d 100644 --- a/doc/README.md +++ b/doc/README.md @@ -1,9 +1,55 @@ # Documentation -This folder contains documentation for the Azure SDK for Python repository: +This folder contains documentation for the Azure SDK for Python repository. -The folder structure is the following -- [sphinx](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/sphinx) : contains the documentation source code for https://azure.github.io/azure-sdk-for-python/ -- [dev](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev) : contains advanced documentation for _developers_ of SDK (not _consumers_ of SDK) +## General -The file [eng_sys_checks](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/eng_sys_checks.md) is a read up as to what a standard `ci.yml` will actually execute. +- [Python Version Support Policy](python_version_support_policy.md) - Supported Python versions and end-of-support dates +- [Engineering System Checks](eng_sys_checks.md) - What a standard `ci.yml` will execute +- [Deprecation Process](deprecation_process.md) - How to deprecate a package +- [Repository Health Status](repo_health_status.md) - Library health status definitions +- [ESRP Release](esrp_release.md) - ESRP release process +- [Request Builders](request_builders.md) - Request builder pattern documentation +- [Send Request](send_request.md) - Send request pattern documentation +- [Analyze Check Versions](analyze_check_versions.md) - Check version analysis +- [Tool Usage Guide](tool_usage_guide.md) - How to use the `azpysdk` CLI tool + +## Developer Documentation + +See the [dev](dev/) folder for documentation aimed at developers of SDK libraries: + +- [Developer Setup](dev/dev_setup.md) - How to create a development environment +- [Release](dev/release.md) - How to release a package +- [Packaging](dev/packaging.md) - How to organize packaging information +- [Changelog Updates](dev/changelog_updates.md) - How to document package changes +- [Testing](dev/tests.md) - How to write unit and functional tests +- [Advanced Testing](dev/tests-advanced.md) - Advanced testing topics +- [Docstrings](dev/docstring.md) - How to document an SDK +- [Type Checking](dev/static_type_checking.md) - Type hints and type checking ([Cheatsheet](dev/static_type_checking_cheat_sheet.md)) +- [Pylint](dev/pylint_checking.md) - Pylint checking guidance +- [Sample Guide](dev/sample_guide.md) - How to write SDK samples +- [Performance Testing](dev/perfstress_tests.md) - How to write and run perf tests +- [Debug Guide](dev/debug_guide.md) - Debugging tips +- [Code Snippets](dev/code_snippets.md) - How to include code snippets in docs + +### Troubleshooting & Common Issues + +- [Common Issues and FAQ](dev/common_issues.md) - Common SDK issues and how to resolve them +- [Test Proxy Troubleshooting](dev/test_proxy_troubleshooting.md) - Test proxy issues + +### Code Generation + +- [Dataplane SDK Generation](dev/dataplane_generation.md) - Generate a dataplane SDK from a TypeSpec definition +- [Management Plane](dev/mgmt/) - Management plane SDK documentation + +### Additional Topics + +- [Customize Long Running Operations](dev/customize_long_running_operation.md) - LRO customization +- [CredScan Process](dev/credscan_process.md) - Credential scanning +- [Recording Migration Guide](dev/recording_migration_guide.md) - Migrating test recordings +- [Engineering Assumptions](dev/engineering_assumptions.md) - SDK engineering assumptions +- [Conda Builds](dev/conda-builds.md) / [Release](dev/conda-release.md) / [Dev Release](dev/conda-release-dev.md) - Conda packaging + +## Sphinx Reference Documentation + +The [sphinx](sphinx/) folder contains the documentation source for https://azure.github.io/azure-sdk-for-python/. diff --git a/doc/dev/README.md b/doc/dev/README.md index 6b48ce6b10fb..8d8ff92a6c08 100644 --- a/doc/dev/README.md +++ b/doc/dev/README.md @@ -11,7 +11,17 @@ Overview of the documents: - [Docstrings](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/docstring.md): How to document an SDK (API View) and our documentation at [MS Docs][ms_docs] and the [azure.github.io][azure_github_io] site. - [Type Hints and Type Checking](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/static_type_checking.md) / [Cheatsheet](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/static_type_checking_cheat_sheet.md): How to add type hints to your library code and run type checking. -The [mgmt](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt) folder contains information specific to management packages (i.e. packages prefixed by `azure-mgmt`) +The [mgmt](mgmt/) folder contains information specific to management packages (i.e. packages prefixed by `azure-mgmt`) + +### Troubleshooting + +- [Common Issues and FAQ](common_issues.md) - Common SDK issues and how to resolve them + +### Code Generation + +- [Dataplane SDK Generation](dataplane_generation.md) - Generate a dataplane SDK from a TypeSpec definition + +### Additional Topics [ms_docs]: https://docs.microsoft.com/python/api/overview/azure/appconfiguration-readme?view=azure-python diff --git a/doc/dev/common_issues.md b/doc/dev/common_issues.md new file mode 100644 index 000000000000..bf6885ab9187 --- /dev/null +++ b/doc/dev/common_issues.md @@ -0,0 +1,176 @@ +# Common Issues and FAQ for Python SDK + +This document clarifies some common misunderstandings about the Python SDK. + +## Table of Contents + +- [How to update an existing resource with create\_or\_update/begin\_create\_or\_update](#how-to-update-an-existing-resource-with-create_or_updatebegin_create_or_update) +- [Different ways to set body parameter of an operation](#different-ways-to-set-body-parameter-of-an-operation) +- [Debug guide](#debug-guide) +- [Build private package with PR](#build-private-package-with-pr) +- [Duplicated models](#duplicated-models) + +## How to update an existing resource with `create_or_update/begin_create_or_update` + +An operation named `create_or_update`/`begin_create_or_update` in the Python SDK usually corresponds to an HTTP `PUT` request. To update a field of an existing resource, you must do the following: + +```python + ... + # Get all fields of the existing resource + agent_pool = client.agent_pools.get( + resource_group_name="rg1", + resource_name="clustername1", + agent_pool_name="agentpool1", + ) + + agent_pool.max_count = 10 + agent_pool.min_count = 1 + # Change any field that you want + #... + + response = client.agent_pools.begin_create_or_update( + resource_group_name="rg1", + resource_name="clustername1", + agent_pool_name="agentpool1", + parameters=agent_pool, + ).result() +``` + +Users may have a question: **Why can't I just set the field directly instead of first retrieving the existing resource?** Here is the [guideline about PUT](https://www.geeksforgeeks.org/difference-between-put-and-patch-request/): + +![image](https://github.com/Azure/azure-sdk-for-python/assets/70930885/a19c6ce3-53bf-472c-a255-cb5e17c00e67) + +So the question becomes: **Why does `PUT` have such strange limitations when updating an existing resource?** To explain, let us consider these scenarios: + +- User A wants to delete a field X, so A sets the field X to `None` and thinks: **Now that I set X to `None`, the service should delete X.** +- User B wants to update a field Y, so B only sets the field Y and thinks: **Now that I didn't set X and it will be `None`, the service should keep X the same as before.** + +Why is there such ambiguity? **It is caused by the meaning of `None`** (in other languages, it may be named `null`/`undefined`/etc.). `None` has two different meanings in nature: **"delete it"** or **"keep it the same as before"**. To eliminate this ambiguity, `PUT` adopts the meaning of "delete it", so if users want to update an existing resource, they have to first get all the fields of the resource and then update the specific field. + +## Different ways to set body parameter of an operation + +Usually, an operation has a body parameter corresponding to the HTTP request body. For Python SDK users, there are two ways to set the body parameter: + +```python +from azure.identity import DefaultAzureCredential +from azure.mgmt.network import NetworkManagementClient + +client = NetworkManagementClient(credential=DefaultAzureCredential(), subscription_id="subid") + +# 1: With a JSON-like object +response = client.public_ip_addresses.begin_create_or_update( + resource_group_name="rg1", + public_ip_address_name="test-ip", + parameters={"location": "eastus", "properties": {"dnsSettings": {"domainNameLabel": "dnslbl"}}}, +).result() +print(response.as_dict()) + +# 2: With a native model +from azure.mgmt.network.models import PublicIPAddress, PublicIPAddressDnsSettings +parameters = PublicIPAddress(location="eastus", dns_settings=PublicIPAddressDnsSettings(domain_name_label="dnslbl")) +response = client.public_ip_addresses.begin_create_or_update( + resource_group_name="rg1", + public_ip_address_name="test-ip", + parameters=parameters, +).result() +print(response.as_dict()) +``` + +In a model, the signature name is snake_case like `domain_name_label`; in a JSON-like object, the name is camelCase like `domainNameLabel`. + +## Debug guide + +This guide helps Python SDK users understand how the SDK calls the REST API. + +1. Copy the following code into your `.py` file: + + ```python + import sys + import logging + + logging.basicConfig(level=logging.DEBUG, + format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', + stream=sys.stdout) + ``` + +2. Set `logging_enable=True` when calling an operation: + + ```python + client.operation_group.operation(..., logging_enable=True) + ``` + +3. Run your `.py` file and you will find the log output on screen. This makes it easy to see the details of how the SDK calls the REST API: + + ![image](https://github.com/Azure/azure-sdk-for-python/assets/70930885/d88fd936-e46f-45a2-a180-1eed4712c41a) + +## Build private package with PR + +This section shows how to build a private package (e.g., `azure-mgmt-devcenter`) locally from a [PR](https://github.com/Azure/azure-sdk-for-python/pull/35049): + +1. Install dependencies: + + ``` + PS D:\dev\azure-sdk-for-python> pip install wheel setuptools build + ``` + +2. Pull the target branch: + + ![image](https://github.com/Azure/azure-sdk-for-python/assets/70930885/a0762b7e-1b80-4a1c-958a-44c53b610928) + + GitHub shows the branch as `azure-sdk:t2-devcenter-2024-04-03-14415`, which contains the remote repo name `azure-sdk` and the branch name `t2-devcenter-2024-04-03-14415`: + + ``` + PS D:\dev\azure-sdk-for-python> git remote add azure-sdk https://github.com/azure-sdk/azure-sdk-for-python.git + PS D:\dev\azure-sdk-for-python> git fetch azure-sdk t2-devcenter-2024-04-03-14415 + PS D:\dev\azure-sdk-for-python> git checkout t2-devcenter-2024-04-03-14415 + ``` + +3. Step into the target package folder where `setup.py` or `pyproject.toml` is located, then run: + + ``` + PS D:\dev\azure-sdk-for-python\sdk\devcenter\azure-mgmt-devcenter> python -m build + ``` + + You will find the built package in the `dist` folder: + + image + +## Duplicated models + +TypeSpec permits defining models with the same name in different namespaces: + +```typespec +namespace Service; + +model Foo { + prop: string; +} + +namespace SubService { + model Foo { + name: string; + } +} +``` + +During Python SDK generation, all models surface in a single Python package namespace. Identical model names collide, so one must be renamed. + +To resolve this, apply the `@clientName` decorator in `client.tsp` to give a distinct Python name: + +```python +# In client.tsp add: +# @@clientName(Service.SubService.Foo, "SubFoo", "python"); + +# Generated Python code (models/_models.py): +class Foo: + ... # from Service.Foo + +class SubFoo: + ... # renamed from Service.SubService.Foo + +# Usage +from service.models import Foo, SubFoo + +foo = Foo(prop="hello") +subfoo = SubFoo(name="world") +``` diff --git a/doc/dev/dataplane_generation.md b/doc/dev/dataplane_generation.md new file mode 100644 index 000000000000..5f3514256edd --- /dev/null +++ b/doc/dev/dataplane_generation.md @@ -0,0 +1,180 @@ +# Dataplane SDK Generation Quick Start (TypeSpec) + +## Support + +For more questions and general overview of the process, please refer to + +## Prerequisites + +- Python 3.9 or later is required + - [download for windows](https://www.python.org/downloads/windows/) + - linux + - sudo apt install python3 + - sudo apt install python3-pip + - sudo apt install python3.{?}-venv explicitly if needed + +- [Node.js 20.x LTS](https://nodejs.org/en/download/) or later is required + +## Setup your repo + +- Fork and clone the [azure-sdk-for-python](https://github.com/Azure/azure-sdk-for-python) repo (we call its name `SDK repo` and its absolute path) + +- Create a branch in SDK repo to work in + +- Make sure your typespec definition is merged into `main` branch of [public rest repo](https://github.com/Azure/azure-rest-api-specs) (we call it `rest repo`) or you already make a PR in `rest repo` so that you could get the github link of your typespec definition which contains commit id (e.g. https://github.com/Azure/azure-rest-api-specs/blob/46ca83821edd120552403d4d11cf1dd22360c0b5/specification/contosowidgetmanager/Contoso.WidgetManager/tspconfig.yaml) + +## Project service name and namespace + +Two key pieces of information for your project are the `service_name` and `namespace`. + +The `service_name` is the short name for the Azure service. The `service_name` should match across all the SDK language repos +and should be name of the directory in the specification folder of the azure-rest-api-specs repo that contains the REST API definition file. +An example is Service Bus, whose API definitions are in the `specification/servicebus` folder of the azure-rest-api-specs repo, +and uses the `service_name` "servicebus". +Not every service follows this convention, but it should be the standard unless there are strong reasons to deviate. + +In Python, a project's `package name` is the name used to publish the package in [PyPI](https://pypi.org/). By default, the package name is derived from the `namespace`, just swapping the `.`s for `-`s. You can override the default `package-name` by passing in a different value in your `tspconfig.yaml`. +For data plane libraries (management plane uses a different convention), the package name could be just `azure-{service_name}`. +An example is "azure-servicebus". + +Some services may need several different packages. For these cases a third component, the `module_name`, is added to the `namespace`, +as `azure.{service_name}.{module_name}`. This change affects the default package name as well, which is needed in these cases. +The `module_name` usually comes from the name of the REST API file itself or one of the directories toward the end of the file path. +An example is the Synapse service, with packages `azure-synapse`, `azure-synapse-accesscontrol`, `azure-synapse-artifacts`, etc. + +## Project folder structure + +Before we start, we probably should get to know the project folder for [SDK repo](https://github.com/Azure/azure-sdk-for-python). + +Normally, the folder structure would be something like: + +- `sdk/{service_name}/{package_name}`: **the PROJECT_ROOT folder** + - `/azure/{service_name}/{module_name}`: folder where generated code is. + - `/tests`: folder of test files + - `/samples`: folder of sample files + - `azure-{service_name}-{module_name}`: package name. Usually, package name is same with part of **${PROJECT_ROOT} folder**. After release, you can find it in pypi. For example: you can find [azure-messaging-webpubsubservice](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/webpubsub/azure-messaging-webpubsubservice) in [pypi](https://pypi.org/project/azure-messaging-webpubsubservice/). + - there are also some other files (like setup.py, README.md, etc.) which are necessary for a complete package. + +More details on the structure of Azure SDK repos is available in the [Azure SDK common repo](https://github.com/Azure/azure-sdk/blob/main/docs/policies/repostructure.md#sdk-directory-layout). + +## How to generate SDK code with Dataplane Codegen + +We are working on to automatically generate everything right now, but currently we still need some manual work to get a releasable package. Here are the steps of how to get the package. + +### 1. Configure python emitter in tspconfig.yaml + +In `rest repo`, there shall be `tspconfig.yaml` where `main.tsp` of your service is. Make sure there are configuration for Python SDK like: + +```yaml +parameters: + "service-dir": + default: "YOUR-SERVICE-DIRECTORY" + +emit: [ + "@azure-tools/typespec-autorest", // this value does not affect python code generation +] + +options: + "@azure-tools/typespec-python": + emitter-output-dir: "{output-dir}/{service-dir}/YOUR-PACKAGE-DIR" + namespace: "YOUR.NAMESPACE.NAME" + flavor: "azure" +``` + +`YOUR.NAMESPACE.NAME` is your namespace name; `YOUR-PACKAGE-DIR` is your package directory; `YOUR-SERVICE-DIRECTORY` is SDK directory name. For example, assume that namespace is "azure.ai.anomalydetector" and you want to put it in folder "azure-sdk-for-python/sdk/anomalydetector", then "YOUR.NAMESPACE.NAME" is "azure.ai.anomalydetector", "YOUR_PACKAGE_DIR" is "azure-ai-anomalydetector" and "YOUR-SERVICE-DIRECTORY" is "sdk/anomalydetector" + +### 2. Run cmd to generate the SDK + +Install `tsp-client` CLI tool: + +```shell +npm install -g @azure-tools/typespec-client-generator-cli +``` + +For initial set up, from the root of the SDK repo, call: + +```shell +tsp-client init -c YOUR_REMOTE_TSPCONFIG_URL +``` + +> An example of YOUR_REMOTE_TSPCONFIG_URL is https://github.com/Azure/azure-rest-api-specs/blob/6bbc511b0a42f9ca9992ef16146a12f5cc2a171a/specification/contosowidgetmanager/Contoso.WidgetManager/tspconfig.yaml + +To update your TypeSpec generated SDK, go to your SDK folder where your tsp-location.yaml is located, call: + +```shell +tsp-client update +``` + +tsp-client will look for a `tsp-location.yaml` file in your local directory. `tsp-location.yaml` contains the configuration information that will be used to sync your TypeSpec project and generate your SDK. Please make sure that the commit is targeting the correct TypeSpec project updates you wish to generate your SDK from. + +## Post-Generation Steps + +The generated code is not enough to release directly. Follow these steps regardless of whether you generated from TypeSpec or Swagger. + +### Write README.md + +Write informative content in README.md for customers. See [webpubsub](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/webpubsub/azure-messaging-webpubsubservice/README.md) for an example. + +### Write Tests + +You should write tests that ensure all APIs fulfil their contract. This is a requirement of the [Azure SDK Guidelines](https://azure.github.io/azure-sdk/general_implementation.html#testing). See the [Python SDK testing guide](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md) for details. + +### Write Samples + +Add samples in the `samples` directory under the project root. Each sample should contain a `README.md` ([example](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/webpubsub/azure-messaging-webpubsubservice/samples/Readme.md)). + +### Create/Update `ci.yml` + +If there's no `ci.yml` file in `sdk/{service_name}/`, add the following template: + +```yaml +# DO NOT EDIT THIS FILE +# This file is generated automatically and any changes will be lost. + +trigger: + branches: + include: + - main + - hotfix/* + - release/* + - restapi* + paths: + include: + - sdk/{service_name}/ + +pr: + branches: + include: + - main + - feature/* + - hotfix/* + - release/* + - restapi* + paths: + include: + - sdk/{service_name}/ + +extends: + template: ../../eng/pipelines/templates/stages/archetype-sdk-client.yml + parameters: + ServiceDirectory: {service_name} + Artifacts: + - name: {package_name} + safeName: {safeName} +``` + +Replace `{service_name}`, `{package_name}`, and `{safeName}` with your values. Usually, `safeName` is derived from the package name by removing all `-`. + +If a `ci.yml` already exists, just add your package's `name` and `safeName` entry. + +### Configure CODEOWNERS + +If adding a new module group (new sub-folder under `sdk/`), add an entry to [CODEOWNERS](https://github.com/Azure/azure-sdk-for-python/blob/main/.github/CODEOWNERS) so GitHub auto-assigns reviewers. + +### Release + +See the [Release Checklist](https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/8/Release-Checklist?anchor=prepare-release-script). Before submitting a PR: + +- **Update CHANGELOG.md** with the changes for the new version and an approximate release date (e.g., `1.0.0b1 (2022-02-02)`). +- **Update the version number** according to the [package version rule](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/package_version/package_version_rule.md). +- **Fix CI failures** — see [CI Troubleshooting](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/ci_troubleshooting.md). diff --git a/doc/dev/mgmt/README.md b/doc/dev/mgmt/README.md index 373281e5612e..845341d170b6 100644 --- a/doc/dev/mgmt/README.md +++ b/doc/dev/mgmt/README.md @@ -3,9 +3,12 @@ The documentation in that folder is intended for developers at Microsoft, or for advanced contributors, in the ARM (Azure Resource Management) world. In order of workflow: -- [swagger_conf.md](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt/swagger_conf.md) : Describe how to configure the different Readme on https://github.com/Azure/azure-rest-api-specs for Python +- [generation.md](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt/generation.md) : How to generate the SDK (TypeSpec is the recommended approach; legacy Swagger/AutoRest instructions are also included) - [tests.md](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt/tests.md) : How to test management SDK (recordings, etc.) -- [mgmt_release.md](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt/mgmt_release.md) : How to finish preapring the package, changelog, version, etc. +- [mgmt_release.md](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt/mgmt_release.md) : How to finish preparing the package, changelog, version, etc. + +Legacy documents: +- [swagger_conf.md](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt/swagger_conf.md) : (Legacy) Describe how to configure Swagger Readmes on https://github.com/Azure/azure-rest-api-specs for Python More implementations / advanced documents: - [changelog_impl.md](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt/changelog_impl.md) : Technical details about the change log tool implementation diff --git a/doc/dev/mgmt/generation.md b/doc/dev/mgmt/generation.md index 4a35b3e45254..c9dd8278a6f8 100644 --- a/doc/dev/mgmt/generation.md +++ b/doc/dev/mgmt/generation.md @@ -1,20 +1,140 @@ # Generation of SDK -Assuming your Swagger are associated with correct Readmes (otherwise see previous chapter [Swagger conf](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt/swagger_conf.md)), this page explains how to generate your packages. - IMPORTANT NOTE: All the commands prefixed by `python` in this page assumes you have loaded the [dev_setup](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/dev_setup.md) in your currently loaded virtual environment. -## Building the code +## Generating from TypeSpec + +### Prerequisites + +#### Setting Up Your Basic Environment +- Python 3.9 or newer is required + - [Download for Windows](https://www.python.org/downloads/windows/) + - For Linux: + - Install Python 3 with `sudo apt install python3` + - Install Python 3 Pip with `sudo apt install python3-pip` + - If needed, install Python 3.{?}-venv with `sudo apt install python3.{?}-venv` + +- [Node.js 20.x LTS](https://nodejs.org/en/download/) or newer is required + +#### Preparing Your Repos + +- Fork and clone the [azure-sdk-for-python](https://github.com/Azure/azure-sdk-for-python) repo (the "SDK repo") +- Fork and clone the [azure-rest-api-specs](https://github.com/Azure/azure-rest-api-specs) repo (the "Rest repo") + +#### Identifying `tspconfig.yaml` + +Find the `tspconfig.yaml` file for your package in the Rest repo. Ensure there's a configuration for the Python SDK similar to the one shown below and in [this example](https://github.com/Azure/azure-rest-api-specs/blob/85ed8fc06e19c902c95b531cbd8a643428d5f28d/specification/contosowidgetmanager/Contoso.Management/tspconfig.yaml#L18-L23): + +```yaml +parameters: + "service-dir": + default: "sdk/SERVICE_DIRECTORY_NAME" +options: + "@azure-tools/typespec-python": + emitter-output-dir: "{output-dir}/{service-dir}/azure-mgmt-NAMESPACE" + namespace: "azure.mgmt.NAMESPACE" + generate-test: true + generate-sample: true + flavor: "azure" +``` + +Replace `SERVICE_DIRECTORY_NAME` and `NAMESPACE` with the actual values for your service. + +#### Installing Necessary Dependencies + +1. Install `typespec-client-generator-cli` globally: +``` +npm install -g @azure-tools/typespec-client-generator-cli +``` + +2. Create a Python virtual environment and activate it: +``` +PS C:\dev\azure-sdk-for-python> python -m venv .venv +PS C:\dev\azure-sdk-for-python> .\.venv\Scripts\Activate.ps1 # Windows + /C/dev/azure-sdk-for-python> source .venv/bin/activate # Linux +``` + +3. Install Python dependencies: +``` +(.venv) PS C:\dev\azure-sdk-for-python> python .\scripts\dev_setup.py -p azure-core +(.venv) PS C:\dev\azure-sdk-for-python> pip install tox +(.venv) PS C:\dev\azure-sdk-for-python> pip install setuptools +``` + +### Generate the SDK + +1. Create a local JSON file named `generatedInput.json` outside the SDK repo with content like: + +```json +{ + "specFolder": "LOCAL_AZURE-REST-API-SPECS_REPO_ROOT", + "headSha": "SHA_OF_AZURE-REST-API-SPECS_REPO", + "repoHttpsUrl": "https://github.com/Azure/azure-rest-api-specs", + "relatedTypeSpecProjectFolder": [ + "specification/SERVICE_DIRECTORY_NAME/PACKAGE_DIRECTORY_NAME/" + ] +} +``` + +Use `git rev-parse HEAD` on the local Rest repo root to get the `headSha` value. + +2. Run the generation command: +``` +(.venv) PS C:\dev\azure-sdk-for-python> python -m packaging_tools.sdk_generator ..\generatedInput.json ..\generatedOutput.json +``` + +3. View information about the generated SDK in `generatedOutput.json`. + +### Key Concepts: Service Name and Namespace + +The `service_name` is the short name for the Azure service. It should match across all SDK language repos and should be the name of the directory in the `specification` folder of the Rest repo that contains the REST API definition file. An example is Service Bus, whose API definitions are in `specification/servicebus`, using the service_name `servicebus`. + +In Python, a project's `package name` is the name used to publish to [PyPI](https://pypi.org/). By default, the package name is derived from the `namespace`, swapping `.`s for `-`s. For management plane libraries, the package_name is typically `azure-mgmt-{service_name}` (e.g., `azure-mgmt-servicebus`). + +### Project Folder Structure + +The standard folder structure in the SDK repo is: + +- `sdk/{service_name}/{package_name}/` — the project root + - `tests/` — manual test files + - `generated_tests/` — auto-generated test files + - `samples/` — manual sample files + - `generated_samples/` — auto-generated sample files + +More details on the structure is available in the [Azure SDK common repo](https://github.com/Azure/azure-sdk/blob/main/docs/policies/repostructure.md#sdk-directory-layout). + +### Additional TypeSpec Resources + +- [Getting Started with TypeSpec Generation (using Copilot)](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/ai/typespec_generation.md) +- [TypeSpec Documentation](https://typespec.io/docs) + +## Post-Generation Steps + +See the [Post-Generation Steps](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/dataplane_generation.md#post-generation-steps) for the steps to follow after generating your SDK code (writing README, tests, samples, ci.yml, CODEOWNERS, and release preparation). + +--- + +## Legacy: Generating from Swagger (AutoRest) + +> **Deprecated:** Swagger-based generation with AutoRest is deprecated. All new SDKs should be generated from [TypeSpec](#generating-from-typespec). The instructions below are preserved for maintaining existing legacy packages only. + +### Swagger Configuration + +Assuming your Swagger are associated with correct Readmes (otherwise see [Swagger conf](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt/swagger_conf.md)), you can use AutoRest to generate packages. + +### Building the Code + +#### Autorest Versioning -### Autorest versioning +> **Historical Note:** The version details below are preserved for reference. AutoRest-based generation is no longer the recommended approach. -A few notes on [Autorest for Python versioning](https://github.com/Azure/autorest.python/blob/master/ChangeLog.md): +A few notes on [Autorest for Python versioning](https://github.com/Azure/autorest.python/blob/main/ChangeLog.md): - Autorest for Python v2.x is deprecated, and should not be used anymore for any generation under any circumstances. - Autorest for Python v3.x is the most currently used one. Should not be used, but still ok if service team are still in v3.x and they want to avoid breaking changes for a given version (rare). - Autorest for Python v4.x is the current recommendation. This generator can generates async code, but this should be disabled with --no-async. No package should be shipped with async based on v4 -- Autorest for Python v5.x is the work in progress based on new runtime called `azure-core` (no `msrest` anymore). To be released in November 2019 (current plan). This version will bring the official async support. +- Autorest for Python v5.x is based on the `azure-core` runtime (no `msrest` anymore). This version brings the official async support. -#### How to recognize what version of autorest was used? +##### How to recognize what version of autorest was used? Autorest doesn't write the version number in the generated code, but a few indicator will tell you what generation is used, just looking at the "models" folder @@ -23,7 +143,7 @@ Autorest doesn't write the version number in the generated code, but a few indic - Autorest v4: Two gigantic model files, one called `_models.py` and the second one `_models_py3.py` - Autorest v5: `paged` file will import base classes from `azure.core` and not `msrest` -### Basics of generation +#### Basics of Generation A basic autorest command line will looks like this: @@ -41,11 +161,11 @@ In practical terms, this is not necessary since the Python SDK has the necessary The common configuration to pass to all generation are located in the [swagger_to_sdk.json file](https://github.com/Azure/azure-sdk-for-python/blob/main/swagger_to_sdk_config.json) -### Automation bot +#### Automation Bot If the automation is doing its job correctly, you should not have to build the SDK, but look for an integration PR for the service in question. This link will give you for instance [the list of all integration PRs](https://github.com/Azure/azure-sdk-for-python/labels/ServicePR). -## Using raw autorest +### Using Raw Autorest If you want to use raw autorest and nothing else, not even Readme, a few tips: @@ -62,7 +182,7 @@ And that's it! You should now have Python code ready to test. Note that this gen This command generate code only. If you want to generate a [wheel](https://pythonwheels.com/) file to share this code, add the `--basic-setup-py` option to generate a basic `setup.py` file and call `python setup.py bdist_wheel`. -### Example +#### Examples ARM management Swagger: @@ -87,7 +207,7 @@ To call Autorest, you need the following options: - If your endpoint is ARM, add `--python --azure-arm=true` - If not, add `--python`. If your client _might_ ask authentication, add `--add-credentials` -## Example +#### More Examples ARM Swagger with MD (preferred syntax): diff --git a/doc/dev/mgmt/mgmt_release.md b/doc/dev/mgmt/mgmt_release.md index d517ccb23a25..6a3f074fac22 100644 --- a/doc/dev/mgmt/mgmt_release.md +++ b/doc/dev/mgmt/mgmt_release.md @@ -21,7 +21,7 @@ IMPORTANT NOTE: All the commands in this page assumes you have loaded the [dev_s ## Manual generation If the automation is not doing its job to create an auto PR, Python has a SwaggerToSdk CLI that can be used to generate SDK by a specific Readme. You need -a virtual environment loaded with at least `eng/tools/azure-sdk-tools` installed. And to manually create a package from Typespec, here's the full direction https://github.com/Azure/azure-sdk-for-python/wiki/Generate-Python-Mgmt-SDK-from-Typespec. +a virtual environment loaded with at least `eng/tools/azure-sdk-tools` installed. And to manually create a package from Typespec, here's the full direction https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt/generation.md. ```shell # Using default configuration (this can be a Github raw link) diff --git a/doc/dev/mgmt/tests.md b/doc/dev/mgmt/tests.md index 08e1f8d0aac4..7f7bfdd4ff87 100644 --- a/doc/dev/mgmt/tests.md +++ b/doc/dev/mgmt/tests.md @@ -147,13 +147,6 @@ Code in the [`azure-sdk-tools/devtools_testutils`][devtools_testutils] directory New tests should be located alongside the packages containing the code they test. For example, the tests for `azure-mgmt-media` are in `azure-mgmt-media/tests`. -There are also legacy tests in the following three locations: - -* azure-servicebus/tests -* azure-servicemanagement-legacy/tests - -For more information about legacy tests, see [Legacy tests](https://github.com/Azure/azure-sdk-for-python/wiki/Legacy-tests). - ## Writing management plane test Management plane SDKs are those that are formatted `azure-mgmt-xxxx`, otherwise the SDK is data plane. Management plane SDKs work against the [Azure Resource Manager APIs][arm_apis], while the data plane SDKs will work against service APIs. This section will demonstrate writing tests using `devtools_testutils` with a few increasingly sophisticated examples to show how to use some of the features of the underlying test frameworks. diff --git a/doc/eng_sys_checks.md b/doc/eng_sys_checks.md index 2c852817e3dc..30364ecb7c94 100644 --- a/doc/eng_sys_checks.md +++ b/doc/eng_sys_checks.md @@ -21,6 +21,7 @@ - [Opt-in to formatting validation](#opt-in-to-formatting-validation) - [Running locally](#running-locally) - [Change log verification](#change-log-verification) + - [CSpell](#cspell) - [PR Validation Checks](#pr-validation-checks) - [PR validation checks](#pr-validation-checks-1) - [whl](#whl) @@ -116,6 +117,8 @@ This is the most useful skip, but the following skip variables are also supporte - Omit checking that a package's keywords are correctly formulated before releasing. - `Skip.Black` - Omit checking `black` in the `analyze` job. +- `Skip.CSpell` + - Omit spell checking in the `analyze` job. ## The pyproject.toml @@ -273,6 +276,22 @@ to opt into the black invocation. Change log verification is added to ensure package has valid change log for current version. Guidelines to properly maintain the change log is documented [here.](https://azure.github.io/azure-sdk/policies_releases.html#change-logs/) +### CSpell + +[`CSpell`](https://cspell.org/) is a spell checker that runs against package source code to catch common spelling errors. It checks Python source files, documentation, and other text content in the package. For more details, see the [Spelling Check Scripts README](https://github.com/Azure/azure-sdk-for-python/blob/main/eng/common/spelling/README.md). + +Spell check configuration can be customized at two levels. Repository-wide terms can be added to [`.vscode/cspell.json`](https://github.com/Azure/azure-sdk-for-python/blob/main/.vscode/cspell.json), while service-specific terms can be added to a `cspell.json` or `cspell.yaml` file in the service directory. In either case, words that are domain-specific or intentionally spelled differently can be added to the `words` list. + +If you encounter a CSpell failure in CI, you can resolve it by: + +1. Fixing the spelling error in your code or documentation. +2. Adding the word to your service-level `cspell.json` or `cspell.yaml` file if the word is intentional (e.g., a domain-specific term). If this file does not exist, you can create it. +3. Adding the word to [`.vscode/cspell.json`](https://github.com/Azure/azure-sdk-for-python/blob/main/.vscode/cspell.json) if it is a common term that applies across the repository. +4. Adding an inline `cspell:ignore` comment for one-off exceptions: + ```python + # cspell:ignore specialword + ``` + ## PR Validation Checks Each pull request runs various tests using `pytest` in addition to all the tests mentioned above in analyze check. Pull request validation performs 3 different types of test: `whl, sdist and depends`. The following section explains the purpose of each of these tests and how to execute them locally. All pull requests are validated on multiple python versions across different platforms. Find the test matrix below. diff --git a/doc/python_version_support_policy.md b/doc/python_version_support_policy.md new file mode 100644 index 000000000000..8aa96b45a695 --- /dev/null +++ b/doc/python_version_support_policy.md @@ -0,0 +1,19 @@ +# Azure SDKs Python Version Support Policy + +This page describes the Python version support policy for the Azure SDK for Python, including end-of-support timelines for each Python version. + +End of support means, in the SDK context, that new features will not be supported for those unsupported Python versions. It will still be possible to install older SDK versions from PyPI as necessary. + +| Python Version | PSF End of Support | SDKs End Of Support | +|----------------|--------------------|---------------------| +| 2.7 ([PEP 373](https://peps.python.org/pep-0373/)) | April 2020 | January 2022 | +| 3.6 ([PEP 494](https://www.python.org/dev/peps/pep-0494/#lifespan)) | December 2021 | August 2022 | +| 3.7 ([PEP 537](https://www.python.org/dev/peps/pep-0537/#lifespan)) | June 2023 | December 2023 | +| 3.8 ([PEP 569](https://www.python.org/dev/peps/pep-0569/#lifespan)) | October 2024 | April 2025 | +| 3.9 ([PEP 596](https://www.python.org/dev/peps/pep-0596/#lifespan)) | October 2025 | April 2026 | +| 3.10 ([PEP 619](https://www.python.org/dev/peps/pep-0619/#lifespan)) | October 2026 | April 2027 | +| 3.11 ([PEP 664](https://www.python.org/dev/peps/pep-0664/#lifespan)) | October 2027 | April 2028 | +| 3.12 ([PEP 693](https://www.python.org/dev/peps/pep-0693/#lifespan)) | October 2028 | April 2029 | +| 3.13 ([PEP 719](https://www.python.org/dev/peps/pep-0719/#lifespan)) | October 2029 | April 2030 | +| 3.14 ([PEP 745](https://www.python.org/dev/peps/pep-0745/#lifespan)) | October 2030 | April 2031 | +| 3.15 ([PEP 790](https://www.python.org/dev/peps/pep-0790/#lifespan)) | October 2031 | April 2032 |