diff --git a/github.com/Masterminds/goutils/.travis.yml b/github.com/Masterminds/goutils/.travis.yml new file mode 100644 index 0000000000..4025e01ec4 --- /dev/null +++ b/github.com/Masterminds/goutils/.travis.yml @@ -0,0 +1,18 @@ +language: go + +go: + - 1.6 + - 1.7 + - 1.8 + - tip + +script: + - go test -v + +notifications: + webhooks: + urls: + - https://webhooks.gitter.im/e/06e3328629952dabe3e0 + on_success: change # options: [always|never|change] default: always + on_failure: always # options: [always|never|change] default: always + on_start: never # options: [always|never|change] default: always diff --git a/github.com/Masterminds/goutils/CHANGELOG.md b/github.com/Masterminds/goutils/CHANGELOG.md new file mode 100644 index 0000000000..d700ec47f2 --- /dev/null +++ b/github.com/Masterminds/goutils/CHANGELOG.md @@ -0,0 +1,8 @@ +# 1.0.1 (2017-05-31) + +## Fixed +- #21: Fix generation of alphanumeric strings (thanks @dbarranco) + +# 1.0.0 (2014-04-30) + +- Initial release. diff --git a/github.com/Masterminds/goutils/LICENSE.txt b/github.com/Masterminds/goutils/LICENSE.txt new file mode 100644 index 0000000000..d645695673 --- /dev/null +++ b/github.com/Masterminds/goutils/LICENSE.txt @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/github.com/Masterminds/goutils/README.md b/github.com/Masterminds/goutils/README.md new file mode 100644 index 0000000000..163ffe72a8 --- /dev/null +++ b/github.com/Masterminds/goutils/README.md @@ -0,0 +1,70 @@ +GoUtils +=========== +[![Stability: Maintenance](https://masterminds.github.io/stability/maintenance.svg)](https://masterminds.github.io/stability/maintenance.html) +[![GoDoc](https://godoc.org/github.com/Masterminds/goutils?status.png)](https://godoc.org/github.com/Masterminds/goutils) [![Build Status](https://travis-ci.org/Masterminds/goutils.svg?branch=master)](https://travis-ci.org/Masterminds/goutils) [![Build status](https://ci.appveyor.com/api/projects/status/sc2b1ew0m7f0aiju?svg=true)](https://ci.appveyor.com/project/mattfarina/goutils) + + +GoUtils provides users with utility functions to manipulate strings in various ways. It is a Go implementation of some +string manipulation libraries of Java Apache Commons. GoUtils includes the following Java Apache Commons classes: +* WordUtils +* RandomStringUtils +* StringUtils (partial implementation) + +## Installation +If you have Go set up on your system, from the GOPATH directory within the command line/terminal, enter this: + + go get github.com/Masterminds/goutils + +If you do not have Go set up on your system, please follow the [Go installation directions from the documenation](http://golang.org/doc/install), and then follow the instructions above to install GoUtils. + + +## Documentation +GoUtils doc is available here: [![GoDoc](https://godoc.org/github.com/Masterminds/goutils?status.png)](https://godoc.org/github.com/Masterminds/goutils) + + +## Usage +The code snippets below show examples of how to use GoUtils. Some functions return errors while others do not. The first instance below, which does not return an error, is the `Initials` function (located within the `wordutils.go` file). + + package main + + import ( + "fmt" + "github.com/Masterminds/goutils" + ) + + func main() { + + // EXAMPLE 1: A goutils function which returns no errors + fmt.Println (goutils.Initials("John Doe Foo")) // Prints out "JDF" + + } +Some functions return errors mainly due to illegal arguements used as parameters. The code example below illustrates how to deal with function that returns an error. In this instance, the function is the `Random` function (located within the `randomstringutils.go` file). + + package main + + import ( + "fmt" + "github.com/Masterminds/goutils" + ) + + func main() { + + // EXAMPLE 2: A goutils function which returns an error + rand1, err1 := goutils.Random (-1, 0, 0, true, true) + + if err1 != nil { + fmt.Println(err1) // Prints out error message because -1 was entered as the first parameter in goutils.Random(...) + } else { + fmt.Println(rand1) + } + + } + +## License +GoUtils is licensed under the Apache License, Version 2.0. Please check the LICENSE.txt file or visit http://www.apache.org/licenses/LICENSE-2.0 for a copy of the license. + +## Issue Reporting +Make suggestions or report issues using the Git issue tracker: https://github.com/Masterminds/goutils/issues + +## Website +* [GoUtils webpage](http://Masterminds.github.io/goutils/) diff --git a/github.com/Masterminds/goutils/appveyor.yml b/github.com/Masterminds/goutils/appveyor.yml new file mode 100644 index 0000000000..657564a847 --- /dev/null +++ b/github.com/Masterminds/goutils/appveyor.yml @@ -0,0 +1,21 @@ +version: build-{build}.{branch} + +clone_folder: C:\gopath\src\github.com\Masterminds\goutils +shallow_clone: true + +environment: + GOPATH: C:\gopath + +platform: + - x64 + +build: off + +install: + - go version + - go env + +test_script: + - go test -v + +deploy: off diff --git a/github.com/Masterminds/goutils/cryptorandomstringutils.go b/github.com/Masterminds/goutils/cryptorandomstringutils.go new file mode 100644 index 0000000000..177dd86584 --- /dev/null +++ b/github.com/Masterminds/goutils/cryptorandomstringutils.go @@ -0,0 +1,251 @@ +/* +Copyright 2014 Alexander Okoli + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package goutils + +import ( + "crypto/rand" + "fmt" + "math" + "math/big" + "regexp" + "unicode" +) + +/* +CryptoRandomNonAlphaNumeric creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of all characters (ASCII/Unicode values between 0 to 2,147,483,647 (math.MaxInt32)). + +Parameter: + count - the length of random string to create + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, CryptoRandom(...) +*/ +func CryptoRandomNonAlphaNumeric(count int) (string, error) { + return CryptoRandomAlphaNumericCustom(count, false, false) +} + +/* +CryptoRandomAscii creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of characters whose ASCII value is between 32 and 126 (inclusive). + +Parameter: + count - the length of random string to create + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, CryptoRandom(...) +*/ +func CryptoRandomAscii(count int) (string, error) { + return CryptoRandom(count, 32, 127, false, false) +} + +/* +CryptoRandomNumeric creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of numeric characters. + +Parameter: + count - the length of random string to create + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, CryptoRandom(...) +*/ +func CryptoRandomNumeric(count int) (string, error) { + return CryptoRandom(count, 0, 0, false, true) +} + +/* +CryptoRandomAlphabetic creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of alpha-numeric characters as indicated by the arguments. + +Parameters: + count - the length of random string to create + letters - if true, generated string may include alphabetic characters + numbers - if true, generated string may include numeric characters + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, CryptoRandom(...) +*/ +func CryptoRandomAlphabetic(count int) (string, error) { + return CryptoRandom(count, 0, 0, true, false) +} + +/* +CryptoRandomAlphaNumeric creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of alpha-numeric characters. + +Parameter: + count - the length of random string to create + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, CryptoRandom(...) +*/ +func CryptoRandomAlphaNumeric(count int) (string, error) { + if count == 0 { + return "", nil + } + RandomString, err := CryptoRandom(count, 0, 0, true, true) + if err != nil { + return "", fmt.Errorf("Error: %s", err) + } + match, err := regexp.MatchString("([0-9]+)", RandomString) + if err != nil { + panic(err) + } + + if !match { + //Get the position between 0 and the length of the string-1 to insert a random number + position := getCryptoRandomInt(count) + //Insert a random number between [0-9] in the position + RandomString = RandomString[:position] + string('0' + getCryptoRandomInt(10)) + RandomString[position + 1:] + return RandomString, err + } + return RandomString, err + +} + +/* +CryptoRandomAlphaNumericCustom creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of alpha-numeric characters as indicated by the arguments. + +Parameters: + count - the length of random string to create + letters - if true, generated string may include alphabetic characters + numbers - if true, generated string may include numeric characters + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, CryptoRandom(...) +*/ +func CryptoRandomAlphaNumericCustom(count int, letters bool, numbers bool) (string, error) { + return CryptoRandom(count, 0, 0, letters, numbers) +} + +/* +CryptoRandom creates a random string based on a variety of options, using using golang's crypto/rand source of randomness. +If the parameters start and end are both 0, start and end are set to ' ' and 'z', the ASCII printable characters, will be used, +unless letters and numbers are both false, in which case, start and end are set to 0 and math.MaxInt32, respectively. +If chars is not nil, characters stored in chars that are between start and end are chosen. + +Parameters: + count - the length of random string to create + start - the position in set of chars (ASCII/Unicode int) to start at + end - the position in set of chars (ASCII/Unicode int) to end before + letters - if true, generated string may include alphabetic characters + numbers - if true, generated string may include numeric characters + chars - the set of chars to choose randoms from. If nil, then it will use the set of all chars. + +Returns: + string - the random string + error - an error stemming from invalid parameters: if count < 0; or the provided chars array is empty; or end <= start; or end > len(chars) +*/ +func CryptoRandom(count int, start int, end int, letters bool, numbers bool, chars ...rune) (string, error) { + if count == 0 { + return "", nil + } else if count < 0 { + err := fmt.Errorf("randomstringutils illegal argument: Requested random string length %v is less than 0.", count) // equiv to err := errors.New("...") + return "", err + } + if chars != nil && len(chars) == 0 { + err := fmt.Errorf("randomstringutils illegal argument: The chars array must not be empty") + return "", err + } + + if start == 0 && end == 0 { + if chars != nil { + end = len(chars) + } else { + if !letters && !numbers { + end = math.MaxInt32 + } else { + end = 'z' + 1 + start = ' ' + } + } + } else { + if end <= start { + err := fmt.Errorf("randomstringutils illegal argument: Parameter end (%v) must be greater than start (%v)", end, start) + return "", err + } + + if chars != nil && end > len(chars) { + err := fmt.Errorf("randomstringutils illegal argument: Parameter end (%v) cannot be greater than len(chars) (%v)", end, len(chars)) + return "", err + } + } + + buffer := make([]rune, count) + gap := end - start + + // high-surrogates range, (\uD800-\uDBFF) = 55296 - 56319 + // low-surrogates range, (\uDC00-\uDFFF) = 56320 - 57343 + + for count != 0 { + count-- + var ch rune + if chars == nil { + ch = rune(getCryptoRandomInt(gap) + int64(start)) + } else { + ch = chars[getCryptoRandomInt(gap) + int64(start)] + } + + if letters && unicode.IsLetter(ch) || numbers && unicode.IsDigit(ch) || !letters && !numbers { + if ch >= 56320 && ch <= 57343 { // low surrogate range + if count == 0 { + count++ + } else { + // Insert low surrogate + buffer[count] = ch + count-- + // Insert high surrogate + buffer[count] = rune(55296 + getCryptoRandomInt(128)) + } + } else if ch >= 55296 && ch <= 56191 { // High surrogates range (Partial) + if count == 0 { + count++ + } else { + // Insert low surrogate + buffer[count] = rune(56320 + getCryptoRandomInt(128)) + count-- + // Insert high surrogate + buffer[count] = ch + } + } else if ch >= 56192 && ch <= 56319 { + // private high surrogate, skip it + count++ + } else { + // not one of the surrogates* + buffer[count] = ch + } + } else { + count++ + } + } + return string(buffer), nil +} + +func getCryptoRandomInt(count int) int64 { + nBig, err := rand.Int(rand.Reader, big.NewInt(int64(count))) + if err != nil { + panic(err) + } + return nBig.Int64() +} diff --git a/github.com/Masterminds/goutils/randomstringutils.go b/github.com/Masterminds/goutils/randomstringutils.go new file mode 100644 index 0000000000..1364e0cafd --- /dev/null +++ b/github.com/Masterminds/goutils/randomstringutils.go @@ -0,0 +1,268 @@ +/* +Copyright 2014 Alexander Okoli + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package goutils + +import ( + "fmt" + "math" + "math/rand" + "regexp" + "time" + "unicode" +) + +// RANDOM provides the time-based seed used to generate random numbers +var RANDOM = rand.New(rand.NewSource(time.Now().UnixNano())) + +/* +RandomNonAlphaNumeric creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of all characters (ASCII/Unicode values between 0 to 2,147,483,647 (math.MaxInt32)). + +Parameter: + count - the length of random string to create + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, RandomSeed(...) +*/ +func RandomNonAlphaNumeric(count int) (string, error) { + return RandomAlphaNumericCustom(count, false, false) +} + +/* +RandomAscii creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of characters whose ASCII value is between 32 and 126 (inclusive). + +Parameter: + count - the length of random string to create + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, RandomSeed(...) +*/ +func RandomAscii(count int) (string, error) { + return Random(count, 32, 127, false, false) +} + +/* +RandomNumeric creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of numeric characters. + +Parameter: + count - the length of random string to create + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, RandomSeed(...) +*/ +func RandomNumeric(count int) (string, error) { + return Random(count, 0, 0, false, true) +} + +/* +RandomAlphabetic creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of alpha-numeric characters as indicated by the arguments. + +Parameters: + count - the length of random string to create + letters - if true, generated string may include alphabetic characters + numbers - if true, generated string may include numeric characters + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, RandomSeed(...) +*/ +func RandomAlphabetic(count int) (string, error) { + return Random(count, 0, 0, true, false) +} + +/* +RandomAlphaNumeric creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of alpha-numeric characters. + +Parameter: + count - the length of random string to create + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, RandomSeed(...) +*/ +func RandomAlphaNumeric(count int) (string, error) { + RandomString, err := Random(count, 0, 0, true, true) + if err != nil { + return "", fmt.Errorf("Error: %s", err) + } + match, err := regexp.MatchString("([0-9]+)", RandomString) + if err != nil { + panic(err) + } + + if !match { + //Get the position between 0 and the length of the string-1 to insert a random number + position := rand.Intn(count) + //Insert a random number between [0-9] in the position + RandomString = RandomString[:position] + string('0'+rand.Intn(10)) + RandomString[position+1:] + return RandomString, err + } + return RandomString, err + +} + +/* +RandomAlphaNumericCustom creates a random string whose length is the number of characters specified. +Characters will be chosen from the set of alpha-numeric characters as indicated by the arguments. + +Parameters: + count - the length of random string to create + letters - if true, generated string may include alphabetic characters + numbers - if true, generated string may include numeric characters + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, RandomSeed(...) +*/ +func RandomAlphaNumericCustom(count int, letters bool, numbers bool) (string, error) { + return Random(count, 0, 0, letters, numbers) +} + +/* +Random creates a random string based on a variety of options, using default source of randomness. +This method has exactly the same semantics as RandomSeed(int, int, int, bool, bool, []char, *rand.Rand), but +instead of using an externally supplied source of randomness, it uses the internal *rand.Rand instance. + +Parameters: + count - the length of random string to create + start - the position in set of chars (ASCII/Unicode int) to start at + end - the position in set of chars (ASCII/Unicode int) to end before + letters - if true, generated string may include alphabetic characters + numbers - if true, generated string may include numeric characters + chars - the set of chars to choose randoms from. If nil, then it will use the set of all chars. + +Returns: + string - the random string + error - an error stemming from an invalid parameter within underlying function, RandomSeed(...) +*/ +func Random(count int, start int, end int, letters bool, numbers bool, chars ...rune) (string, error) { + return RandomSeed(count, start, end, letters, numbers, chars, RANDOM) +} + +/* +RandomSeed creates a random string based on a variety of options, using supplied source of randomness. +If the parameters start and end are both 0, start and end are set to ' ' and 'z', the ASCII printable characters, will be used, +unless letters and numbers are both false, in which case, start and end are set to 0 and math.MaxInt32, respectively. +If chars is not nil, characters stored in chars that are between start and end are chosen. +This method accepts a user-supplied *rand.Rand instance to use as a source of randomness. By seeding a single *rand.Rand instance +with a fixed seed and using it for each call, the same random sequence of strings can be generated repeatedly and predictably. + +Parameters: + count - the length of random string to create + start - the position in set of chars (ASCII/Unicode decimals) to start at + end - the position in set of chars (ASCII/Unicode decimals) to end before + letters - if true, generated string may include alphabetic characters + numbers - if true, generated string may include numeric characters + chars - the set of chars to choose randoms from. If nil, then it will use the set of all chars. + random - a source of randomness. + +Returns: + string - the random string + error - an error stemming from invalid parameters: if count < 0; or the provided chars array is empty; or end <= start; or end > len(chars) +*/ +func RandomSeed(count int, start int, end int, letters bool, numbers bool, chars []rune, random *rand.Rand) (string, error) { + + if count == 0 { + return "", nil + } else if count < 0 { + err := fmt.Errorf("randomstringutils illegal argument: Requested random string length %v is less than 0.", count) // equiv to err := errors.New("...") + return "", err + } + if chars != nil && len(chars) == 0 { + err := fmt.Errorf("randomstringutils illegal argument: The chars array must not be empty") + return "", err + } + + if start == 0 && end == 0 { + if chars != nil { + end = len(chars) + } else { + if !letters && !numbers { + end = math.MaxInt32 + } else { + end = 'z' + 1 + start = ' ' + } + } + } else { + if end <= start { + err := fmt.Errorf("randomstringutils illegal argument: Parameter end (%v) must be greater than start (%v)", end, start) + return "", err + } + + if chars != nil && end > len(chars) { + err := fmt.Errorf("randomstringutils illegal argument: Parameter end (%v) cannot be greater than len(chars) (%v)", end, len(chars)) + return "", err + } + } + + buffer := make([]rune, count) + gap := end - start + + // high-surrogates range, (\uD800-\uDBFF) = 55296 - 56319 + // low-surrogates range, (\uDC00-\uDFFF) = 56320 - 57343 + + for count != 0 { + count-- + var ch rune + if chars == nil { + ch = rune(random.Intn(gap) + start) + } else { + ch = chars[random.Intn(gap)+start] + } + + if letters && unicode.IsLetter(ch) || numbers && unicode.IsDigit(ch) || !letters && !numbers { + if ch >= 56320 && ch <= 57343 { // low surrogate range + if count == 0 { + count++ + } else { + // Insert low surrogate + buffer[count] = ch + count-- + // Insert high surrogate + buffer[count] = rune(55296 + random.Intn(128)) + } + } else if ch >= 55296 && ch <= 56191 { // High surrogates range (Partial) + if count == 0 { + count++ + } else { + // Insert low surrogate + buffer[count] = rune(56320 + random.Intn(128)) + count-- + // Insert high surrogate + buffer[count] = ch + } + } else if ch >= 56192 && ch <= 56319 { + // private high surrogate, skip it + count++ + } else { + // not one of the surrogates* + buffer[count] = ch + } + } else { + count++ + } + } + return string(buffer), nil +} diff --git a/github.com/Masterminds/goutils/stringutils.go b/github.com/Masterminds/goutils/stringutils.go new file mode 100644 index 0000000000..5037c4516b --- /dev/null +++ b/github.com/Masterminds/goutils/stringutils.go @@ -0,0 +1,224 @@ +/* +Copyright 2014 Alexander Okoli + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package goutils + +import ( + "bytes" + "fmt" + "strings" + "unicode" +) + +// Typically returned by functions where a searched item cannot be found +const INDEX_NOT_FOUND = -1 + +/* +Abbreviate abbreviates a string using ellipses. This will turn the string "Now is the time for all good men" into "Now is the time for..." + +Specifically, the algorithm is as follows: + + - If str is less than maxWidth characters long, return it. + - Else abbreviate it to (str[0:maxWidth - 3] + "..."). + - If maxWidth is less than 4, return an illegal argument error. + - In no case will it return a string of length greater than maxWidth. + +Parameters: + str - the string to check + maxWidth - maximum length of result string, must be at least 4 + +Returns: + string - abbreviated string + error - if the width is too small +*/ +func Abbreviate(str string, maxWidth int) (string, error) { + return AbbreviateFull(str, 0, maxWidth) +} + +/* +AbbreviateFull abbreviates a string using ellipses. This will turn the string "Now is the time for all good men" into "...is the time for..." +This function works like Abbreviate(string, int), but allows you to specify a "left edge" offset. Note that this left edge is not +necessarily going to be the leftmost character in the result, or the first character following the ellipses, but it will appear +somewhere in the result. +In no case will it return a string of length greater than maxWidth. + +Parameters: + str - the string to check + offset - left edge of source string + maxWidth - maximum length of result string, must be at least 4 + +Returns: + string - abbreviated string + error - if the width is too small +*/ +func AbbreviateFull(str string, offset int, maxWidth int) (string, error) { + if str == "" { + return "", nil + } + if maxWidth < 4 { + err := fmt.Errorf("stringutils illegal argument: Minimum abbreviation width is 4") + return "", err + } + if len(str) <= maxWidth { + return str, nil + } + if offset > len(str) { + offset = len(str) + } + if len(str)-offset < (maxWidth - 3) { // 15 - 5 < 10 - 3 = 10 < 7 + offset = len(str) - (maxWidth - 3) + } + abrevMarker := "..." + if offset <= 4 { + return str[0:maxWidth-3] + abrevMarker, nil // str.substring(0, maxWidth - 3) + abrevMarker; + } + if maxWidth < 7 { + err := fmt.Errorf("stringutils illegal argument: Minimum abbreviation width with offset is 7") + return "", err + } + if (offset + maxWidth - 3) < len(str) { // 5 + (10-3) < 15 = 12 < 15 + abrevStr, _ := Abbreviate(str[offset:len(str)], (maxWidth - 3)) + return abrevMarker + abrevStr, nil // abrevMarker + abbreviate(str.substring(offset), maxWidth - 3); + } + return abrevMarker + str[(len(str)-(maxWidth-3)):len(str)], nil // abrevMarker + str.substring(str.length() - (maxWidth - 3)); +} + +/* +DeleteWhiteSpace deletes all whitespaces from a string as defined by unicode.IsSpace(rune). +It returns the string without whitespaces. + +Parameter: + str - the string to delete whitespace from, may be nil + +Returns: + the string without whitespaces +*/ +func DeleteWhiteSpace(str string) string { + if str == "" { + return str + } + sz := len(str) + var chs bytes.Buffer + count := 0 + for i := 0; i < sz; i++ { + ch := rune(str[i]) + if !unicode.IsSpace(ch) { + chs.WriteRune(ch) + count++ + } + } + if count == sz { + return str + } + return chs.String() +} + +/* +IndexOfDifference compares two strings, and returns the index at which the strings begin to differ. + +Parameters: + str1 - the first string + str2 - the second string + +Returns: + the index where str1 and str2 begin to differ; -1 if they are equal +*/ +func IndexOfDifference(str1 string, str2 string) int { + if str1 == str2 { + return INDEX_NOT_FOUND + } + if IsEmpty(str1) || IsEmpty(str2) { + return 0 + } + var i int + for i = 0; i < len(str1) && i < len(str2); i++ { + if rune(str1[i]) != rune(str2[i]) { + break + } + } + if i < len(str2) || i < len(str1) { + return i + } + return INDEX_NOT_FOUND +} + +/* +IsBlank checks if a string is whitespace or empty (""). Observe the following behavior: + + goutils.IsBlank("") = true + goutils.IsBlank(" ") = true + goutils.IsBlank("bob") = false + goutils.IsBlank(" bob ") = false + +Parameter: + str - the string to check + +Returns: + true - if the string is whitespace or empty ("") +*/ +func IsBlank(str string) bool { + strLen := len(str) + if str == "" || strLen == 0 { + return true + } + for i := 0; i < strLen; i++ { + if unicode.IsSpace(rune(str[i])) == false { + return false + } + } + return true +} + +/* +IndexOf returns the index of the first instance of sub in str, with the search beginning from the +index start point specified. -1 is returned if sub is not present in str. + +An empty string ("") will return -1 (INDEX_NOT_FOUND). A negative start position is treated as zero. +A start position greater than the string length returns -1. + +Parameters: + str - the string to check + sub - the substring to find + start - the start position; negative treated as zero + +Returns: + the first index where the sub string was found (always >= start) +*/ +func IndexOf(str string, sub string, start int) int { + + if start < 0 { + start = 0 + } + + if len(str) < start { + return INDEX_NOT_FOUND + } + + if IsEmpty(str) || IsEmpty(sub) { + return INDEX_NOT_FOUND + } + + partialIndex := strings.Index(str[start:len(str)], sub) + if partialIndex == -1 { + return INDEX_NOT_FOUND + } + return partialIndex + start +} + +// IsEmpty checks if a string is empty (""). Returns true if empty, and false otherwise. +func IsEmpty(str string) bool { + return len(str) == 0 +} diff --git a/github.com/Masterminds/goutils/wordutils.go b/github.com/Masterminds/goutils/wordutils.go new file mode 100644 index 0000000000..034cad8e21 --- /dev/null +++ b/github.com/Masterminds/goutils/wordutils.go @@ -0,0 +1,357 @@ +/* +Copyright 2014 Alexander Okoli + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +/* +Package goutils provides utility functions to manipulate strings in various ways. +The code snippets below show examples of how to use goutils. Some functions return +errors while others do not, so usage would vary as a result. + +Example: + + package main + + import ( + "fmt" + "github.com/aokoli/goutils" + ) + + func main() { + + // EXAMPLE 1: A goutils function which returns no errors + fmt.Println (goutils.Initials("John Doe Foo")) // Prints out "JDF" + + + + // EXAMPLE 2: A goutils function which returns an error + rand1, err1 := goutils.Random (-1, 0, 0, true, true) + + if err1 != nil { + fmt.Println(err1) // Prints out error message because -1 was entered as the first parameter in goutils.Random(...) + } else { + fmt.Println(rand1) + } + } +*/ +package goutils + +import ( + "bytes" + "strings" + "unicode" +) + +// VERSION indicates the current version of goutils +const VERSION = "1.0.0" + +/* +Wrap wraps a single line of text, identifying words by ' '. +New lines will be separated by '\n'. Very long words, such as URLs will not be wrapped. +Leading spaces on a new line are stripped. Trailing spaces are not stripped. + +Parameters: + str - the string to be word wrapped + wrapLength - the column (a column can fit only one character) to wrap the words at, less than 1 is treated as 1 + +Returns: + a line with newlines inserted +*/ +func Wrap(str string, wrapLength int) string { + return WrapCustom(str, wrapLength, "", false) +} + +/* +WrapCustom wraps a single line of text, identifying words by ' '. +Leading spaces on a new line are stripped. Trailing spaces are not stripped. + +Parameters: + str - the string to be word wrapped + wrapLength - the column number (a column can fit only one character) to wrap the words at, less than 1 is treated as 1 + newLineStr - the string to insert for a new line, "" uses '\n' + wrapLongWords - true if long words (such as URLs) should be wrapped + +Returns: + a line with newlines inserted +*/ +func WrapCustom(str string, wrapLength int, newLineStr string, wrapLongWords bool) string { + + if str == "" { + return "" + } + if newLineStr == "" { + newLineStr = "\n" // TODO Assumes "\n" is seperator. Explore SystemUtils.LINE_SEPARATOR from Apache Commons + } + if wrapLength < 1 { + wrapLength = 1 + } + + inputLineLength := len(str) + offset := 0 + + var wrappedLine bytes.Buffer + + for inputLineLength-offset > wrapLength { + + if rune(str[offset]) == ' ' { + offset++ + continue + } + + end := wrapLength + offset + 1 + spaceToWrapAt := strings.LastIndex(str[offset:end], " ") + offset + + if spaceToWrapAt >= offset { + // normal word (not longer than wrapLength) + wrappedLine.WriteString(str[offset:spaceToWrapAt]) + wrappedLine.WriteString(newLineStr) + offset = spaceToWrapAt + 1 + + } else { + // long word or URL + if wrapLongWords { + end := wrapLength + offset + // long words are wrapped one line at a time + wrappedLine.WriteString(str[offset:end]) + wrappedLine.WriteString(newLineStr) + offset += wrapLength + } else { + // long words aren't wrapped, just extended beyond limit + end := wrapLength + offset + index := strings.IndexRune(str[end:len(str)], ' ') + if index == -1 { + wrappedLine.WriteString(str[offset:len(str)]) + offset = inputLineLength + } else { + spaceToWrapAt = index + end + wrappedLine.WriteString(str[offset:spaceToWrapAt]) + wrappedLine.WriteString(newLineStr) + offset = spaceToWrapAt + 1 + } + } + } + } + + wrappedLine.WriteString(str[offset:len(str)]) + + return wrappedLine.String() + +} + +/* +Capitalize capitalizes all the delimiter separated words in a string. Only the first letter of each word is changed. +To convert the rest of each word to lowercase at the same time, use CapitalizeFully(str string, delimiters ...rune). +The delimiters represent a set of characters understood to separate words. The first string character +and the first non-delimiter character after a delimiter will be capitalized. A "" input string returns "". +Capitalization uses the Unicode title case, normally equivalent to upper case. + +Parameters: + str - the string to capitalize + delimiters - set of characters to determine capitalization, exclusion of this parameter means whitespace would be delimeter + +Returns: + capitalized string +*/ +func Capitalize(str string, delimiters ...rune) string { + + var delimLen int + + if delimiters == nil { + delimLen = -1 + } else { + delimLen = len(delimiters) + } + + if str == "" || delimLen == 0 { + return str + } + + buffer := []rune(str) + capitalizeNext := true + for i := 0; i < len(buffer); i++ { + ch := buffer[i] + if isDelimiter(ch, delimiters...) { + capitalizeNext = true + } else if capitalizeNext { + buffer[i] = unicode.ToTitle(ch) + capitalizeNext = false + } + } + return string(buffer) + +} + +/* +CapitalizeFully converts all the delimiter separated words in a string into capitalized words, that is each word is made up of a +titlecase character and then a series of lowercase characters. The delimiters represent a set of characters understood +to separate words. The first string character and the first non-delimiter character after a delimiter will be capitalized. +Capitalization uses the Unicode title case, normally equivalent to upper case. + +Parameters: + str - the string to capitalize fully + delimiters - set of characters to determine capitalization, exclusion of this parameter means whitespace would be delimeter + +Returns: + capitalized string +*/ +func CapitalizeFully(str string, delimiters ...rune) string { + + var delimLen int + + if delimiters == nil { + delimLen = -1 + } else { + delimLen = len(delimiters) + } + + if str == "" || delimLen == 0 { + return str + } + str = strings.ToLower(str) + return Capitalize(str, delimiters...) +} + +/* +Uncapitalize uncapitalizes all the whitespace separated words in a string. Only the first letter of each word is changed. +The delimiters represent a set of characters understood to separate words. The first string character and the first non-delimiter +character after a delimiter will be uncapitalized. Whitespace is defined by unicode.IsSpace(char). + +Parameters: + str - the string to uncapitalize fully + delimiters - set of characters to determine capitalization, exclusion of this parameter means whitespace would be delimeter + +Returns: + uncapitalized string +*/ +func Uncapitalize(str string, delimiters ...rune) string { + + var delimLen int + + if delimiters == nil { + delimLen = -1 + } else { + delimLen = len(delimiters) + } + + if str == "" || delimLen == 0 { + return str + } + + buffer := []rune(str) + uncapitalizeNext := true // TODO Always makes capitalize/un apply to first char. + for i := 0; i < len(buffer); i++ { + ch := buffer[i] + if isDelimiter(ch, delimiters...) { + uncapitalizeNext = true + } else if uncapitalizeNext { + buffer[i] = unicode.ToLower(ch) + uncapitalizeNext = false + } + } + return string(buffer) +} + +/* +SwapCase swaps the case of a string using a word based algorithm. + +Conversion algorithm: + + Upper case character converts to Lower case + Title case character converts to Lower case + Lower case character after Whitespace or at start converts to Title case + Other Lower case character converts to Upper case + Whitespace is defined by unicode.IsSpace(char). + +Parameters: + str - the string to swap case + +Returns: + the changed string +*/ +func SwapCase(str string) string { + if str == "" { + return str + } + buffer := []rune(str) + + whitespace := true + + for i := 0; i < len(buffer); i++ { + ch := buffer[i] + if unicode.IsUpper(ch) { + buffer[i] = unicode.ToLower(ch) + whitespace = false + } else if unicode.IsTitle(ch) { + buffer[i] = unicode.ToLower(ch) + whitespace = false + } else if unicode.IsLower(ch) { + if whitespace { + buffer[i] = unicode.ToTitle(ch) + whitespace = false + } else { + buffer[i] = unicode.ToUpper(ch) + } + } else { + whitespace = unicode.IsSpace(ch) + } + } + return string(buffer) +} + +/* +Initials extracts the initial letters from each word in the string. The first letter of the string and all first +letters after the defined delimiters are returned as a new string. Their case is not changed. If the delimiters +parameter is excluded, then Whitespace is used. Whitespace is defined by unicode.IsSpacea(char). An empty delimiter array returns an empty string. + +Parameters: + str - the string to get initials from + delimiters - set of characters to determine words, exclusion of this parameter means whitespace would be delimeter +Returns: + string of initial letters +*/ +func Initials(str string, delimiters ...rune) string { + if str == "" { + return str + } + if delimiters != nil && len(delimiters) == 0 { + return "" + } + strLen := len(str) + var buf bytes.Buffer + lastWasGap := true + for i := 0; i < strLen; i++ { + ch := rune(str[i]) + + if isDelimiter(ch, delimiters...) { + lastWasGap = true + } else if lastWasGap { + buf.WriteRune(ch) + lastWasGap = false + } + } + return buf.String() +} + +// private function (lower case func name) +func isDelimiter(ch rune, delimiters ...rune) bool { + if delimiters == nil { + return unicode.IsSpace(ch) + } + for _, delimiter := range delimiters { + if ch == delimiter { + return true + } + } + return false +} diff --git a/github.com/Masterminds/semver/.travis.yml b/github.com/Masterminds/semver/.travis.yml new file mode 100644 index 0000000000..096369d44d --- /dev/null +++ b/github.com/Masterminds/semver/.travis.yml @@ -0,0 +1,29 @@ +language: go + +go: + - 1.6.x + - 1.7.x + - 1.8.x + - 1.9.x + - 1.10.x + - 1.11.x + - 1.12.x + - tip + +# Setting sudo access to false will let Travis CI use containers rather than +# VMs to run the tests. For more details see: +# - http://docs.travis-ci.com/user/workers/container-based-infrastructure/ +# - http://docs.travis-ci.com/user/workers/standard-infrastructure/ +sudo: false + +script: + - make setup + - make test + +notifications: + webhooks: + urls: + - https://webhooks.gitter.im/e/06e3328629952dabe3e0 + on_success: change # options: [always|never|change] default: always + on_failure: always # options: [always|never|change] default: always + on_start: never # options: [always|never|change] default: always diff --git a/github.com/Masterminds/semver/CHANGELOG.md b/github.com/Masterminds/semver/CHANGELOG.md new file mode 100644 index 0000000000..e405c9a84d --- /dev/null +++ b/github.com/Masterminds/semver/CHANGELOG.md @@ -0,0 +1,109 @@ +# 1.5.0 (2019-09-11) + +## Added + +- #103: Add basic fuzzing for `NewVersion()` (thanks @jesse-c) + +## Changed + +- #82: Clarify wildcard meaning in range constraints and update tests for it (thanks @greysteil) +- #83: Clarify caret operator range for pre-1.0.0 dependencies (thanks @greysteil) +- #72: Adding docs comment pointing to vert for a cli +- #71: Update the docs on pre-release comparator handling +- #89: Test with new go versions (thanks @thedevsaddam) +- #87: Added $ to ValidPrerelease for better validation (thanks @jeremycarroll) + +## Fixed + +- #78: Fix unchecked error in example code (thanks @ravron) +- #70: Fix the handling of pre-releases and the 0.0.0 release edge case +- #97: Fixed copyright file for proper display on GitHub +- #107: Fix handling prerelease when sorting alphanum and num +- #109: Fixed where Validate sometimes returns wrong message on error + +# 1.4.2 (2018-04-10) + +## Changed +- #72: Updated the docs to point to vert for a console appliaction +- #71: Update the docs on pre-release comparator handling + +## Fixed +- #70: Fix the handling of pre-releases and the 0.0.0 release edge case + +# 1.4.1 (2018-04-02) + +## Fixed +- Fixed #64: Fix pre-release precedence issue (thanks @uudashr) + +# 1.4.0 (2017-10-04) + +## Changed +- #61: Update NewVersion to parse ints with a 64bit int size (thanks @zknill) + +# 1.3.1 (2017-07-10) + +## Fixed +- Fixed #57: number comparisons in prerelease sometimes inaccurate + +# 1.3.0 (2017-05-02) + +## Added +- #45: Added json (un)marshaling support (thanks @mh-cbon) +- Stability marker. See https://masterminds.github.io/stability/ + +## Fixed +- #51: Fix handling of single digit tilde constraint (thanks @dgodd) + +## Changed +- #55: The godoc icon moved from png to svg + +# 1.2.3 (2017-04-03) + +## Fixed +- #46: Fixed 0.x.x and 0.0.x in constraints being treated as * + +# Release 1.2.2 (2016-12-13) + +## Fixed +- #34: Fixed issue where hyphen range was not working with pre-release parsing. + +# Release 1.2.1 (2016-11-28) + +## Fixed +- #24: Fixed edge case issue where constraint "> 0" does not handle "0.0.1-alpha" + properly. + +# Release 1.2.0 (2016-11-04) + +## Added +- #20: Added MustParse function for versions (thanks @adamreese) +- #15: Added increment methods on versions (thanks @mh-cbon) + +## Fixed +- Issue #21: Per the SemVer spec (section 9) a pre-release is unstable and + might not satisfy the intended compatibility. The change here ignores pre-releases + on constraint checks (e.g., ~ or ^) when a pre-release is not part of the + constraint. For example, `^1.2.3` will ignore pre-releases while + `^1.2.3-alpha` will include them. + +# Release 1.1.1 (2016-06-30) + +## Changed +- Issue #9: Speed up version comparison performance (thanks @sdboyer) +- Issue #8: Added benchmarks (thanks @sdboyer) +- Updated Go Report Card URL to new location +- Updated Readme to add code snippet formatting (thanks @mh-cbon) +- Updating tagging to v[SemVer] structure for compatibility with other tools. + +# Release 1.1.0 (2016-03-11) + +- Issue #2: Implemented validation to provide reasons a versions failed a + constraint. + +# Release 1.0.1 (2015-12-31) + +- Fixed #1: * constraint failing on valid versions. + +# Release 1.0.0 (2015-10-20) + +- Initial release diff --git a/github.com/Masterminds/semver/LICENSE.txt b/github.com/Masterminds/semver/LICENSE.txt new file mode 100644 index 0000000000..9ff7da9c48 --- /dev/null +++ b/github.com/Masterminds/semver/LICENSE.txt @@ -0,0 +1,19 @@ +Copyright (C) 2014-2019, Matt Butcher and Matt Farina + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. diff --git a/github.com/Masterminds/semver/Makefile b/github.com/Masterminds/semver/Makefile new file mode 100644 index 0000000000..a7a1b4e36d --- /dev/null +++ b/github.com/Masterminds/semver/Makefile @@ -0,0 +1,36 @@ +.PHONY: setup +setup: + go get -u gopkg.in/alecthomas/gometalinter.v1 + gometalinter.v1 --install + +.PHONY: test +test: validate lint + @echo "==> Running tests" + go test -v + +.PHONY: validate +validate: + @echo "==> Running static validations" + @gometalinter.v1 \ + --disable-all \ + --enable deadcode \ + --severity deadcode:error \ + --enable gofmt \ + --enable gosimple \ + --enable ineffassign \ + --enable misspell \ + --enable vet \ + --tests \ + --vendor \ + --deadline 60s \ + ./... || exit_code=1 + +.PHONY: lint +lint: + @echo "==> Running linters" + @gometalinter.v1 \ + --disable-all \ + --enable golint \ + --vendor \ + --deadline 60s \ + ./... || : diff --git a/github.com/Masterminds/semver/README.md b/github.com/Masterminds/semver/README.md new file mode 100644 index 0000000000..1b52d2f436 --- /dev/null +++ b/github.com/Masterminds/semver/README.md @@ -0,0 +1,194 @@ +# SemVer + +The `semver` package provides the ability to work with [Semantic Versions](http://semver.org) in Go. Specifically it provides the ability to: + +* Parse semantic versions +* Sort semantic versions +* Check if a semantic version fits within a set of constraints +* Optionally work with a `v` prefix + +[![Stability: +Active](https://masterminds.github.io/stability/active.svg)](https://masterminds.github.io/stability/active.html) +[![Build Status](https://travis-ci.org/Masterminds/semver.svg)](https://travis-ci.org/Masterminds/semver) [![Build status](https://ci.appveyor.com/api/projects/status/jfk66lib7hb985k8/branch/master?svg=true&passingText=windows%20build%20passing&failingText=windows%20build%20failing)](https://ci.appveyor.com/project/mattfarina/semver/branch/master) [![GoDoc](https://godoc.org/github.com/Masterminds/semver?status.svg)](https://godoc.org/github.com/Masterminds/semver) [![Go Report Card](https://goreportcard.com/badge/github.com/Masterminds/semver)](https://goreportcard.com/report/github.com/Masterminds/semver) + +If you are looking for a command line tool for version comparisons please see +[vert](https://github.com/Masterminds/vert) which uses this library. + +## Parsing Semantic Versions + +To parse a semantic version use the `NewVersion` function. For example, + +```go + v, err := semver.NewVersion("1.2.3-beta.1+build345") +``` + +If there is an error the version wasn't parseable. The version object has methods +to get the parts of the version, compare it to other versions, convert the +version back into a string, and get the original string. For more details +please see the [documentation](https://godoc.org/github.com/Masterminds/semver). + +## Sorting Semantic Versions + +A set of versions can be sorted using the [`sort`](https://golang.org/pkg/sort/) +package from the standard library. For example, + +```go + raw := []string{"1.2.3", "1.0", "1.3", "2", "0.4.2",} + vs := make([]*semver.Version, len(raw)) + for i, r := range raw { + v, err := semver.NewVersion(r) + if err != nil { + t.Errorf("Error parsing version: %s", err) + } + + vs[i] = v + } + + sort.Sort(semver.Collection(vs)) +``` + +## Checking Version Constraints + +Checking a version against version constraints is one of the most featureful +parts of the package. + +```go + c, err := semver.NewConstraint(">= 1.2.3") + if err != nil { + // Handle constraint not being parseable. + } + + v, _ := semver.NewVersion("1.3") + if err != nil { + // Handle version not being parseable. + } + // Check if the version meets the constraints. The a variable will be true. + a := c.Check(v) +``` + +## Basic Comparisons + +There are two elements to the comparisons. First, a comparison string is a list +of comma separated and comparisons. These are then separated by || separated or +comparisons. For example, `">= 1.2, < 3.0.0 || >= 4.2.3"` is looking for a +comparison that's greater than or equal to 1.2 and less than 3.0.0 or is +greater than or equal to 4.2.3. + +The basic comparisons are: + +* `=`: equal (aliased to no operator) +* `!=`: not equal +* `>`: greater than +* `<`: less than +* `>=`: greater than or equal to +* `<=`: less than or equal to + +## Working With Pre-release Versions + +Pre-releases, for those not familiar with them, are used for software releases +prior to stable or generally available releases. Examples of pre-releases include +development, alpha, beta, and release candidate releases. A pre-release may be +a version such as `1.2.3-beta.1` while the stable release would be `1.2.3`. In the +order of precidence, pre-releases come before their associated releases. In this +example `1.2.3-beta.1 < 1.2.3`. + +According to the Semantic Version specification pre-releases may not be +API compliant with their release counterpart. It says, + +> A pre-release version indicates that the version is unstable and might not satisfy the intended compatibility requirements as denoted by its associated normal version. + +SemVer comparisons without a pre-release comparator will skip pre-release versions. +For example, `>=1.2.3` will skip pre-releases when looking at a list of releases +while `>=1.2.3-0` will evaluate and find pre-releases. + +The reason for the `0` as a pre-release version in the example comparison is +because pre-releases can only contain ASCII alphanumerics and hyphens (along with +`.` separators), per the spec. Sorting happens in ASCII sort order, again per the spec. The lowest character is a `0` in ASCII sort order (see an [ASCII Table](http://www.asciitable.com/)) + +Understanding ASCII sort ordering is important because A-Z comes before a-z. That +means `>=1.2.3-BETA` will return `1.2.3-alpha`. What you might expect from case +sensitivity doesn't apply here. This is due to ASCII sort ordering which is what +the spec specifies. + +## Hyphen Range Comparisons + +There are multiple methods to handle ranges and the first is hyphens ranges. +These look like: + +* `1.2 - 1.4.5` which is equivalent to `>= 1.2, <= 1.4.5` +* `2.3.4 - 4.5` which is equivalent to `>= 2.3.4, <= 4.5` + +## Wildcards In Comparisons + +The `x`, `X`, and `*` characters can be used as a wildcard character. This works +for all comparison operators. When used on the `=` operator it falls +back to the pack level comparison (see tilde below). For example, + +* `1.2.x` is equivalent to `>= 1.2.0, < 1.3.0` +* `>= 1.2.x` is equivalent to `>= 1.2.0` +* `<= 2.x` is equivalent to `< 3` +* `*` is equivalent to `>= 0.0.0` + +## Tilde Range Comparisons (Patch) + +The tilde (`~`) comparison operator is for patch level ranges when a minor +version is specified and major level changes when the minor number is missing. +For example, + +* `~1.2.3` is equivalent to `>= 1.2.3, < 1.3.0` +* `~1` is equivalent to `>= 1, < 2` +* `~2.3` is equivalent to `>= 2.3, < 2.4` +* `~1.2.x` is equivalent to `>= 1.2.0, < 1.3.0` +* `~1.x` is equivalent to `>= 1, < 2` + +## Caret Range Comparisons (Major) + +The caret (`^`) comparison operator is for major level changes. This is useful +when comparisons of API versions as a major change is API breaking. For example, + +* `^1.2.3` is equivalent to `>= 1.2.3, < 2.0.0` +* `^0.0.1` is equivalent to `>= 0.0.1, < 1.0.0` +* `^1.2.x` is equivalent to `>= 1.2.0, < 2.0.0` +* `^2.3` is equivalent to `>= 2.3, < 3` +* `^2.x` is equivalent to `>= 2.0.0, < 3` + +# Validation + +In addition to testing a version against a constraint, a version can be validated +against a constraint. When validation fails a slice of errors containing why a +version didn't meet the constraint is returned. For example, + +```go + c, err := semver.NewConstraint("<= 1.2.3, >= 1.4") + if err != nil { + // Handle constraint not being parseable. + } + + v, _ := semver.NewVersion("1.3") + if err != nil { + // Handle version not being parseable. + } + + // Validate a version against a constraint. + a, msgs := c.Validate(v) + // a is false + for _, m := range msgs { + fmt.Println(m) + + // Loops over the errors which would read + // "1.3 is greater than 1.2.3" + // "1.3 is less than 1.4" + } +``` + +# Fuzzing + + [dvyukov/go-fuzz](https://github.com/dvyukov/go-fuzz) is used for fuzzing. + +1. `go-fuzz-build` +2. `go-fuzz -workdir=fuzz` + +# Contribute + +If you find an issue or want to contribute please file an [issue](https://github.com/Masterminds/semver/issues) +or [create a pull request](https://github.com/Masterminds/semver/pulls). diff --git a/github.com/Masterminds/semver/appveyor.yml b/github.com/Masterminds/semver/appveyor.yml new file mode 100644 index 0000000000..b2778df15a --- /dev/null +++ b/github.com/Masterminds/semver/appveyor.yml @@ -0,0 +1,44 @@ +version: build-{build}.{branch} + +clone_folder: C:\gopath\src\github.com\Masterminds\semver +shallow_clone: true + +environment: + GOPATH: C:\gopath + +platform: + - x64 + +install: + - go version + - go env + - go get -u gopkg.in/alecthomas/gometalinter.v1 + - set PATH=%PATH%;%GOPATH%\bin + - gometalinter.v1.exe --install + +build_script: + - go install -v ./... + +test_script: + - "gometalinter.v1 \ + --disable-all \ + --enable deadcode \ + --severity deadcode:error \ + --enable gofmt \ + --enable gosimple \ + --enable ineffassign \ + --enable misspell \ + --enable vet \ + --tests \ + --vendor \ + --deadline 60s \ + ./... || exit_code=1" + - "gometalinter.v1 \ + --disable-all \ + --enable golint \ + --vendor \ + --deadline 60s \ + ./... || :" + - go test -v + +deploy: off diff --git a/github.com/Masterminds/semver/collection.go b/github.com/Masterminds/semver/collection.go new file mode 100644 index 0000000000..a78235895f --- /dev/null +++ b/github.com/Masterminds/semver/collection.go @@ -0,0 +1,24 @@ +package semver + +// Collection is a collection of Version instances and implements the sort +// interface. See the sort package for more details. +// https://golang.org/pkg/sort/ +type Collection []*Version + +// Len returns the length of a collection. The number of Version instances +// on the slice. +func (c Collection) Len() int { + return len(c) +} + +// Less is needed for the sort interface to compare two Version objects on the +// slice. If checks if one is less than the other. +func (c Collection) Less(i, j int) bool { + return c[i].LessThan(c[j]) +} + +// Swap is needed for the sort interface to replace the Version objects +// at two different positions in the slice. +func (c Collection) Swap(i, j int) { + c[i], c[j] = c[j], c[i] +} diff --git a/github.com/Masterminds/semver/constraints.go b/github.com/Masterminds/semver/constraints.go new file mode 100644 index 0000000000..b94b93413f --- /dev/null +++ b/github.com/Masterminds/semver/constraints.go @@ -0,0 +1,423 @@ +package semver + +import ( + "errors" + "fmt" + "regexp" + "strings" +) + +// Constraints is one or more constraint that a semantic version can be +// checked against. +type Constraints struct { + constraints [][]*constraint +} + +// NewConstraint returns a Constraints instance that a Version instance can +// be checked against. If there is a parse error it will be returned. +func NewConstraint(c string) (*Constraints, error) { + + // Rewrite - ranges into a comparison operation. + c = rewriteRange(c) + + ors := strings.Split(c, "||") + or := make([][]*constraint, len(ors)) + for k, v := range ors { + cs := strings.Split(v, ",") + result := make([]*constraint, len(cs)) + for i, s := range cs { + pc, err := parseConstraint(s) + if err != nil { + return nil, err + } + + result[i] = pc + } + or[k] = result + } + + o := &Constraints{constraints: or} + return o, nil +} + +// Check tests if a version satisfies the constraints. +func (cs Constraints) Check(v *Version) bool { + // loop over the ORs and check the inner ANDs + for _, o := range cs.constraints { + joy := true + for _, c := range o { + if !c.check(v) { + joy = false + break + } + } + + if joy { + return true + } + } + + return false +} + +// Validate checks if a version satisfies a constraint. If not a slice of +// reasons for the failure are returned in addition to a bool. +func (cs Constraints) Validate(v *Version) (bool, []error) { + // loop over the ORs and check the inner ANDs + var e []error + + // Capture the prerelease message only once. When it happens the first time + // this var is marked + var prerelesase bool + for _, o := range cs.constraints { + joy := true + for _, c := range o { + // Before running the check handle the case there the version is + // a prerelease and the check is not searching for prereleases. + if c.con.pre == "" && v.pre != "" { + if !prerelesase { + em := fmt.Errorf("%s is a prerelease version and the constraint is only looking for release versions", v) + e = append(e, em) + prerelesase = true + } + joy = false + + } else { + + if !c.check(v) { + em := fmt.Errorf(c.msg, v, c.orig) + e = append(e, em) + joy = false + } + } + } + + if joy { + return true, []error{} + } + } + + return false, e +} + +var constraintOps map[string]cfunc +var constraintMsg map[string]string +var constraintRegex *regexp.Regexp + +func init() { + constraintOps = map[string]cfunc{ + "": constraintTildeOrEqual, + "=": constraintTildeOrEqual, + "!=": constraintNotEqual, + ">": constraintGreaterThan, + "<": constraintLessThan, + ">=": constraintGreaterThanEqual, + "=>": constraintGreaterThanEqual, + "<=": constraintLessThanEqual, + "=<": constraintLessThanEqual, + "~": constraintTilde, + "~>": constraintTilde, + "^": constraintCaret, + } + + constraintMsg = map[string]string{ + "": "%s is not equal to %s", + "=": "%s is not equal to %s", + "!=": "%s is equal to %s", + ">": "%s is less than or equal to %s", + "<": "%s is greater than or equal to %s", + ">=": "%s is less than %s", + "=>": "%s is less than %s", + "<=": "%s is greater than %s", + "=<": "%s is greater than %s", + "~": "%s does not have same major and minor version as %s", + "~>": "%s does not have same major and minor version as %s", + "^": "%s does not have same major version as %s", + } + + ops := make([]string, 0, len(constraintOps)) + for k := range constraintOps { + ops = append(ops, regexp.QuoteMeta(k)) + } + + constraintRegex = regexp.MustCompile(fmt.Sprintf( + `^\s*(%s)\s*(%s)\s*$`, + strings.Join(ops, "|"), + cvRegex)) + + constraintRangeRegex = regexp.MustCompile(fmt.Sprintf( + `\s*(%s)\s+-\s+(%s)\s*`, + cvRegex, cvRegex)) +} + +// An individual constraint +type constraint struct { + // The callback function for the restraint. It performs the logic for + // the constraint. + function cfunc + + msg string + + // The version used in the constraint check. For example, if a constraint + // is '<= 2.0.0' the con a version instance representing 2.0.0. + con *Version + + // The original parsed version (e.g., 4.x from != 4.x) + orig string + + // When an x is used as part of the version (e.g., 1.x) + minorDirty bool + dirty bool + patchDirty bool +} + +// Check if a version meets the constraint +func (c *constraint) check(v *Version) bool { + return c.function(v, c) +} + +type cfunc func(v *Version, c *constraint) bool + +func parseConstraint(c string) (*constraint, error) { + m := constraintRegex.FindStringSubmatch(c) + if m == nil { + return nil, fmt.Errorf("improper constraint: %s", c) + } + + ver := m[2] + orig := ver + minorDirty := false + patchDirty := false + dirty := false + if isX(m[3]) { + ver = "0.0.0" + dirty = true + } else if isX(strings.TrimPrefix(m[4], ".")) || m[4] == "" { + minorDirty = true + dirty = true + ver = fmt.Sprintf("%s.0.0%s", m[3], m[6]) + } else if isX(strings.TrimPrefix(m[5], ".")) { + dirty = true + patchDirty = true + ver = fmt.Sprintf("%s%s.0%s", m[3], m[4], m[6]) + } + + con, err := NewVersion(ver) + if err != nil { + + // The constraintRegex should catch any regex parsing errors. So, + // we should never get here. + return nil, errors.New("constraint Parser Error") + } + + cs := &constraint{ + function: constraintOps[m[1]], + msg: constraintMsg[m[1]], + con: con, + orig: orig, + minorDirty: minorDirty, + patchDirty: patchDirty, + dirty: dirty, + } + return cs, nil +} + +// Constraint functions +func constraintNotEqual(v *Version, c *constraint) bool { + if c.dirty { + + // If there is a pre-release on the version but the constraint isn't looking + // for them assume that pre-releases are not compatible. See issue 21 for + // more details. + if v.Prerelease() != "" && c.con.Prerelease() == "" { + return false + } + + if c.con.Major() != v.Major() { + return true + } + if c.con.Minor() != v.Minor() && !c.minorDirty { + return true + } else if c.minorDirty { + return false + } + + return false + } + + return !v.Equal(c.con) +} + +func constraintGreaterThan(v *Version, c *constraint) bool { + + // If there is a pre-release on the version but the constraint isn't looking + // for them assume that pre-releases are not compatible. See issue 21 for + // more details. + if v.Prerelease() != "" && c.con.Prerelease() == "" { + return false + } + + return v.Compare(c.con) == 1 +} + +func constraintLessThan(v *Version, c *constraint) bool { + // If there is a pre-release on the version but the constraint isn't looking + // for them assume that pre-releases are not compatible. See issue 21 for + // more details. + if v.Prerelease() != "" && c.con.Prerelease() == "" { + return false + } + + if !c.dirty { + return v.Compare(c.con) < 0 + } + + if v.Major() > c.con.Major() { + return false + } else if v.Minor() > c.con.Minor() && !c.minorDirty { + return false + } + + return true +} + +func constraintGreaterThanEqual(v *Version, c *constraint) bool { + + // If there is a pre-release on the version but the constraint isn't looking + // for them assume that pre-releases are not compatible. See issue 21 for + // more details. + if v.Prerelease() != "" && c.con.Prerelease() == "" { + return false + } + + return v.Compare(c.con) >= 0 +} + +func constraintLessThanEqual(v *Version, c *constraint) bool { + // If there is a pre-release on the version but the constraint isn't looking + // for them assume that pre-releases are not compatible. See issue 21 for + // more details. + if v.Prerelease() != "" && c.con.Prerelease() == "" { + return false + } + + if !c.dirty { + return v.Compare(c.con) <= 0 + } + + if v.Major() > c.con.Major() { + return false + } else if v.Minor() > c.con.Minor() && !c.minorDirty { + return false + } + + return true +} + +// ~*, ~>* --> >= 0.0.0 (any) +// ~2, ~2.x, ~2.x.x, ~>2, ~>2.x ~>2.x.x --> >=2.0.0, <3.0.0 +// ~2.0, ~2.0.x, ~>2.0, ~>2.0.x --> >=2.0.0, <2.1.0 +// ~1.2, ~1.2.x, ~>1.2, ~>1.2.x --> >=1.2.0, <1.3.0 +// ~1.2.3, ~>1.2.3 --> >=1.2.3, <1.3.0 +// ~1.2.0, ~>1.2.0 --> >=1.2.0, <1.3.0 +func constraintTilde(v *Version, c *constraint) bool { + // If there is a pre-release on the version but the constraint isn't looking + // for them assume that pre-releases are not compatible. See issue 21 for + // more details. + if v.Prerelease() != "" && c.con.Prerelease() == "" { + return false + } + + if v.LessThan(c.con) { + return false + } + + // ~0.0.0 is a special case where all constraints are accepted. It's + // equivalent to >= 0.0.0. + if c.con.Major() == 0 && c.con.Minor() == 0 && c.con.Patch() == 0 && + !c.minorDirty && !c.patchDirty { + return true + } + + if v.Major() != c.con.Major() { + return false + } + + if v.Minor() != c.con.Minor() && !c.minorDirty { + return false + } + + return true +} + +// When there is a .x (dirty) status it automatically opts in to ~. Otherwise +// it's a straight = +func constraintTildeOrEqual(v *Version, c *constraint) bool { + // If there is a pre-release on the version but the constraint isn't looking + // for them assume that pre-releases are not compatible. See issue 21 for + // more details. + if v.Prerelease() != "" && c.con.Prerelease() == "" { + return false + } + + if c.dirty { + c.msg = constraintMsg["~"] + return constraintTilde(v, c) + } + + return v.Equal(c.con) +} + +// ^* --> (any) +// ^2, ^2.x, ^2.x.x --> >=2.0.0, <3.0.0 +// ^2.0, ^2.0.x --> >=2.0.0, <3.0.0 +// ^1.2, ^1.2.x --> >=1.2.0, <2.0.0 +// ^1.2.3 --> >=1.2.3, <2.0.0 +// ^1.2.0 --> >=1.2.0, <2.0.0 +func constraintCaret(v *Version, c *constraint) bool { + // If there is a pre-release on the version but the constraint isn't looking + // for them assume that pre-releases are not compatible. See issue 21 for + // more details. + if v.Prerelease() != "" && c.con.Prerelease() == "" { + return false + } + + if v.LessThan(c.con) { + return false + } + + if v.Major() != c.con.Major() { + return false + } + + return true +} + +var constraintRangeRegex *regexp.Regexp + +const cvRegex string = `v?([0-9|x|X|\*]+)(\.[0-9|x|X|\*]+)?(\.[0-9|x|X|\*]+)?` + + `(-([0-9A-Za-z\-]+(\.[0-9A-Za-z\-]+)*))?` + + `(\+([0-9A-Za-z\-]+(\.[0-9A-Za-z\-]+)*))?` + +func isX(x string) bool { + switch x { + case "x", "*", "X": + return true + default: + return false + } +} + +func rewriteRange(i string) string { + m := constraintRangeRegex.FindAllStringSubmatch(i, -1) + if m == nil { + return i + } + o := i + for _, v := range m { + t := fmt.Sprintf(">= %s, <= %s", v[1], v[11]) + o = strings.Replace(o, v[0], t, 1) + } + + return o +} diff --git a/github.com/Masterminds/semver/doc.go b/github.com/Masterminds/semver/doc.go new file mode 100644 index 0000000000..6a6c24c6d6 --- /dev/null +++ b/github.com/Masterminds/semver/doc.go @@ -0,0 +1,115 @@ +/* +Package semver provides the ability to work with Semantic Versions (http://semver.org) in Go. + +Specifically it provides the ability to: + + * Parse semantic versions + * Sort semantic versions + * Check if a semantic version fits within a set of constraints + * Optionally work with a `v` prefix + +Parsing Semantic Versions + +To parse a semantic version use the `NewVersion` function. For example, + + v, err := semver.NewVersion("1.2.3-beta.1+build345") + +If there is an error the version wasn't parseable. The version object has methods +to get the parts of the version, compare it to other versions, convert the +version back into a string, and get the original string. For more details +please see the documentation at https://godoc.org/github.com/Masterminds/semver. + +Sorting Semantic Versions + +A set of versions can be sorted using the `sort` package from the standard library. +For example, + + raw := []string{"1.2.3", "1.0", "1.3", "2", "0.4.2",} + vs := make([]*semver.Version, len(raw)) + for i, r := range raw { + v, err := semver.NewVersion(r) + if err != nil { + t.Errorf("Error parsing version: %s", err) + } + + vs[i] = v + } + + sort.Sort(semver.Collection(vs)) + +Checking Version Constraints + +Checking a version against version constraints is one of the most featureful +parts of the package. + + c, err := semver.NewConstraint(">= 1.2.3") + if err != nil { + // Handle constraint not being parseable. + } + + v, err := semver.NewVersion("1.3") + if err != nil { + // Handle version not being parseable. + } + // Check if the version meets the constraints. The a variable will be true. + a := c.Check(v) + +Basic Comparisons + +There are two elements to the comparisons. First, a comparison string is a list +of comma separated and comparisons. These are then separated by || separated or +comparisons. For example, `">= 1.2, < 3.0.0 || >= 4.2.3"` is looking for a +comparison that's greater than or equal to 1.2 and less than 3.0.0 or is +greater than or equal to 4.2.3. + +The basic comparisons are: + + * `=`: equal (aliased to no operator) + * `!=`: not equal + * `>`: greater than + * `<`: less than + * `>=`: greater than or equal to + * `<=`: less than or equal to + +Hyphen Range Comparisons + +There are multiple methods to handle ranges and the first is hyphens ranges. +These look like: + + * `1.2 - 1.4.5` which is equivalent to `>= 1.2, <= 1.4.5` + * `2.3.4 - 4.5` which is equivalent to `>= 2.3.4, <= 4.5` + +Wildcards In Comparisons + +The `x`, `X`, and `*` characters can be used as a wildcard character. This works +for all comparison operators. When used on the `=` operator it falls +back to the pack level comparison (see tilde below). For example, + + * `1.2.x` is equivalent to `>= 1.2.0, < 1.3.0` + * `>= 1.2.x` is equivalent to `>= 1.2.0` + * `<= 2.x` is equivalent to `<= 3` + * `*` is equivalent to `>= 0.0.0` + +Tilde Range Comparisons (Patch) + +The tilde (`~`) comparison operator is for patch level ranges when a minor +version is specified and major level changes when the minor number is missing. +For example, + + * `~1.2.3` is equivalent to `>= 1.2.3, < 1.3.0` + * `~1` is equivalent to `>= 1, < 2` + * `~2.3` is equivalent to `>= 2.3, < 2.4` + * `~1.2.x` is equivalent to `>= 1.2.0, < 1.3.0` + * `~1.x` is equivalent to `>= 1, < 2` + +Caret Range Comparisons (Major) + +The caret (`^`) comparison operator is for major level changes. This is useful +when comparisons of API versions as a major change is API breaking. For example, + + * `^1.2.3` is equivalent to `>= 1.2.3, < 2.0.0` + * `^1.2.x` is equivalent to `>= 1.2.0, < 2.0.0` + * `^2.3` is equivalent to `>= 2.3, < 3` + * `^2.x` is equivalent to `>= 2.0.0, < 3` +*/ +package semver diff --git a/github.com/Masterminds/semver/version.go b/github.com/Masterminds/semver/version.go new file mode 100644 index 0000000000..400d4f9341 --- /dev/null +++ b/github.com/Masterminds/semver/version.go @@ -0,0 +1,425 @@ +package semver + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "regexp" + "strconv" + "strings" +) + +// The compiled version of the regex created at init() is cached here so it +// only needs to be created once. +var versionRegex *regexp.Regexp +var validPrereleaseRegex *regexp.Regexp + +var ( + // ErrInvalidSemVer is returned a version is found to be invalid when + // being parsed. + ErrInvalidSemVer = errors.New("Invalid Semantic Version") + + // ErrInvalidMetadata is returned when the metadata is an invalid format + ErrInvalidMetadata = errors.New("Invalid Metadata string") + + // ErrInvalidPrerelease is returned when the pre-release is an invalid format + ErrInvalidPrerelease = errors.New("Invalid Prerelease string") +) + +// SemVerRegex is the regular expression used to parse a semantic version. +const SemVerRegex string = `v?([0-9]+)(\.[0-9]+)?(\.[0-9]+)?` + + `(-([0-9A-Za-z\-]+(\.[0-9A-Za-z\-]+)*))?` + + `(\+([0-9A-Za-z\-]+(\.[0-9A-Za-z\-]+)*))?` + +// ValidPrerelease is the regular expression which validates +// both prerelease and metadata values. +const ValidPrerelease string = `^([0-9A-Za-z\-]+(\.[0-9A-Za-z\-]+)*)$` + +// Version represents a single semantic version. +type Version struct { + major, minor, patch int64 + pre string + metadata string + original string +} + +func init() { + versionRegex = regexp.MustCompile("^" + SemVerRegex + "$") + validPrereleaseRegex = regexp.MustCompile(ValidPrerelease) +} + +// NewVersion parses a given version and returns an instance of Version or +// an error if unable to parse the version. +func NewVersion(v string) (*Version, error) { + m := versionRegex.FindStringSubmatch(v) + if m == nil { + return nil, ErrInvalidSemVer + } + + sv := &Version{ + metadata: m[8], + pre: m[5], + original: v, + } + + var temp int64 + temp, err := strconv.ParseInt(m[1], 10, 64) + if err != nil { + return nil, fmt.Errorf("Error parsing version segment: %s", err) + } + sv.major = temp + + if m[2] != "" { + temp, err = strconv.ParseInt(strings.TrimPrefix(m[2], "."), 10, 64) + if err != nil { + return nil, fmt.Errorf("Error parsing version segment: %s", err) + } + sv.minor = temp + } else { + sv.minor = 0 + } + + if m[3] != "" { + temp, err = strconv.ParseInt(strings.TrimPrefix(m[3], "."), 10, 64) + if err != nil { + return nil, fmt.Errorf("Error parsing version segment: %s", err) + } + sv.patch = temp + } else { + sv.patch = 0 + } + + return sv, nil +} + +// MustParse parses a given version and panics on error. +func MustParse(v string) *Version { + sv, err := NewVersion(v) + if err != nil { + panic(err) + } + return sv +} + +// String converts a Version object to a string. +// Note, if the original version contained a leading v this version will not. +// See the Original() method to retrieve the original value. Semantic Versions +// don't contain a leading v per the spec. Instead it's optional on +// implementation. +func (v *Version) String() string { + var buf bytes.Buffer + + fmt.Fprintf(&buf, "%d.%d.%d", v.major, v.minor, v.patch) + if v.pre != "" { + fmt.Fprintf(&buf, "-%s", v.pre) + } + if v.metadata != "" { + fmt.Fprintf(&buf, "+%s", v.metadata) + } + + return buf.String() +} + +// Original returns the original value passed in to be parsed. +func (v *Version) Original() string { + return v.original +} + +// Major returns the major version. +func (v *Version) Major() int64 { + return v.major +} + +// Minor returns the minor version. +func (v *Version) Minor() int64 { + return v.minor +} + +// Patch returns the patch version. +func (v *Version) Patch() int64 { + return v.patch +} + +// Prerelease returns the pre-release version. +func (v *Version) Prerelease() string { + return v.pre +} + +// Metadata returns the metadata on the version. +func (v *Version) Metadata() string { + return v.metadata +} + +// originalVPrefix returns the original 'v' prefix if any. +func (v *Version) originalVPrefix() string { + + // Note, only lowercase v is supported as a prefix by the parser. + if v.original != "" && v.original[:1] == "v" { + return v.original[:1] + } + return "" +} + +// IncPatch produces the next patch version. +// If the current version does not have prerelease/metadata information, +// it unsets metadata and prerelease values, increments patch number. +// If the current version has any of prerelease or metadata information, +// it unsets both values and keeps curent patch value +func (v Version) IncPatch() Version { + vNext := v + // according to http://semver.org/#spec-item-9 + // Pre-release versions have a lower precedence than the associated normal version. + // according to http://semver.org/#spec-item-10 + // Build metadata SHOULD be ignored when determining version precedence. + if v.pre != "" { + vNext.metadata = "" + vNext.pre = "" + } else { + vNext.metadata = "" + vNext.pre = "" + vNext.patch = v.patch + 1 + } + vNext.original = v.originalVPrefix() + "" + vNext.String() + return vNext +} + +// IncMinor produces the next minor version. +// Sets patch to 0. +// Increments minor number. +// Unsets metadata. +// Unsets prerelease status. +func (v Version) IncMinor() Version { + vNext := v + vNext.metadata = "" + vNext.pre = "" + vNext.patch = 0 + vNext.minor = v.minor + 1 + vNext.original = v.originalVPrefix() + "" + vNext.String() + return vNext +} + +// IncMajor produces the next major version. +// Sets patch to 0. +// Sets minor to 0. +// Increments major number. +// Unsets metadata. +// Unsets prerelease status. +func (v Version) IncMajor() Version { + vNext := v + vNext.metadata = "" + vNext.pre = "" + vNext.patch = 0 + vNext.minor = 0 + vNext.major = v.major + 1 + vNext.original = v.originalVPrefix() + "" + vNext.String() + return vNext +} + +// SetPrerelease defines the prerelease value. +// Value must not include the required 'hypen' prefix. +func (v Version) SetPrerelease(prerelease string) (Version, error) { + vNext := v + if len(prerelease) > 0 && !validPrereleaseRegex.MatchString(prerelease) { + return vNext, ErrInvalidPrerelease + } + vNext.pre = prerelease + vNext.original = v.originalVPrefix() + "" + vNext.String() + return vNext, nil +} + +// SetMetadata defines metadata value. +// Value must not include the required 'plus' prefix. +func (v Version) SetMetadata(metadata string) (Version, error) { + vNext := v + if len(metadata) > 0 && !validPrereleaseRegex.MatchString(metadata) { + return vNext, ErrInvalidMetadata + } + vNext.metadata = metadata + vNext.original = v.originalVPrefix() + "" + vNext.String() + return vNext, nil +} + +// LessThan tests if one version is less than another one. +func (v *Version) LessThan(o *Version) bool { + return v.Compare(o) < 0 +} + +// GreaterThan tests if one version is greater than another one. +func (v *Version) GreaterThan(o *Version) bool { + return v.Compare(o) > 0 +} + +// Equal tests if two versions are equal to each other. +// Note, versions can be equal with different metadata since metadata +// is not considered part of the comparable version. +func (v *Version) Equal(o *Version) bool { + return v.Compare(o) == 0 +} + +// Compare compares this version to another one. It returns -1, 0, or 1 if +// the version smaller, equal, or larger than the other version. +// +// Versions are compared by X.Y.Z. Build metadata is ignored. Prerelease is +// lower than the version without a prerelease. +func (v *Version) Compare(o *Version) int { + // Compare the major, minor, and patch version for differences. If a + // difference is found return the comparison. + if d := compareSegment(v.Major(), o.Major()); d != 0 { + return d + } + if d := compareSegment(v.Minor(), o.Minor()); d != 0 { + return d + } + if d := compareSegment(v.Patch(), o.Patch()); d != 0 { + return d + } + + // At this point the major, minor, and patch versions are the same. + ps := v.pre + po := o.Prerelease() + + if ps == "" && po == "" { + return 0 + } + if ps == "" { + return 1 + } + if po == "" { + return -1 + } + + return comparePrerelease(ps, po) +} + +// UnmarshalJSON implements JSON.Unmarshaler interface. +func (v *Version) UnmarshalJSON(b []byte) error { + var s string + if err := json.Unmarshal(b, &s); err != nil { + return err + } + temp, err := NewVersion(s) + if err != nil { + return err + } + v.major = temp.major + v.minor = temp.minor + v.patch = temp.patch + v.pre = temp.pre + v.metadata = temp.metadata + v.original = temp.original + temp = nil + return nil +} + +// MarshalJSON implements JSON.Marshaler interface. +func (v *Version) MarshalJSON() ([]byte, error) { + return json.Marshal(v.String()) +} + +func compareSegment(v, o int64) int { + if v < o { + return -1 + } + if v > o { + return 1 + } + + return 0 +} + +func comparePrerelease(v, o string) int { + + // split the prelease versions by their part. The separator, per the spec, + // is a . + sparts := strings.Split(v, ".") + oparts := strings.Split(o, ".") + + // Find the longer length of the parts to know how many loop iterations to + // go through. + slen := len(sparts) + olen := len(oparts) + + l := slen + if olen > slen { + l = olen + } + + // Iterate over each part of the prereleases to compare the differences. + for i := 0; i < l; i++ { + // Since the lentgh of the parts can be different we need to create + // a placeholder. This is to avoid out of bounds issues. + stemp := "" + if i < slen { + stemp = sparts[i] + } + + otemp := "" + if i < olen { + otemp = oparts[i] + } + + d := comparePrePart(stemp, otemp) + if d != 0 { + return d + } + } + + // Reaching here means two versions are of equal value but have different + // metadata (the part following a +). They are not identical in string form + // but the version comparison finds them to be equal. + return 0 +} + +func comparePrePart(s, o string) int { + // Fastpath if they are equal + if s == o { + return 0 + } + + // When s or o are empty we can use the other in an attempt to determine + // the response. + if s == "" { + if o != "" { + return -1 + } + return 1 + } + + if o == "" { + if s != "" { + return 1 + } + return -1 + } + + // When comparing strings "99" is greater than "103". To handle + // cases like this we need to detect numbers and compare them. According + // to the semver spec, numbers are always positive. If there is a - at the + // start like -99 this is to be evaluated as an alphanum. numbers always + // have precedence over alphanum. Parsing as Uints because negative numbers + // are ignored. + + oi, n1 := strconv.ParseUint(o, 10, 64) + si, n2 := strconv.ParseUint(s, 10, 64) + + // The case where both are strings compare the strings + if n1 != nil && n2 != nil { + if s > o { + return 1 + } + return -1 + } else if n1 != nil { + // o is a string and s is a number + return -1 + } else if n2 != nil { + // s is a string and o is a number + return 1 + } + // Both are numbers + if si > oi { + return 1 + } + return -1 + +} diff --git a/github.com/Masterminds/semver/version_fuzz.go b/github.com/Masterminds/semver/version_fuzz.go new file mode 100644 index 0000000000..b42bcd62b9 --- /dev/null +++ b/github.com/Masterminds/semver/version_fuzz.go @@ -0,0 +1,10 @@ +// +build gofuzz + +package semver + +func Fuzz(data []byte) int { + if _, err := NewVersion(string(data)); err != nil { + return 0 + } + return 1 +} diff --git a/github.com/Masterminds/sprig/.gitignore b/github.com/Masterminds/sprig/.gitignore new file mode 100644 index 0000000000..5e3002f88f --- /dev/null +++ b/github.com/Masterminds/sprig/.gitignore @@ -0,0 +1,2 @@ +vendor/ +/.glide diff --git a/github.com/Masterminds/sprig/.travis.yml b/github.com/Masterminds/sprig/.travis.yml new file mode 100644 index 0000000000..b9da8b825b --- /dev/null +++ b/github.com/Masterminds/sprig/.travis.yml @@ -0,0 +1,26 @@ +language: go + +go: + - 1.9.x + - 1.10.x + - 1.11.x + - 1.12.x + - 1.13.x + - tip + +# Setting sudo access to false will let Travis CI use containers rather than +# VMs to run the tests. For more details see: +# - http://docs.travis-ci.com/user/workers/container-based-infrastructure/ +# - http://docs.travis-ci.com/user/workers/standard-infrastructure/ +sudo: false + +script: + - make setup test + +notifications: + webhooks: + urls: + - https://webhooks.gitter.im/e/06e3328629952dabe3e0 + on_success: change # options: [always|never|change] default: always + on_failure: always # options: [always|never|change] default: always + on_start: never # options: [always|never|change] default: always diff --git a/github.com/Masterminds/sprig/CHANGELOG.md b/github.com/Masterminds/sprig/CHANGELOG.md new file mode 100644 index 0000000000..6a79fbde46 --- /dev/null +++ b/github.com/Masterminds/sprig/CHANGELOG.md @@ -0,0 +1,282 @@ +# Changelog + +## Release 2.22.0 (2019-10-02) + +### Added + +- #173: Added getHostByName function to resolve dns names to ips (thanks @fcgravalos) +- #195: Added deepCopy function for use with dicts + +### Changed + +- Updated merge and mergeOverwrite documentation to explain copying and how to + use deepCopy with it + +## Release 2.21.0 (2019-09-18) + +### Added + +- #122: Added encryptAES/decryptAES functions (thanks @n0madic) +- #128: Added toDecimal support (thanks @Dean-Coakley) +- #169: Added list contcat (thanks @astorath) +- #174: Added deepEqual function (thanks @bonifaido) +- #170: Added url parse and join functions (thanks @astorath) + +### Changed + +- #171: Updated glide config for Google UUID to v1 and to add ranges to semver and testify + +### Fixed + +- #172: Fix semver wildcard example (thanks @piepmatz) +- #175: Fix dateInZone doc example (thanks @s3than) + +## Release 2.20.0 (2019-06-18) + +### Added + +- #164: Adding function to get unix epoch for a time (@mattfarina) +- #166: Adding tests for date_in_zone (@mattfarina) + +### Changed + +- #144: Fix function comments based on best practices from Effective Go (@CodeLingoTeam) +- #150: Handles pointer type for time.Time in "htmlDate" (@mapreal19) +- #161, #157, #160, #153, #158, #156, #155, #159, #152 documentation updates (@badeadan) + +### Fixed + +## Release 2.19.0 (2019-03-02) + +IMPORTANT: This release reverts a change from 2.18.0 + +In the previous release (2.18), we prematurely merged a partial change to the crypto functions that led to creating two sets of crypto functions (I blame @technosophos -- since that's me). This release rolls back that change, and does what was originally intended: It alters the existing crypto functions to use secure random. + +We debated whether this classifies as a change worthy of major revision, but given the proximity to the last release, we have decided that treating 2.18 as a faulty release is the correct course of action. We apologize for any inconvenience. + +### Changed + +- Fix substr panic 35fb796 (Alexey igrychev) +- Remove extra period 1eb7729 (Matthew Lorimor) +- Make random string functions use crypto by default 6ceff26 (Matthew Lorimor) +- README edits/fixes/suggestions 08fe136 (Lauri Apple) + + +## Release 2.18.0 (2019-02-12) + +### Added + +- Added mergeOverwrite function +- cryptographic functions that use secure random (see fe1de12) + +### Changed + +- Improve documentation of regexMatch function, resolves #139 90b89ce (Jan Tagscherer) +- Handle has for nil list 9c10885 (Daniel Cohen) +- Document behaviour of mergeOverwrite fe0dbe9 (Lukas Rieder) +- doc: adds missing documentation. 4b871e6 (Fernandez Ludovic) +- Replace outdated goutils imports 01893d2 (Matthew Lorimor) +- Surface crypto secure random strings from goutils fe1de12 (Matthew Lorimor) +- Handle untyped nil values as paramters to string functions 2b2ec8f (Morten Torkildsen) + +### Fixed + +- Fix dict merge issue and provide mergeOverwrite .dst .src1 to overwrite from src -> dst 4c59c12 (Lukas Rieder) +- Fix substr var names and comments d581f80 (Dean Coakley) +- Fix substr documentation 2737203 (Dean Coakley) + +## Release 2.17.1 (2019-01-03) + +### Fixed + +The 2.17.0 release did not have a version pinned for xstrings, which caused compilation failures when xstrings < 1.2 was used. This adds the correct version string to glide.yaml. + +## Release 2.17.0 (2019-01-03) + +### Added + +- adds alder32sum function and test 6908fc2 (marshallford) +- Added kebabcase function ca331a1 (Ilyes512) + +### Changed + +- Update goutils to 1.1.0 4e1125d (Matt Butcher) + +### Fixed + +- Fix 'has' documentation e3f2a85 (dean-coakley) +- docs(dict): fix typo in pick example dc424f9 (Dustin Specker) +- fixes spelling errors... not sure how that happened 4cf188a (marshallford) + +## Release 2.16.0 (2018-08-13) + +### Added + +- add splitn function fccb0b0 (Helgi Þorbjörnsson) +- Add slice func df28ca7 (gongdo) +- Generate serial number a3bdffd (Cody Coons) +- Extract values of dict with values function df39312 (Lawrence Jones) + +### Changed + +- Modify panic message for list.slice ae38335 (gongdo) +- Minor improvement in code quality - Removed an unreachable piece of code at defaults.go#L26:6 - Resolve formatting issues. 5834241 (Abhishek Kashyap) +- Remove duplicated documentation 1d97af1 (Matthew Fisher) +- Test on go 1.11 49df809 (Helgi Þormar Þorbjörnsson) + +### Fixed + +- Fix file permissions c5f40b5 (gongdo) +- Fix example for buildCustomCert 7779e0d (Tin Lam) + +## Release 2.15.0 (2018-04-02) + +### Added + +- #68 and #69: Add json helpers to docs (thanks @arunvelsriram) +- #66: Add ternary function (thanks @binoculars) +- #67: Allow keys function to take multiple dicts (thanks @binoculars) +- #89: Added sha1sum to crypto function (thanks @benkeil) +- #81: Allow customizing Root CA that used by genSignedCert (thanks @chenzhiwei) +- #92: Add travis testing for go 1.10 +- #93: Adding appveyor config for windows testing + +### Changed + +- #90: Updating to more recent dependencies +- #73: replace satori/go.uuid with google/uuid (thanks @petterw) + +### Fixed + +- #76: Fixed documentation typos (thanks @Thiht) +- Fixed rounding issue on the `ago` function. Note, the removes support for Go 1.8 and older + +## Release 2.14.1 (2017-12-01) + +### Fixed + +- #60: Fix typo in function name documentation (thanks @neil-ca-moore) +- #61: Removing line with {{ due to blocking github pages genertion +- #64: Update the list functions to handle int, string, and other slices for compatibility + +## Release 2.14.0 (2017-10-06) + +This new version of Sprig adds a set of functions for generating and working with SSL certificates. + +- `genCA` generates an SSL Certificate Authority +- `genSelfSignedCert` generates an SSL self-signed certificate +- `genSignedCert` generates an SSL certificate and key based on a given CA + +## Release 2.13.0 (2017-09-18) + +This release adds new functions, including: + +- `regexMatch`, `regexFindAll`, `regexFind`, `regexReplaceAll`, `regexReplaceAllLiteral`, and `regexSplit` to work with regular expressions +- `floor`, `ceil`, and `round` math functions +- `toDate` converts a string to a date +- `nindent` is just like `indent` but also prepends a new line +- `ago` returns the time from `time.Now` + +### Added + +- #40: Added basic regex functionality (thanks @alanquillin) +- #41: Added ceil floor and round functions (thanks @alanquillin) +- #48: Added toDate function (thanks @andreynering) +- #50: Added nindent function (thanks @binoculars) +- #46: Added ago function (thanks @slayer) + +### Changed + +- #51: Updated godocs to include new string functions (thanks @curtisallen) +- #49: Added ability to merge multiple dicts (thanks @binoculars) + +## Release 2.12.0 (2017-05-17) + +- `snakecase`, `camelcase`, and `shuffle` are three new string functions +- `fail` allows you to bail out of a template render when conditions are not met + +## Release 2.11.0 (2017-05-02) + +- Added `toJson` and `toPrettyJson` +- Added `merge` +- Refactored documentation + +## Release 2.10.0 (2017-03-15) + +- Added `semver` and `semverCompare` for Semantic Versions +- `list` replaces `tuple` +- Fixed issue with `join` +- Added `first`, `last`, `intial`, `rest`, `prepend`, `append`, `toString`, `toStrings`, `sortAlpha`, `reverse`, `coalesce`, `pluck`, `pick`, `compact`, `keys`, `omit`, `uniq`, `has`, `without` + +## Release 2.9.0 (2017-02-23) + +- Added `splitList` to split a list +- Added crypto functions of `genPrivateKey` and `derivePassword` + +## Release 2.8.0 (2016-12-21) + +- Added access to several path functions (`base`, `dir`, `clean`, `ext`, and `abs`) +- Added functions for _mutating_ dictionaries (`set`, `unset`, `hasKey`) + +## Release 2.7.0 (2016-12-01) + +- Added `sha256sum` to generate a hash of an input +- Added functions to convert a numeric or string to `int`, `int64`, `float64` + +## Release 2.6.0 (2016-10-03) + +- Added a `uuidv4` template function for generating UUIDs inside of a template. + +## Release 2.5.0 (2016-08-19) + +- New `trimSuffix`, `trimPrefix`, `hasSuffix`, and `hasPrefix` functions +- New aliases have been added for a few functions that didn't follow the naming conventions (`trimAll` and `abbrevBoth`) +- `trimall` and `abbrevboth` (notice the case) are deprecated and will be removed in 3.0.0 + +## Release 2.4.0 (2016-08-16) + +- Adds two functions: `until` and `untilStep` + +## Release 2.3.0 (2016-06-21) + +- cat: Concatenate strings with whitespace separators. +- replace: Replace parts of a string: `replace " " "-" "Me First"` renders "Me-First" +- plural: Format plurals: `len "foo" | plural "one foo" "many foos"` renders "many foos" +- indent: Indent blocks of text in a way that is sensitive to "\n" characters. + +## Release 2.2.0 (2016-04-21) + +- Added a `genPrivateKey` function (Thanks @bacongobbler) + +## Release 2.1.0 (2016-03-30) + +- `default` now prints the default value when it does not receive a value down the pipeline. It is much safer now to do `{{.Foo | default "bar"}}`. +- Added accessors for "hermetic" functions. These return only functions that, when given the same input, produce the same output. + +## Release 2.0.0 (2016-03-29) + +Because we switched from `int` to `int64` as the return value for all integer math functions, the library's major version number has been incremented. + +- `min` complements `max` (formerly `biggest`) +- `empty` indicates that a value is the empty value for its type +- `tuple` creates a tuple inside of a template: `{{$t := tuple "a", "b" "c"}}` +- `dict` creates a dictionary inside of a template `{{$d := dict "key1" "val1" "key2" "val2"}}` +- Date formatters have been added for HTML dates (as used in `date` input fields) +- Integer math functions can convert from a number of types, including `string` (via `strconv.ParseInt`). + +## Release 1.2.0 (2016-02-01) + +- Added quote and squote +- Added b32enc and b32dec +- add now takes varargs +- biggest now takes varargs + +## Release 1.1.0 (2015-12-29) + +- Added #4: Added contains function. strings.Contains, but with the arguments + switched to simplify common pipelines. (thanks krancour) +- Added Travis-CI testing support + +## Release 1.0.0 (2015-12-23) + +- Initial release diff --git a/github.com/Masterminds/sprig/LICENSE.txt b/github.com/Masterminds/sprig/LICENSE.txt new file mode 100644 index 0000000000..5c95accc2e --- /dev/null +++ b/github.com/Masterminds/sprig/LICENSE.txt @@ -0,0 +1,20 @@ +Sprig +Copyright (C) 2013 Masterminds + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN +THE SOFTWARE. diff --git a/github.com/Masterminds/sprig/Makefile b/github.com/Masterminds/sprig/Makefile new file mode 100644 index 0000000000..63a93fdf79 --- /dev/null +++ b/github.com/Masterminds/sprig/Makefile @@ -0,0 +1,13 @@ + +HAS_GLIDE := $(shell command -v glide;) + +.PHONY: test +test: + go test -v . + +.PHONY: setup +setup: +ifndef HAS_GLIDE + go get -u github.com/Masterminds/glide +endif + glide install diff --git a/github.com/Masterminds/sprig/README.md b/github.com/Masterminds/sprig/README.md new file mode 100644 index 0000000000..b70569585f --- /dev/null +++ b/github.com/Masterminds/sprig/README.md @@ -0,0 +1,78 @@ +# Sprig: Template functions for Go templates +[![Stability: Sustained](https://masterminds.github.io/stability/sustained.svg)](https://masterminds.github.io/stability/sustained.html) +[![Build Status](https://travis-ci.org/Masterminds/sprig.svg?branch=master)](https://travis-ci.org/Masterminds/sprig) + +The Go language comes with a [built-in template +language](http://golang.org/pkg/text/template/), but not +very many template functions. Sprig is a library that provides more than 100 commonly +used template functions. + +It is inspired by the template functions found in +[Twig](http://twig.sensiolabs.org/documentation) and in various +JavaScript libraries, such as [underscore.js](http://underscorejs.org/). + +## Usage + +**Template developers**: Please use Sprig's [function documentation](http://masterminds.github.io/sprig/) for +detailed instructions and code snippets for the >100 template functions available. + +**Go developers**: If you'd like to include Sprig as a library in your program, +our API documentation is available [at GoDoc.org](http://godoc.org/github.com/Masterminds/sprig). + +For standard usage, read on. + +### Load the Sprig library + +To load the Sprig `FuncMap`: + +```go + +import ( + "github.com/Masterminds/sprig" + "html/template" +) + +// This example illustrates that the FuncMap *must* be set before the +// templates themselves are loaded. +tpl := template.Must( + template.New("base").Funcs(sprig.FuncMap()).ParseGlob("*.html") +) + + +``` + +### Calling the functions inside of templates + +By convention, all functions are lowercase. This seems to follow the Go +idiom for template functions (as opposed to template methods, which are +TitleCase). For example, this: + +``` +{{ "hello!" | upper | repeat 5 }} +``` + +produces this: + +``` +HELLO!HELLO!HELLO!HELLO!HELLO! +``` + +## Principles Driving Our Function Selection + +We followed these principles to decide which functions to add and how to implement them: + +- Use template functions to build layout. The following + types of operations are within the domain of template functions: + - Formatting + - Layout + - Simple type conversions + - Utilities that assist in handling common formatting and layout needs (e.g. arithmetic) +- Template functions should not return errors unless there is no way to print + a sensible value. For example, converting a string to an integer should not + produce an error if conversion fails. Instead, it should display a default + value. +- Simple math is necessary for grid layouts, pagers, and so on. Complex math + (anything other than arithmetic) should be done outside of templates. +- Template functions only deal with the data passed into them. They never retrieve + data from a source. +- Finally, do not override core Go template functions. diff --git a/github.com/Masterminds/sprig/appveyor.yml b/github.com/Masterminds/sprig/appveyor.yml new file mode 100644 index 0000000000..d545a987a3 --- /dev/null +++ b/github.com/Masterminds/sprig/appveyor.yml @@ -0,0 +1,26 @@ + +version: build-{build}.{branch} + +clone_folder: C:\gopath\src\github.com\Masterminds\sprig +shallow_clone: true + +environment: + GOPATH: C:\gopath + +platform: + - x64 + +install: + - go get -u github.com/Masterminds/glide + - set PATH=%GOPATH%\bin;%PATH% + - go version + - go env + +build_script: + - glide install + - go install ./... + +test_script: + - go test -v + +deploy: off diff --git a/github.com/Masterminds/sprig/crypto.go b/github.com/Masterminds/sprig/crypto.go new file mode 100644 index 0000000000..7a418ba88d --- /dev/null +++ b/github.com/Masterminds/sprig/crypto.go @@ -0,0 +1,502 @@ +package sprig + +import ( + "bytes" + "crypto/aes" + "crypto/cipher" + "crypto/dsa" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/hmac" + "crypto/rand" + "crypto/rsa" + "crypto/sha1" + "crypto/sha256" + "crypto/x509" + "crypto/x509/pkix" + "encoding/asn1" + "encoding/base64" + "encoding/binary" + "encoding/hex" + "encoding/pem" + "errors" + "fmt" + "io" + "hash/adler32" + "math/big" + "net" + "time" + + "github.com/google/uuid" + "golang.org/x/crypto/scrypt" +) + +func sha256sum(input string) string { + hash := sha256.Sum256([]byte(input)) + return hex.EncodeToString(hash[:]) +} + +func sha1sum(input string) string { + hash := sha1.Sum([]byte(input)) + return hex.EncodeToString(hash[:]) +} + +func adler32sum(input string) string { + hash := adler32.Checksum([]byte(input)) + return fmt.Sprintf("%d", hash) +} + +// uuidv4 provides a safe and secure UUID v4 implementation +func uuidv4() string { + return fmt.Sprintf("%s", uuid.New()) +} + +var master_password_seed = "com.lyndir.masterpassword" + +var password_type_templates = map[string][][]byte{ + "maximum": {[]byte("anoxxxxxxxxxxxxxxxxx"), []byte("axxxxxxxxxxxxxxxxxno")}, + "long": {[]byte("CvcvnoCvcvCvcv"), []byte("CvcvCvcvnoCvcv"), []byte("CvcvCvcvCvcvno"), []byte("CvccnoCvcvCvcv"), []byte("CvccCvcvnoCvcv"), + []byte("CvccCvcvCvcvno"), []byte("CvcvnoCvccCvcv"), []byte("CvcvCvccnoCvcv"), []byte("CvcvCvccCvcvno"), []byte("CvcvnoCvcvCvcc"), + []byte("CvcvCvcvnoCvcc"), []byte("CvcvCvcvCvccno"), []byte("CvccnoCvccCvcv"), []byte("CvccCvccnoCvcv"), []byte("CvccCvccCvcvno"), + []byte("CvcvnoCvccCvcc"), []byte("CvcvCvccnoCvcc"), []byte("CvcvCvccCvccno"), []byte("CvccnoCvcvCvcc"), []byte("CvccCvcvnoCvcc"), + []byte("CvccCvcvCvccno")}, + "medium": {[]byte("CvcnoCvc"), []byte("CvcCvcno")}, + "short": {[]byte("Cvcn")}, + "basic": {[]byte("aaanaaan"), []byte("aannaaan"), []byte("aaannaaa")}, + "pin": {[]byte("nnnn")}, +} + +var template_characters = map[byte]string{ + 'V': "AEIOU", + 'C': "BCDFGHJKLMNPQRSTVWXYZ", + 'v': "aeiou", + 'c': "bcdfghjklmnpqrstvwxyz", + 'A': "AEIOUBCDFGHJKLMNPQRSTVWXYZ", + 'a': "AEIOUaeiouBCDFGHJKLMNPQRSTVWXYZbcdfghjklmnpqrstvwxyz", + 'n': "0123456789", + 'o': "@&%?,=[]_:-+*$#!'^~;()/.", + 'x': "AEIOUaeiouBCDFGHJKLMNPQRSTVWXYZbcdfghjklmnpqrstvwxyz0123456789!@#$%^&*()", +} + +func derivePassword(counter uint32, password_type, password, user, site string) string { + var templates = password_type_templates[password_type] + if templates == nil { + return fmt.Sprintf("cannot find password template %s", password_type) + } + + var buffer bytes.Buffer + buffer.WriteString(master_password_seed) + binary.Write(&buffer, binary.BigEndian, uint32(len(user))) + buffer.WriteString(user) + + salt := buffer.Bytes() + key, err := scrypt.Key([]byte(password), salt, 32768, 8, 2, 64) + if err != nil { + return fmt.Sprintf("failed to derive password: %s", err) + } + + buffer.Truncate(len(master_password_seed)) + binary.Write(&buffer, binary.BigEndian, uint32(len(site))) + buffer.WriteString(site) + binary.Write(&buffer, binary.BigEndian, counter) + + var hmacv = hmac.New(sha256.New, key) + hmacv.Write(buffer.Bytes()) + var seed = hmacv.Sum(nil) + var temp = templates[int(seed[0])%len(templates)] + + buffer.Truncate(0) + for i, element := range temp { + pass_chars := template_characters[element] + pass_char := pass_chars[int(seed[i+1])%len(pass_chars)] + buffer.WriteByte(pass_char) + } + + return buffer.String() +} + +func generatePrivateKey(typ string) string { + var priv interface{} + var err error + switch typ { + case "", "rsa": + // good enough for government work + priv, err = rsa.GenerateKey(rand.Reader, 4096) + case "dsa": + key := new(dsa.PrivateKey) + // again, good enough for government work + if err = dsa.GenerateParameters(&key.Parameters, rand.Reader, dsa.L2048N256); err != nil { + return fmt.Sprintf("failed to generate dsa params: %s", err) + } + err = dsa.GenerateKey(key, rand.Reader) + priv = key + case "ecdsa": + // again, good enough for government work + priv, err = ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + default: + return "Unknown type " + typ + } + if err != nil { + return fmt.Sprintf("failed to generate private key: %s", err) + } + + return string(pem.EncodeToMemory(pemBlockForKey(priv))) +} + +type DSAKeyFormat struct { + Version int + P, Q, G, Y, X *big.Int +} + +func pemBlockForKey(priv interface{}) *pem.Block { + switch k := priv.(type) { + case *rsa.PrivateKey: + return &pem.Block{Type: "RSA PRIVATE KEY", Bytes: x509.MarshalPKCS1PrivateKey(k)} + case *dsa.PrivateKey: + val := DSAKeyFormat{ + P: k.P, Q: k.Q, G: k.G, + Y: k.Y, X: k.X, + } + bytes, _ := asn1.Marshal(val) + return &pem.Block{Type: "DSA PRIVATE KEY", Bytes: bytes} + case *ecdsa.PrivateKey: + b, _ := x509.MarshalECPrivateKey(k) + return &pem.Block{Type: "EC PRIVATE KEY", Bytes: b} + default: + return nil + } +} + +type certificate struct { + Cert string + Key string +} + +func buildCustomCertificate(b64cert string, b64key string) (certificate, error) { + crt := certificate{} + + cert, err := base64.StdEncoding.DecodeString(b64cert) + if err != nil { + return crt, errors.New("unable to decode base64 certificate") + } + + key, err := base64.StdEncoding.DecodeString(b64key) + if err != nil { + return crt, errors.New("unable to decode base64 private key") + } + + decodedCert, _ := pem.Decode(cert) + if decodedCert == nil { + return crt, errors.New("unable to decode certificate") + } + _, err = x509.ParseCertificate(decodedCert.Bytes) + if err != nil { + return crt, fmt.Errorf( + "error parsing certificate: decodedCert.Bytes: %s", + err, + ) + } + + decodedKey, _ := pem.Decode(key) + if decodedKey == nil { + return crt, errors.New("unable to decode key") + } + _, err = x509.ParsePKCS1PrivateKey(decodedKey.Bytes) + if err != nil { + return crt, fmt.Errorf( + "error parsing prive key: decodedKey.Bytes: %s", + err, + ) + } + + crt.Cert = string(cert) + crt.Key = string(key) + + return crt, nil +} + +func generateCertificateAuthority( + cn string, + daysValid int, +) (certificate, error) { + ca := certificate{} + + template, err := getBaseCertTemplate(cn, nil, nil, daysValid) + if err != nil { + return ca, err + } + // Override KeyUsage and IsCA + template.KeyUsage = x509.KeyUsageKeyEncipherment | + x509.KeyUsageDigitalSignature | + x509.KeyUsageCertSign + template.IsCA = true + + priv, err := rsa.GenerateKey(rand.Reader, 2048) + if err != nil { + return ca, fmt.Errorf("error generating rsa key: %s", err) + } + + ca.Cert, ca.Key, err = getCertAndKey(template, priv, template, priv) + if err != nil { + return ca, err + } + + return ca, nil +} + +func generateSelfSignedCertificate( + cn string, + ips []interface{}, + alternateDNS []interface{}, + daysValid int, +) (certificate, error) { + cert := certificate{} + + template, err := getBaseCertTemplate(cn, ips, alternateDNS, daysValid) + if err != nil { + return cert, err + } + + priv, err := rsa.GenerateKey(rand.Reader, 2048) + if err != nil { + return cert, fmt.Errorf("error generating rsa key: %s", err) + } + + cert.Cert, cert.Key, err = getCertAndKey(template, priv, template, priv) + if err != nil { + return cert, err + } + + return cert, nil +} + +func generateSignedCertificate( + cn string, + ips []interface{}, + alternateDNS []interface{}, + daysValid int, + ca certificate, +) (certificate, error) { + cert := certificate{} + + decodedSignerCert, _ := pem.Decode([]byte(ca.Cert)) + if decodedSignerCert == nil { + return cert, errors.New("unable to decode certificate") + } + signerCert, err := x509.ParseCertificate(decodedSignerCert.Bytes) + if err != nil { + return cert, fmt.Errorf( + "error parsing certificate: decodedSignerCert.Bytes: %s", + err, + ) + } + decodedSignerKey, _ := pem.Decode([]byte(ca.Key)) + if decodedSignerKey == nil { + return cert, errors.New("unable to decode key") + } + signerKey, err := x509.ParsePKCS1PrivateKey(decodedSignerKey.Bytes) + if err != nil { + return cert, fmt.Errorf( + "error parsing prive key: decodedSignerKey.Bytes: %s", + err, + ) + } + + template, err := getBaseCertTemplate(cn, ips, alternateDNS, daysValid) + if err != nil { + return cert, err + } + + priv, err := rsa.GenerateKey(rand.Reader, 2048) + if err != nil { + return cert, fmt.Errorf("error generating rsa key: %s", err) + } + + cert.Cert, cert.Key, err = getCertAndKey( + template, + priv, + signerCert, + signerKey, + ) + if err != nil { + return cert, err + } + + return cert, nil +} + +func getCertAndKey( + template *x509.Certificate, + signeeKey *rsa.PrivateKey, + parent *x509.Certificate, + signingKey *rsa.PrivateKey, +) (string, string, error) { + derBytes, err := x509.CreateCertificate( + rand.Reader, + template, + parent, + &signeeKey.PublicKey, + signingKey, + ) + if err != nil { + return "", "", fmt.Errorf("error creating certificate: %s", err) + } + + certBuffer := bytes.Buffer{} + if err := pem.Encode( + &certBuffer, + &pem.Block{Type: "CERTIFICATE", Bytes: derBytes}, + ); err != nil { + return "", "", fmt.Errorf("error pem-encoding certificate: %s", err) + } + + keyBuffer := bytes.Buffer{} + if err := pem.Encode( + &keyBuffer, + &pem.Block{ + Type: "RSA PRIVATE KEY", + Bytes: x509.MarshalPKCS1PrivateKey(signeeKey), + }, + ); err != nil { + return "", "", fmt.Errorf("error pem-encoding key: %s", err) + } + + return string(certBuffer.Bytes()), string(keyBuffer.Bytes()), nil +} + +func getBaseCertTemplate( + cn string, + ips []interface{}, + alternateDNS []interface{}, + daysValid int, +) (*x509.Certificate, error) { + ipAddresses, err := getNetIPs(ips) + if err != nil { + return nil, err + } + dnsNames, err := getAlternateDNSStrs(alternateDNS) + if err != nil { + return nil, err + } + serialNumberUpperBound := new(big.Int).Lsh(big.NewInt(1), 128) + serialNumber, err := rand.Int(rand.Reader, serialNumberUpperBound) + if err != nil { + return nil, err + } + return &x509.Certificate{ + SerialNumber: serialNumber, + Subject: pkix.Name{ + CommonName: cn, + }, + IPAddresses: ipAddresses, + DNSNames: dnsNames, + NotBefore: time.Now(), + NotAfter: time.Now().Add(time.Hour * 24 * time.Duration(daysValid)), + KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature, + ExtKeyUsage: []x509.ExtKeyUsage{ + x509.ExtKeyUsageServerAuth, + x509.ExtKeyUsageClientAuth, + }, + BasicConstraintsValid: true, + }, nil +} + +func getNetIPs(ips []interface{}) ([]net.IP, error) { + if ips == nil { + return []net.IP{}, nil + } + var ipStr string + var ok bool + var netIP net.IP + netIPs := make([]net.IP, len(ips)) + for i, ip := range ips { + ipStr, ok = ip.(string) + if !ok { + return nil, fmt.Errorf("error parsing ip: %v is not a string", ip) + } + netIP = net.ParseIP(ipStr) + if netIP == nil { + return nil, fmt.Errorf("error parsing ip: %s", ipStr) + } + netIPs[i] = netIP + } + return netIPs, nil +} + +func getAlternateDNSStrs(alternateDNS []interface{}) ([]string, error) { + if alternateDNS == nil { + return []string{}, nil + } + var dnsStr string + var ok bool + alternateDNSStrs := make([]string, len(alternateDNS)) + for i, dns := range alternateDNS { + dnsStr, ok = dns.(string) + if !ok { + return nil, fmt.Errorf( + "error processing alternate dns name: %v is not a string", + dns, + ) + } + alternateDNSStrs[i] = dnsStr + } + return alternateDNSStrs, nil +} + +func encryptAES(password string, plaintext string) (string, error) { + if plaintext == "" { + return "", nil + } + + key := make([]byte, 32) + copy(key, []byte(password)) + block, err := aes.NewCipher(key) + if err != nil { + return "", err + } + + content := []byte(plaintext) + blockSize := block.BlockSize() + padding := blockSize - len(content)%blockSize + padtext := bytes.Repeat([]byte{byte(padding)}, padding) + content = append(content, padtext...) + + ciphertext := make([]byte, aes.BlockSize+len(content)) + + iv := ciphertext[:aes.BlockSize] + if _, err := io.ReadFull(rand.Reader, iv); err != nil { + return "", err + } + + mode := cipher.NewCBCEncrypter(block, iv) + mode.CryptBlocks(ciphertext[aes.BlockSize:], content) + + return base64.StdEncoding.EncodeToString(ciphertext), nil +} + +func decryptAES(password string, crypt64 string) (string, error) { + if crypt64 == "" { + return "", nil + } + + key := make([]byte, 32) + copy(key, []byte(password)) + + crypt, err := base64.StdEncoding.DecodeString(crypt64) + if err != nil { + return "", err + } + + block, err := aes.NewCipher(key) + if err != nil { + return "", err + } + + iv := crypt[:aes.BlockSize] + crypt = crypt[aes.BlockSize:] + decrypted := make([]byte, len(crypt)) + mode := cipher.NewCBCDecrypter(block, iv) + mode.CryptBlocks(decrypted, crypt) + + return string(decrypted[:len(decrypted)-int(decrypted[len(decrypted)-1])]), nil +} diff --git a/github.com/Masterminds/sprig/date.go b/github.com/Masterminds/sprig/date.go new file mode 100644 index 0000000000..d1d6155d72 --- /dev/null +++ b/github.com/Masterminds/sprig/date.go @@ -0,0 +1,83 @@ +package sprig + +import ( + "strconv" + "time" +) + +// Given a format and a date, format the date string. +// +// Date can be a `time.Time` or an `int, int32, int64`. +// In the later case, it is treated as seconds since UNIX +// epoch. +func date(fmt string, date interface{}) string { + return dateInZone(fmt, date, "Local") +} + +func htmlDate(date interface{}) string { + return dateInZone("2006-01-02", date, "Local") +} + +func htmlDateInZone(date interface{}, zone string) string { + return dateInZone("2006-01-02", date, zone) +} + +func dateInZone(fmt string, date interface{}, zone string) string { + var t time.Time + switch date := date.(type) { + default: + t = time.Now() + case time.Time: + t = date + case *time.Time: + t = *date + case int64: + t = time.Unix(date, 0) + case int: + t = time.Unix(int64(date), 0) + case int32: + t = time.Unix(int64(date), 0) + } + + loc, err := time.LoadLocation(zone) + if err != nil { + loc, _ = time.LoadLocation("UTC") + } + + return t.In(loc).Format(fmt) +} + +func dateModify(fmt string, date time.Time) time.Time { + d, err := time.ParseDuration(fmt) + if err != nil { + return date + } + return date.Add(d) +} + +func dateAgo(date interface{}) string { + var t time.Time + + switch date := date.(type) { + default: + t = time.Now() + case time.Time: + t = date + case int64: + t = time.Unix(date, 0) + case int: + t = time.Unix(int64(date), 0) + } + // Drop resolution to seconds + duration := time.Since(t).Round(time.Second) + return duration.String() +} + +func toDate(fmt, str string) time.Time { + t, _ := time.ParseInLocation(fmt, str, time.Local) + return t +} + +func unixEpoch(date time.Time) string { + return strconv.FormatInt(date.Unix(), 10) +} diff --git a/github.com/Masterminds/sprig/defaults.go b/github.com/Masterminds/sprig/defaults.go new file mode 100644 index 0000000000..ed6a8ab291 --- /dev/null +++ b/github.com/Masterminds/sprig/defaults.go @@ -0,0 +1,83 @@ +package sprig + +import ( + "encoding/json" + "reflect" +) + +// dfault checks whether `given` is set, and returns default if not set. +// +// This returns `d` if `given` appears not to be set, and `given` otherwise. +// +// For numeric types 0 is unset. +// For strings, maps, arrays, and slices, len() = 0 is considered unset. +// For bool, false is unset. +// Structs are never considered unset. +// +// For everything else, including pointers, a nil value is unset. +func dfault(d interface{}, given ...interface{}) interface{} { + + if empty(given) || empty(given[0]) { + return d + } + return given[0] +} + +// empty returns true if the given value has the zero value for its type. +func empty(given interface{}) bool { + g := reflect.ValueOf(given) + if !g.IsValid() { + return true + } + + // Basically adapted from text/template.isTrue + switch g.Kind() { + default: + return g.IsNil() + case reflect.Array, reflect.Slice, reflect.Map, reflect.String: + return g.Len() == 0 + case reflect.Bool: + return g.Bool() == false + case reflect.Complex64, reflect.Complex128: + return g.Complex() == 0 + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + return g.Int() == 0 + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + return g.Uint() == 0 + case reflect.Float32, reflect.Float64: + return g.Float() == 0 + case reflect.Struct: + return false + } +} + +// coalesce returns the first non-empty value. +func coalesce(v ...interface{}) interface{} { + for _, val := range v { + if !empty(val) { + return val + } + } + return nil +} + +// toJson encodes an item into a JSON string +func toJson(v interface{}) string { + output, _ := json.Marshal(v) + return string(output) +} + +// toPrettyJson encodes an item into a pretty (indented) JSON string +func toPrettyJson(v interface{}) string { + output, _ := json.MarshalIndent(v, "", " ") + return string(output) +} + +// ternary returns the first value if the last value is true, otherwise returns the second value. +func ternary(vt interface{}, vf interface{}, v bool) interface{} { + if v { + return vt + } + + return vf +} diff --git a/github.com/Masterminds/sprig/dict.go b/github.com/Masterminds/sprig/dict.go new file mode 100644 index 0000000000..738405b433 --- /dev/null +++ b/github.com/Masterminds/sprig/dict.go @@ -0,0 +1,119 @@ +package sprig + +import ( + "github.com/imdario/mergo" + "github.com/mitchellh/copystructure" +) + +func set(d map[string]interface{}, key string, value interface{}) map[string]interface{} { + d[key] = value + return d +} + +func unset(d map[string]interface{}, key string) map[string]interface{} { + delete(d, key) + return d +} + +func hasKey(d map[string]interface{}, key string) bool { + _, ok := d[key] + return ok +} + +func pluck(key string, d ...map[string]interface{}) []interface{} { + res := []interface{}{} + for _, dict := range d { + if val, ok := dict[key]; ok { + res = append(res, val) + } + } + return res +} + +func keys(dicts ...map[string]interface{}) []string { + k := []string{} + for _, dict := range dicts { + for key := range dict { + k = append(k, key) + } + } + return k +} + +func pick(dict map[string]interface{}, keys ...string) map[string]interface{} { + res := map[string]interface{}{} + for _, k := range keys { + if v, ok := dict[k]; ok { + res[k] = v + } + } + return res +} + +func omit(dict map[string]interface{}, keys ...string) map[string]interface{} { + res := map[string]interface{}{} + + omit := make(map[string]bool, len(keys)) + for _, k := range keys { + omit[k] = true + } + + for k, v := range dict { + if _, ok := omit[k]; !ok { + res[k] = v + } + } + return res +} + +func dict(v ...interface{}) map[string]interface{} { + dict := map[string]interface{}{} + lenv := len(v) + for i := 0; i < lenv; i += 2 { + key := strval(v[i]) + if i+1 >= lenv { + dict[key] = "" + continue + } + dict[key] = v[i+1] + } + return dict +} + +func merge(dst map[string]interface{}, srcs ...map[string]interface{}) interface{} { + for _, src := range srcs { + if err := mergo.Merge(&dst, src); err != nil { + // Swallow errors inside of a template. + return "" + } + } + return dst +} + +func mergeOverwrite(dst map[string]interface{}, srcs ...map[string]interface{}) interface{} { + for _, src := range srcs { + if err := mergo.MergeWithOverwrite(&dst, src); err != nil { + // Swallow errors inside of a template. + return "" + } + } + return dst +} + +func values(dict map[string]interface{}) []interface{} { + values := []interface{}{} + for _, value := range dict { + values = append(values, value) + } + + return values +} + +func deepCopy(i interface{}) interface{} { + c, err := copystructure.Copy(i) + if err != nil { + panic("deepCopy error: " + err.Error()) + } + + return c +} diff --git a/github.com/Masterminds/sprig/doc.go b/github.com/Masterminds/sprig/doc.go new file mode 100644 index 0000000000..8f8f1d7370 --- /dev/null +++ b/github.com/Masterminds/sprig/doc.go @@ -0,0 +1,19 @@ +/* +Sprig: Template functions for Go. + +This package contains a number of utility functions for working with data +inside of Go `html/template` and `text/template` files. + +To add these functions, use the `template.Funcs()` method: + + t := templates.New("foo").Funcs(sprig.FuncMap()) + +Note that you should add the function map before you parse any template files. + + In several cases, Sprig reverses the order of arguments from the way they + appear in the standard library. This is to make it easier to pipe + arguments into functions. + +See http://masterminds.github.io/sprig/ for more detailed documentation on each of the available functions. +*/ +package sprig diff --git a/github.com/Masterminds/sprig/functions.go b/github.com/Masterminds/sprig/functions.go new file mode 100644 index 0000000000..7b5b0af86c --- /dev/null +++ b/github.com/Masterminds/sprig/functions.go @@ -0,0 +1,306 @@ +package sprig + +import ( + "errors" + "html/template" + "os" + "path" + "reflect" + "strconv" + "strings" + ttemplate "text/template" + "time" + + util "github.com/Masterminds/goutils" + "github.com/huandu/xstrings" +) + +// Produce the function map. +// +// Use this to pass the functions into the template engine: +// +// tpl := template.New("foo").Funcs(sprig.FuncMap())) +// +func FuncMap() template.FuncMap { + return HtmlFuncMap() +} + +// HermeticTxtFuncMap returns a 'text/template'.FuncMap with only repeatable functions. +func HermeticTxtFuncMap() ttemplate.FuncMap { + r := TxtFuncMap() + for _, name := range nonhermeticFunctions { + delete(r, name) + } + return r +} + +// HermeticHtmlFuncMap returns an 'html/template'.Funcmap with only repeatable functions. +func HermeticHtmlFuncMap() template.FuncMap { + r := HtmlFuncMap() + for _, name := range nonhermeticFunctions { + delete(r, name) + } + return r +} + +// TxtFuncMap returns a 'text/template'.FuncMap +func TxtFuncMap() ttemplate.FuncMap { + return ttemplate.FuncMap(GenericFuncMap()) +} + +// HtmlFuncMap returns an 'html/template'.Funcmap +func HtmlFuncMap() template.FuncMap { + return template.FuncMap(GenericFuncMap()) +} + +// GenericFuncMap returns a copy of the basic function map as a map[string]interface{}. +func GenericFuncMap() map[string]interface{} { + gfm := make(map[string]interface{}, len(genericMap)) + for k, v := range genericMap { + gfm[k] = v + } + return gfm +} + +// These functions are not guaranteed to evaluate to the same result for given input, because they +// refer to the environemnt or global state. +var nonhermeticFunctions = []string{ + // Date functions + "date", + "date_in_zone", + "date_modify", + "now", + "htmlDate", + "htmlDateInZone", + "dateInZone", + "dateModify", + + // Strings + "randAlphaNum", + "randAlpha", + "randAscii", + "randNumeric", + "uuidv4", + + // OS + "env", + "expandenv", + + // Network + "getHostByName", +} + +var genericMap = map[string]interface{}{ + "hello": func() string { return "Hello!" }, + + // Date functions + "date": date, + "date_in_zone": dateInZone, + "date_modify": dateModify, + "now": func() time.Time { return time.Now() }, + "htmlDate": htmlDate, + "htmlDateInZone": htmlDateInZone, + "dateInZone": dateInZone, + "dateModify": dateModify, + "ago": dateAgo, + "toDate": toDate, + "unixEpoch": unixEpoch, + + // Strings + "abbrev": abbrev, + "abbrevboth": abbrevboth, + "trunc": trunc, + "trim": strings.TrimSpace, + "upper": strings.ToUpper, + "lower": strings.ToLower, + "title": strings.Title, + "untitle": untitle, + "substr": substring, + // Switch order so that "foo" | repeat 5 + "repeat": func(count int, str string) string { return strings.Repeat(str, count) }, + // Deprecated: Use trimAll. + "trimall": func(a, b string) string { return strings.Trim(b, a) }, + // Switch order so that "$foo" | trimall "$" + "trimAll": func(a, b string) string { return strings.Trim(b, a) }, + "trimSuffix": func(a, b string) string { return strings.TrimSuffix(b, a) }, + "trimPrefix": func(a, b string) string { return strings.TrimPrefix(b, a) }, + "nospace": util.DeleteWhiteSpace, + "initials": initials, + "randAlphaNum": randAlphaNumeric, + "randAlpha": randAlpha, + "randAscii": randAscii, + "randNumeric": randNumeric, + "swapcase": util.SwapCase, + "shuffle": xstrings.Shuffle, + "snakecase": xstrings.ToSnakeCase, + "camelcase": xstrings.ToCamelCase, + "kebabcase": xstrings.ToKebabCase, + "wrap": func(l int, s string) string { return util.Wrap(s, l) }, + "wrapWith": func(l int, sep, str string) string { return util.WrapCustom(str, l, sep, true) }, + // Switch order so that "foobar" | contains "foo" + "contains": func(substr string, str string) bool { return strings.Contains(str, substr) }, + "hasPrefix": func(substr string, str string) bool { return strings.HasPrefix(str, substr) }, + "hasSuffix": func(substr string, str string) bool { return strings.HasSuffix(str, substr) }, + "quote": quote, + "squote": squote, + "cat": cat, + "indent": indent, + "nindent": nindent, + "replace": replace, + "plural": plural, + "sha1sum": sha1sum, + "sha256sum": sha256sum, + "adler32sum": adler32sum, + "toString": strval, + + // Wrap Atoi to stop errors. + "atoi": func(a string) int { i, _ := strconv.Atoi(a); return i }, + "int64": toInt64, + "int": toInt, + "float64": toFloat64, + "toDecimal": toDecimal, + + //"gt": func(a, b int) bool {return a > b}, + //"gte": func(a, b int) bool {return a >= b}, + //"lt": func(a, b int) bool {return a < b}, + //"lte": func(a, b int) bool {return a <= b}, + + // split "/" foo/bar returns map[int]string{0: foo, 1: bar} + "split": split, + "splitList": func(sep, orig string) []string { return strings.Split(orig, sep) }, + // splitn "/" foo/bar/fuu returns map[int]string{0: foo, 1: bar/fuu} + "splitn": splitn, + "toStrings": strslice, + + "until": until, + "untilStep": untilStep, + + // VERY basic arithmetic. + "add1": func(i interface{}) int64 { return toInt64(i) + 1 }, + "add": func(i ...interface{}) int64 { + var a int64 = 0 + for _, b := range i { + a += toInt64(b) + } + return a + }, + "sub": func(a, b interface{}) int64 { return toInt64(a) - toInt64(b) }, + "div": func(a, b interface{}) int64 { return toInt64(a) / toInt64(b) }, + "mod": func(a, b interface{}) int64 { return toInt64(a) % toInt64(b) }, + "mul": func(a interface{}, v ...interface{}) int64 { + val := toInt64(a) + for _, b := range v { + val = val * toInt64(b) + } + return val + }, + "biggest": max, + "max": max, + "min": min, + "ceil": ceil, + "floor": floor, + "round": round, + + // string slices. Note that we reverse the order b/c that's better + // for template processing. + "join": join, + "sortAlpha": sortAlpha, + + // Defaults + "default": dfault, + "empty": empty, + "coalesce": coalesce, + "compact": compact, + "deepCopy": deepCopy, + "toJson": toJson, + "toPrettyJson": toPrettyJson, + "ternary": ternary, + + // Reflection + "typeOf": typeOf, + "typeIs": typeIs, + "typeIsLike": typeIsLike, + "kindOf": kindOf, + "kindIs": kindIs, + "deepEqual": reflect.DeepEqual, + + // OS: + "env": func(s string) string { return os.Getenv(s) }, + "expandenv": func(s string) string { return os.ExpandEnv(s) }, + + // Network: + "getHostByName": getHostByName, + + // File Paths: + "base": path.Base, + "dir": path.Dir, + "clean": path.Clean, + "ext": path.Ext, + "isAbs": path.IsAbs, + + // Encoding: + "b64enc": base64encode, + "b64dec": base64decode, + "b32enc": base32encode, + "b32dec": base32decode, + + // Data Structures: + "tuple": list, // FIXME: with the addition of append/prepend these are no longer immutable. + "list": list, + "dict": dict, + "set": set, + "unset": unset, + "hasKey": hasKey, + "pluck": pluck, + "keys": keys, + "pick": pick, + "omit": omit, + "merge": merge, + "mergeOverwrite": mergeOverwrite, + "values": values, + + "append": push, "push": push, + "prepend": prepend, + "first": first, + "rest": rest, + "last": last, + "initial": initial, + "reverse": reverse, + "uniq": uniq, + "without": without, + "has": has, + "slice": slice, + "concat": concat, + + // Crypto: + "genPrivateKey": generatePrivateKey, + "derivePassword": derivePassword, + "buildCustomCert": buildCustomCertificate, + "genCA": generateCertificateAuthority, + "genSelfSignedCert": generateSelfSignedCertificate, + "genSignedCert": generateSignedCertificate, + "encryptAES": encryptAES, + "decryptAES": decryptAES, + + // UUIDs: + "uuidv4": uuidv4, + + // SemVer: + "semver": semver, + "semverCompare": semverCompare, + + // Flow Control: + "fail": func(msg string) (string, error) { return "", errors.New(msg) }, + + // Regex + "regexMatch": regexMatch, + "regexFindAll": regexFindAll, + "regexFind": regexFind, + "regexReplaceAll": regexReplaceAll, + "regexReplaceAllLiteral": regexReplaceAllLiteral, + "regexSplit": regexSplit, + + // URLs: + "urlParse": urlParse, + "urlJoin": urlJoin, +} diff --git a/github.com/Masterminds/sprig/glide.yaml b/github.com/Masterminds/sprig/glide.yaml new file mode 100644 index 0000000000..f317d2b2b1 --- /dev/null +++ b/github.com/Masterminds/sprig/glide.yaml @@ -0,0 +1,19 @@ +package: github.com/Masterminds/sprig +import: +- package: github.com/Masterminds/goutils + version: ^1.0.0 +- package: github.com/google/uuid + version: ^1.0.0 +- package: golang.org/x/crypto + subpackages: + - scrypt +- package: github.com/Masterminds/semver + version: ^v1.2.2 +- package: github.com/stretchr/testify + version: ^v1.2.2 +- package: github.com/imdario/mergo + version: ~0.3.7 +- package: github.com/huandu/xstrings + version: ^1.2 +- package: github.com/mitchellh/copystructure + version: ^1.0.0 diff --git a/github.com/Masterminds/sprig/list.go b/github.com/Masterminds/sprig/list.go new file mode 100644 index 0000000000..c0381bbb65 --- /dev/null +++ b/github.com/Masterminds/sprig/list.go @@ -0,0 +1,311 @@ +package sprig + +import ( + "fmt" + "reflect" + "sort" +) + +// Reflection is used in these functions so that slices and arrays of strings, +// ints, and other types not implementing []interface{} can be worked with. +// For example, this is useful if you need to work on the output of regexs. + +func list(v ...interface{}) []interface{} { + return v +} + +func push(list interface{}, v interface{}) []interface{} { + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + + l := l2.Len() + nl := make([]interface{}, l) + for i := 0; i < l; i++ { + nl[i] = l2.Index(i).Interface() + } + + return append(nl, v) + + default: + panic(fmt.Sprintf("Cannot push on type %s", tp)) + } +} + +func prepend(list interface{}, v interface{}) []interface{} { + //return append([]interface{}{v}, list...) + + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + + l := l2.Len() + nl := make([]interface{}, l) + for i := 0; i < l; i++ { + nl[i] = l2.Index(i).Interface() + } + + return append([]interface{}{v}, nl...) + + default: + panic(fmt.Sprintf("Cannot prepend on type %s", tp)) + } +} + +func last(list interface{}) interface{} { + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + + l := l2.Len() + if l == 0 { + return nil + } + + return l2.Index(l - 1).Interface() + default: + panic(fmt.Sprintf("Cannot find last on type %s", tp)) + } +} + +func first(list interface{}) interface{} { + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + + l := l2.Len() + if l == 0 { + return nil + } + + return l2.Index(0).Interface() + default: + panic(fmt.Sprintf("Cannot find first on type %s", tp)) + } +} + +func rest(list interface{}) []interface{} { + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + + l := l2.Len() + if l == 0 { + return nil + } + + nl := make([]interface{}, l-1) + for i := 1; i < l; i++ { + nl[i-1] = l2.Index(i).Interface() + } + + return nl + default: + panic(fmt.Sprintf("Cannot find rest on type %s", tp)) + } +} + +func initial(list interface{}) []interface{} { + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + + l := l2.Len() + if l == 0 { + return nil + } + + nl := make([]interface{}, l-1) + for i := 0; i < l-1; i++ { + nl[i] = l2.Index(i).Interface() + } + + return nl + default: + panic(fmt.Sprintf("Cannot find initial on type %s", tp)) + } +} + +func sortAlpha(list interface{}) []string { + k := reflect.Indirect(reflect.ValueOf(list)).Kind() + switch k { + case reflect.Slice, reflect.Array: + a := strslice(list) + s := sort.StringSlice(a) + s.Sort() + return s + } + return []string{strval(list)} +} + +func reverse(v interface{}) []interface{} { + tp := reflect.TypeOf(v).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(v) + + l := l2.Len() + // We do not sort in place because the incoming array should not be altered. + nl := make([]interface{}, l) + for i := 0; i < l; i++ { + nl[l-i-1] = l2.Index(i).Interface() + } + + return nl + default: + panic(fmt.Sprintf("Cannot find reverse on type %s", tp)) + } +} + +func compact(list interface{}) []interface{} { + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + + l := l2.Len() + nl := []interface{}{} + var item interface{} + for i := 0; i < l; i++ { + item = l2.Index(i).Interface() + if !empty(item) { + nl = append(nl, item) + } + } + + return nl + default: + panic(fmt.Sprintf("Cannot compact on type %s", tp)) + } +} + +func uniq(list interface{}) []interface{} { + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + + l := l2.Len() + dest := []interface{}{} + var item interface{} + for i := 0; i < l; i++ { + item = l2.Index(i).Interface() + if !inList(dest, item) { + dest = append(dest, item) + } + } + + return dest + default: + panic(fmt.Sprintf("Cannot find uniq on type %s", tp)) + } +} + +func inList(haystack []interface{}, needle interface{}) bool { + for _, h := range haystack { + if reflect.DeepEqual(needle, h) { + return true + } + } + return false +} + +func without(list interface{}, omit ...interface{}) []interface{} { + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + + l := l2.Len() + res := []interface{}{} + var item interface{} + for i := 0; i < l; i++ { + item = l2.Index(i).Interface() + if !inList(omit, item) { + res = append(res, item) + } + } + + return res + default: + panic(fmt.Sprintf("Cannot find without on type %s", tp)) + } +} + +func has(needle interface{}, haystack interface{}) bool { + if haystack == nil { + return false + } + tp := reflect.TypeOf(haystack).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(haystack) + var item interface{} + l := l2.Len() + for i := 0; i < l; i++ { + item = l2.Index(i).Interface() + if reflect.DeepEqual(needle, item) { + return true + } + } + + return false + default: + panic(fmt.Sprintf("Cannot find has on type %s", tp)) + } +} + +// $list := [1, 2, 3, 4, 5] +// slice $list -> list[0:5] = list[:] +// slice $list 0 3 -> list[0:3] = list[:3] +// slice $list 3 5 -> list[3:5] +// slice $list 3 -> list[3:5] = list[3:] +func slice(list interface{}, indices ...interface{}) interface{} { + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + + l := l2.Len() + if l == 0 { + return nil + } + + var start, end int + if len(indices) > 0 { + start = toInt(indices[0]) + } + if len(indices) < 2 { + end = l + } else { + end = toInt(indices[1]) + } + + return l2.Slice(start, end).Interface() + default: + panic(fmt.Sprintf("list should be type of slice or array but %s", tp)) + } +} + +func concat(lists ...interface{}) interface{} { + var res []interface{} + for _, list := range lists { + tp := reflect.TypeOf(list).Kind() + switch tp { + case reflect.Slice, reflect.Array: + l2 := reflect.ValueOf(list) + for i := 0; i < l2.Len(); i++ { + res = append(res, l2.Index(i).Interface()) + } + default: + panic(fmt.Sprintf("Cannot concat type %s as list", tp)) + } + } + return res +} diff --git a/github.com/Masterminds/sprig/network.go b/github.com/Masterminds/sprig/network.go new file mode 100644 index 0000000000..d786cc7363 --- /dev/null +++ b/github.com/Masterminds/sprig/network.go @@ -0,0 +1,12 @@ +package sprig + +import ( + "math/rand" + "net" +) + +func getHostByName(name string) string { + addrs, _ := net.LookupHost(name) + //TODO: add error handing when release v3 cames out + return addrs[rand.Intn(len(addrs))] +} diff --git a/github.com/Masterminds/sprig/numeric.go b/github.com/Masterminds/sprig/numeric.go new file mode 100644 index 0000000000..f4af4af2a7 --- /dev/null +++ b/github.com/Masterminds/sprig/numeric.go @@ -0,0 +1,169 @@ +package sprig + +import ( + "fmt" + "math" + "reflect" + "strconv" +) + +// toFloat64 converts 64-bit floats +func toFloat64(v interface{}) float64 { + if str, ok := v.(string); ok { + iv, err := strconv.ParseFloat(str, 64) + if err != nil { + return 0 + } + return iv + } + + val := reflect.Indirect(reflect.ValueOf(v)) + switch val.Kind() { + case reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Int: + return float64(val.Int()) + case reflect.Uint8, reflect.Uint16, reflect.Uint32: + return float64(val.Uint()) + case reflect.Uint, reflect.Uint64: + return float64(val.Uint()) + case reflect.Float32, reflect.Float64: + return val.Float() + case reflect.Bool: + if val.Bool() == true { + return 1 + } + return 0 + default: + return 0 + } +} + +func toInt(v interface{}) int { + //It's not optimal. Bud I don't want duplicate toInt64 code. + return int(toInt64(v)) +} + +// toInt64 converts integer types to 64-bit integers +func toInt64(v interface{}) int64 { + if str, ok := v.(string); ok { + iv, err := strconv.ParseInt(str, 10, 64) + if err != nil { + return 0 + } + return iv + } + + val := reflect.Indirect(reflect.ValueOf(v)) + switch val.Kind() { + case reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Int: + return val.Int() + case reflect.Uint8, reflect.Uint16, reflect.Uint32: + return int64(val.Uint()) + case reflect.Uint, reflect.Uint64: + tv := val.Uint() + if tv <= math.MaxInt64 { + return int64(tv) + } + // TODO: What is the sensible thing to do here? + return math.MaxInt64 + case reflect.Float32, reflect.Float64: + return int64(val.Float()) + case reflect.Bool: + if val.Bool() == true { + return 1 + } + return 0 + default: + return 0 + } +} + +func max(a interface{}, i ...interface{}) int64 { + aa := toInt64(a) + for _, b := range i { + bb := toInt64(b) + if bb > aa { + aa = bb + } + } + return aa +} + +func min(a interface{}, i ...interface{}) int64 { + aa := toInt64(a) + for _, b := range i { + bb := toInt64(b) + if bb < aa { + aa = bb + } + } + return aa +} + +func until(count int) []int { + step := 1 + if count < 0 { + step = -1 + } + return untilStep(0, count, step) +} + +func untilStep(start, stop, step int) []int { + v := []int{} + + if stop < start { + if step >= 0 { + return v + } + for i := start; i > stop; i += step { + v = append(v, i) + } + return v + } + + if step <= 0 { + return v + } + for i := start; i < stop; i += step { + v = append(v, i) + } + return v +} + +func floor(a interface{}) float64 { + aa := toFloat64(a) + return math.Floor(aa) +} + +func ceil(a interface{}) float64 { + aa := toFloat64(a) + return math.Ceil(aa) +} + +func round(a interface{}, p int, r_opt ...float64) float64 { + roundOn := .5 + if len(r_opt) > 0 { + roundOn = r_opt[0] + } + val := toFloat64(a) + places := toFloat64(p) + + var round float64 + pow := math.Pow(10, places) + digit := pow * val + _, div := math.Modf(digit) + if div >= roundOn { + round = math.Ceil(digit) + } else { + round = math.Floor(digit) + } + return round / pow +} + +// converts unix octal to decimal +func toDecimal(v interface{}) int64 { + result, err := strconv.ParseInt(fmt.Sprint(v), 8, 64) + if err != nil { + return 0 + } + return result +} diff --git a/github.com/Masterminds/sprig/reflect.go b/github.com/Masterminds/sprig/reflect.go new file mode 100644 index 0000000000..8a65c132f0 --- /dev/null +++ b/github.com/Masterminds/sprig/reflect.go @@ -0,0 +1,28 @@ +package sprig + +import ( + "fmt" + "reflect" +) + +// typeIs returns true if the src is the type named in target. +func typeIs(target string, src interface{}) bool { + return target == typeOf(src) +} + +func typeIsLike(target string, src interface{}) bool { + t := typeOf(src) + return target == t || "*"+target == t +} + +func typeOf(src interface{}) string { + return fmt.Sprintf("%T", src) +} + +func kindIs(target string, src interface{}) bool { + return target == kindOf(src) +} + +func kindOf(src interface{}) string { + return reflect.ValueOf(src).Kind().String() +} diff --git a/github.com/Masterminds/sprig/regex.go b/github.com/Masterminds/sprig/regex.go new file mode 100644 index 0000000000..2016f66336 --- /dev/null +++ b/github.com/Masterminds/sprig/regex.go @@ -0,0 +1,35 @@ +package sprig + +import ( + "regexp" +) + +func regexMatch(regex string, s string) bool { + match, _ := regexp.MatchString(regex, s) + return match +} + +func regexFindAll(regex string, s string, n int) []string { + r := regexp.MustCompile(regex) + return r.FindAllString(s, n) +} + +func regexFind(regex string, s string) string { + r := regexp.MustCompile(regex) + return r.FindString(s) +} + +func regexReplaceAll(regex string, s string, repl string) string { + r := regexp.MustCompile(regex) + return r.ReplaceAllString(s, repl) +} + +func regexReplaceAllLiteral(regex string, s string, repl string) string { + r := regexp.MustCompile(regex) + return r.ReplaceAllLiteralString(s, repl) +} + +func regexSplit(regex string, s string, n int) []string { + r := regexp.MustCompile(regex) + return r.Split(s, n) +} diff --git a/github.com/Masterminds/sprig/semver.go b/github.com/Masterminds/sprig/semver.go new file mode 100644 index 0000000000..c2bf8a1fdf --- /dev/null +++ b/github.com/Masterminds/sprig/semver.go @@ -0,0 +1,23 @@ +package sprig + +import ( + sv2 "github.com/Masterminds/semver" +) + +func semverCompare(constraint, version string) (bool, error) { + c, err := sv2.NewConstraint(constraint) + if err != nil { + return false, err + } + + v, err := sv2.NewVersion(version) + if err != nil { + return false, err + } + + return c.Check(v), nil +} + +func semver(version string) (*sv2.Version, error) { + return sv2.NewVersion(version) +} diff --git a/github.com/Masterminds/sprig/strings.go b/github.com/Masterminds/sprig/strings.go new file mode 100644 index 0000000000..943fa3e8ad --- /dev/null +++ b/github.com/Masterminds/sprig/strings.go @@ -0,0 +1,233 @@ +package sprig + +import ( + "encoding/base32" + "encoding/base64" + "fmt" + "reflect" + "strconv" + "strings" + + util "github.com/Masterminds/goutils" +) + +func base64encode(v string) string { + return base64.StdEncoding.EncodeToString([]byte(v)) +} + +func base64decode(v string) string { + data, err := base64.StdEncoding.DecodeString(v) + if err != nil { + return err.Error() + } + return string(data) +} + +func base32encode(v string) string { + return base32.StdEncoding.EncodeToString([]byte(v)) +} + +func base32decode(v string) string { + data, err := base32.StdEncoding.DecodeString(v) + if err != nil { + return err.Error() + } + return string(data) +} + +func abbrev(width int, s string) string { + if width < 4 { + return s + } + r, _ := util.Abbreviate(s, width) + return r +} + +func abbrevboth(left, right int, s string) string { + if right < 4 || left > 0 && right < 7 { + return s + } + r, _ := util.AbbreviateFull(s, left, right) + return r +} +func initials(s string) string { + // Wrap this just to eliminate the var args, which templates don't do well. + return util.Initials(s) +} + +func randAlphaNumeric(count int) string { + // It is not possible, it appears, to actually generate an error here. + r, _ := util.CryptoRandomAlphaNumeric(count) + return r +} + +func randAlpha(count int) string { + r, _ := util.CryptoRandomAlphabetic(count) + return r +} + +func randAscii(count int) string { + r, _ := util.CryptoRandomAscii(count) + return r +} + +func randNumeric(count int) string { + r, _ := util.CryptoRandomNumeric(count) + return r +} + +func untitle(str string) string { + return util.Uncapitalize(str) +} + +func quote(str ...interface{}) string { + out := make([]string, 0, len(str)) + for _, s := range str { + if s != nil { + out = append(out, fmt.Sprintf("%q", strval(s))) + } + } + return strings.Join(out, " ") +} + +func squote(str ...interface{}) string { + out := make([]string, 0, len(str)) + for _, s := range str { + if s != nil { + out = append(out, fmt.Sprintf("'%v'", s)) + } + } + return strings.Join(out, " ") +} + +func cat(v ...interface{}) string { + v = removeNilElements(v) + r := strings.TrimSpace(strings.Repeat("%v ", len(v))) + return fmt.Sprintf(r, v...) +} + +func indent(spaces int, v string) string { + pad := strings.Repeat(" ", spaces) + return pad + strings.Replace(v, "\n", "\n"+pad, -1) +} + +func nindent(spaces int, v string) string { + return "\n" + indent(spaces, v) +} + +func replace(old, new, src string) string { + return strings.Replace(src, old, new, -1) +} + +func plural(one, many string, count int) string { + if count == 1 { + return one + } + return many +} + +func strslice(v interface{}) []string { + switch v := v.(type) { + case []string: + return v + case []interface{}: + b := make([]string, 0, len(v)) + for _, s := range v { + if s != nil { + b = append(b, strval(s)) + } + } + return b + default: + val := reflect.ValueOf(v) + switch val.Kind() { + case reflect.Array, reflect.Slice: + l := val.Len() + b := make([]string, 0, l) + for i := 0; i < l; i++ { + value := val.Index(i).Interface() + if value != nil { + b = append(b, strval(value)) + } + } + return b + default: + if v == nil { + return []string{} + } else { + return []string{strval(v)} + } + } + } +} + +func removeNilElements(v []interface{}) []interface{} { + newSlice := make([]interface{}, 0, len(v)) + for _, i := range v { + if i != nil { + newSlice = append(newSlice, i) + } + } + return newSlice +} + +func strval(v interface{}) string { + switch v := v.(type) { + case string: + return v + case []byte: + return string(v) + case error: + return v.Error() + case fmt.Stringer: + return v.String() + default: + return fmt.Sprintf("%v", v) + } +} + +func trunc(c int, s string) string { + if len(s) <= c { + return s + } + return s[0:c] +} + +func join(sep string, v interface{}) string { + return strings.Join(strslice(v), sep) +} + +func split(sep, orig string) map[string]string { + parts := strings.Split(orig, sep) + res := make(map[string]string, len(parts)) + for i, v := range parts { + res["_"+strconv.Itoa(i)] = v + } + return res +} + +func splitn(sep string, n int, orig string) map[string]string { + parts := strings.SplitN(orig, sep, n) + res := make(map[string]string, len(parts)) + for i, v := range parts { + res["_"+strconv.Itoa(i)] = v + } + return res +} + +// substring creates a substring of the given string. +// +// If start is < 0, this calls string[:end]. +// +// If start is >= 0 and end < 0 or end bigger than s length, this calls string[start:] +// +// Otherwise, this calls string[start, end]. +func substring(start, end int, s string) string { + if start < 0 { + return s[:end] + } + if end < 0 || end > len(s) { + return s[start:] + } + return s[start:end] +} diff --git a/github.com/Masterminds/sprig/url.go b/github.com/Masterminds/sprig/url.go new file mode 100644 index 0000000000..5f22d801f9 --- /dev/null +++ b/github.com/Masterminds/sprig/url.go @@ -0,0 +1,66 @@ +package sprig + +import ( + "fmt" + "net/url" + "reflect" +) + +func dictGetOrEmpty(dict map[string]interface{}, key string) string { + value, ok := dict[key]; if !ok { + return "" + } + tp := reflect.TypeOf(value).Kind() + if tp != reflect.String { + panic(fmt.Sprintf("unable to parse %s key, must be of type string, but %s found", key, tp.String())) + } + return reflect.ValueOf(value).String() +} + +// parses given URL to return dict object +func urlParse(v string) map[string]interface{} { + dict := map[string]interface{}{} + parsedUrl, err := url.Parse(v) + if err != nil { + panic(fmt.Sprintf("unable to parse url: %s", err)) + } + dict["scheme"] = parsedUrl.Scheme + dict["host"] = parsedUrl.Host + dict["hostname"] = parsedUrl.Hostname() + dict["path"] = parsedUrl.Path + dict["query"] = parsedUrl.RawQuery + dict["opaque"] = parsedUrl.Opaque + dict["fragment"] = parsedUrl.Fragment + if parsedUrl.User != nil { + dict["userinfo"] = parsedUrl.User.String() + } else { + dict["userinfo"] = "" + } + + return dict +} + +// join given dict to URL string +func urlJoin(d map[string]interface{}) string { + resUrl := url.URL{ + Scheme: dictGetOrEmpty(d, "scheme"), + Host: dictGetOrEmpty(d, "host"), + Path: dictGetOrEmpty(d, "path"), + RawQuery: dictGetOrEmpty(d, "query"), + Opaque: dictGetOrEmpty(d, "opaque"), + Fragment: dictGetOrEmpty(d, "fragment"), + + } + userinfo := dictGetOrEmpty(d, "userinfo") + var user *url.Userinfo = nil + if userinfo != "" { + tempUrl, err := url.Parse(fmt.Sprintf("proto://%s@host", userinfo)) + if err != nil { + panic(fmt.Sprintf("unable to parse userinfo in dict: %s", err)) + } + user = tempUrl.User + } + + resUrl.User = user + return resUrl.String() +} diff --git a/github.com/PuerkitoBio/goquery/.travis.yml b/github.com/PuerkitoBio/goquery/.travis.yml index cc1402d5cf..27c3ce8136 100644 --- a/github.com/PuerkitoBio/goquery/.travis.yml +++ b/github.com/PuerkitoBio/goquery/.travis.yml @@ -1,7 +1,6 @@ language: go go: - - 1.1 - 1.2.x - 1.3.x - 1.4.x @@ -10,7 +9,9 @@ go: - 1.7.x - 1.8.x - 1.9.x - - "1.10.x" + - 1.10.x - 1.11.x + - 1.12.x + - 1.13.x - tip diff --git a/github.com/PuerkitoBio/goquery/README.md b/github.com/PuerkitoBio/goquery/README.md index 84f9af39e3..0bc221d163 100644 --- a/github.com/PuerkitoBio/goquery/README.md +++ b/github.com/PuerkitoBio/goquery/README.md @@ -37,6 +37,7 @@ Please note that because of the net/html dependency, goquery requires Go1.1+. **Note that goquery's API is now stable, and will not break.** +* **2020-02-04 (v1.5.1)** : Update module dependencies. * **2018-11-15 (v1.5.0)** : Go module support (thanks @Zaba505). * **2018-06-07 (v1.4.1)** : Add `NewDocumentFromReader` examples. * **2018-03-24 (v1.4.0)** : Deprecate `NewDocument(url)` and `NewDocumentFromResponse(response)`. @@ -138,9 +139,12 @@ func main() { - [Goq][goq], an HTML deserialization and scraping library based on goquery and struct tags. - [andybalholm/cascadia][cascadia], the CSS selector library used by goquery. - [suntong/cascadia][cascadiacli], a command-line interface to the cascadia CSS selector library, useful to test selectors. -- [asciimoo/colly](https://github.com/asciimoo/colly), a lightning fast and elegant Scraping Framework +- [gocolly/colly](https://github.com/gocolly/colly), a lightning fast and elegant Scraping Framework - [gnulnx/goperf](https://github.com/gnulnx/goperf), a website performance test tool that also fetches static assets. - [MontFerret/ferret](https://github.com/MontFerret/ferret), declarative web scraping. +- [tacusci/berrycms](https://github.com/tacusci/berrycms), a modern simple to use CMS with easy to write plugins +- [Dataflow kit](https://github.com/slotix/dataflowkit), Web Scraping framework for Gophers. +- [Geziyor](https://github.com/geziyor/geziyor), a fast web crawling & scraping framework for Go. Supports JS rendering. ## Support diff --git a/github.com/PuerkitoBio/goquery/go.mod b/github.com/PuerkitoBio/goquery/go.mod index 2fa1332a58..95826ad366 100644 --- a/github.com/PuerkitoBio/goquery/go.mod +++ b/github.com/PuerkitoBio/goquery/go.mod @@ -1,6 +1,8 @@ module github.com/PuerkitoBio/goquery require ( - github.com/andybalholm/cascadia v1.0.0 - golang.org/x/net v0.0.0-20181114220301-adae6a3d119a + github.com/andybalholm/cascadia v1.1.0 + golang.org/x/net v0.0.0-20200202094626-16171245cfb2 ) + +go 1.13 diff --git a/github.com/PuerkitoBio/goquery/go.sum b/github.com/PuerkitoBio/goquery/go.sum index 11c5757546..bc79107e59 100644 --- a/github.com/PuerkitoBio/goquery/go.sum +++ b/github.com/PuerkitoBio/goquery/go.sum @@ -1,5 +1,8 @@ -github.com/andybalholm/cascadia v1.0.0 h1:hOCXnnZ5A+3eVDX8pvgl4kofXv2ELss0bKcqRySc45o= -github.com/andybalholm/cascadia v1.0.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y= +github.com/andybalholm/cascadia v1.1.0 h1:BuuO6sSfQNFRu1LppgbD25Hr2vLYW25JvxHs5zzsLTo= +github.com/andybalholm/cascadia v1.1.0/go.mod h1:GsXiBklL0woXo1j/WYWtSYYC4ouU9PqHO0sqidkEA4Y= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= -golang.org/x/net v0.0.0-20181114220301-adae6a3d119a h1:gOpx8G595UYyvj8UK4+OFyY4rx037g3fmfhe5SasG3U= -golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20200202094626-16171245cfb2 h1:CCH4IOTTfewWjGOlSp+zGcjutRKlBEZQ6wTn8ozI/nI= +golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= diff --git a/github.com/alexbrainman/sspi/LICENSE b/github.com/alexbrainman/sspi/LICENSE new file mode 100644 index 0000000000..7448756763 --- /dev/null +++ b/github.com/alexbrainman/sspi/LICENSE @@ -0,0 +1,27 @@ +Copyright (c) 2012 The Go Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/github.com/alexbrainman/sspi/README.md b/github.com/alexbrainman/sspi/README.md new file mode 100644 index 0000000000..848b6b586b --- /dev/null +++ b/github.com/alexbrainman/sspi/README.md @@ -0,0 +1 @@ +This repository holds Go packages for accessing Security Support Provider Interface on Windows. diff --git a/github.com/alexbrainman/sspi/buffer.go b/github.com/alexbrainman/sspi/buffer.go new file mode 100644 index 0000000000..f4b0ef326d --- /dev/null +++ b/github.com/alexbrainman/sspi/buffer.go @@ -0,0 +1,57 @@ +// Copyright 2015 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// +build windows + +package sspi + +import ( + "io" + "unsafe" +) + +func (b *SecBuffer) Set(buftype uint32, data []byte) { + b.BufferType = buftype + if len(data) > 0 { + b.Buffer = &data[0] + b.BufferSize = uint32(len(data)) + } else { + b.Buffer = nil + b.BufferSize = 0 + } +} + +func (b *SecBuffer) Free() error { + if b.Buffer == nil { + return nil + } + return FreeContextBuffer((*byte)(unsafe.Pointer(b.Buffer))) +} + +func (b *SecBuffer) Bytes() []byte { + if b.Buffer == nil || b.BufferSize <= 0 { + return nil + } + return (*[2 << 20]byte)(unsafe.Pointer(b.Buffer))[:b.BufferSize] +} + +func (b *SecBuffer) WriteAll(w io.Writer) (int, error) { + if b.BufferSize == 0 || b.Buffer == nil { + return 0, nil + } + data := b.Bytes() + total := 0 + for { + n, err := w.Write(data) + total += n + if err != nil { + return total, err + } + if n >= len(data) { + break + } + data = data[n:] + } + return total, nil +} diff --git a/github.com/alexbrainman/sspi/mksyscall.go b/github.com/alexbrainman/sspi/mksyscall.go new file mode 100644 index 0000000000..19e1195984 --- /dev/null +++ b/github.com/alexbrainman/sspi/mksyscall.go @@ -0,0 +1,7 @@ +// Copyright 2018 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package sspi + +//go:generate go run $GOROOT/src/syscall/mksyscall_windows.go -systemdll=false -output=zsyscall_windows.go syscall.go diff --git a/github.com/alexbrainman/sspi/negotiate/negotiate.go b/github.com/alexbrainman/sspi/negotiate/negotiate.go new file mode 100644 index 0000000000..e04126087e --- /dev/null +++ b/github.com/alexbrainman/sspi/negotiate/negotiate.go @@ -0,0 +1,462 @@ +// Copyright 2016 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// +build windows + +// Package negotiate provides access to the Microsoft Negotiate SSP Package. +// +package negotiate + +import ( + "errors" + "syscall" + "time" + "unsafe" + + "github.com/alexbrainman/sspi" +) + +// TODO: maybe (if possible) move all winapi related out of sspi and into sspi/internal/winapi + +// PackageInfo contains Negotiate SSP package description. +var PackageInfo *sspi.PackageInfo + +func init() { + var err error + PackageInfo, err = sspi.QueryPackageInfo(sspi.NEGOSSP_NAME) + if err != nil { + panic("failed to fetch Negotiate package info: " + err.Error()) + } +} + +func acquireCredentials(principalName string, creduse uint32, ai *sspi.SEC_WINNT_AUTH_IDENTITY) (*sspi.Credentials, error) { + c, err := sspi.AcquireCredentials(principalName, sspi.NEGOSSP_NAME, creduse, (*byte)(unsafe.Pointer(ai))) + if err != nil { + return nil, err + } + return c, nil +} + +// AcquireCurrentUserCredentials acquires credentials of currently +// logged on user. These will be used by the client to authenticate +// itself to the server. It will also be used by the server +// to impersonate the user. +func AcquireCurrentUserCredentials() (*sspi.Credentials, error) { + return acquireCredentials("", sspi.SECPKG_CRED_OUTBOUND, nil) +} + +// TODO: see if I can share this common ntlm and negotiate code + +// AcquireUserCredentials acquires credentials of user described by +// domain, username and password. These will be used by the client to +// authenticate itself to the server. It will also be used by the +// server to impersonate the user. +func AcquireUserCredentials(domain, username, password string) (*sspi.Credentials, error) { + if len(username) == 0 { + return nil, errors.New("username parameter cannot be empty") + } + d, err := syscall.UTF16FromString(domain) + if err != nil { + return nil, err + } + u, err := syscall.UTF16FromString(username) + if err != nil { + return nil, err + } + p, err := syscall.UTF16FromString(password) + if err != nil { + return nil, err + } + ai := sspi.SEC_WINNT_AUTH_IDENTITY{ + User: &u[0], + UserLength: uint32(len(u) - 1), // do not count terminating 0 + Domain: &d[0], + DomainLength: uint32(len(d) - 1), // do not count terminating 0 + Password: &p[0], + PasswordLength: uint32(len(p) - 1), // do not count terminating 0 + Flags: sspi.SEC_WINNT_AUTH_IDENTITY_UNICODE, + } + return acquireCredentials("", sspi.SECPKG_CRED_OUTBOUND, &ai) +} + +// AcquireServerCredentials acquires server credentials that will +// be used to authenticate clients. +// The principalName parameter is passed to the underlying call to +// the winapi AcquireCredentialsHandle function (and specifies the +// name of the principal whose credentials the underlying handle +// will reference). +// As a special case, using an empty string for the principal name +// will require the credential of the user under whose security context +// the current process is running. +func AcquireServerCredentials(principalName string) (*sspi.Credentials, error) { + return acquireCredentials(principalName, sspi.SECPKG_CRED_INBOUND, nil) +} + +func updateContext(c *sspi.Context, dst, src []byte, targetName *uint16) (authCompleted bool, n int, err error) { + var inBuf, outBuf [1]sspi.SecBuffer + inBuf[0].Set(sspi.SECBUFFER_TOKEN, src) + inBufs := &sspi.SecBufferDesc{ + Version: sspi.SECBUFFER_VERSION, + BuffersCount: 1, + Buffers: &inBuf[0], + } + outBuf[0].Set(sspi.SECBUFFER_TOKEN, dst) + outBufs := &sspi.SecBufferDesc{ + Version: sspi.SECBUFFER_VERSION, + BuffersCount: 1, + Buffers: &outBuf[0], + } + ret := c.Update(targetName, outBufs, inBufs) + switch ret { + case sspi.SEC_E_OK: + // session established -> return success + return true, int(outBuf[0].BufferSize), nil + case sspi.SEC_I_COMPLETE_NEEDED, sspi.SEC_I_COMPLETE_AND_CONTINUE: + ret = sspi.CompleteAuthToken(c.Handle, outBufs) + if ret != sspi.SEC_E_OK { + return false, 0, ret + } + case sspi.SEC_I_CONTINUE_NEEDED: + default: + return false, 0, ret + } + return false, int(outBuf[0].BufferSize), nil +} + +func makeSignature(c *sspi.Context, msg []byte, qop, seqno uint32) ([]byte, error) { + _, maxSignature, _, _, err := c.Sizes() + if err != nil { + return nil, err + } + + if maxSignature == 0 { + return nil, errors.New("integrity services are not requested or unavailable") + } + + var b [2]sspi.SecBuffer + b[0].Set(sspi.SECBUFFER_DATA, msg) + b[1].Set(sspi.SECBUFFER_TOKEN, make([]byte, maxSignature)) + + ret := sspi.MakeSignature(c.Handle, qop, sspi.NewSecBufferDesc(b[:]), seqno) + if ret != sspi.SEC_E_OK { + return nil, ret + } + + return b[1].Bytes(), nil +} + +func encryptMessage(c *sspi.Context, msg []byte, qop, seqno uint32) ([]byte, error) { + _ /*maxToken*/, maxSignature, cBlockSize, cSecurityTrailer, err := c.Sizes() + if err != nil { + return nil, err + } + + if maxSignature == 0 { + return nil, errors.New("integrity services are not requested or unavailable") + } + + var b [3]sspi.SecBuffer + b[0].Set(sspi.SECBUFFER_TOKEN, make([]byte, cSecurityTrailer)) + b[1].Set(sspi.SECBUFFER_DATA, msg) + b[2].Set(sspi.SECBUFFER_PADDING, make([]byte, cBlockSize)) + + ret := sspi.EncryptMessage(c.Handle, qop, sspi.NewSecBufferDesc(b[:]), seqno) + if ret != sspi.SEC_E_OK { + return nil, ret + } + + r0, r1, r2 := b[0].Bytes(), b[1].Bytes(), b[2].Bytes() + res := make([]byte, 0, len(r0)+len(r1)+len(r2)) + res = append(res, r0...) + res = append(res, r1...) + res = append(res, r2...) + + return res, nil +} + +func decryptMessage(c *sspi.Context, msg []byte, seqno uint32) (uint32, []byte, error) { + var b [2]sspi.SecBuffer + b[0].Set(sspi.SECBUFFER_STREAM, msg) + b[1].Set(sspi.SECBUFFER_DATA, []byte{}) + + var qop uint32 + ret := sspi.DecryptMessage(c.Handle, sspi.NewSecBufferDesc(b[:]), seqno, &qop) + if ret != sspi.SEC_E_OK { + return qop, nil, ret + } + + return qop, b[1].Bytes(), nil +} + +func verifySignature(c *sspi.Context, msg, token []byte, seqno uint32) (uint32, error) { + var b [2]sspi.SecBuffer + b[0].Set(sspi.SECBUFFER_DATA, msg) + b[1].Set(sspi.SECBUFFER_TOKEN, token) + + var qop uint32 + + ret := sspi.VerifySignature(c.Handle, sspi.NewSecBufferDesc(b[:]), seqno, &qop) + if ret != sspi.SEC_E_OK { + return 0, ret + } + + return qop, nil +} + +// ClientContext is used by the client to manage all steps of Negotiate negotiation. +type ClientContext struct { + sctxt *sspi.Context + targetName *uint16 +} + +// NewClientContext creates a new client context. It uses client +// credentials cred generated by AcquireCurrentUserCredentials or +// AcquireUserCredentials and SPN to start a client Negotiate +// negotiation sequence. targetName is the service principal name +// (SPN) or the security context of the destination server. +// NewClientContext returns a new token to be sent to the server. +func NewClientContext(cred *sspi.Credentials, targetName string) (cc *ClientContext, outputToken []byte, err error) { + return NewClientContextWithFlags(cred, targetName, sspi.ISC_REQ_CONNECTION) +} + +// NewClientContextWithFlags creates a new client context. It uses client +// credentials cred generated by AcquireCurrentUserCredentials or +// AcquireUserCredentials and SPN to start a client Negotiate +// negotiation sequence. targetName is the service principal name +// (SPN) or the security context of the destination server. +// The flags parameter is used to indicate requests for the context +// (for example sspi.ISC_REQ_CONFIDENTIALITY|sspi.ISC_REQ_REPLAY_DETECT) +// NewClientContextWithFlags returns a new token to be sent to the server. +func NewClientContextWithFlags(cred *sspi.Credentials, targetName string, flags uint32) (cc *ClientContext, outputToken []byte, err error) { + var tname *uint16 + if len(targetName) > 0 { + p, err2 := syscall.UTF16FromString(targetName) + if err2 != nil { + return nil, nil, err2 + } + if len(p) > 0 { + tname = &p[0] + } + } + otoken := make([]byte, PackageInfo.MaxToken) + c := sspi.NewClientContext(cred, flags) + + authCompleted, n, err2 := updateContext(c, otoken, nil, tname) + if err2 != nil { + return nil, nil, err2 + } + if authCompleted { + c.Release() + return nil, nil, errors.New("negotiate authentication should not be completed yet") + } + if n == 0 { + c.Release() + return nil, nil, errors.New("negotiate token should not be empty") + } + otoken = otoken[:n] + return &ClientContext{sctxt: c, targetName: tname}, otoken, nil +} + +// Release free up resources associated with client context c. +func (c *ClientContext) Release() error { + if c == nil { + return nil + } + return c.sctxt.Release() +} + +// Expiry returns c expiry time. +func (c *ClientContext) Expiry() time.Time { + return c.sctxt.Expiry() +} + +// Update advances client part of Negotiate negotiation c. It uses +// token received from the server and returns true if client part +// of authentication is complete. It also returns new token to be +// sent to the server. +func (c *ClientContext) Update(token []byte) (authCompleted bool, outputToken []byte, err error) { + otoken := make([]byte, PackageInfo.MaxToken) + authDone, n, err2 := updateContext(c.sctxt, otoken, token, c.targetName) + if err2 != nil { + return false, nil, err2 + } + if n == 0 && !authDone { + return false, nil, errors.New("negotiate token should not be empty") + } + otoken = otoken[:n] + return authDone, otoken, nil +} + +// Sizes queries the client context for the sizes used in per-message +// functions. It returns the maximum token size used in authentication +// exchanges, the maximum signature size, the preferred integral size of +// messages, the size of any security trailer, and any error. +func (c *ClientContext) Sizes() (uint32, uint32, uint32, uint32, error) { + return c.sctxt.Sizes() +} + +// MakeSignature uses the established client context to create a signature +// for the given message using the provided quality of protection flags and +// sequence number. It returns the signature token in addition to any error. +func (c *ClientContext) MakeSignature(msg []byte, qop, seqno uint32) ([]byte, error) { + return makeSignature(c.sctxt, msg, qop, seqno) +} + +// VerifySignature uses the established client context and signature token +// to check that the provided message hasn't been tampered or received out +// of sequence. It returns any quality of protection flags and any error +// that occurred. +func (c *ClientContext) VerifySignature(msg, token []byte, seqno uint32) (uint32, error) { + return verifySignature(c.sctxt, msg, token, seqno) +} + +// EncryptMessage uses the established client context to encrypt a message +// using the provided quality of protection flags and sequence number. +// It returns the signature token in addition to any error. +// IMPORTANT: the input msg parameter is updated in place by the low-level windows api +// so must be copied if the initial content should not be modified. +func (c *ClientContext) EncryptMessage(msg []byte, qop, seqno uint32) ([]byte, error) { + return encryptMessage(c.sctxt, msg, qop, seqno) +} + +// DecryptMessage uses the established client context to decrypt a message +// using the provided sequence number. +// It returns the quality of protection flag and the decrypted message in addition to any error. +func (c *ClientContext) DecryptMessage(msg []byte, seqno uint32) (uint32, []byte, error) { + return decryptMessage(c.sctxt, msg, seqno) +} + +// VerifyFlags determines if all flags used to construct the client context +// were honored (see NewClientContextWithFlags). It should be called after c.Update. +func (c *ClientContext) VerifyFlags() error { + return c.sctxt.VerifyFlags() +} + +// VerifySelectiveFlags determines if the given flags were honored (see NewClientContextWithFlags). +// It should be called after c.Update. +func (c *ClientContext) VerifySelectiveFlags(flags uint32) error { + return c.sctxt.VerifySelectiveFlags(flags) +} + +// ServerContext is used by the server to manage all steps of Negotiate +// negotiation. Once authentication is completed the context can be +// used to impersonate client. +type ServerContext struct { + sctxt *sspi.Context +} + +// NewServerContext creates new server context. It uses server +// credentials created by AcquireServerCredentials and token from +// the client to start server Negotiate negotiation sequence. +// It also returns new token to be sent to the client. +func NewServerContext(cred *sspi.Credentials, token []byte) (sc *ServerContext, authDone bool, outputToken []byte, err error) { + otoken := make([]byte, PackageInfo.MaxToken) + c := sspi.NewServerContext(cred, sspi.ASC_REQ_CONNECTION) + authDone, n, err2 := updateContext(c, otoken, token, nil) + if err2 != nil { + return nil, false, nil, err2 + } + otoken = otoken[:n] + return &ServerContext{sctxt: c}, authDone, otoken, nil +} + +// Release free up resources associated with server context c. +func (c *ServerContext) Release() error { + if c == nil { + return nil + } + return c.sctxt.Release() +} + +// Expiry returns c expiry time. +func (c *ServerContext) Expiry() time.Time { + return c.sctxt.Expiry() +} + +// Update advances server part of Negotiate negotiation c. It uses +// token received from the client and returns true if server part +// of authentication is complete. It also returns new token to be +// sent to the client. +func (c *ServerContext) Update(token []byte) (authCompleted bool, outputToken []byte, err error) { + otoken := make([]byte, PackageInfo.MaxToken) + authDone, n, err2 := updateContext(c.sctxt, otoken, token, nil) + if err2 != nil { + return false, nil, err2 + } + if n == 0 && !authDone { + return false, nil, errors.New("negotiate token should not be empty") + } + otoken = otoken[:n] + return authDone, otoken, nil +} + +const _SECPKG_ATTR_NATIVE_NAMES = 13 + +type _SecPkgContext_NativeNames struct { + ClientName *uint16 + ServerName *uint16 +} + +// GetUsername returns the username corresponding to the authenticated client +func (c *ServerContext) GetUsername() (string, error) { + var ns _SecPkgContext_NativeNames + ret := sspi.QueryContextAttributes(c.sctxt.Handle, _SECPKG_ATTR_NATIVE_NAMES, (*byte)(unsafe.Pointer(&ns))) + if ret != sspi.SEC_E_OK { + return "", ret + } + sspi.FreeContextBuffer((*byte)(unsafe.Pointer(ns.ServerName))) + defer sspi.FreeContextBuffer((*byte)(unsafe.Pointer(ns.ClientName))) + return syscall.UTF16ToString((*[2 << 20]uint16)(unsafe.Pointer(ns.ClientName))[:]), nil +} + +// ImpersonateUser changes current OS thread user. New user is +// the user as specified by client credentials. +func (c *ServerContext) ImpersonateUser() error { + return c.sctxt.ImpersonateUser() +} + +// RevertToSelf stops impersonation. It changes current OS thread +// user to what it was before ImpersonateUser was executed. +func (c *ServerContext) RevertToSelf() error { + return c.sctxt.RevertToSelf() +} + +// Sizes queries the server context for the sizes used in per-message +// functions. It returns the maximum token size used in authentication +// exchanges, the maximum signature size, the preferred integral size of +// messages, the size of any security trailer, and any error. +func (c *ServerContext) Sizes() (uint32, uint32, uint32, uint32, error) { + return c.sctxt.Sizes() +} + +// MakeSignature uses the established server context to create a signature +// for the given message using the provided quality of protection flags and +// sequence number. It returns the signature token in addition to any error. +func (c *ServerContext) MakeSignature(msg []byte, qop, seqno uint32) ([]byte, error) { + return makeSignature(c.sctxt, msg, qop, seqno) +} + +// VerifySignature uses the established server context and signature token +// to check that the provided message hasn't been tampered or received out +// of sequence. It returns any quality of protection flags and any error +// that occurred. +func (c *ServerContext) VerifySignature(msg, token []byte, seqno uint32) (uint32, error) { + return verifySignature(c.sctxt, msg, token, seqno) +} + +// EncryptMessage uses the established server context to encrypt a message +// using the provided quality of protection flags and sequence number. +// It returns the signature token in addition to any error. +// IMPORTANT: the input msg parameter is updated in place by the low-level windows api +// so must be copied if the initial content should not be modified. +func (c *ServerContext) EncryptMessage(msg []byte, qop, seqno uint32) ([]byte, error) { + return encryptMessage(c.sctxt, msg, qop, seqno) +} + +// DecryptMessage uses the established server context to decrypt a message +// using the provided sequence number. +// It returns the quality of protection flag and the decrypted message in addition to any error. +func (c *ServerContext) DecryptMessage(msg []byte, seqno uint32) (uint32, []byte, error) { + return decryptMessage(c.sctxt, msg, seqno) +} diff --git a/github.com/alexbrainman/sspi/sspi.go b/github.com/alexbrainman/sspi/sspi.go new file mode 100644 index 0000000000..04f20b75ae --- /dev/null +++ b/github.com/alexbrainman/sspi/sspi.go @@ -0,0 +1,226 @@ +// Copyright 2015 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// +build windows + +package sspi + +import ( + "fmt" + "syscall" + "time" + "unsafe" +) + +// TODO: add documentation + +type PackageInfo struct { + Capabilities uint32 + Version uint16 + RPCID uint16 + MaxToken uint32 + Name string + Comment string +} + +func QueryPackageInfo(pkgname string) (*PackageInfo, error) { + name, err := syscall.UTF16PtrFromString(pkgname) + if err != nil { + return nil, err + } + var pi *SecPkgInfo + ret := QuerySecurityPackageInfo(name, &pi) + if ret != SEC_E_OK { + return nil, ret + } + defer FreeContextBuffer((*byte)(unsafe.Pointer(pi))) + + return &PackageInfo{ + Capabilities: pi.Capabilities, + Version: pi.Version, + RPCID: pi.RPCID, + MaxToken: pi.MaxToken, + Name: syscall.UTF16ToString((*[2 << 12]uint16)(unsafe.Pointer(pi.Name))[:]), + Comment: syscall.UTF16ToString((*[2 << 12]uint16)(unsafe.Pointer(pi.Comment))[:]), + }, nil +} + +type Credentials struct { + Handle CredHandle + expiry syscall.Filetime +} + +// AcquireCredentials calls the windows AcquireCredentialsHandle function and +// returns Credentials containing a security handle that can be used for +// InitializeSecurityContext or AcceptSecurityContext operations. +// As a special case, passing an empty string as the principal parameter will +// pass a null string to the underlying function. +func AcquireCredentials(principal string, pkgname string, creduse uint32, authdata *byte) (*Credentials, error) { + var principalName *uint16 + if principal != "" { + var err error + principalName, err = syscall.UTF16PtrFromString(principal) + if err != nil { + return nil, err + } + } + name, err := syscall.UTF16PtrFromString(pkgname) + if err != nil { + return nil, err + } + var c Credentials + ret := AcquireCredentialsHandle(principalName, name, creduse, nil, authdata, 0, 0, &c.Handle, &c.expiry) + if ret != SEC_E_OK { + return nil, ret + } + return &c, nil +} + +func (c *Credentials) Release() error { + if c == nil { + return nil + } + ret := FreeCredentialsHandle(&c.Handle) + if ret != SEC_E_OK { + return ret + } + return nil +} + +func (c *Credentials) Expiry() time.Time { + return time.Unix(0, c.expiry.Nanoseconds()) +} + +// TODO: add functions to display and manage RequestedFlags and EstablishedFlags fields. +// TODO: maybe get rid of RequestedFlags and EstablishedFlags fields, and replace them with input parameter for New...Context and return value of Update (instead of current bool parameter). + +type updateFunc func(c *Context, targname *uint16, h, newh *CtxtHandle, out, in *SecBufferDesc) syscall.Errno + +type Context struct { + Cred *Credentials + Handle *CtxtHandle + handle CtxtHandle + updFn updateFunc + expiry syscall.Filetime + RequestedFlags uint32 + EstablishedFlags uint32 +} + +func NewClientContext(cred *Credentials, flags uint32) *Context { + return &Context{ + Cred: cred, + updFn: initialize, + RequestedFlags: flags, + } +} + +func NewServerContext(cred *Credentials, flags uint32) *Context { + return &Context{ + Cred: cred, + updFn: accept, + RequestedFlags: flags, + } +} + +func initialize(c *Context, targname *uint16, h, newh *CtxtHandle, out, in *SecBufferDesc) syscall.Errno { + return InitializeSecurityContext(&c.Cred.Handle, h, targname, c.RequestedFlags, + 0, SECURITY_NATIVE_DREP, in, 0, newh, out, &c.EstablishedFlags, &c.expiry) +} + +func accept(c *Context, targname *uint16, h, newh *CtxtHandle, out, in *SecBufferDesc) syscall.Errno { + return AcceptSecurityContext(&c.Cred.Handle, h, in, c.RequestedFlags, + SECURITY_NATIVE_DREP, newh, out, &c.EstablishedFlags, &c.expiry) +} + +func (c *Context) Update(targname *uint16, out, in *SecBufferDesc) syscall.Errno { + h := c.Handle + if c.Handle == nil { + c.Handle = &c.handle + } + return c.updFn(c, targname, h, c.Handle, out, in) +} + +func (c *Context) Release() error { + if c == nil { + return nil + } + ret := DeleteSecurityContext(c.Handle) + if ret != SEC_E_OK { + return ret + } + return nil +} + +func (c *Context) Expiry() time.Time { + return time.Unix(0, c.expiry.Nanoseconds()) +} + +// TODO: add comment to function doco that this "impersonation" is applied to current OS thread. +func (c *Context) ImpersonateUser() error { + ret := ImpersonateSecurityContext(c.Handle) + if ret != SEC_E_OK { + return ret + } + return nil +} + +func (c *Context) RevertToSelf() error { + ret := RevertSecurityContext(c.Handle) + if ret != SEC_E_OK { + return ret + } + return nil +} + +// Sizes queries the context for the sizes used in per-message functions. +// It returns the maximum token size used in authentication exchanges, the +// maximum signature size, the preferred integral size of messages, the +// size of any security trailer, and any error. +func (c *Context) Sizes() (uint32, uint32, uint32, uint32, error) { + var s _SecPkgContext_Sizes + ret := QueryContextAttributes(c.Handle, _SECPKG_ATTR_SIZES, (*byte)(unsafe.Pointer(&s))) + if ret != SEC_E_OK { + return 0, 0, 0, 0, ret + } + return s.MaxToken, s.MaxSignature, s.BlockSize, s.SecurityTrailer, nil +} + +// VerifyFlags determines if all flags used to construct the context +// were honored (see NewClientContext). It should be called after c.Update. +func (c *Context) VerifyFlags() error { + return c.VerifySelectiveFlags(c.RequestedFlags) +} + +// VerifySelectiveFlags determines if the given flags were honored (see NewClientContext). +// It should be called after c.Update. +func (c *Context) VerifySelectiveFlags(flags uint32) error { + if valid, missing, extra := verifySelectiveFlags(flags, c.RequestedFlags); !valid { + return fmt.Errorf("sspi: invalid flags check: desired=%b requested=%b missing=%b extra=%b", flags, c.RequestedFlags, missing, extra) + } + if valid, missing, extra := verifySelectiveFlags(flags, c.EstablishedFlags); !valid { + return fmt.Errorf("sspi: invalid flags: desired=%b established=%b missing=%b extra=%b", flags, c.EstablishedFlags, missing, extra) + } + return nil +} + +// verifySelectiveFlags determines if all bits requested in flags are set in establishedFlags. +// missing represents the bits set in flags that are not set in establishedFlags. +// extra represents the bits set in establishedFlags that are not set in flags. +// valid is true and missing is zero when establishedFlags has all of the requested flags. +func verifySelectiveFlags(flags, establishedFlags uint32) (valid bool, missing, extra uint32) { + missing = flags&establishedFlags ^ flags + extra = flags | establishedFlags ^ flags + valid = missing == 0 + return valid, missing, extra +} + +// NewSecBufferDesc returns an initialized SecBufferDesc describing the +// provided SecBuffer. +func NewSecBufferDesc(b []SecBuffer) *SecBufferDesc { + return &SecBufferDesc{ + Version: SECBUFFER_VERSION, + BuffersCount: uint32(len(b)), + Buffers: &b[0], + } +} diff --git a/github.com/alexbrainman/sspi/syscall.go b/github.com/alexbrainman/sspi/syscall.go new file mode 100644 index 0000000000..04660df2ae --- /dev/null +++ b/github.com/alexbrainman/sspi/syscall.go @@ -0,0 +1,174 @@ +// Copyright 2015 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// +build windows + +package sspi + +import ( + "syscall" +) + +const ( + SEC_E_OK = syscall.Errno(0) + + SEC_I_COMPLETE_AND_CONTINUE = syscall.Errno(590612) + SEC_I_COMPLETE_NEEDED = syscall.Errno(590611) + SEC_I_CONTINUE_NEEDED = syscall.Errno(590610) + + SEC_E_LOGON_DENIED = syscall.Errno(0x8009030c) + SEC_E_CONTEXT_EXPIRED = syscall.Errno(0x80090317) // not sure if the value is valid + SEC_E_INCOMPLETE_MESSAGE = syscall.Errno(0x80090318) + + NTLMSP_NAME = "NTLM" + MICROSOFT_KERBEROS_NAME = "Kerberos" + NEGOSSP_NAME = "Negotiate" + UNISP_NAME = "Microsoft Unified Security Protocol Provider" + + _SECPKG_ATTR_SIZES = 0 + _SECPKG_ATTR_NAMES = 1 + _SECPKG_ATTR_LIFESPAN = 2 + _SECPKG_ATTR_DCE_INFO = 3 + _SECPKG_ATTR_STREAM_SIZES = 4 + _SECPKG_ATTR_KEY_INFO = 5 + _SECPKG_ATTR_AUTHORITY = 6 + _SECPKG_ATTR_PROTO_INFO = 7 + _SECPKG_ATTR_PASSWORD_EXPIRY = 8 + _SECPKG_ATTR_SESSION_KEY = 9 + _SECPKG_ATTR_PACKAGE_INFO = 10 + _SECPKG_ATTR_USER_FLAGS = 11 + _SECPKG_ATTR_NEGOTIATION_INFO = 12 + _SECPKG_ATTR_NATIVE_NAMES = 13 + _SECPKG_ATTR_FLAGS = 14 +) + +type SecPkgInfo struct { + Capabilities uint32 + Version uint16 + RPCID uint16 + MaxToken uint32 + Name *uint16 + Comment *uint16 +} + +type _SecPkgContext_Sizes struct { + MaxToken uint32 + MaxSignature uint32 + BlockSize uint32 + SecurityTrailer uint32 +} + +//sys QuerySecurityPackageInfo(pkgname *uint16, pkginfo **SecPkgInfo) (ret syscall.Errno) = secur32.QuerySecurityPackageInfoW +//sys FreeContextBuffer(buf *byte) (ret syscall.Errno) = secur32.FreeContextBuffer + +const ( + SECPKG_CRED_INBOUND = 1 + SECPKG_CRED_OUTBOUND = 2 + SECPKG_CRED_BOTH = (SECPKG_CRED_OUTBOUND | SECPKG_CRED_INBOUND) + + SEC_WINNT_AUTH_IDENTITY_UNICODE = 0x2 +) + +type SEC_WINNT_AUTH_IDENTITY struct { + User *uint16 + UserLength uint32 + Domain *uint16 + DomainLength uint32 + Password *uint16 + PasswordLength uint32 + Flags uint32 +} + +type LUID struct { + LowPart uint32 + HighPart int32 +} + +type CredHandle struct { + Lower uintptr + Upper uintptr +} + +//sys AcquireCredentialsHandle(principal *uint16, pkgname *uint16, creduse uint32, logonid *LUID, authdata *byte, getkeyfn uintptr, getkeyarg uintptr, handle *CredHandle, expiry *syscall.Filetime) (ret syscall.Errno) = secur32.AcquireCredentialsHandleW +//sys FreeCredentialsHandle(handle *CredHandle) (ret syscall.Errno) = secur32.FreeCredentialsHandle + +const ( + SECURITY_NATIVE_DREP = 16 + + SECBUFFER_DATA = 1 + SECBUFFER_TOKEN = 2 + SECBUFFER_PKG_PARAMS = 3 + SECBUFFER_MISSING = 4 + SECBUFFER_EXTRA = 5 + SECBUFFER_STREAM_TRAILER = 6 + SECBUFFER_STREAM_HEADER = 7 + SECBUFFER_PADDING = 9 + SECBUFFER_STREAM = 10 + SECBUFFER_READONLY = 0x80000000 + SECBUFFER_ATTRMASK = 0xf0000000 + SECBUFFER_VERSION = 0 + SECBUFFER_EMPTY = 0 + + ISC_REQ_DELEGATE = 1 + ISC_REQ_MUTUAL_AUTH = 2 + ISC_REQ_REPLAY_DETECT = 4 + ISC_REQ_SEQUENCE_DETECT = 8 + ISC_REQ_CONFIDENTIALITY = 16 + ISC_REQ_USE_SESSION_KEY = 32 + ISC_REQ_PROMPT_FOR_CREDS = 64 + ISC_REQ_USE_SUPPLIED_CREDS = 128 + ISC_REQ_ALLOCATE_MEMORY = 256 + ISC_REQ_USE_DCE_STYLE = 512 + ISC_REQ_DATAGRAM = 1024 + ISC_REQ_CONNECTION = 2048 + ISC_REQ_EXTENDED_ERROR = 16384 + ISC_REQ_STREAM = 32768 + ISC_REQ_INTEGRITY = 65536 + ISC_REQ_MANUAL_CRED_VALIDATION = 524288 + ISC_REQ_HTTP = 268435456 + + ASC_REQ_DELEGATE = 1 + ASC_REQ_MUTUAL_AUTH = 2 + ASC_REQ_REPLAY_DETECT = 4 + ASC_REQ_SEQUENCE_DETECT = 8 + ASC_REQ_CONFIDENTIALITY = 16 + ASC_REQ_USE_SESSION_KEY = 32 + ASC_REQ_ALLOCATE_MEMORY = 256 + ASC_REQ_USE_DCE_STYLE = 512 + ASC_REQ_DATAGRAM = 1024 + ASC_REQ_CONNECTION = 2048 + ASC_REQ_EXTENDED_ERROR = 32768 + ASC_REQ_STREAM = 65536 + ASC_REQ_INTEGRITY = 131072 +) + +type CtxtHandle struct { + Lower uintptr + Upper uintptr +} + +type SecBuffer struct { + BufferSize uint32 + BufferType uint32 + Buffer *byte +} + +type SecBufferDesc struct { + Version uint32 + BuffersCount uint32 + Buffers *SecBuffer +} + +//sys InitializeSecurityContext(credential *CredHandle, context *CtxtHandle, targname *uint16, contextreq uint32, reserved1 uint32, targdatarep uint32, input *SecBufferDesc, reserved2 uint32, newcontext *CtxtHandle, output *SecBufferDesc, contextattr *uint32, expiry *syscall.Filetime) (ret syscall.Errno) = secur32.InitializeSecurityContextW +//sys AcceptSecurityContext(credential *CredHandle, context *CtxtHandle, input *SecBufferDesc, contextreq uint32, targdatarep uint32, newcontext *CtxtHandle, output *SecBufferDesc, contextattr *uint32, expiry *syscall.Filetime) (ret syscall.Errno) = secur32.AcceptSecurityContext +//sys CompleteAuthToken(context *CtxtHandle, token *SecBufferDesc) (ret syscall.Errno) = secur32.CompleteAuthToken +//sys DeleteSecurityContext(context *CtxtHandle) (ret syscall.Errno) = secur32.DeleteSecurityContext +//sys ImpersonateSecurityContext(context *CtxtHandle) (ret syscall.Errno) = secur32.ImpersonateSecurityContext +//sys RevertSecurityContext(context *CtxtHandle) (ret syscall.Errno) = secur32.RevertSecurityContext +//sys QueryContextAttributes(context *CtxtHandle, attribute uint32, buf *byte) (ret syscall.Errno) = secur32.QueryContextAttributesW +//sys EncryptMessage(context *CtxtHandle, qop uint32, message *SecBufferDesc, messageseqno uint32) (ret syscall.Errno) = secur32.EncryptMessage +//sys DecryptMessage(context *CtxtHandle, message *SecBufferDesc, messageseqno uint32, qop *uint32) (ret syscall.Errno) = secur32.DecryptMessage +//sys ApplyControlToken(context *CtxtHandle, input *SecBufferDesc) (ret syscall.Errno) = secur32.ApplyControlToken +//sys MakeSignature(context *CtxtHandle, qop uint32, message *SecBufferDesc, messageseqno uint32) (ret syscall.Errno) = secur32.MakeSignature +//sys VerifySignature(context *CtxtHandle, message *SecBufferDesc, messageseqno uint32, qop *uint32) (ret syscall.Errno) = secur32.VerifySignature diff --git a/github.com/alexbrainman/sspi/zsyscall_windows.go b/github.com/alexbrainman/sspi/zsyscall_windows.go new file mode 100644 index 0000000000..55e8209970 --- /dev/null +++ b/github.com/alexbrainman/sspi/zsyscall_windows.go @@ -0,0 +1,152 @@ +// MACHINE GENERATED BY 'go generate' COMMAND; DO NOT EDIT + +package sspi + +import ( + "syscall" + "unsafe" +) + +var _ unsafe.Pointer + +// Do the interface allocations only once for common +// Errno values. +const ( + errnoERROR_IO_PENDING = 997 +) + +var ( + errERROR_IO_PENDING error = syscall.Errno(errnoERROR_IO_PENDING) +) + +// errnoErr returns common boxed Errno values, to prevent +// allocations at runtime. +func errnoErr(e syscall.Errno) error { + switch e { + case 0: + return nil + case errnoERROR_IO_PENDING: + return errERROR_IO_PENDING + } + // TODO: add more here, after collecting data on the common + // error values see on Windows. (perhaps when running + // all.bat?) + return e +} + +var ( + modsecur32 = syscall.NewLazyDLL("secur32.dll") + + procQuerySecurityPackageInfoW = modsecur32.NewProc("QuerySecurityPackageInfoW") + procFreeContextBuffer = modsecur32.NewProc("FreeContextBuffer") + procAcquireCredentialsHandleW = modsecur32.NewProc("AcquireCredentialsHandleW") + procFreeCredentialsHandle = modsecur32.NewProc("FreeCredentialsHandle") + procInitializeSecurityContextW = modsecur32.NewProc("InitializeSecurityContextW") + procAcceptSecurityContext = modsecur32.NewProc("AcceptSecurityContext") + procCompleteAuthToken = modsecur32.NewProc("CompleteAuthToken") + procDeleteSecurityContext = modsecur32.NewProc("DeleteSecurityContext") + procImpersonateSecurityContext = modsecur32.NewProc("ImpersonateSecurityContext") + procRevertSecurityContext = modsecur32.NewProc("RevertSecurityContext") + procQueryContextAttributesW = modsecur32.NewProc("QueryContextAttributesW") + procEncryptMessage = modsecur32.NewProc("EncryptMessage") + procDecryptMessage = modsecur32.NewProc("DecryptMessage") + procApplyControlToken = modsecur32.NewProc("ApplyControlToken") + procMakeSignature = modsecur32.NewProc("MakeSignature") + procVerifySignature = modsecur32.NewProc("VerifySignature") +) + +func QuerySecurityPackageInfo(pkgname *uint16, pkginfo **SecPkgInfo) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall(procQuerySecurityPackageInfoW.Addr(), 2, uintptr(unsafe.Pointer(pkgname)), uintptr(unsafe.Pointer(pkginfo)), 0) + ret = syscall.Errno(r0) + return +} + +func FreeContextBuffer(buf *byte) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall(procFreeContextBuffer.Addr(), 1, uintptr(unsafe.Pointer(buf)), 0, 0) + ret = syscall.Errno(r0) + return +} + +func AcquireCredentialsHandle(principal *uint16, pkgname *uint16, creduse uint32, logonid *LUID, authdata *byte, getkeyfn uintptr, getkeyarg uintptr, handle *CredHandle, expiry *syscall.Filetime) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall9(procAcquireCredentialsHandleW.Addr(), 9, uintptr(unsafe.Pointer(principal)), uintptr(unsafe.Pointer(pkgname)), uintptr(creduse), uintptr(unsafe.Pointer(logonid)), uintptr(unsafe.Pointer(authdata)), uintptr(getkeyfn), uintptr(getkeyarg), uintptr(unsafe.Pointer(handle)), uintptr(unsafe.Pointer(expiry))) + ret = syscall.Errno(r0) + return +} + +func FreeCredentialsHandle(handle *CredHandle) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall(procFreeCredentialsHandle.Addr(), 1, uintptr(unsafe.Pointer(handle)), 0, 0) + ret = syscall.Errno(r0) + return +} + +func InitializeSecurityContext(credential *CredHandle, context *CtxtHandle, targname *uint16, contextreq uint32, reserved1 uint32, targdatarep uint32, input *SecBufferDesc, reserved2 uint32, newcontext *CtxtHandle, output *SecBufferDesc, contextattr *uint32, expiry *syscall.Filetime) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall12(procInitializeSecurityContextW.Addr(), 12, uintptr(unsafe.Pointer(credential)), uintptr(unsafe.Pointer(context)), uintptr(unsafe.Pointer(targname)), uintptr(contextreq), uintptr(reserved1), uintptr(targdatarep), uintptr(unsafe.Pointer(input)), uintptr(reserved2), uintptr(unsafe.Pointer(newcontext)), uintptr(unsafe.Pointer(output)), uintptr(unsafe.Pointer(contextattr)), uintptr(unsafe.Pointer(expiry))) + ret = syscall.Errno(r0) + return +} + +func AcceptSecurityContext(credential *CredHandle, context *CtxtHandle, input *SecBufferDesc, contextreq uint32, targdatarep uint32, newcontext *CtxtHandle, output *SecBufferDesc, contextattr *uint32, expiry *syscall.Filetime) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall9(procAcceptSecurityContext.Addr(), 9, uintptr(unsafe.Pointer(credential)), uintptr(unsafe.Pointer(context)), uintptr(unsafe.Pointer(input)), uintptr(contextreq), uintptr(targdatarep), uintptr(unsafe.Pointer(newcontext)), uintptr(unsafe.Pointer(output)), uintptr(unsafe.Pointer(contextattr)), uintptr(unsafe.Pointer(expiry))) + ret = syscall.Errno(r0) + return +} + +func CompleteAuthToken(context *CtxtHandle, token *SecBufferDesc) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall(procCompleteAuthToken.Addr(), 2, uintptr(unsafe.Pointer(context)), uintptr(unsafe.Pointer(token)), 0) + ret = syscall.Errno(r0) + return +} + +func DeleteSecurityContext(context *CtxtHandle) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall(procDeleteSecurityContext.Addr(), 1, uintptr(unsafe.Pointer(context)), 0, 0) + ret = syscall.Errno(r0) + return +} + +func ImpersonateSecurityContext(context *CtxtHandle) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall(procImpersonateSecurityContext.Addr(), 1, uintptr(unsafe.Pointer(context)), 0, 0) + ret = syscall.Errno(r0) + return +} + +func RevertSecurityContext(context *CtxtHandle) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall(procRevertSecurityContext.Addr(), 1, uintptr(unsafe.Pointer(context)), 0, 0) + ret = syscall.Errno(r0) + return +} + +func QueryContextAttributes(context *CtxtHandle, attribute uint32, buf *byte) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall(procQueryContextAttributesW.Addr(), 3, uintptr(unsafe.Pointer(context)), uintptr(attribute), uintptr(unsafe.Pointer(buf))) + ret = syscall.Errno(r0) + return +} + +func EncryptMessage(context *CtxtHandle, qop uint32, message *SecBufferDesc, messageseqno uint32) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall6(procEncryptMessage.Addr(), 4, uintptr(unsafe.Pointer(context)), uintptr(qop), uintptr(unsafe.Pointer(message)), uintptr(messageseqno), 0, 0) + ret = syscall.Errno(r0) + return +} + +func DecryptMessage(context *CtxtHandle, message *SecBufferDesc, messageseqno uint32, qop *uint32) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall6(procDecryptMessage.Addr(), 4, uintptr(unsafe.Pointer(context)), uintptr(unsafe.Pointer(message)), uintptr(messageseqno), uintptr(unsafe.Pointer(qop)), 0, 0) + ret = syscall.Errno(r0) + return +} + +func ApplyControlToken(context *CtxtHandle, input *SecBufferDesc) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall(procApplyControlToken.Addr(), 2, uintptr(unsafe.Pointer(context)), uintptr(unsafe.Pointer(input)), 0) + ret = syscall.Errno(r0) + return +} + +func MakeSignature(context *CtxtHandle, qop uint32, message *SecBufferDesc, messageseqno uint32) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall6(procMakeSignature.Addr(), 4, uintptr(unsafe.Pointer(context)), uintptr(qop), uintptr(unsafe.Pointer(message)), uintptr(messageseqno), 0, 0) + ret = syscall.Errno(r0) + return +} + +func VerifySignature(context *CtxtHandle, message *SecBufferDesc, messageseqno uint32, qop *uint32) (ret syscall.Errno) { + r0, _, _ := syscall.Syscall6(procVerifySignature.Addr(), 4, uintptr(unsafe.Pointer(context)), uintptr(unsafe.Pointer(message)), uintptr(messageseqno), uintptr(unsafe.Pointer(qop)), 0, 0) + ret = syscall.Errno(r0) + return +} diff --git a/github.com/aws/aws-sdk-go/NOTICE.txt b/github.com/aws/aws-sdk-go/NOTICE.txt index 5f14d1162e..899129ecc4 100644 --- a/github.com/aws/aws-sdk-go/NOTICE.txt +++ b/github.com/aws/aws-sdk-go/NOTICE.txt @@ -1,3 +1,3 @@ AWS SDK for Go -Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved. +Copyright 2015 Amazon.com, Inc. or its affiliates. All Rights Reserved. Copyright 2014-2015 Stripe, Inc. diff --git a/github.com/aws/aws-sdk-go/aws/arn/arn.go b/github.com/aws/aws-sdk-go/aws/arn/arn.go new file mode 100644 index 0000000000..1c49674290 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/arn/arn.go @@ -0,0 +1,93 @@ +// Package arn provides a parser for interacting with Amazon Resource Names. +package arn + +import ( + "errors" + "strings" +) + +const ( + arnDelimiter = ":" + arnSections = 6 + arnPrefix = "arn:" + + // zero-indexed + sectionPartition = 1 + sectionService = 2 + sectionRegion = 3 + sectionAccountID = 4 + sectionResource = 5 + + // errors + invalidPrefix = "arn: invalid prefix" + invalidSections = "arn: not enough sections" +) + +// ARN captures the individual fields of an Amazon Resource Name. +// See http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html for more information. +type ARN struct { + // The partition that the resource is in. For standard AWS regions, the partition is "aws". If you have resources in + // other partitions, the partition is "aws-partitionname". For example, the partition for resources in the China + // (Beijing) region is "aws-cn". + Partition string + + // The service namespace that identifies the AWS product (for example, Amazon S3, IAM, or Amazon RDS). For a list of + // namespaces, see + // http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#genref-aws-service-namespaces. + Service string + + // The region the resource resides in. Note that the ARNs for some resources do not require a region, so this + // component might be omitted. + Region string + + // The ID of the AWS account that owns the resource, without the hyphens. For example, 123456789012. Note that the + // ARNs for some resources don't require an account number, so this component might be omitted. + AccountID string + + // The content of this part of the ARN varies by service. It often includes an indicator of the type of resource — + // for example, an IAM user or Amazon RDS database - followed by a slash (/) or a colon (:), followed by the + // resource name itself. Some services allows paths for resource names, as described in + // http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arns-paths. + Resource string +} + +// Parse parses an ARN into its constituent parts. +// +// Some example ARNs: +// arn:aws:elasticbeanstalk:us-east-1:123456789012:environment/My App/MyEnvironment +// arn:aws:iam::123456789012:user/David +// arn:aws:rds:eu-west-1:123456789012:db:mysql-db +// arn:aws:s3:::my_corporate_bucket/exampleobject.png +func Parse(arn string) (ARN, error) { + if !strings.HasPrefix(arn, arnPrefix) { + return ARN{}, errors.New(invalidPrefix) + } + sections := strings.SplitN(arn, arnDelimiter, arnSections) + if len(sections) != arnSections { + return ARN{}, errors.New(invalidSections) + } + return ARN{ + Partition: sections[sectionPartition], + Service: sections[sectionService], + Region: sections[sectionRegion], + AccountID: sections[sectionAccountID], + Resource: sections[sectionResource], + }, nil +} + +// IsARN returns whether the given string is an ARN by looking for +// whether the string starts with "arn:" and contains the correct number +// of sections delimited by colons(:). +func IsARN(arn string) bool { + return strings.HasPrefix(arn, arnPrefix) && strings.Count(arn, ":") >= arnSections-1 +} + +// String returns the canonical representation of the ARN +func (arn ARN) String() string { + return arnPrefix + + arn.Partition + arnDelimiter + + arn.Service + arnDelimiter + + arn.Region + arnDelimiter + + arn.AccountID + arnDelimiter + + arn.Resource +} diff --git a/github.com/aws/aws-sdk-go/aws/awserr/error.go b/github.com/aws/aws-sdk-go/aws/awserr/error.go index 56fdfc2bfc..99849c0e19 100644 --- a/github.com/aws/aws-sdk-go/aws/awserr/error.go +++ b/github.com/aws/aws-sdk-go/aws/awserr/error.go @@ -138,8 +138,27 @@ type RequestFailure interface { RequestID() string } -// NewRequestFailure returns a new request error wrapper for the given Error -// provided. +// NewRequestFailure returns a wrapped error with additional information for +// request status code, and service requestID. +// +// Should be used to wrap all request which involve service requests. Even if +// the request failed without a service response, but had an HTTP status code +// that may be meaningful. func NewRequestFailure(err Error, statusCode int, reqID string) RequestFailure { return newRequestError(err, statusCode, reqID) } + +// UnmarshalError provides the interface for the SDK failing to unmarshal data. +type UnmarshalError interface { + awsError + Bytes() []byte +} + +// NewUnmarshalError returns an initialized UnmarshalError error wrapper adding +// the bytes that fail to unmarshal to the error. +func NewUnmarshalError(err error, msg string, bytes []byte) UnmarshalError { + return &unmarshalError{ + awsError: New("UnmarshalError", msg, err), + bytes: bytes, + } +} diff --git a/github.com/aws/aws-sdk-go/aws/awserr/types.go b/github.com/aws/aws-sdk-go/aws/awserr/types.go index 0202a008f5..9cf7eaf400 100644 --- a/github.com/aws/aws-sdk-go/aws/awserr/types.go +++ b/github.com/aws/aws-sdk-go/aws/awserr/types.go @@ -1,6 +1,9 @@ package awserr -import "fmt" +import ( + "encoding/hex" + "fmt" +) // SprintError returns a string of the formatted error code. // @@ -119,6 +122,7 @@ type requestError struct { awsError statusCode int requestID string + bytes []byte } // newRequestError returns a wrapped error with additional information for @@ -170,6 +174,29 @@ func (r requestError) OrigErrs() []error { return []error{r.OrigErr()} } +type unmarshalError struct { + awsError + bytes []byte +} + +// Error returns the string representation of the error. +// Satisfies the error interface. +func (e unmarshalError) Error() string { + extra := hex.Dump(e.bytes) + return SprintError(e.Code(), e.Message(), extra, e.OrigErr()) +} + +// String returns the string representation of the error. +// Alias for Error to satisfy the stringer interface. +func (e unmarshalError) String() string { + return e.Error() +} + +// Bytes returns the bytes that failed to unmarshal. +func (e unmarshalError) Bytes() []byte { + return e.bytes +} + // An error list that satisfies the golang interface type errorList []error @@ -181,7 +208,7 @@ func (e errorList) Error() string { // How do we want to handle the array size being zero if size := len(e); size > 0 { for i := 0; i < size; i++ { - msg += fmt.Sprintf("%s", e[i].Error()) + msg += e[i].Error() // We check the next index to see if it is within the slice. // If it is, then we append a newline. We do this, because unit tests // could be broken with the additional '\n' diff --git a/github.com/aws/aws-sdk-go/aws/awsutil/equal.go b/github.com/aws/aws-sdk-go/aws/awsutil/equal.go index 59fa4a558a..142a7a01c5 100644 --- a/github.com/aws/aws-sdk-go/aws/awsutil/equal.go +++ b/github.com/aws/aws-sdk-go/aws/awsutil/equal.go @@ -15,7 +15,7 @@ func DeepEqual(a, b interface{}) bool { rb := reflect.Indirect(reflect.ValueOf(b)) if raValid, rbValid := ra.IsValid(), rb.IsValid(); !raValid && !rbValid { - // If the elements are both nil, and of the same type the are equal + // If the elements are both nil, and of the same type they are equal // If they are of different types they are not equal return reflect.TypeOf(a) == reflect.TypeOf(b) } else if raValid != rbValid { diff --git a/github.com/aws/aws-sdk-go/aws/awsutil/path_value.go b/github.com/aws/aws-sdk-go/aws/awsutil/path_value.go index 11c52c3896..a4eb6a7f43 100644 --- a/github.com/aws/aws-sdk-go/aws/awsutil/path_value.go +++ b/github.com/aws/aws-sdk-go/aws/awsutil/path_value.go @@ -70,7 +70,7 @@ func rValuesAtPath(v interface{}, path string, createPath, caseSensitive, nilTer value = value.FieldByNameFunc(func(name string) bool { if c == name { return true - } else if !caseSensitive && strings.ToLower(name) == strings.ToLower(c) { + } else if !caseSensitive && strings.EqualFold(name, c) { return true } return false @@ -185,13 +185,12 @@ func ValuesAtPath(i interface{}, path string) ([]interface{}, error) { // SetValueAtPath sets a value at the case insensitive lexical path inside // of a structure. func SetValueAtPath(i interface{}, path string, v interface{}) { - if rvals := rValuesAtPath(i, path, true, false, v == nil); rvals != nil { - for _, rval := range rvals { - if rval.Kind() == reflect.Ptr && rval.IsNil() { - continue - } - setValue(rval, v) + rvals := rValuesAtPath(i, path, true, false, v == nil) + for _, rval := range rvals { + if rval.Kind() == reflect.Ptr && rval.IsNil() { + continue } + setValue(rval, v) } } diff --git a/github.com/aws/aws-sdk-go/aws/client/client.go b/github.com/aws/aws-sdk-go/aws/client/client.go index 7096053840..03334d6920 100644 --- a/github.com/aws/aws-sdk-go/aws/client/client.go +++ b/github.com/aws/aws-sdk-go/aws/client/client.go @@ -12,6 +12,7 @@ import ( type Config struct { Config *aws.Config Handlers request.Handlers + PartitionID string Endpoint string SigningRegion string SigningName string @@ -64,7 +65,7 @@ func New(cfg aws.Config, info metadata.ClientInfo, handlers request.Handlers, op default: maxRetries := aws.IntValue(cfg.MaxRetries) if cfg.MaxRetries == nil || maxRetries == aws.UseServiceDefaultRetries { - maxRetries = 3 + maxRetries = DefaultRetryerMaxNumRetries } svc.Retryer = DefaultRetryer{NumMaxRetries: maxRetries} } diff --git a/github.com/aws/aws-sdk-go/aws/client/default_retryer.go b/github.com/aws/aws-sdk-go/aws/client/default_retryer.go index a397b0d044..9f6af19dd4 100644 --- a/github.com/aws/aws-sdk-go/aws/client/default_retryer.go +++ b/github.com/aws/aws-sdk-go/aws/client/default_retryer.go @@ -1,6 +1,7 @@ package client import ( + "math" "strconv" "time" @@ -9,82 +10,142 @@ import ( ) // DefaultRetryer implements basic retry logic using exponential backoff for -// most services. If you want to implement custom retry logic, implement the -// request.Retryer interface or create a structure type that composes this -// struct and override the specific methods. For example, to override only -// the MaxRetries method: +// most services. If you want to implement custom retry logic, you can implement the +// request.Retryer interface. // -// type retryer struct { -// client.DefaultRetryer -// } -// -// // This implementation always has 100 max retries -// func (d retryer) MaxRetries() int { return 100 } type DefaultRetryer struct { + // Num max Retries is the number of max retries that will be performed. + // By default, this is zero. NumMaxRetries int + + // MinRetryDelay is the minimum retry delay after which retry will be performed. + // If not set, the value is 0ns. + MinRetryDelay time.Duration + + // MinThrottleRetryDelay is the minimum retry delay when throttled. + // If not set, the value is 0ns. + MinThrottleDelay time.Duration + + // MaxRetryDelay is the maximum retry delay before which retry must be performed. + // If not set, the value is 0ns. + MaxRetryDelay time.Duration + + // MaxThrottleDelay is the maximum retry delay when throttled. + // If not set, the value is 0ns. + MaxThrottleDelay time.Duration } +const ( + // DefaultRetryerMaxNumRetries sets maximum number of retries + DefaultRetryerMaxNumRetries = 3 + + // DefaultRetryerMinRetryDelay sets minimum retry delay + DefaultRetryerMinRetryDelay = 30 * time.Millisecond + + // DefaultRetryerMinThrottleDelay sets minimum delay when throttled + DefaultRetryerMinThrottleDelay = 500 * time.Millisecond + + // DefaultRetryerMaxRetryDelay sets maximum retry delay + DefaultRetryerMaxRetryDelay = 300 * time.Second + + // DefaultRetryerMaxThrottleDelay sets maximum delay when throttled + DefaultRetryerMaxThrottleDelay = 300 * time.Second +) + // MaxRetries returns the number of maximum returns the service will use to make // an individual API request. func (d DefaultRetryer) MaxRetries() int { return d.NumMaxRetries } +// setRetryerDefaults sets the default values of the retryer if not set +func (d *DefaultRetryer) setRetryerDefaults() { + if d.MinRetryDelay == 0 { + d.MinRetryDelay = DefaultRetryerMinRetryDelay + } + if d.MaxRetryDelay == 0 { + d.MaxRetryDelay = DefaultRetryerMaxRetryDelay + } + if d.MinThrottleDelay == 0 { + d.MinThrottleDelay = DefaultRetryerMinThrottleDelay + } + if d.MaxThrottleDelay == 0 { + d.MaxThrottleDelay = DefaultRetryerMaxThrottleDelay + } +} + // RetryRules returns the delay duration before retrying this request again func (d DefaultRetryer) RetryRules(r *request.Request) time.Duration { - // Set the upper limit of delay in retrying at ~five minutes - minTime := 30 - throttle := d.shouldThrottle(r) - if throttle { - if delay, ok := getRetryDelay(r); ok { - return delay - } - minTime = 500 + // if number of max retries is zero, no retries will be performed. + if d.NumMaxRetries == 0 { + return 0 + } + + // Sets default value for retryer members + d.setRetryerDefaults() + + // minDelay is the minimum retryer delay + minDelay := d.MinRetryDelay + + var initialDelay time.Duration + + isThrottle := r.IsErrorThrottle() + if isThrottle { + if delay, ok := getRetryAfterDelay(r); ok { + initialDelay = delay + } + minDelay = d.MinThrottleDelay } retryCount := r.RetryCount - if throttle && retryCount > 8 { - retryCount = 8 - } else if retryCount > 13 { - retryCount = 13 + + // maxDelay the maximum retryer delay + maxDelay := d.MaxRetryDelay + + if isThrottle { + maxDelay = d.MaxThrottleDelay + } + + var delay time.Duration + + // Logic to cap the retry count based on the minDelay provided + actualRetryCount := int(math.Log2(float64(minDelay))) + 1 + if actualRetryCount < 63-retryCount { + delay = time.Duration(1< maxDelay { + delay = getJitterDelay(maxDelay / 2) + } + } else { + delay = getJitterDelay(maxDelay / 2) } + return delay + initialDelay +} - delay := (1 << uint(retryCount)) * (sdkrand.SeededRand.Intn(minTime) + minTime) - return time.Duration(delay) * time.Millisecond +// getJitterDelay returns a jittered delay for retry +func getJitterDelay(duration time.Duration) time.Duration { + return time.Duration(sdkrand.SeededRand.Int63n(int64(duration)) + int64(duration)) } // ShouldRetry returns true if the request should be retried. func (d DefaultRetryer) ShouldRetry(r *request.Request) bool { + + // ShouldRetry returns false if number of max retries is 0. + if d.NumMaxRetries == 0 { + return false + } + // If one of the other handlers already set the retry state // we don't want to override it based on the service's state if r.Retryable != nil { return *r.Retryable } - - if r.HTTPResponse.StatusCode >= 500 && r.HTTPResponse.StatusCode != 501 { - return true - } - return r.IsErrorRetryable() || d.shouldThrottle(r) -} - -// ShouldThrottle returns true if the request should be throttled. -func (d DefaultRetryer) shouldThrottle(r *request.Request) bool { - switch r.HTTPResponse.StatusCode { - case 429: - case 502: - case 503: - case 504: - default: - return r.IsErrorThrottle() - } - - return true + return r.IsErrorRetryable() || r.IsErrorThrottle() } // This will look in the Retry-After header, RFC 7231, for how long // it will wait before attempting another request -func getRetryDelay(r *request.Request) (time.Duration, bool) { +func getRetryAfterDelay(r *request.Request) (time.Duration, bool) { if !canUseRetryAfterHeader(r) { return 0, false } diff --git a/github.com/aws/aws-sdk-go/aws/client/logger.go b/github.com/aws/aws-sdk-go/aws/client/logger.go index ce9fb896d9..8958c32d4e 100644 --- a/github.com/aws/aws-sdk-go/aws/client/logger.go +++ b/github.com/aws/aws-sdk-go/aws/client/logger.go @@ -67,10 +67,14 @@ func logRequest(r *request.Request) { if !bodySeekable { r.SetReaderBody(aws.ReadSeekCloser(r.HTTPRequest.Body)) } - // Reset the request body because dumpRequest will re-wrap the r.HTTPRequest's - // Body as a NoOpCloser and will not be reset after read by the HTTP - // client reader. - r.ResetBody() + // Reset the request body because dumpRequest will re-wrap the + // r.HTTPRequest's Body as a NoOpCloser and will not be reset after + // read by the HTTP client reader. + if err := r.Error; err != nil { + r.Config.Logger.Log(fmt.Sprintf(logReqErrMsg, + r.ClientInfo.ServiceName, r.Operation.Name, err)) + return + } } r.Config.Logger.Log(fmt.Sprintf(logReqMsg, @@ -118,6 +122,12 @@ var LogHTTPResponseHandler = request.NamedHandler{ func logResponse(r *request.Request) { lw := &logWriter{r.Config.Logger, bytes.NewBuffer(nil)} + if r.HTTPResponse == nil { + lw.Logger.Log(fmt.Sprintf(logRespErrMsg, + r.ClientInfo.ServiceName, r.Operation.Name, "request's HTTPResponse is nil")) + return + } + logBody := r.Config.LogLevel.Matches(aws.LogDebugWithHTTPBody) if logBody { r.HTTPResponse.Body = &teeReaderCloser{ diff --git a/github.com/aws/aws-sdk-go/aws/client/metadata/client_info.go b/github.com/aws/aws-sdk-go/aws/client/metadata/client_info.go index 920e9fddf8..0c48f72e08 100644 --- a/github.com/aws/aws-sdk-go/aws/client/metadata/client_info.go +++ b/github.com/aws/aws-sdk-go/aws/client/metadata/client_info.go @@ -5,6 +5,7 @@ type ClientInfo struct { ServiceName string ServiceID string APIVersion string + PartitionID string Endpoint string SigningName string SigningRegion string diff --git a/github.com/aws/aws-sdk-go/aws/client/no_op_retryer.go b/github.com/aws/aws-sdk-go/aws/client/no_op_retryer.go new file mode 100644 index 0000000000..881d575f01 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/client/no_op_retryer.go @@ -0,0 +1,28 @@ +package client + +import ( + "time" + + "github.com/aws/aws-sdk-go/aws/request" +) + +// NoOpRetryer provides a retryer that performs no retries. +// It should be used when we do not want retries to be performed. +type NoOpRetryer struct{} + +// MaxRetries returns the number of maximum returns the service will use to make +// an individual API; For NoOpRetryer the MaxRetries will always be zero. +func (d NoOpRetryer) MaxRetries() int { + return 0 +} + +// ShouldRetry will always return false for NoOpRetryer, as it should never retry. +func (d NoOpRetryer) ShouldRetry(_ *request.Request) bool { + return false +} + +// RetryRules returns the delay duration before retrying this request again; +// since NoOpRetryer does not retry, RetryRules always returns 0. +func (d NoOpRetryer) RetryRules(_ *request.Request) time.Duration { + return 0 +} diff --git a/github.com/aws/aws-sdk-go/aws/config.go b/github.com/aws/aws-sdk-go/aws/config.go index 10634d173d..3b809e8478 100644 --- a/github.com/aws/aws-sdk-go/aws/config.go +++ b/github.com/aws/aws-sdk-go/aws/config.go @@ -20,7 +20,7 @@ type RequestRetryer interface{} // A Config provides service configuration for service clients. By default, // all clients will use the defaults.DefaultConfig structure. // -// // Create Session with MaxRetry configuration to be shared by multiple +// // Create Session with MaxRetries configuration to be shared by multiple // // service clients. // sess := session.Must(session.NewSession(&aws.Config{ // MaxRetries: aws.Int(3), @@ -43,7 +43,7 @@ type Config struct { // An optional endpoint URL (hostname only or fully qualified URI) // that overrides the default generated endpoint for a client. Set this - // to `""` to use the default generated endpoint. + // to `nil` or the value to `""` to use the default generated endpoint. // // Note: You must still provide a `Region` value when specifying an // endpoint for a client. @@ -138,7 +138,7 @@ type Config struct { // `ExpectContinueTimeout` for information on adjusting the continue wait // timeout. https://golang.org/pkg/net/http/#Transport // - // You should use this flag to disble 100-Continue if you experience issues + // You should use this flag to disable 100-Continue if you experience issues // with proxies or third party S3 compatible services. S3Disable100Continue *bool @@ -161,6 +161,17 @@ type Config struct { // on GetObject API calls. S3DisableContentMD5Validation *bool + // Set this to `true` to have the S3 service client to use the region specified + // in the ARN, when an ARN is provided as an argument to a bucket parameter. + S3UseARNRegion *bool + + // Set this to `true` to enable the SDK to unmarshal API response header maps to + // normalized lower case map keys. + // + // For example S3's X-Amz-Meta prefixed header will be unmarshaled to lower case + // Metadata member's map keys. The value of the header in the map is unaffected. + LowerCaseHeaderMaps *bool + // Set this to `true` to disable the EC2Metadata client from overriding the // default http.Client's Timeout. This is helpful if you do not want the // EC2Metadata client to create a new http.Client. This options is only @@ -172,7 +183,7 @@ type Config struct { // // Example: // sess := session.Must(session.NewSession(aws.NewConfig() - // .WithEC2MetadataDiableTimeoutOverride(true))) + // .WithEC2MetadataDisableTimeoutOverride(true))) // // svc := s3.New(sess) // @@ -183,7 +194,7 @@ type Config struct { // both IPv4 and IPv6 addressing. // // Setting this for a service which does not support dual stack will fail - // to make requets. It is not recommended to set this value on the session + // to make requests. It is not recommended to set this value on the session // as it will apply to all service clients created with the session. Even // services which don't support dual stack endpoints. // @@ -227,6 +238,7 @@ type Config struct { // EnableEndpointDiscovery will allow for endpoint discovery on operations that // have the definition in its model. By default, endpoint discovery is off. + // To use EndpointDiscovery, Endpoint should be unset or set to an empty string. // // Example: // sess := session.Must(session.NewSession(&aws.Config{ @@ -246,12 +258,18 @@ type Config struct { // Disabling this feature is useful when you want to use local endpoints // for testing that do not support the modeled host prefix pattern. DisableEndpointHostPrefix *bool + + // STSRegionalEndpoint will enable regional or legacy endpoint resolving + STSRegionalEndpoint endpoints.STSRegionalEndpoint + + // S3UsEast1RegionalEndpoint will enable regional or legacy endpoint resolving + S3UsEast1RegionalEndpoint endpoints.S3UsEast1RegionalEndpoint } // NewConfig returns a new Config pointer that can be chained with builder // methods to set multiple configuration values inline without using pointers. // -// // Create Session with MaxRetry configuration to be shared by multiple +// // Create Session with MaxRetries configuration to be shared by multiple // // service clients. // sess := session.Must(session.NewSession(aws.NewConfig(). // WithMaxRetries(3), @@ -379,6 +397,13 @@ func (c *Config) WithS3DisableContentMD5Validation(enable bool) *Config { } +// WithS3UseARNRegion sets a config S3UseARNRegion value and +// returning a Config pointer for chaining +func (c *Config) WithS3UseARNRegion(enable bool) *Config { + c.S3UseARNRegion = &enable + return c +} + // WithUseDualStack sets a config UseDualStack value returning a Config // pointer for chaining. func (c *Config) WithUseDualStack(enable bool) *Config { @@ -420,6 +445,20 @@ func (c *Config) MergeIn(cfgs ...*Config) { } } +// WithSTSRegionalEndpoint will set whether or not to use regional endpoint flag +// when resolving the endpoint for a service +func (c *Config) WithSTSRegionalEndpoint(sre endpoints.STSRegionalEndpoint) *Config { + c.STSRegionalEndpoint = sre + return c +} + +// WithS3UsEast1RegionalEndpoint will set whether or not to use regional endpoint flag +// when resolving the endpoint for a service +func (c *Config) WithS3UsEast1RegionalEndpoint(sre endpoints.S3UsEast1RegionalEndpoint) *Config { + c.S3UsEast1RegionalEndpoint = sre + return c +} + func mergeInConfig(dst *Config, other *Config) { if other == nil { return @@ -493,6 +532,10 @@ func mergeInConfig(dst *Config, other *Config) { dst.S3DisableContentMD5Validation = other.S3DisableContentMD5Validation } + if other.S3UseARNRegion != nil { + dst.S3UseARNRegion = other.S3UseARNRegion + } + if other.UseDualStack != nil { dst.UseDualStack = other.UseDualStack } @@ -520,6 +563,14 @@ func mergeInConfig(dst *Config, other *Config) { if other.DisableEndpointHostPrefix != nil { dst.DisableEndpointHostPrefix = other.DisableEndpointHostPrefix } + + if other.STSRegionalEndpoint != endpoints.UnsetSTSEndpoint { + dst.STSRegionalEndpoint = other.STSRegionalEndpoint + } + + if other.S3UsEast1RegionalEndpoint != endpoints.UnsetS3UsEast1Endpoint { + dst.S3UsEast1RegionalEndpoint = other.S3UsEast1RegionalEndpoint + } } // Copy will return a shallow copy of the Config object. If any additional diff --git a/github.com/aws/aws-sdk-go/aws/context.go b/github.com/aws/aws-sdk-go/aws/context_1_5.go similarity index 58% rename from github.com/aws/aws-sdk-go/aws/context.go rename to github.com/aws/aws-sdk-go/aws/context_1_5.go index 79f426853b..2866f9a7fb 100644 --- a/github.com/aws/aws-sdk-go/aws/context.go +++ b/github.com/aws/aws-sdk-go/aws/context_1_5.go @@ -1,8 +1,8 @@ +// +build !go1.9 + package aws -import ( - "time" -) +import "time" // Context is an copy of the Go v1.7 stdlib's context.Context interface. // It is represented as a SDK interface to enable you to use the "WithContext" @@ -35,37 +35,3 @@ type Context interface { // functions. Value(key interface{}) interface{} } - -// BackgroundContext returns a context that will never be canceled, has no -// values, and no deadline. This context is used by the SDK to provide -// backwards compatibility with non-context API operations and functionality. -// -// Go 1.6 and before: -// This context function is equivalent to context.Background in the Go stdlib. -// -// Go 1.7 and later: -// The context returned will be the value returned by context.Background() -// -// See https://golang.org/pkg/context for more information on Contexts. -func BackgroundContext() Context { - return backgroundCtx -} - -// SleepWithContext will wait for the timer duration to expire, or the context -// is canceled. Which ever happens first. If the context is canceled the Context's -// error will be returned. -// -// Expects Context to always return a non-nil error if the Done channel is closed. -func SleepWithContext(ctx Context, dur time.Duration) error { - t := time.NewTimer(dur) - defer t.Stop() - - select { - case <-t.C: - break - case <-ctx.Done(): - return ctx.Err() - } - - return nil -} diff --git a/github.com/aws/aws-sdk-go/aws/context_1_7.go b/github.com/aws/aws-sdk-go/aws/context_1_7.go deleted file mode 100644 index 064f75c925..0000000000 --- a/github.com/aws/aws-sdk-go/aws/context_1_7.go +++ /dev/null @@ -1,9 +0,0 @@ -// +build go1.7 - -package aws - -import "context" - -var ( - backgroundCtx = context.Background() -) diff --git a/github.com/aws/aws-sdk-go/aws/context_1_9.go b/github.com/aws/aws-sdk-go/aws/context_1_9.go new file mode 100644 index 0000000000..3718b26e10 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/context_1_9.go @@ -0,0 +1,11 @@ +// +build go1.9 + +package aws + +import "context" + +// Context is an alias of the Go stdlib's context.Context interface. +// It can be used within the SDK's API operation "WithContext" methods. +// +// See https://golang.org/pkg/context on how to use contexts. +type Context = context.Context diff --git a/github.com/aws/aws-sdk-go/aws/context_background_1_5.go b/github.com/aws/aws-sdk-go/aws/context_background_1_5.go new file mode 100644 index 0000000000..2f9446333a --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/context_background_1_5.go @@ -0,0 +1,22 @@ +// +build !go1.7 + +package aws + +import ( + "github.com/aws/aws-sdk-go/internal/context" +) + +// BackgroundContext returns a context that will never be canceled, has no +// values, and no deadline. This context is used by the SDK to provide +// backwards compatibility with non-context API operations and functionality. +// +// Go 1.6 and before: +// This context function is equivalent to context.Background in the Go stdlib. +// +// Go 1.7 and later: +// The context returned will be the value returned by context.Background() +// +// See https://golang.org/pkg/context for more information on Contexts. +func BackgroundContext() Context { + return context.BackgroundCtx +} diff --git a/github.com/aws/aws-sdk-go/aws/context_background_1_7.go b/github.com/aws/aws-sdk-go/aws/context_background_1_7.go new file mode 100644 index 0000000000..9c29f29af1 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/context_background_1_7.go @@ -0,0 +1,20 @@ +// +build go1.7 + +package aws + +import "context" + +// BackgroundContext returns a context that will never be canceled, has no +// values, and no deadline. This context is used by the SDK to provide +// backwards compatibility with non-context API operations and functionality. +// +// Go 1.6 and before: +// This context function is equivalent to context.Background in the Go stdlib. +// +// Go 1.7 and later: +// The context returned will be the value returned by context.Background() +// +// See https://golang.org/pkg/context for more information on Contexts. +func BackgroundContext() Context { + return context.Background() +} diff --git a/github.com/aws/aws-sdk-go/aws/context_sleep.go b/github.com/aws/aws-sdk-go/aws/context_sleep.go new file mode 100644 index 0000000000..304fd15612 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/context_sleep.go @@ -0,0 +1,24 @@ +package aws + +import ( + "time" +) + +// SleepWithContext will wait for the timer duration to expire, or the context +// is canceled. Which ever happens first. If the context is canceled the Context's +// error will be returned. +// +// Expects Context to always return a non-nil error if the Done channel is closed. +func SleepWithContext(ctx Context, dur time.Duration) error { + t := time.NewTimer(dur) + defer t.Stop() + + select { + case <-t.C: + break + case <-ctx.Done(): + return ctx.Err() + } + + return nil +} diff --git a/github.com/aws/aws-sdk-go/aws/convert_types.go b/github.com/aws/aws-sdk-go/aws/convert_types.go index ff5d58e068..4e076c1837 100644 --- a/github.com/aws/aws-sdk-go/aws/convert_types.go +++ b/github.com/aws/aws-sdk-go/aws/convert_types.go @@ -179,6 +179,242 @@ func IntValueMap(src map[string]*int) map[string]int { return dst } +// Uint returns a pointer to the uint value passed in. +func Uint(v uint) *uint { + return &v +} + +// UintValue returns the value of the uint pointer passed in or +// 0 if the pointer is nil. +func UintValue(v *uint) uint { + if v != nil { + return *v + } + return 0 +} + +// UintSlice converts a slice of uint values uinto a slice of +// uint pointers +func UintSlice(src []uint) []*uint { + dst := make([]*uint, len(src)) + for i := 0; i < len(src); i++ { + dst[i] = &(src[i]) + } + return dst +} + +// UintValueSlice converts a slice of uint pointers uinto a slice of +// uint values +func UintValueSlice(src []*uint) []uint { + dst := make([]uint, len(src)) + for i := 0; i < len(src); i++ { + if src[i] != nil { + dst[i] = *(src[i]) + } + } + return dst +} + +// UintMap converts a string map of uint values uinto a string +// map of uint pointers +func UintMap(src map[string]uint) map[string]*uint { + dst := make(map[string]*uint) + for k, val := range src { + v := val + dst[k] = &v + } + return dst +} + +// UintValueMap converts a string map of uint pointers uinto a string +// map of uint values +func UintValueMap(src map[string]*uint) map[string]uint { + dst := make(map[string]uint) + for k, val := range src { + if val != nil { + dst[k] = *val + } + } + return dst +} + +// Int8 returns a pointer to the int8 value passed in. +func Int8(v int8) *int8 { + return &v +} + +// Int8Value returns the value of the int8 pointer passed in or +// 0 if the pointer is nil. +func Int8Value(v *int8) int8 { + if v != nil { + return *v + } + return 0 +} + +// Int8Slice converts a slice of int8 values into a slice of +// int8 pointers +func Int8Slice(src []int8) []*int8 { + dst := make([]*int8, len(src)) + for i := 0; i < len(src); i++ { + dst[i] = &(src[i]) + } + return dst +} + +// Int8ValueSlice converts a slice of int8 pointers into a slice of +// int8 values +func Int8ValueSlice(src []*int8) []int8 { + dst := make([]int8, len(src)) + for i := 0; i < len(src); i++ { + if src[i] != nil { + dst[i] = *(src[i]) + } + } + return dst +} + +// Int8Map converts a string map of int8 values into a string +// map of int8 pointers +func Int8Map(src map[string]int8) map[string]*int8 { + dst := make(map[string]*int8) + for k, val := range src { + v := val + dst[k] = &v + } + return dst +} + +// Int8ValueMap converts a string map of int8 pointers into a string +// map of int8 values +func Int8ValueMap(src map[string]*int8) map[string]int8 { + dst := make(map[string]int8) + for k, val := range src { + if val != nil { + dst[k] = *val + } + } + return dst +} + +// Int16 returns a pointer to the int16 value passed in. +func Int16(v int16) *int16 { + return &v +} + +// Int16Value returns the value of the int16 pointer passed in or +// 0 if the pointer is nil. +func Int16Value(v *int16) int16 { + if v != nil { + return *v + } + return 0 +} + +// Int16Slice converts a slice of int16 values into a slice of +// int16 pointers +func Int16Slice(src []int16) []*int16 { + dst := make([]*int16, len(src)) + for i := 0; i < len(src); i++ { + dst[i] = &(src[i]) + } + return dst +} + +// Int16ValueSlice converts a slice of int16 pointers into a slice of +// int16 values +func Int16ValueSlice(src []*int16) []int16 { + dst := make([]int16, len(src)) + for i := 0; i < len(src); i++ { + if src[i] != nil { + dst[i] = *(src[i]) + } + } + return dst +} + +// Int16Map converts a string map of int16 values into a string +// map of int16 pointers +func Int16Map(src map[string]int16) map[string]*int16 { + dst := make(map[string]*int16) + for k, val := range src { + v := val + dst[k] = &v + } + return dst +} + +// Int16ValueMap converts a string map of int16 pointers into a string +// map of int16 values +func Int16ValueMap(src map[string]*int16) map[string]int16 { + dst := make(map[string]int16) + for k, val := range src { + if val != nil { + dst[k] = *val + } + } + return dst +} + +// Int32 returns a pointer to the int32 value passed in. +func Int32(v int32) *int32 { + return &v +} + +// Int32Value returns the value of the int32 pointer passed in or +// 0 if the pointer is nil. +func Int32Value(v *int32) int32 { + if v != nil { + return *v + } + return 0 +} + +// Int32Slice converts a slice of int32 values into a slice of +// int32 pointers +func Int32Slice(src []int32) []*int32 { + dst := make([]*int32, len(src)) + for i := 0; i < len(src); i++ { + dst[i] = &(src[i]) + } + return dst +} + +// Int32ValueSlice converts a slice of int32 pointers into a slice of +// int32 values +func Int32ValueSlice(src []*int32) []int32 { + dst := make([]int32, len(src)) + for i := 0; i < len(src); i++ { + if src[i] != nil { + dst[i] = *(src[i]) + } + } + return dst +} + +// Int32Map converts a string map of int32 values into a string +// map of int32 pointers +func Int32Map(src map[string]int32) map[string]*int32 { + dst := make(map[string]*int32) + for k, val := range src { + v := val + dst[k] = &v + } + return dst +} + +// Int32ValueMap converts a string map of int32 pointers into a string +// map of int32 values +func Int32ValueMap(src map[string]*int32) map[string]int32 { + dst := make(map[string]int32) + for k, val := range src { + if val != nil { + dst[k] = *val + } + } + return dst +} + // Int64 returns a pointer to the int64 value passed in. func Int64(v int64) *int64 { return &v @@ -238,6 +474,301 @@ func Int64ValueMap(src map[string]*int64) map[string]int64 { return dst } +// Uint8 returns a pointer to the uint8 value passed in. +func Uint8(v uint8) *uint8 { + return &v +} + +// Uint8Value returns the value of the uint8 pointer passed in or +// 0 if the pointer is nil. +func Uint8Value(v *uint8) uint8 { + if v != nil { + return *v + } + return 0 +} + +// Uint8Slice converts a slice of uint8 values into a slice of +// uint8 pointers +func Uint8Slice(src []uint8) []*uint8 { + dst := make([]*uint8, len(src)) + for i := 0; i < len(src); i++ { + dst[i] = &(src[i]) + } + return dst +} + +// Uint8ValueSlice converts a slice of uint8 pointers into a slice of +// uint8 values +func Uint8ValueSlice(src []*uint8) []uint8 { + dst := make([]uint8, len(src)) + for i := 0; i < len(src); i++ { + if src[i] != nil { + dst[i] = *(src[i]) + } + } + return dst +} + +// Uint8Map converts a string map of uint8 values into a string +// map of uint8 pointers +func Uint8Map(src map[string]uint8) map[string]*uint8 { + dst := make(map[string]*uint8) + for k, val := range src { + v := val + dst[k] = &v + } + return dst +} + +// Uint8ValueMap converts a string map of uint8 pointers into a string +// map of uint8 values +func Uint8ValueMap(src map[string]*uint8) map[string]uint8 { + dst := make(map[string]uint8) + for k, val := range src { + if val != nil { + dst[k] = *val + } + } + return dst +} + +// Uint16 returns a pointer to the uint16 value passed in. +func Uint16(v uint16) *uint16 { + return &v +} + +// Uint16Value returns the value of the uint16 pointer passed in or +// 0 if the pointer is nil. +func Uint16Value(v *uint16) uint16 { + if v != nil { + return *v + } + return 0 +} + +// Uint16Slice converts a slice of uint16 values into a slice of +// uint16 pointers +func Uint16Slice(src []uint16) []*uint16 { + dst := make([]*uint16, len(src)) + for i := 0; i < len(src); i++ { + dst[i] = &(src[i]) + } + return dst +} + +// Uint16ValueSlice converts a slice of uint16 pointers into a slice of +// uint16 values +func Uint16ValueSlice(src []*uint16) []uint16 { + dst := make([]uint16, len(src)) + for i := 0; i < len(src); i++ { + if src[i] != nil { + dst[i] = *(src[i]) + } + } + return dst +} + +// Uint16Map converts a string map of uint16 values into a string +// map of uint16 pointers +func Uint16Map(src map[string]uint16) map[string]*uint16 { + dst := make(map[string]*uint16) + for k, val := range src { + v := val + dst[k] = &v + } + return dst +} + +// Uint16ValueMap converts a string map of uint16 pointers into a string +// map of uint16 values +func Uint16ValueMap(src map[string]*uint16) map[string]uint16 { + dst := make(map[string]uint16) + for k, val := range src { + if val != nil { + dst[k] = *val + } + } + return dst +} + +// Uint32 returns a pointer to the uint32 value passed in. +func Uint32(v uint32) *uint32 { + return &v +} + +// Uint32Value returns the value of the uint32 pointer passed in or +// 0 if the pointer is nil. +func Uint32Value(v *uint32) uint32 { + if v != nil { + return *v + } + return 0 +} + +// Uint32Slice converts a slice of uint32 values into a slice of +// uint32 pointers +func Uint32Slice(src []uint32) []*uint32 { + dst := make([]*uint32, len(src)) + for i := 0; i < len(src); i++ { + dst[i] = &(src[i]) + } + return dst +} + +// Uint32ValueSlice converts a slice of uint32 pointers into a slice of +// uint32 values +func Uint32ValueSlice(src []*uint32) []uint32 { + dst := make([]uint32, len(src)) + for i := 0; i < len(src); i++ { + if src[i] != nil { + dst[i] = *(src[i]) + } + } + return dst +} + +// Uint32Map converts a string map of uint32 values into a string +// map of uint32 pointers +func Uint32Map(src map[string]uint32) map[string]*uint32 { + dst := make(map[string]*uint32) + for k, val := range src { + v := val + dst[k] = &v + } + return dst +} + +// Uint32ValueMap converts a string map of uint32 pointers into a string +// map of uint32 values +func Uint32ValueMap(src map[string]*uint32) map[string]uint32 { + dst := make(map[string]uint32) + for k, val := range src { + if val != nil { + dst[k] = *val + } + } + return dst +} + +// Uint64 returns a pointer to the uint64 value passed in. +func Uint64(v uint64) *uint64 { + return &v +} + +// Uint64Value returns the value of the uint64 pointer passed in or +// 0 if the pointer is nil. +func Uint64Value(v *uint64) uint64 { + if v != nil { + return *v + } + return 0 +} + +// Uint64Slice converts a slice of uint64 values into a slice of +// uint64 pointers +func Uint64Slice(src []uint64) []*uint64 { + dst := make([]*uint64, len(src)) + for i := 0; i < len(src); i++ { + dst[i] = &(src[i]) + } + return dst +} + +// Uint64ValueSlice converts a slice of uint64 pointers into a slice of +// uint64 values +func Uint64ValueSlice(src []*uint64) []uint64 { + dst := make([]uint64, len(src)) + for i := 0; i < len(src); i++ { + if src[i] != nil { + dst[i] = *(src[i]) + } + } + return dst +} + +// Uint64Map converts a string map of uint64 values into a string +// map of uint64 pointers +func Uint64Map(src map[string]uint64) map[string]*uint64 { + dst := make(map[string]*uint64) + for k, val := range src { + v := val + dst[k] = &v + } + return dst +} + +// Uint64ValueMap converts a string map of uint64 pointers into a string +// map of uint64 values +func Uint64ValueMap(src map[string]*uint64) map[string]uint64 { + dst := make(map[string]uint64) + for k, val := range src { + if val != nil { + dst[k] = *val + } + } + return dst +} + +// Float32 returns a pointer to the float32 value passed in. +func Float32(v float32) *float32 { + return &v +} + +// Float32Value returns the value of the float32 pointer passed in or +// 0 if the pointer is nil. +func Float32Value(v *float32) float32 { + if v != nil { + return *v + } + return 0 +} + +// Float32Slice converts a slice of float32 values into a slice of +// float32 pointers +func Float32Slice(src []float32) []*float32 { + dst := make([]*float32, len(src)) + for i := 0; i < len(src); i++ { + dst[i] = &(src[i]) + } + return dst +} + +// Float32ValueSlice converts a slice of float32 pointers into a slice of +// float32 values +func Float32ValueSlice(src []*float32) []float32 { + dst := make([]float32, len(src)) + for i := 0; i < len(src); i++ { + if src[i] != nil { + dst[i] = *(src[i]) + } + } + return dst +} + +// Float32Map converts a string map of float32 values into a string +// map of float32 pointers +func Float32Map(src map[string]float32) map[string]*float32 { + dst := make(map[string]*float32) + for k, val := range src { + v := val + dst[k] = &v + } + return dst +} + +// Float32ValueMap converts a string map of float32 pointers into a string +// map of float32 values +func Float32ValueMap(src map[string]*float32) map[string]float32 { + dst := make(map[string]float32) + for k, val := range src { + if val != nil { + dst[k] = *val + } + } + return dst +} + // Float64 returns a pointer to the float64 value passed in. func Float64(v float64) *float64 { return &v diff --git a/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go b/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go index f8853d78af..d95a5eb540 100644 --- a/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go +++ b/github.com/aws/aws-sdk-go/aws/corehandlers/handlers.go @@ -159,9 +159,9 @@ func handleSendError(r *request.Request, err error) { Body: ioutil.NopCloser(bytes.NewReader([]byte{})), } } - // Catch all other request errors. - r.Error = awserr.New("RequestError", "send request failed", err) - r.Retryable = aws.Bool(true) // network errors are retryable + // Catch all request errors, and let the default retrier determine + // if the error is retryable. + r.Error = awserr.New(request.ErrCodeRequestError, "send request failed", err) // Override the error with a context canceled error, if that was canceled. ctx := r.Context() @@ -184,37 +184,39 @@ var ValidateResponseHandler = request.NamedHandler{Name: "core.ValidateResponseH // AfterRetryHandler performs final checks to determine if the request should // be retried and how long to delay. -var AfterRetryHandler = request.NamedHandler{Name: "core.AfterRetryHandler", Fn: func(r *request.Request) { - // If one of the other handlers already set the retry state - // we don't want to override it based on the service's state - if r.Retryable == nil || aws.BoolValue(r.Config.EnforceShouldRetryCheck) { - r.Retryable = aws.Bool(r.ShouldRetry(r)) - } +var AfterRetryHandler = request.NamedHandler{ + Name: "core.AfterRetryHandler", + Fn: func(r *request.Request) { + // If one of the other handlers already set the retry state + // we don't want to override it based on the service's state + if r.Retryable == nil || aws.BoolValue(r.Config.EnforceShouldRetryCheck) { + r.Retryable = aws.Bool(r.ShouldRetry(r)) + } - if r.WillRetry() { - r.RetryDelay = r.RetryRules(r) + if r.WillRetry() { + r.RetryDelay = r.RetryRules(r) - if sleepFn := r.Config.SleepDelay; sleepFn != nil { - // Support SleepDelay for backwards compatibility and testing - sleepFn(r.RetryDelay) - } else if err := aws.SleepWithContext(r.Context(), r.RetryDelay); err != nil { - r.Error = awserr.New(request.CanceledErrorCode, - "request context canceled", err) - r.Retryable = aws.Bool(false) - return - } + if sleepFn := r.Config.SleepDelay; sleepFn != nil { + // Support SleepDelay for backwards compatibility and testing + sleepFn(r.RetryDelay) + } else if err := aws.SleepWithContext(r.Context(), r.RetryDelay); err != nil { + r.Error = awserr.New(request.CanceledErrorCode, + "request context canceled", err) + r.Retryable = aws.Bool(false) + return + } - // when the expired token exception occurs the credentials - // need to be expired locally so that the next request to - // get credentials will trigger a credentials refresh. - if r.IsErrorExpired() { - r.Config.Credentials.Expire() - } + // when the expired token exception occurs the credentials + // need to be expired locally so that the next request to + // get credentials will trigger a credentials refresh. + if r.IsErrorExpired() { + r.Config.Credentials.Expire() + } - r.RetryCount++ - r.Error = nil - } -}} + r.RetryCount++ + r.Error = nil + } + }} // ValidateEndpointHandler is a request handler to validate a request had the // appropriate Region and Endpoint set. Will set r.Error if the endpoint or @@ -223,6 +225,8 @@ var ValidateEndpointHandler = request.NamedHandler{Name: "core.ValidateEndpointH if r.ClientInfo.SigningRegion == "" && aws.StringValue(r.Config.Region) == "" { r.Error = aws.ErrMissingRegion } else if r.ClientInfo.Endpoint == "" { + // Was any endpoint provided by the user, or one was derived by the + // SDK's endpoint resolver? r.Error = aws.ErrMissingEndpoint } }} diff --git a/github.com/aws/aws-sdk-go/aws/credentials/context_background_go1.5.go b/github.com/aws/aws-sdk-go/aws/credentials/context_background_go1.5.go new file mode 100644 index 0000000000..5852b26487 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/credentials/context_background_go1.5.go @@ -0,0 +1,22 @@ +// +build !go1.7 + +package credentials + +import ( + "github.com/aws/aws-sdk-go/internal/context" +) + +// backgroundContext returns a context that will never be canceled, has no +// values, and no deadline. This context is used by the SDK to provide +// backwards compatibility with non-context API operations and functionality. +// +// Go 1.6 and before: +// This context function is equivalent to context.Background in the Go stdlib. +// +// Go 1.7 and later: +// The context returned will be the value returned by context.Background() +// +// See https://golang.org/pkg/context for more information on Contexts. +func backgroundContext() Context { + return context.BackgroundCtx +} diff --git a/github.com/aws/aws-sdk-go/aws/credentials/context_background_go1.7.go b/github.com/aws/aws-sdk-go/aws/credentials/context_background_go1.7.go new file mode 100644 index 0000000000..388b215418 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/credentials/context_background_go1.7.go @@ -0,0 +1,20 @@ +// +build go1.7 + +package credentials + +import "context" + +// backgroundContext returns a context that will never be canceled, has no +// values, and no deadline. This context is used by the SDK to provide +// backwards compatibility with non-context API operations and functionality. +// +// Go 1.6 and before: +// This context function is equivalent to context.Background in the Go stdlib. +// +// Go 1.7 and later: +// The context returned will be the value returned by context.Background() +// +// See https://golang.org/pkg/context for more information on Contexts. +func backgroundContext() Context { + return context.Background() +} diff --git a/github.com/aws/aws-sdk-go/aws/credentials/context_go1.5.go b/github.com/aws/aws-sdk-go/aws/credentials/context_go1.5.go new file mode 100644 index 0000000000..8152a864ad --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/credentials/context_go1.5.go @@ -0,0 +1,39 @@ +// +build !go1.9 + +package credentials + +import "time" + +// Context is an copy of the Go v1.7 stdlib's context.Context interface. +// It is represented as a SDK interface to enable you to use the "WithContext" +// API methods with Go v1.6 and a Context type such as golang.org/x/net/context. +// +// This type, aws.Context, and context.Context are equivalent. +// +// See https://golang.org/pkg/context on how to use contexts. +type Context interface { + // Deadline returns the time when work done on behalf of this context + // should be canceled. Deadline returns ok==false when no deadline is + // set. Successive calls to Deadline return the same results. + Deadline() (deadline time.Time, ok bool) + + // Done returns a channel that's closed when work done on behalf of this + // context should be canceled. Done may return nil if this context can + // never be canceled. Successive calls to Done return the same value. + Done() <-chan struct{} + + // Err returns a non-nil error value after Done is closed. Err returns + // Canceled if the context was canceled or DeadlineExceeded if the + // context's deadline passed. No other values for Err are defined. + // After Done is closed, successive calls to Err return the same value. + Err() error + + // Value returns the value associated with this context for key, or nil + // if no value is associated with key. Successive calls to Value with + // the same key returns the same result. + // + // Use context values only for request-scoped data that transits + // processes and API boundaries, not for passing optional parameters to + // functions. + Value(key interface{}) interface{} +} diff --git a/github.com/aws/aws-sdk-go/aws/credentials/context_go1.9.go b/github.com/aws/aws-sdk-go/aws/credentials/context_go1.9.go new file mode 100644 index 0000000000..4356edb3d5 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/credentials/context_go1.9.go @@ -0,0 +1,13 @@ +// +build go1.9 + +package credentials + +import "context" + +// Context is an alias of the Go stdlib's context.Context interface. +// It can be used within the SDK's API operation "WithContext" methods. +// +// This type, aws.Context, and context.Context are equivalent. +// +// See https://golang.org/pkg/context on how to use contexts. +type Context = context.Context diff --git a/github.com/aws/aws-sdk-go/aws/credentials/credentials.go b/github.com/aws/aws-sdk-go/aws/credentials/credentials.go index 894bbc7f82..9f8fd92a50 100644 --- a/github.com/aws/aws-sdk-go/aws/credentials/credentials.go +++ b/github.com/aws/aws-sdk-go/aws/credentials/credentials.go @@ -50,9 +50,11 @@ package credentials import ( "fmt" - "github.com/aws/aws-sdk-go/aws/awserr" - "sync" + "sync/atomic" "time" + + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/internal/sync/singleflight" ) // AnonymousCredentials is an empty Credential object that can be used as @@ -83,6 +85,12 @@ type Value struct { ProviderName string } +// HasKeys returns if the credentials Value has both AccessKeyID and +// SecretAccessKey value set. +func (v Value) HasKeys() bool { + return len(v.AccessKeyID) != 0 && len(v.SecretAccessKey) != 0 +} + // A Provider is the interface for any component which will provide credentials // Value. A provider is required to manage its own Expired state, and what to // be expired means. @@ -99,6 +107,13 @@ type Provider interface { IsExpired() bool } +// ProviderWithContext is a Provider that can retrieve credentials with a Context +type ProviderWithContext interface { + Provider + + RetrieveWithContext(Context) (Value, error) +} + // An Expirer is an interface that Providers can implement to expose the expiration // time, if known. If the Provider cannot accurately provide this info, // it should not implement this interface. @@ -190,24 +205,24 @@ func (e *Expiry) ExpiresAt() time.Time { // first instance of the credentials Value. All calls to Get() after that // will return the cached credentials Value until IsExpired() returns true. type Credentials struct { - creds Value - forceRefresh bool - - m sync.RWMutex + creds atomic.Value + sf singleflight.Group provider Provider } // NewCredentials returns a pointer to a new Credentials with the provider set. func NewCredentials(provider Provider) *Credentials { - return &Credentials{ - provider: provider, - forceRefresh: true, + c := &Credentials{ + provider: provider, } + c.creds.Store(Value{}) + return c } -// Get returns the credentials value, or error if the credentials Value failed -// to be retrieved. +// GetWithContext returns the credentials value, or error if the credentials +// Value failed to be retrieved. Will return early if the passed in context is +// canceled. // // Will return the cached credentials Value if it has not expired. If the // credentials Value has expired the Provider's Retrieve() will be called @@ -215,31 +230,56 @@ func NewCredentials(provider Provider) *Credentials { // // If Credentials.Expire() was called the credentials Value will be force // expired, and the next call to Get() will cause them to be refreshed. -func (c *Credentials) Get() (Value, error) { - // Check the cached credentials first with just the read lock. - c.m.RLock() - if !c.isExpired() { - creds := c.creds - c.m.RUnlock() - return creds, nil +// +// Passed in Context is equivalent to aws.Context, and context.Context. +func (c *Credentials) GetWithContext(ctx Context) (Value, error) { + if curCreds := c.creds.Load(); !c.isExpired(curCreds) { + return curCreds.(Value), nil + } + + // Cannot pass context down to the actual retrieve, because the first + // context would cancel the whole group when there is not direct + // association of items in the group. + resCh := c.sf.DoChan("", func() (interface{}, error) { + return c.singleRetrieve(&suppressedContext{ctx}) + }) + select { + case res := <-resCh: + return res.Val.(Value), res.Err + case <-ctx.Done(): + return Value{}, awserr.New("RequestCanceled", + "request context canceled", ctx.Err()) } - c.m.RUnlock() - - // Credentials are expired need to retrieve the credentials taking the full - // lock. - c.m.Lock() - defer c.m.Unlock() - - if c.isExpired() { - creds, err := c.provider.Retrieve() - if err != nil { - return Value{}, err - } - c.creds = creds - c.forceRefresh = false +} + +func (c *Credentials) singleRetrieve(ctx Context) (creds interface{}, err error) { + if curCreds := c.creds.Load(); !c.isExpired(curCreds) { + return curCreds.(Value), nil } - return c.creds, nil + if p, ok := c.provider.(ProviderWithContext); ok { + creds, err = p.RetrieveWithContext(ctx) + } else { + creds, err = c.provider.Retrieve() + } + if err == nil { + c.creds.Store(creds) + } + + return creds, err +} + +// Get returns the credentials value, or error if the credentials Value failed +// to be retrieved. +// +// Will return the cached credentials Value if it has not expired. If the +// credentials Value has expired the Provider's Retrieve() will be called +// to refresh the credentials. +// +// If Credentials.Expire() was called the credentials Value will be force +// expired, and the next call to Get() will cause them to be refreshed. +func (c *Credentials) Get() (Value, error) { + return c.GetWithContext(backgroundContext()) } // Expire expires the credentials and forces them to be retrieved on the @@ -248,10 +288,7 @@ func (c *Credentials) Get() (Value, error) { // This will override the Provider's expired state, and force Credentials // to call the Provider's Retrieve(). func (c *Credentials) Expire() { - c.m.Lock() - defer c.m.Unlock() - - c.forceRefresh = true + c.creds.Store(Value{}) } // IsExpired returns if the credentials are no longer valid, and need @@ -260,33 +297,43 @@ func (c *Credentials) Expire() { // If the Credentials were forced to be expired with Expire() this will // reflect that override. func (c *Credentials) IsExpired() bool { - c.m.RLock() - defer c.m.RUnlock() - - return c.isExpired() + return c.isExpired(c.creds.Load()) } // isExpired helper method wrapping the definition of expired credentials. -func (c *Credentials) isExpired() bool { - return c.forceRefresh || c.provider.IsExpired() +func (c *Credentials) isExpired(creds interface{}) bool { + return creds == nil || creds.(Value) == Value{} || c.provider.IsExpired() } // ExpiresAt provides access to the functionality of the Expirer interface of // the underlying Provider, if it supports that interface. Otherwise, it returns // an error. func (c *Credentials) ExpiresAt() (time.Time, error) { - c.m.RLock() - defer c.m.RUnlock() - expirer, ok := c.provider.(Expirer) if !ok { return time.Time{}, awserr.New("ProviderNotExpirer", - fmt.Sprintf("provider %s does not support ExpiresAt()", c.creds.ProviderName), + fmt.Sprintf("provider %s does not support ExpiresAt()", c.creds.Load().(Value).ProviderName), nil) } - if c.forceRefresh { + if c.creds.Load().(Value) == (Value{}) { // set expiration time to the distant past return time.Time{}, nil } return expirer.ExpiresAt(), nil } + +type suppressedContext struct { + Context +} + +func (s *suppressedContext) Deadline() (deadline time.Time, ok bool) { + return time.Time{}, false +} + +func (s *suppressedContext) Done() <-chan struct{} { + return nil +} + +func (s *suppressedContext) Err() error { + return nil +} diff --git a/github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds/ec2_role_provider.go b/github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds/ec2_role_provider.go index 0ed791be64..92af5b7250 100644 --- a/github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds/ec2_role_provider.go +++ b/github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds/ec2_role_provider.go @@ -7,10 +7,12 @@ import ( "strings" "time" + "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/client" "github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/ec2metadata" + "github.com/aws/aws-sdk-go/aws/request" "github.com/aws/aws-sdk-go/internal/sdkuri" ) @@ -86,7 +88,14 @@ func NewCredentialsWithClient(client *ec2metadata.EC2Metadata, options ...func(* // Error will be returned if the request fails, or unable to extract // the desired credentials. func (m *EC2RoleProvider) Retrieve() (credentials.Value, error) { - credsList, err := requestCredList(m.Client) + return m.RetrieveWithContext(aws.BackgroundContext()) +} + +// RetrieveWithContext retrieves credentials from the EC2 service. +// Error will be returned if the request fails, or unable to extract +// the desired credentials. +func (m *EC2RoleProvider) RetrieveWithContext(ctx credentials.Context) (credentials.Value, error) { + credsList, err := requestCredList(ctx, m.Client) if err != nil { return credentials.Value{ProviderName: ProviderName}, err } @@ -96,7 +105,7 @@ func (m *EC2RoleProvider) Retrieve() (credentials.Value, error) { } credsName := credsList[0] - roleCreds, err := requestCred(m.Client, credsName) + roleCreds, err := requestCred(ctx, m.Client, credsName) if err != nil { return credentials.Value{ProviderName: ProviderName}, err } @@ -129,8 +138,8 @@ const iamSecurityCredsPath = "iam/security-credentials/" // requestCredList requests a list of credentials from the EC2 service. // If there are no credentials, or there is an error making or receiving the request -func requestCredList(client *ec2metadata.EC2Metadata) ([]string, error) { - resp, err := client.GetMetadata(iamSecurityCredsPath) +func requestCredList(ctx aws.Context, client *ec2metadata.EC2Metadata) ([]string, error) { + resp, err := client.GetMetadataWithContext(ctx, iamSecurityCredsPath) if err != nil { return nil, awserr.New("EC2RoleRequestError", "no EC2 instance role found", err) } @@ -142,7 +151,8 @@ func requestCredList(client *ec2metadata.EC2Metadata) ([]string, error) { } if err := s.Err(); err != nil { - return nil, awserr.New("SerializationError", "failed to read EC2 instance role from metadata service", err) + return nil, awserr.New(request.ErrCodeSerialization, + "failed to read EC2 instance role from metadata service", err) } return credsList, nil @@ -152,8 +162,8 @@ func requestCredList(client *ec2metadata.EC2Metadata) ([]string, error) { // // If the credentials cannot be found, or there is an error reading the response // and error will be returned. -func requestCred(client *ec2metadata.EC2Metadata, credsName string) (ec2RoleCredRespBody, error) { - resp, err := client.GetMetadata(sdkuri.PathJoin(iamSecurityCredsPath, credsName)) +func requestCred(ctx aws.Context, client *ec2metadata.EC2Metadata, credsName string) (ec2RoleCredRespBody, error) { + resp, err := client.GetMetadataWithContext(ctx, sdkuri.PathJoin(iamSecurityCredsPath, credsName)) if err != nil { return ec2RoleCredRespBody{}, awserr.New("EC2RoleRequestError", @@ -164,7 +174,7 @@ func requestCred(client *ec2metadata.EC2Metadata, credsName string) (ec2RoleCred respCreds := ec2RoleCredRespBody{} if err := json.NewDecoder(strings.NewReader(resp)).Decode(&respCreds); err != nil { return ec2RoleCredRespBody{}, - awserr.New("SerializationError", + awserr.New(request.ErrCodeSerialization, fmt.Sprintf("failed to decode %s EC2 instance role credentials", credsName), err) } diff --git a/github.com/aws/aws-sdk-go/aws/credentials/endpointcreds/provider.go b/github.com/aws/aws-sdk-go/aws/credentials/endpointcreds/provider.go index ace5131382..785f30d8e6 100644 --- a/github.com/aws/aws-sdk-go/aws/credentials/endpointcreds/provider.go +++ b/github.com/aws/aws-sdk-go/aws/credentials/endpointcreds/provider.go @@ -39,6 +39,7 @@ import ( "github.com/aws/aws-sdk-go/aws/client/metadata" "github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/private/protocol/json/jsonutil" ) // ProviderName is the name of the credentials provider. @@ -97,8 +98,8 @@ func NewProviderClient(cfg aws.Config, handlers request.Handlers, endpoint strin return p } -// NewCredentialsClient returns a Credentials wrapper for retrieving credentials -// from an arbitrary endpoint concurrently. The client will request the +// NewCredentialsClient returns a pointer to a new Credentials object +// wrapping the endpoint credentials Provider. func NewCredentialsClient(cfg aws.Config, handlers request.Handlers, endpoint string, options ...func(*Provider)) *credentials.Credentials { return credentials.NewCredentials(NewProviderClient(cfg, handlers, endpoint, options...)) } @@ -115,7 +116,13 @@ func (p *Provider) IsExpired() bool { // Retrieve will attempt to request the credentials from the endpoint the Provider // was configured for. And error will be returned if the retrieval fails. func (p *Provider) Retrieve() (credentials.Value, error) { - resp, err := p.getCredentials() + return p.RetrieveWithContext(aws.BackgroundContext()) +} + +// RetrieveWithContext will attempt to request the credentials from the endpoint the Provider +// was configured for. And error will be returned if the retrieval fails. +func (p *Provider) RetrieveWithContext(ctx credentials.Context) (credentials.Value, error) { + resp, err := p.getCredentials(ctx) if err != nil { return credentials.Value{ProviderName: ProviderName}, awserr.New("CredentialsEndpointError", "failed to load credentials", err) @@ -147,7 +154,7 @@ type errorOutput struct { Message string `json:"message"` } -func (p *Provider) getCredentials() (*getCredentialsOutput, error) { +func (p *Provider) getCredentials(ctx aws.Context) (*getCredentialsOutput, error) { op := &request.Operation{ Name: "GetCredentials", HTTPMethod: "GET", @@ -155,6 +162,7 @@ func (p *Provider) getCredentials() (*getCredentialsOutput, error) { out := &getCredentialsOutput{} req := p.Client.NewRequest(op, nil, out) + req.SetContext(ctx) req.HTTPRequest.Header.Set("Accept", "application/json") if authToken := p.AuthorizationToken; len(authToken) != 0 { req.HTTPRequest.Header.Set("Authorization", authToken) @@ -174,7 +182,7 @@ func unmarshalHandler(r *request.Request) { out := r.Data.(*getCredentialsOutput) if err := json.NewDecoder(r.HTTPResponse.Body).Decode(&out); err != nil { - r.Error = awserr.New("SerializationError", + r.Error = awserr.New(request.ErrCodeSerialization, "failed to decode endpoint credentials", err, ) @@ -185,11 +193,15 @@ func unmarshalError(r *request.Request) { defer r.HTTPResponse.Body.Close() var errOut errorOutput - if err := json.NewDecoder(r.HTTPResponse.Body).Decode(&errOut); err != nil { - r.Error = awserr.New("SerializationError", - "failed to decode endpoint credentials", - err, + err := jsonutil.UnmarshalJSONError(&errOut, r.HTTPResponse.Body) + if err != nil { + r.Error = awserr.NewRequestFailure( + awserr.New(request.ErrCodeSerialization, + "failed to decode error message", err), + r.HTTPResponse.StatusCode, + r.RequestID, ) + return } // Response body format is not consistent between metadata endpoints. diff --git a/github.com/aws/aws-sdk-go/aws/credentials/processcreds/provider.go b/github.com/aws/aws-sdk-go/aws/credentials/processcreds/provider.go index 1980c8c140..e624836002 100644 --- a/github.com/aws/aws-sdk-go/aws/credentials/processcreds/provider.go +++ b/github.com/aws/aws-sdk-go/aws/credentials/processcreds/provider.go @@ -90,6 +90,7 @@ import ( "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/credentials" + "github.com/aws/aws-sdk-go/internal/sdkio" ) const ( @@ -142,7 +143,7 @@ const ( // DefaultBufSize limits buffer size from growing to an enormous // amount due to a faulty process. - DefaultBufSize = 1024 + DefaultBufSize = int(8 * sdkio.KibiByte) // DefaultTimeout default limit on time a process can run. DefaultTimeout = time.Duration(1) * time.Minute diff --git a/github.com/aws/aws-sdk-go/aws/credentials/shared_credentials_provider.go b/github.com/aws/aws-sdk-go/aws/credentials/shared_credentials_provider.go index e155149581..22b5c5d9f3 100644 --- a/github.com/aws/aws-sdk-go/aws/credentials/shared_credentials_provider.go +++ b/github.com/aws/aws-sdk-go/aws/credentials/shared_credentials_provider.go @@ -17,8 +17,9 @@ var ( ErrSharedCredentialsHomeNotFound = awserr.New("UserHomeNotFound", "user home directory not found.", nil) ) -// A SharedCredentialsProvider retrieves credentials from the current user's home -// directory, and keeps track if those credentials are expired. +// A SharedCredentialsProvider retrieves access key pair (access key ID, +// secret access key, and session token if present) credentials from the current +// user's home directory, and keeps track if those credentials are expired. // // Profile ini file example: $HOME/.aws/credentials type SharedCredentialsProvider struct { diff --git a/github.com/aws/aws-sdk-go/aws/credentials/static_provider.go b/github.com/aws/aws-sdk-go/aws/credentials/static_provider.go index 531139e397..cbba1e3d56 100644 --- a/github.com/aws/aws-sdk-go/aws/credentials/static_provider.go +++ b/github.com/aws/aws-sdk-go/aws/credentials/static_provider.go @@ -19,7 +19,9 @@ type StaticProvider struct { } // NewStaticCredentials returns a pointer to a new Credentials object -// wrapping a static credentials value provider. +// wrapping a static credentials value provider. Token is only required +// for temporary security credentials retrieved via STS, otherwise an empty +// string can be passed for this parameter. func NewStaticCredentials(id, secret, token string) *Credentials { return NewCredentials(&StaticProvider{Value: Value{ AccessKeyID: id, diff --git a/github.com/aws/aws-sdk-go/aws/credentials/stscreds/assume_role_provider.go b/github.com/aws/aws-sdk-go/aws/credentials/stscreds/assume_role_provider.go index 4108e433e6..6846ef6f80 100644 --- a/github.com/aws/aws-sdk-go/aws/credentials/stscreds/assume_role_provider.go +++ b/github.com/aws/aws-sdk-go/aws/credentials/stscreds/assume_role_provider.go @@ -80,16 +80,19 @@ package stscreds import ( "fmt" + "os" "time" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/client" "github.com/aws/aws-sdk-go/aws/credentials" + "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/internal/sdkrand" "github.com/aws/aws-sdk-go/service/sts" ) -// StdinTokenProvider will prompt on stdout and read from stdin for a string value. +// StdinTokenProvider will prompt on stderr and read from stdin for a string value. // An error is returned if reading from stdin fails. // // Use this function go read MFA tokens from stdin. The function makes no attempt @@ -102,7 +105,7 @@ import ( // Will wait forever until something is provided on the stdin. func StdinTokenProvider() (string, error) { var v string - fmt.Printf("Assume Role MFA token code: ") + fmt.Fprintf(os.Stderr, "Assume Role MFA token code: ") _, err := fmt.Scanln(&v) return v, err @@ -116,6 +119,10 @@ type AssumeRoler interface { AssumeRole(input *sts.AssumeRoleInput) (*sts.AssumeRoleOutput, error) } +type assumeRolerWithContext interface { + AssumeRoleWithContext(aws.Context, *sts.AssumeRoleInput, ...request.Option) (*sts.AssumeRoleOutput, error) +} + // DefaultDuration is the default amount of time in minutes that the credentials // will be valid for. var DefaultDuration = time.Duration(15) * time.Minute @@ -142,6 +149,13 @@ type AssumeRoleProvider struct { // Session name, if you wish to reuse the credentials elsewhere. RoleSessionName string + // Optional, you can pass tag key-value pairs to your session. These tags are called session tags. + Tags []*sts.Tag + + // A list of keys for session tags that you want to set as transitive. + // If you set a tag key as transitive, the corresponding key and value passes to subsequent sessions in a role chain. + TransitiveTagKeys []*string + // Expiry duration of the STS credentials. Defaults to 15 minutes if not set. Duration time.Duration @@ -155,6 +169,29 @@ type AssumeRoleProvider struct { // size. Policy *string + // The ARNs of IAM managed policies you want to use as managed session policies. + // The policies must exist in the same account as the role. + // + // This parameter is optional. You can provide up to 10 managed policy ARNs. + // However, the plain text that you use for both inline and managed session + // policies can't exceed 2,048 characters. + // + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. + // + // Passing policies to this operation returns new temporary credentials. The + // resulting session's permissions are the intersection of the role's identity-based + // policy and the session policies. You can use the role's temporary credentials + // in subsequent AWS API calls to access resources in the account that owns + // the role. You cannot use session policies to grant more permissions than + // those allowed by the identity-based policy of the role that is being assumed. + // For more information, see Session Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) + // in the IAM User Guide. + PolicyArns []*sts.PolicyDescriptorType + // The identification number of the MFA device that is associated with the user // who is making the AssumeRole call. Specify this value if the trust policy // of the role being assumed includes a condition that requires MFA authentication. @@ -193,6 +230,18 @@ type AssumeRoleProvider struct { // // If ExpiryWindow is 0 or less it will be ignored. ExpiryWindow time.Duration + + // MaxJitterFrac reduces the effective Duration of each credential requested + // by a random percentage between 0 and MaxJitterFraction. MaxJitterFrac must + // have a value between 0 and 1. Any other value may lead to expected behavior. + // With a MaxJitterFrac value of 0, default) will no jitter will be used. + // + // For example, with a Duration of 30m and a MaxJitterFrac of 0.1, the + // AssumeRole call will be made with an arbitrary Duration between 27m and + // 30m. + // + // MaxJitterFrac should not be negative. + MaxJitterFrac float64 } // NewCredentials returns a pointer to a new Credentials object wrapping the @@ -244,7 +293,11 @@ func NewCredentialsWithClient(svc AssumeRoler, roleARN string, options ...func(* // Retrieve generates a new set of temporary credentials using STS. func (p *AssumeRoleProvider) Retrieve() (credentials.Value, error) { + return p.RetrieveWithContext(aws.BackgroundContext()) +} +// RetrieveWithContext generates a new set of temporary credentials using STS. +func (p *AssumeRoleProvider) RetrieveWithContext(ctx credentials.Context) (credentials.Value, error) { // Apply defaults where parameters are not set. if p.RoleSessionName == "" { // Try to work out a role name that will hopefully end up unique. @@ -254,11 +307,15 @@ func (p *AssumeRoleProvider) Retrieve() (credentials.Value, error) { // Expire as often as AWS permits. p.Duration = DefaultDuration } + jitter := time.Duration(sdkrand.SeededRand.Float64() * p.MaxJitterFrac * float64(p.Duration)) input := &sts.AssumeRoleInput{ - DurationSeconds: aws.Int64(int64(p.Duration / time.Second)), - RoleArn: aws.String(p.RoleARN), - RoleSessionName: aws.String(p.RoleSessionName), - ExternalId: p.ExternalID, + DurationSeconds: aws.Int64(int64((p.Duration - jitter) / time.Second)), + RoleArn: aws.String(p.RoleARN), + RoleSessionName: aws.String(p.RoleSessionName), + ExternalId: p.ExternalID, + Tags: p.Tags, + PolicyArns: p.PolicyArns, + TransitiveTagKeys: p.TransitiveTagKeys, } if p.Policy != nil { input.Policy = p.Policy @@ -281,7 +338,15 @@ func (p *AssumeRoleProvider) Retrieve() (credentials.Value, error) { } } - roleOutput, err := p.Client.AssumeRole(input) + var roleOutput *sts.AssumeRoleOutput + var err error + + if c, ok := p.Client.(assumeRolerWithContext); ok { + roleOutput, err = c.AssumeRoleWithContext(ctx, input) + } else { + roleOutput, err = p.Client.AssumeRole(input) + } + if err != nil { return credentials.Value{ProviderName: ProviderName}, err } diff --git a/github.com/aws/aws-sdk-go/aws/credentials/stscreds/web_identity_provider.go b/github.com/aws/aws-sdk-go/aws/credentials/stscreds/web_identity_provider.go new file mode 100644 index 0000000000..6feb262b2f --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/credentials/stscreds/web_identity_provider.go @@ -0,0 +1,135 @@ +package stscreds + +import ( + "fmt" + "io/ioutil" + "strconv" + "time" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/aws/client" + "github.com/aws/aws-sdk-go/aws/credentials" + "github.com/aws/aws-sdk-go/service/sts" + "github.com/aws/aws-sdk-go/service/sts/stsiface" +) + +const ( + // ErrCodeWebIdentity will be used as an error code when constructing + // a new error to be returned during session creation or retrieval. + ErrCodeWebIdentity = "WebIdentityErr" + + // WebIdentityProviderName is the web identity provider name + WebIdentityProviderName = "WebIdentityCredentials" +) + +// now is used to return a time.Time object representing +// the current time. This can be used to easily test and +// compare test values. +var now = time.Now + +// TokenFetcher shuold return WebIdentity token bytes or an error +type TokenFetcher interface { + FetchToken(credentials.Context) ([]byte, error) +} + +// FetchTokenPath is a path to a WebIdentity token file +type FetchTokenPath string + +// FetchToken returns a token by reading from the filesystem +func (f FetchTokenPath) FetchToken(ctx credentials.Context) ([]byte, error) { + data, err := ioutil.ReadFile(string(f)) + if err != nil { + errMsg := fmt.Sprintf("unable to read file at %s", f) + return nil, awserr.New(ErrCodeWebIdentity, errMsg, err) + } + return data, nil +} + +// WebIdentityRoleProvider is used to retrieve credentials using +// an OIDC token. +type WebIdentityRoleProvider struct { + credentials.Expiry + PolicyArns []*sts.PolicyDescriptorType + + client stsiface.STSAPI + ExpiryWindow time.Duration + + tokenFetcher TokenFetcher + roleARN string + roleSessionName string +} + +// NewWebIdentityCredentials will return a new set of credentials with a given +// configuration, role arn, and token file path. +func NewWebIdentityCredentials(c client.ConfigProvider, roleARN, roleSessionName, path string) *credentials.Credentials { + svc := sts.New(c) + p := NewWebIdentityRoleProvider(svc, roleARN, roleSessionName, path) + return credentials.NewCredentials(p) +} + +// NewWebIdentityRoleProvider will return a new WebIdentityRoleProvider with the +// provided stsiface.STSAPI +func NewWebIdentityRoleProvider(svc stsiface.STSAPI, roleARN, roleSessionName, path string) *WebIdentityRoleProvider { + return NewWebIdentityRoleProviderWithToken(svc, roleARN, roleSessionName, FetchTokenPath(path)) +} + +// NewWebIdentityRoleProviderWithToken will return a new WebIdentityRoleProvider with the +// provided stsiface.STSAPI and a TokenFetcher +func NewWebIdentityRoleProviderWithToken(svc stsiface.STSAPI, roleARN, roleSessionName string, tokenFetcher TokenFetcher) *WebIdentityRoleProvider { + return &WebIdentityRoleProvider{ + client: svc, + tokenFetcher: tokenFetcher, + roleARN: roleARN, + roleSessionName: roleSessionName, + } +} + +// Retrieve will attempt to assume a role from a token which is located at +// 'WebIdentityTokenFilePath' specified destination and if that is empty an +// error will be returned. +func (p *WebIdentityRoleProvider) Retrieve() (credentials.Value, error) { + return p.RetrieveWithContext(aws.BackgroundContext()) +} + +// RetrieveWithContext will attempt to assume a role from a token which is located at +// 'WebIdentityTokenFilePath' specified destination and if that is empty an +// error will be returned. +func (p *WebIdentityRoleProvider) RetrieveWithContext(ctx credentials.Context) (credentials.Value, error) { + b, err := p.tokenFetcher.FetchToken(ctx) + if err != nil { + return credentials.Value{}, awserr.New(ErrCodeWebIdentity, "failed fetching WebIdentity token: ", err) + } + + sessionName := p.roleSessionName + if len(sessionName) == 0 { + // session name is used to uniquely identify a session. This simply + // uses unix time in nanoseconds to uniquely identify sessions. + sessionName = strconv.FormatInt(now().UnixNano(), 10) + } + req, resp := p.client.AssumeRoleWithWebIdentityRequest(&sts.AssumeRoleWithWebIdentityInput{ + PolicyArns: p.PolicyArns, + RoleArn: &p.roleARN, + RoleSessionName: &sessionName, + WebIdentityToken: aws.String(string(b)), + }) + + req.SetContext(ctx) + + // InvalidIdentityToken error is a temporary error that can occur + // when assuming an Role with a JWT web identity token. + req.RetryErrorCodes = append(req.RetryErrorCodes, sts.ErrCodeInvalidIdentityTokenException) + if err := req.Send(); err != nil { + return credentials.Value{}, awserr.New(ErrCodeWebIdentity, "failed to retrieve credentials", err) + } + + p.SetExpiration(aws.TimeValue(resp.Credentials.Expiration), p.ExpiryWindow) + + value := credentials.Value{ + AccessKeyID: aws.StringValue(resp.Credentials.AccessKeyId), + SecretAccessKey: aws.StringValue(resp.Credentials.SecretAccessKey), + SessionToken: aws.StringValue(resp.Credentials.SessionToken), + ProviderName: WebIdentityProviderName, + } + return value, nil +} diff --git a/github.com/aws/aws-sdk-go/aws/csm/doc.go b/github.com/aws/aws-sdk-go/aws/csm/doc.go index 152d785b36..25a66d1dda 100644 --- a/github.com/aws/aws-sdk-go/aws/csm/doc.go +++ b/github.com/aws/aws-sdk-go/aws/csm/doc.go @@ -1,30 +1,61 @@ -// Package csm provides Client Side Monitoring (CSM) which enables sending metrics -// via UDP connection. Using the Start function will enable the reporting of -// metrics on a given port. If Start is called, with different parameters, again, -// a panic will occur. +// Package csm provides the Client Side Monitoring (CSM) client which enables +// sending metrics via UDP connection to the CSM agent. This package provides +// control options, and configuration for the CSM client. The client can be +// controlled manually, or automatically via the SDK's Session configuration. // -// Pause can be called to pause any metrics publishing on a given port. Sessions -// that have had their handlers modified via InjectHandlers may still be used. -// However, the handlers will act as a no-op meaning no metrics will be published. +// Enabling CSM client via SDK's Session configuration +// +// The CSM client can be enabled automatically via SDK's Session configuration. +// The SDK's session configuration enables the CSM client if the AWS_CSM_PORT +// environment variable is set to a non-empty value. +// +// The configuration options for the CSM client via the SDK's session +// configuration are: +// +// * AWS_CSM_PORT= +// The port number the CSM agent will receive metrics on. +// +// * AWS_CSM_HOST= +// The hostname, or IP address the CSM agent will receive metrics on. +// Without port number. +// +// Manually enabling the CSM client +// +// The CSM client can be started, paused, and resumed manually. The Start +// function will enable the CSM client to publish metrics to the CSM agent. It +// is safe to call Start concurrently, but if Start is called additional times +// with different ClientID or address it will panic. // -// Example: // r, err := csm.Start("clientID", ":31000") // if err != nil { // panic(fmt.Errorf("failed starting CSM: %v", err)) // } // +// When controlling the CSM client manually, you must also inject its request +// handlers into the SDK's Session configuration for the SDK's API clients to +// publish metrics. +// // sess, err := session.NewSession(&aws.Config{}) // if err != nil { // panic(fmt.Errorf("failed loading session: %v", err)) // } // +// // Add CSM client's metric publishing request handlers to the SDK's +// // Session Configuration. // r.InjectHandlers(&sess.Handlers) // -// client := s3.New(sess) -// resp, err := client.GetObject(&s3.GetObjectInput{ -// Bucket: aws.String("bucket"), -// Key: aws.String("key"), -// }) +// Controlling CSM client +// +// Once the CSM client has been enabled the Get function will return a Reporter +// value that you can use to pause and resume the metrics published to the CSM +// agent. If Get function is called before the reporter is enabled with the +// Start function or via SDK's Session configuration nil will be returned. +// +// The Pause method can be called to stop the CSM client publishing metrics to +// the CSM agent. The Continue method will resume metric publishing. +// +// // Get the CSM client Reporter. +// r := csm.Get() // // // Will pause monitoring // r.Pause() @@ -35,12 +66,4 @@ // // // Resume monitoring // r.Continue() -// -// Start returns a Reporter that is used to enable or disable monitoring. If -// access to the Reporter is required later, calling Get will return the Reporter -// singleton. -// -// Example: -// r := csm.Get() -// r.Continue() package csm diff --git a/github.com/aws/aws-sdk-go/aws/csm/enable.go b/github.com/aws/aws-sdk-go/aws/csm/enable.go index 2f0c6eac9a..4b19e2800e 100644 --- a/github.com/aws/aws-sdk-go/aws/csm/enable.go +++ b/github.com/aws/aws-sdk-go/aws/csm/enable.go @@ -2,6 +2,7 @@ package csm import ( "fmt" + "strings" "sync" ) @@ -9,19 +10,40 @@ var ( lock sync.Mutex ) -// Client side metric handler names const ( - APICallMetricHandlerName = "awscsm.SendAPICallMetric" - APICallAttemptMetricHandlerName = "awscsm.SendAPICallAttemptMetric" + // DefaultPort is used when no port is specified. + DefaultPort = "31000" + + // DefaultHost is the host that will be used when none is specified. + DefaultHost = "127.0.0.1" ) -// Start will start the a long running go routine to capture +// AddressWithDefaults returns a CSM address built from the host and port +// values. If the host or port is not set, default values will be used +// instead. If host is "localhost" it will be replaced with "127.0.0.1". +func AddressWithDefaults(host, port string) string { + if len(host) == 0 || strings.EqualFold(host, "localhost") { + host = DefaultHost + } + + if len(port) == 0 { + port = DefaultPort + } + + // Only IP6 host can contain a colon + if strings.Contains(host, ":") { + return "[" + host + "]:" + port + } + + return host + ":" + port +} + +// Start will start a long running go routine to capture // client side metrics. Calling start multiple time will only // start the metric listener once and will panic if a different // client ID or port is passed in. // -// Example: -// r, err := csm.Start("clientID", "127.0.0.1:8094") +// r, err := csm.Start("clientID", "127.0.0.1:31000") // if err != nil { // panic(fmt.Errorf("expected no error, but received %v", err)) // } diff --git a/github.com/aws/aws-sdk-go/aws/csm/metric_chan.go b/github.com/aws/aws-sdk-go/aws/csm/metric_chan.go index 514fc3739a..82a3e345e9 100644 --- a/github.com/aws/aws-sdk-go/aws/csm/metric_chan.go +++ b/github.com/aws/aws-sdk-go/aws/csm/metric_chan.go @@ -16,25 +16,26 @@ var ( type metricChan struct { ch chan metric - paused int64 + paused *int64 } func newMetricChan(size int) metricChan { return metricChan{ - ch: make(chan metric, size), + ch: make(chan metric, size), + paused: new(int64), } } func (ch *metricChan) Pause() { - atomic.StoreInt64(&ch.paused, pausedEnum) + atomic.StoreInt64(ch.paused, pausedEnum) } func (ch *metricChan) Continue() { - atomic.StoreInt64(&ch.paused, runningEnum) + atomic.StoreInt64(ch.paused, runningEnum) } func (ch *metricChan) IsPaused() bool { - v := atomic.LoadInt64(&ch.paused) + v := atomic.LoadInt64(ch.paused) return v == pausedEnum } diff --git a/github.com/aws/aws-sdk-go/aws/csm/reporter.go b/github.com/aws/aws-sdk-go/aws/csm/reporter.go index 0b5571acfb..835bcd49cb 100644 --- a/github.com/aws/aws-sdk-go/aws/csm/reporter.go +++ b/github.com/aws/aws-sdk-go/aws/csm/reporter.go @@ -10,11 +10,6 @@ import ( "github.com/aws/aws-sdk-go/aws/request" ) -const ( - // DefaultPort is used when no port is specified - DefaultPort = "31000" -) - // Reporter will gather metrics of API requests made and // send those metrics to the CSM endpoint. type Reporter struct { @@ -71,7 +66,6 @@ func (rep *Reporter) sendAPICallAttemptMetric(r *request.Request) { XAmzRequestID: aws.String(r.RequestID), - AttemptCount: aws.Int(r.RetryCount + 1), AttemptLatency: aws.Int(int(now.Sub(r.AttemptTime).Nanoseconds() / int64(time.Millisecond))), AccessKey: aws.String(creds.AccessKeyID), } @@ -95,8 +89,8 @@ func getMetricException(err awserr.Error) metricException { code := err.Code() switch code { - case "RequestError", - "SerializationError", + case request.ErrCodeRequestError, + request.ErrCodeSerialization, request.CanceledErrorCode: return sdkException{ requestException{exception: code, message: msg}, @@ -123,7 +117,7 @@ func (rep *Reporter) sendAPICallMetric(r *request.Request) { Type: aws.String("ApiCall"), AttemptCount: aws.Int(r.RetryCount + 1), Region: r.Config.Region, - Latency: aws.Int(int(time.Now().Sub(r.Time) / time.Millisecond)), + Latency: aws.Int(int(time.Since(r.Time) / time.Millisecond)), XAmzRequestID: aws.String(r.RequestID), MaxRetriesExceeded: aws.Int(boolIntValue(r.RetryCount >= r.MaxRetries())), } @@ -190,8 +184,9 @@ func (rep *Reporter) start() { } } -// Pause will pause the metric channel preventing any new metrics from -// being added. +// Pause will pause the metric channel preventing any new metrics from being +// added. It is safe to call concurrently with other calls to Pause, but if +// called concurently with Continue can lead to unexpected state. func (rep *Reporter) Pause() { lock.Lock() defer lock.Unlock() @@ -203,8 +198,9 @@ func (rep *Reporter) Pause() { rep.close() } -// Continue will reopen the metric channel and allow for monitoring -// to be resumed. +// Continue will reopen the metric channel and allow for monitoring to be +// resumed. It is safe to call concurrently with other calls to Continue, but +// if called concurently with Pause can lead to unexpected state. func (rep *Reporter) Continue() { lock.Lock() defer lock.Unlock() @@ -219,10 +215,18 @@ func (rep *Reporter) Continue() { rep.metricsCh.Continue() } +// Client side metric handler names +const ( + APICallMetricHandlerName = "awscsm.SendAPICallMetric" + APICallAttemptMetricHandlerName = "awscsm.SendAPICallAttemptMetric" +) + // InjectHandlers will will enable client side metrics and inject the proper // handlers to handle how metrics are sent. // -// Example: +// InjectHandlers is NOT safe to call concurrently. Calling InjectHandlers +// multiple times may lead to unexpected behavior, (e.g. duplicate metrics). +// // // Start must be called in order to inject the correct handlers // r, err := csm.Start("clientID", "127.0.0.1:8094") // if err != nil { diff --git a/github.com/aws/aws-sdk-go/aws/ec2metadata/api.go b/github.com/aws/aws-sdk-go/aws/ec2metadata/api.go index 88e2fc7073..a716c021cf 100644 --- a/github.com/aws/aws-sdk-go/aws/ec2metadata/api.go +++ b/github.com/aws/aws-sdk-go/aws/ec2metadata/api.go @@ -4,34 +4,87 @@ import ( "encoding/json" "fmt" "net/http" + "strconv" "strings" "time" + "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/request" "github.com/aws/aws-sdk-go/internal/sdkuri" ) +// getToken uses the duration to return a token for EC2 metadata service, +// or an error if the request failed. +func (c *EC2Metadata) getToken(ctx aws.Context, duration time.Duration) (tokenOutput, error) { + op := &request.Operation{ + Name: "GetToken", + HTTPMethod: "PUT", + HTTPPath: "/api/token", + } + + var output tokenOutput + req := c.NewRequest(op, nil, &output) + req.SetContext(ctx) + + // remove the fetch token handler from the request handlers to avoid infinite recursion + req.Handlers.Sign.RemoveByName(fetchTokenHandlerName) + + // Swap the unmarshalMetadataHandler with unmarshalTokenHandler on this request. + req.Handlers.Unmarshal.Swap(unmarshalMetadataHandlerName, unmarshalTokenHandler) + + ttl := strconv.FormatInt(int64(duration/time.Second), 10) + req.HTTPRequest.Header.Set(ttlHeader, ttl) + + err := req.Send() + + // Errors with bad request status should be returned. + if err != nil { + err = awserr.NewRequestFailure( + awserr.New(req.HTTPResponse.Status, http.StatusText(req.HTTPResponse.StatusCode), err), + req.HTTPResponse.StatusCode, req.RequestID) + } + + return output, err +} + // GetMetadata uses the path provided to request information from the EC2 -// instance metdata service. The content will be returned as a string, or +// instance metadata service. The content will be returned as a string, or // error if the request failed. func (c *EC2Metadata) GetMetadata(p string) (string, error) { + return c.GetMetadataWithContext(aws.BackgroundContext(), p) +} + +// GetMetadataWithContext uses the path provided to request information from the EC2 +// instance metadata service. The content will be returned as a string, or +// error if the request failed. +func (c *EC2Metadata) GetMetadataWithContext(ctx aws.Context, p string) (string, error) { op := &request.Operation{ Name: "GetMetadata", HTTPMethod: "GET", HTTPPath: sdkuri.PathJoin("/meta-data", p), } - output := &metadataOutput{} + req := c.NewRequest(op, nil, output) - return output.Content, req.Send() + req.SetContext(ctx) + + err := req.Send() + return output.Content, err } // GetUserData returns the userdata that was configured for the service. If // there is no user-data setup for the EC2 instance a "NotFoundError" error // code will be returned. func (c *EC2Metadata) GetUserData() (string, error) { + return c.GetUserDataWithContext(aws.BackgroundContext()) +} + +// GetUserDataWithContext returns the userdata that was configured for the service. If +// there is no user-data setup for the EC2 instance a "NotFoundError" error +// code will be returned. +func (c *EC2Metadata) GetUserDataWithContext(ctx aws.Context) (string, error) { op := &request.Operation{ Name: "GetUserData", HTTPMethod: "GET", @@ -40,19 +93,23 @@ func (c *EC2Metadata) GetUserData() (string, error) { output := &metadataOutput{} req := c.NewRequest(op, nil, output) - req.Handlers.UnmarshalError.PushBack(func(r *request.Request) { - if r.HTTPResponse.StatusCode == http.StatusNotFound { - r.Error = awserr.New("NotFoundError", "user-data not found", r.Error) - } - }) + req.SetContext(ctx) - return output.Content, req.Send() + err := req.Send() + return output.Content, err } // GetDynamicData uses the path provided to request information from the EC2 // instance metadata service for dynamic data. The content will be returned // as a string, or error if the request failed. func (c *EC2Metadata) GetDynamicData(p string) (string, error) { + return c.GetDynamicDataWithContext(aws.BackgroundContext(), p) +} + +// GetDynamicDataWithContext uses the path provided to request information from the EC2 +// instance metadata service for dynamic data. The content will be returned +// as a string, or error if the request failed. +func (c *EC2Metadata) GetDynamicDataWithContext(ctx aws.Context, p string) (string, error) { op := &request.Operation{ Name: "GetDynamicData", HTTPMethod: "GET", @@ -61,15 +118,24 @@ func (c *EC2Metadata) GetDynamicData(p string) (string, error) { output := &metadataOutput{} req := c.NewRequest(op, nil, output) + req.SetContext(ctx) - return output.Content, req.Send() + err := req.Send() + return output.Content, err } // GetInstanceIdentityDocument retrieves an identity document describing an // instance. Error is returned if the request fails or is unable to parse // the response. func (c *EC2Metadata) GetInstanceIdentityDocument() (EC2InstanceIdentityDocument, error) { - resp, err := c.GetDynamicData("instance-identity/document") + return c.GetInstanceIdentityDocumentWithContext(aws.BackgroundContext()) +} + +// GetInstanceIdentityDocumentWithContext retrieves an identity document describing an +// instance. Error is returned if the request fails or is unable to parse +// the response. +func (c *EC2Metadata) GetInstanceIdentityDocumentWithContext(ctx aws.Context) (EC2InstanceIdentityDocument, error) { + resp, err := c.GetDynamicDataWithContext(ctx, "instance-identity/document") if err != nil { return EC2InstanceIdentityDocument{}, awserr.New("EC2MetadataRequestError", @@ -79,7 +145,7 @@ func (c *EC2Metadata) GetInstanceIdentityDocument() (EC2InstanceIdentityDocument doc := EC2InstanceIdentityDocument{} if err := json.NewDecoder(strings.NewReader(resp)).Decode(&doc); err != nil { return EC2InstanceIdentityDocument{}, - awserr.New("SerializationError", + awserr.New(request.ErrCodeSerialization, "failed to decode EC2 instance identity document", err) } @@ -88,7 +154,12 @@ func (c *EC2Metadata) GetInstanceIdentityDocument() (EC2InstanceIdentityDocument // IAMInfo retrieves IAM info from the metadata API func (c *EC2Metadata) IAMInfo() (EC2IAMInfo, error) { - resp, err := c.GetMetadata("iam/info") + return c.IAMInfoWithContext(aws.BackgroundContext()) +} + +// IAMInfoWithContext retrieves IAM info from the metadata API +func (c *EC2Metadata) IAMInfoWithContext(ctx aws.Context) (EC2IAMInfo, error) { + resp, err := c.GetMetadataWithContext(ctx, "iam/info") if err != nil { return EC2IAMInfo{}, awserr.New("EC2MetadataRequestError", @@ -98,7 +169,7 @@ func (c *EC2Metadata) IAMInfo() (EC2IAMInfo, error) { info := EC2IAMInfo{} if err := json.NewDecoder(strings.NewReader(resp)).Decode(&info); err != nil { return EC2IAMInfo{}, - awserr.New("SerializationError", + awserr.New(request.ErrCodeSerialization, "failed to decode EC2 IAM info", err) } @@ -113,24 +184,36 @@ func (c *EC2Metadata) IAMInfo() (EC2IAMInfo, error) { // Region returns the region the instance is running in. func (c *EC2Metadata) Region() (string, error) { - resp, err := c.GetMetadata("placement/availability-zone") + return c.RegionWithContext(aws.BackgroundContext()) +} + +// RegionWithContext returns the region the instance is running in. +func (c *EC2Metadata) RegionWithContext(ctx aws.Context) (string, error) { + ec2InstanceIdentityDocument, err := c.GetInstanceIdentityDocumentWithContext(ctx) if err != nil { return "", err } - - if len(resp) == 0 { - return "", awserr.New("EC2MetadataError", "invalid Region response", nil) + // extract region from the ec2InstanceIdentityDocument + region := ec2InstanceIdentityDocument.Region + if len(region) == 0 { + return "", awserr.New("EC2MetadataError", "invalid region received for ec2metadata instance", nil) } - - // returns region without the suffix. Eg: us-west-2a becomes us-west-2 - return resp[:len(resp)-1], nil + // returns region + return region, nil } // Available returns if the application has access to the EC2 Metadata service. // Can be used to determine if application is running within an EC2 Instance and // the metadata service is available. func (c *EC2Metadata) Available() bool { - if _, err := c.GetMetadata("instance-id"); err != nil { + return c.AvailableWithContext(aws.BackgroundContext()) +} + +// AvailableWithContext returns if the application has access to the EC2 Metadata service. +// Can be used to determine if application is running within an EC2 Instance and +// the metadata service is available. +func (c *EC2Metadata) AvailableWithContext(ctx aws.Context) bool { + if _, err := c.GetMetadataWithContext(ctx, "instance-id"); err != nil { return false } @@ -149,18 +232,19 @@ type EC2IAMInfo struct { // An EC2InstanceIdentityDocument provides the shape for unmarshaling // an instance identity document type EC2InstanceIdentityDocument struct { - DevpayProductCodes []string `json:"devpayProductCodes"` - AvailabilityZone string `json:"availabilityZone"` - PrivateIP string `json:"privateIp"` - Version string `json:"version"` - Region string `json:"region"` - InstanceID string `json:"instanceId"` - BillingProducts []string `json:"billingProducts"` - InstanceType string `json:"instanceType"` - AccountID string `json:"accountId"` - PendingTime time.Time `json:"pendingTime"` - ImageID string `json:"imageId"` - KernelID string `json:"kernelId"` - RamdiskID string `json:"ramdiskId"` - Architecture string `json:"architecture"` + DevpayProductCodes []string `json:"devpayProductCodes"` + MarketplaceProductCodes []string `json:"marketplaceProductCodes"` + AvailabilityZone string `json:"availabilityZone"` + PrivateIP string `json:"privateIp"` + Version string `json:"version"` + Region string `json:"region"` + InstanceID string `json:"instanceId"` + BillingProducts []string `json:"billingProducts"` + InstanceType string `json:"instanceType"` + AccountID string `json:"accountId"` + PendingTime time.Time `json:"pendingTime"` + ImageID string `json:"imageId"` + KernelID string `json:"kernelId"` + RamdiskID string `json:"ramdiskId"` + Architecture string `json:"architecture"` } diff --git a/github.com/aws/aws-sdk-go/aws/ec2metadata/service.go b/github.com/aws/aws-sdk-go/aws/ec2metadata/service.go index 7d1f66e4e8..b8b2940d74 100644 --- a/github.com/aws/aws-sdk-go/aws/ec2metadata/service.go +++ b/github.com/aws/aws-sdk-go/aws/ec2metadata/service.go @@ -13,6 +13,7 @@ import ( "io" "net/http" "os" + "strconv" "strings" "time" @@ -24,9 +25,25 @@ import ( "github.com/aws/aws-sdk-go/aws/request" ) -// ServiceName is the name of the service. -const ServiceName = "ec2metadata" -const disableServiceEnvVar = "AWS_EC2_METADATA_DISABLED" +const ( + // ServiceName is the name of the service. + ServiceName = "ec2metadata" + disableServiceEnvVar = "AWS_EC2_METADATA_DISABLED" + + // Headers for Token and TTL + ttlHeader = "x-aws-ec2-metadata-token-ttl-seconds" + tokenHeader = "x-aws-ec2-metadata-token" + + // Named Handler constants + fetchTokenHandlerName = "FetchTokenHandler" + unmarshalMetadataHandlerName = "unmarshalMetadataHandler" + unmarshalTokenHandlerName = "unmarshalTokenHandler" + enableTokenProviderHandlerName = "enableTokenProviderHandler" + + // TTL constants + defaultTTL = 21600 * time.Second + ttlExpirationWindow = 30 * time.Second +) // A EC2Metadata is an EC2 Metadata service Client. type EC2Metadata struct { @@ -63,8 +80,10 @@ func NewClient(cfg aws.Config, handlers request.Handlers, endpoint, signingRegio // use a shorter timeout than default because the metadata // service is local if it is running, and to fail faster // if not running on an ec2 instance. - Timeout: 5 * time.Second, + Timeout: 1 * time.Second, } + // max number of retries on the client operation + cfg.MaxRetries = aws.Int(2) } svc := &EC2Metadata{ @@ -80,18 +99,35 @@ func NewClient(cfg aws.Config, handlers request.Handlers, endpoint, signingRegio ), } - svc.Handlers.Unmarshal.PushBack(unmarshalHandler) + // token provider instance + tp := newTokenProvider(svc, defaultTTL) + + // NamedHandler for fetching token + svc.Handlers.Sign.PushBackNamed(request.NamedHandler{ + Name: fetchTokenHandlerName, + Fn: tp.fetchTokenHandler, + }) + // NamedHandler for enabling token provider + svc.Handlers.Complete.PushBackNamed(request.NamedHandler{ + Name: enableTokenProviderHandlerName, + Fn: tp.enableTokenProviderHandler, + }) + + svc.Handlers.Unmarshal.PushBackNamed(unmarshalHandler) svc.Handlers.UnmarshalError.PushBack(unmarshalError) svc.Handlers.Validate.Clear() svc.Handlers.Validate.PushBack(validateEndpointHandler) // Disable the EC2 Metadata service if the environment variable is set. - // This shortcirctes the service's functionality to always fail to send + // This short-circuits the service's functionality to always fail to send // requests. if strings.ToLower(os.Getenv(disableServiceEnvVar)) == "true" { svc.Handlers.Send.SwapNamed(request.NamedHandler{ Name: corehandlers.SendHandler.Name, Fn: func(r *request.Request) { + r.HTTPResponse = &http.Response{ + Header: http.Header{}, + } r.Error = awserr.New( request.CanceledErrorCode, "EC2 IMDS access disabled via "+disableServiceEnvVar+" env var", @@ -104,7 +140,6 @@ func NewClient(cfg aws.Config, handlers request.Handlers, endpoint, signingRegio for _, option := range opts { option(svc.Client) } - return svc } @@ -116,30 +151,74 @@ type metadataOutput struct { Content string } -func unmarshalHandler(r *request.Request) { - defer r.HTTPResponse.Body.Close() - b := &bytes.Buffer{} - if _, err := io.Copy(b, r.HTTPResponse.Body); err != nil { - r.Error = awserr.New("SerializationError", "unable to unmarshal EC2 metadata respose", err) - return - } +type tokenOutput struct { + Token string + TTL time.Duration +} - if data, ok := r.Data.(*metadataOutput); ok { - data.Content = b.String() - } +// unmarshal token handler is used to parse the response of a getToken operation +var unmarshalTokenHandler = request.NamedHandler{ + Name: unmarshalTokenHandlerName, + Fn: func(r *request.Request) { + defer r.HTTPResponse.Body.Close() + var b bytes.Buffer + if _, err := io.Copy(&b, r.HTTPResponse.Body); err != nil { + r.Error = awserr.NewRequestFailure(awserr.New(request.ErrCodeSerialization, + "unable to unmarshal EC2 metadata response", err), r.HTTPResponse.StatusCode, r.RequestID) + return + } + + v := r.HTTPResponse.Header.Get(ttlHeader) + data, ok := r.Data.(*tokenOutput) + if !ok { + return + } + + data.Token = b.String() + // TTL is in seconds + i, err := strconv.ParseInt(v, 10, 64) + if err != nil { + r.Error = awserr.NewRequestFailure(awserr.New(request.ParamFormatErrCode, + "unable to parse EC2 token TTL response", err), r.HTTPResponse.StatusCode, r.RequestID) + return + } + t := time.Duration(i) * time.Second + data.TTL = t + }, +} + +var unmarshalHandler = request.NamedHandler{ + Name: unmarshalMetadataHandlerName, + Fn: func(r *request.Request) { + defer r.HTTPResponse.Body.Close() + var b bytes.Buffer + if _, err := io.Copy(&b, r.HTTPResponse.Body); err != nil { + r.Error = awserr.NewRequestFailure(awserr.New(request.ErrCodeSerialization, + "unable to unmarshal EC2 metadata response", err), r.HTTPResponse.StatusCode, r.RequestID) + return + } + + if data, ok := r.Data.(*metadataOutput); ok { + data.Content = b.String() + } + }, } func unmarshalError(r *request.Request) { defer r.HTTPResponse.Body.Close() - b := &bytes.Buffer{} - if _, err := io.Copy(b, r.HTTPResponse.Body); err != nil { - r.Error = awserr.New("SerializationError", "unable to unmarshal EC2 metadata error respose", err) + var b bytes.Buffer + + if _, err := io.Copy(&b, r.HTTPResponse.Body); err != nil { + r.Error = awserr.NewRequestFailure( + awserr.New(request.ErrCodeSerialization, "unable to unmarshal EC2 metadata error response", err), + r.HTTPResponse.StatusCode, r.RequestID) return } // Response body format is not consistent between metadata endpoints. // Grab the error message as a string and include that as the source error - r.Error = awserr.New("EC2MetadataError", "failed to make EC2Metadata request", errors.New(b.String())) + r.Error = awserr.NewRequestFailure(awserr.New("EC2MetadataError", "failed to make EC2Metadata request", errors.New(b.String())), + r.HTTPResponse.StatusCode, r.RequestID) } func validateEndpointHandler(r *request.Request) { diff --git a/github.com/aws/aws-sdk-go/aws/ec2metadata/token_provider.go b/github.com/aws/aws-sdk-go/aws/ec2metadata/token_provider.go new file mode 100644 index 0000000000..d0a3a020d8 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/ec2metadata/token_provider.go @@ -0,0 +1,92 @@ +package ec2metadata + +import ( + "net/http" + "sync/atomic" + "time" + + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/aws/credentials" + "github.com/aws/aws-sdk-go/aws/request" +) + +// A tokenProvider struct provides access to EC2Metadata client +// and atomic instance of a token, along with configuredTTL for it. +// tokenProvider also provides an atomic flag to disable the +// fetch token operation. +// The disabled member will use 0 as false, and 1 as true. +type tokenProvider struct { + client *EC2Metadata + token atomic.Value + configuredTTL time.Duration + disabled uint32 +} + +// A ec2Token struct helps use of token in EC2 Metadata service ops +type ec2Token struct { + token string + credentials.Expiry +} + +// newTokenProvider provides a pointer to a tokenProvider instance +func newTokenProvider(c *EC2Metadata, duration time.Duration) *tokenProvider { + return &tokenProvider{client: c, configuredTTL: duration} +} + +// fetchTokenHandler fetches token for EC2Metadata service client by default. +func (t *tokenProvider) fetchTokenHandler(r *request.Request) { + + // short-circuits to insecure data flow if tokenProvider is disabled. + if v := atomic.LoadUint32(&t.disabled); v == 1 { + return + } + + if ec2Token, ok := t.token.Load().(ec2Token); ok && !ec2Token.IsExpired() { + r.HTTPRequest.Header.Set(tokenHeader, ec2Token.token) + return + } + + output, err := t.client.getToken(r.Context(), t.configuredTTL) + + if err != nil { + + // change the disabled flag on token provider to true, + // when error is request timeout error. + if requestFailureError, ok := err.(awserr.RequestFailure); ok { + switch requestFailureError.StatusCode() { + case http.StatusForbidden, http.StatusNotFound, http.StatusMethodNotAllowed: + atomic.StoreUint32(&t.disabled, 1) + case http.StatusBadRequest: + r.Error = requestFailureError + } + + // Check if request timed out while waiting for response + if e, ok := requestFailureError.OrigErr().(awserr.Error); ok { + if e.Code() == request.ErrCodeRequestError { + atomic.StoreUint32(&t.disabled, 1) + } + } + } + return + } + + newToken := ec2Token{ + token: output.Token, + } + newToken.SetExpiration(time.Now().Add(output.TTL), ttlExpirationWindow) + t.token.Store(newToken) + + // Inject token header to the request. + if ec2Token, ok := t.token.Load().(ec2Token); ok { + r.HTTPRequest.Header.Set(tokenHeader, ec2Token.token) + } +} + +// enableTokenProviderHandler enables the token provider +func (t *tokenProvider) enableTokenProviderHandler(r *request.Request) { + // If the error code status is 401, we enable the token provider + if e, ok := r.Error.(awserr.RequestFailure); ok && e != nil && + e.StatusCode() == http.StatusUnauthorized { + atomic.StoreUint32(&t.disabled, 0) + } +} diff --git a/github.com/aws/aws-sdk-go/aws/endpoints/decode.go b/github.com/aws/aws-sdk-go/aws/endpoints/decode.go index 87b9ff3ffe..654fb1ad52 100644 --- a/github.com/aws/aws-sdk-go/aws/endpoints/decode.go +++ b/github.com/aws/aws-sdk-go/aws/endpoints/decode.go @@ -83,6 +83,7 @@ func decodeV3Endpoints(modelDef modelDefinition, opts DecodeModelOptions) (Resol p := &ps[i] custAddEC2Metadata(p) custAddS3DualStack(p) + custRegionalS3(p) custRmIotDataService(p) custFixAppAutoscalingChina(p) custFixAppAutoscalingUsGov(p) @@ -92,7 +93,7 @@ func decodeV3Endpoints(modelDef modelDefinition, opts DecodeModelOptions) (Resol } func custAddS3DualStack(p *partition) { - if p.ID != "aws" { + if !(p.ID == "aws" || p.ID == "aws-cn" || p.ID == "aws-us-gov") { return } @@ -100,6 +101,33 @@ func custAddS3DualStack(p *partition) { custAddDualstack(p, "s3-control") } +func custRegionalS3(p *partition) { + if p.ID != "aws" { + return + } + + service, ok := p.Services["s3"] + if !ok { + return + } + + // If global endpoint already exists no customization needed. + if _, ok := service.Endpoints["aws-global"]; ok { + return + } + + service.PartitionEndpoint = "aws-global" + service.Endpoints["us-east-1"] = endpoint{} + service.Endpoints["aws-global"] = endpoint{ + Hostname: "s3.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + } + + p.Services["s3"] = service +} + func custAddDualstack(p *partition, svcName string) { s, ok := p.Services[svcName] if !ok { diff --git a/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go b/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go index 636f614f16..bc5fb73f91 100644 --- a/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go +++ b/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go @@ -11,21 +11,27 @@ const ( AwsPartitionID = "aws" // AWS Standard partition. AwsCnPartitionID = "aws-cn" // AWS China partition. AwsUsGovPartitionID = "aws-us-gov" // AWS GovCloud (US) partition. + AwsIsoPartitionID = "aws-iso" // AWS ISO (US) partition. + AwsIsoBPartitionID = "aws-iso-b" // AWS ISOB (US) partition. ) // AWS Standard partition's regions. const ( + AfSouth1RegionID = "af-south-1" // Africa (Cape Town). + ApEast1RegionID = "ap-east-1" // Asia Pacific (Hong Kong). ApNortheast1RegionID = "ap-northeast-1" // Asia Pacific (Tokyo). ApNortheast2RegionID = "ap-northeast-2" // Asia Pacific (Seoul). ApSouth1RegionID = "ap-south-1" // Asia Pacific (Mumbai). ApSoutheast1RegionID = "ap-southeast-1" // Asia Pacific (Singapore). ApSoutheast2RegionID = "ap-southeast-2" // Asia Pacific (Sydney). CaCentral1RegionID = "ca-central-1" // Canada (Central). - EuCentral1RegionID = "eu-central-1" // EU (Frankfurt). - EuNorth1RegionID = "eu-north-1" // EU (Stockholm). - EuWest1RegionID = "eu-west-1" // EU (Ireland). - EuWest2RegionID = "eu-west-2" // EU (London). - EuWest3RegionID = "eu-west-3" // EU (Paris). + EuCentral1RegionID = "eu-central-1" // Europe (Frankfurt). + EuNorth1RegionID = "eu-north-1" // Europe (Stockholm). + EuSouth1RegionID = "eu-south-1" // Europe (Milan). + EuWest1RegionID = "eu-west-1" // Europe (Ireland). + EuWest2RegionID = "eu-west-2" // Europe (London). + EuWest3RegionID = "eu-west-3" // Europe (Paris). + MeSouth1RegionID = "me-south-1" // Middle East (Bahrain). SaEast1RegionID = "sa-east-1" // South America (Sao Paulo). UsEast1RegionID = "us-east-1" // US East (N. Virginia). UsEast2RegionID = "us-east-2" // US East (Ohio). @@ -42,11 +48,21 @@ const ( // AWS GovCloud (US) partition's regions. const ( UsGovEast1RegionID = "us-gov-east-1" // AWS GovCloud (US-East). - UsGovWest1RegionID = "us-gov-west-1" // AWS GovCloud (US). + UsGovWest1RegionID = "us-gov-west-1" // AWS GovCloud (US-West). +) + +// AWS ISO (US) partition's regions. +const ( + UsIsoEast1RegionID = "us-iso-east-1" // US ISO East. +) + +// AWS ISOB (US) partition's regions. +const ( + UsIsobEast1RegionID = "us-isob-east-1" // US ISOB East (Ohio). ) // DefaultResolver returns an Endpoint resolver that will be able -// to resolve endpoints for: AWS Standard, AWS China, and AWS GovCloud (US). +// to resolve endpoints for: AWS Standard, AWS China, AWS GovCloud (US), AWS ISO (US), and AWS ISOB (US). // // Use DefaultPartitions() to get the list of the default partitions. func DefaultResolver() Resolver { @@ -54,7 +70,7 @@ func DefaultResolver() Resolver { } // DefaultPartitions returns a list of the partitions the SDK is bundled -// with. The available partitions are: AWS Standard, AWS China, and AWS GovCloud (US). +// with. The available partitions are: AWS Standard, AWS China, AWS GovCloud (US), AWS ISO (US), and AWS ISOB (US). // // partitions := endpoints.DefaultPartitions // for _, p := range partitions { @@ -68,6 +84,8 @@ var defaultPartitions = partitions{ awsPartition, awscnPartition, awsusgovPartition, + awsisoPartition, + awsisobPartition, } // AwsPartition returns the Resolver for AWS Standard. @@ -81,7 +99,7 @@ var awsPartition = partition{ DNSSuffix: "amazonaws.com", RegionRegex: regionRegex{ Regexp: func() *regexp.Regexp { - reg, _ := regexp.Compile("^(us|eu|ap|sa|ca)\\-\\w+\\-\\d+$") + reg, _ := regexp.Compile("^(us|eu|ap|sa|ca|me|af)\\-\\w+\\-\\d+$") return reg }(), }, @@ -91,6 +109,12 @@ var awsPartition = partition{ SignatureVersions: []string{"v4"}, }, Regions: regions{ + "af-south-1": region{ + Description: "Africa (Cape Town)", + }, + "ap-east-1": region{ + Description: "Asia Pacific (Hong Kong)", + }, "ap-northeast-1": region{ Description: "Asia Pacific (Tokyo)", }, @@ -110,19 +134,25 @@ var awsPartition = partition{ Description: "Canada (Central)", }, "eu-central-1": region{ - Description: "EU (Frankfurt)", + Description: "Europe (Frankfurt)", }, "eu-north-1": region{ - Description: "EU (Stockholm)", + Description: "Europe (Stockholm)", + }, + "eu-south-1": region{ + Description: "Europe (Milan)", }, "eu-west-1": region{ - Description: "EU (Ireland)", + Description: "Europe (Ireland)", }, "eu-west-2": region{ - Description: "EU (London)", + Description: "Europe (London)", }, "eu-west-3": region{ - Description: "EU (Paris)", + Description: "Europe (Paris)", + }, + "me-south-1": region{ + Description: "Middle East (Bahrain)", }, "sa-east-1": region{ Description: "South America (Sao Paulo)", @@ -147,9 +177,11 @@ var awsPartition = partition{ "us-east-1": endpoint{}, }, }, - "acm": service{ + "access-analyzer": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -158,9 +190,11 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, @@ -168,163 +202,67 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "acm-pca": service{ - Defaults: endpoint{ - Protocols: []string{"https"}, - }, - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "api.mediatailor": service{ - - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - }, - }, - "api.pricing": service{ - Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "pricing", - }, - }, - Endpoints: endpoints{ - "ap-south-1": endpoint{}, - "us-east-1": endpoint{}, - }, - }, - "api.sagemaker": service{ - - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "apigateway": service{ + "acm": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "application-autoscaling": service{ - Defaults: endpoint{ - Hostname: "autoscaling.{region}.amazonaws.com", - Protocols: []string{"http", "https"}, - CredentialScope: credentialScope{ - Service: "application-autoscaling", + "ca-central-1-fips": endpoint{ + Hostname: "acm-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, }, - }, - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "appstream2": service{ - Defaults: endpoint{ - Protocols: []string{"https"}, - CredentialScope: credentialScope{ - Service: "appstream", + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-1-fips": endpoint{ + Hostname: "acm-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "acm-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "acm-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "acm-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, }, - }, - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "appsync": service{ - - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "athena": service{ - - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, }, }, - "autoscaling": service{ + "acm-pca": service{ Defaults: endpoint{ - Protocols: []string{"http", "https"}, + Protocols: []string{"https"}, }, Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -336,39 +274,48 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "fips-ca-central-1": endpoint{ + Hostname: "acm-pca-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "acm-pca-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "acm-pca-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "acm-pca-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "acm-pca-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "autoscaling-plans": service{ + "api.detective": service{ Defaults: endpoint{ - Hostname: "autoscaling.{region}.amazonaws.com", - Protocols: []string{"http", "https"}, - CredentialScope: credentialScope{ - Service: "autoscaling-plans", - }, - }, - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + Protocols: []string{"https"}, }, - }, - "batch": service{ - Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, @@ -377,6 +324,7 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, @@ -387,76 +335,253 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "budgets": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, + "api.ecr": service{ Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "budgets.amazonaws.com", + "af-south-1": endpoint{ + Hostname: "api.ecr.af-south-1.amazonaws.com", CredentialScope: credentialScope{ - Region: "us-east-1", + Region: "af-south-1", }, }, - }, - }, - "ce": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, - - Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "ce.us-east-1.amazonaws.com", + "ap-east-1": endpoint{ + Hostname: "api.ecr.ap-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-east-1", + }, + }, + "ap-northeast-1": endpoint{ + Hostname: "api.ecr.ap-northeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-1", + }, + }, + "ap-northeast-2": endpoint{ + Hostname: "api.ecr.ap-northeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-2", + }, + }, + "ap-south-1": endpoint{ + Hostname: "api.ecr.ap-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-south-1", + }, + }, + "ap-southeast-1": endpoint{ + Hostname: "api.ecr.ap-southeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-1", + }, + }, + "ap-southeast-2": endpoint{ + Hostname: "api.ecr.ap-southeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-2", + }, + }, + "ca-central-1": endpoint{ + Hostname: "api.ecr.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "eu-central-1": endpoint{ + Hostname: "api.ecr.eu-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-central-1", + }, + }, + "eu-north-1": endpoint{ + Hostname: "api.ecr.eu-north-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-north-1", + }, + }, + "eu-south-1": endpoint{ + Hostname: "api.ecr.eu-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-south-1", + }, + }, + "eu-west-1": endpoint{ + Hostname: "api.ecr.eu-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-1", + }, + }, + "eu-west-2": endpoint{ + Hostname: "api.ecr.eu-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-2", + }, + }, + "eu-west-3": endpoint{ + Hostname: "api.ecr.eu-west-3.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-3", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "ecr-fips.us-east-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, - }, - }, - "chime": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, - Defaults: endpoint{ - SSLCommonName: "service.chime.aws.amazon.com", - Protocols: []string{"https"}, - }, - Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "service.chime.aws.amazon.com", - Protocols: []string{"https"}, + "fips-us-east-2": endpoint{ + Hostname: "ecr-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "ecr-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "ecr-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{ + Hostname: "api.ecr.me-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "me-south-1", + }, + }, + "sa-east-1": endpoint{ + Hostname: "api.ecr.sa-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "sa-east-1", + }, + }, + "us-east-1": endpoint{ + Hostname: "api.ecr.us-east-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, + "us-east-2": endpoint{ + Hostname: "api.ecr.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{ + Hostname: "api.ecr.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{ + Hostname: "api.ecr.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, }, }, - "cloud9": service{ + "api.elastic-inference": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{ + Hostname: "api.elastic-inference.ap-northeast-1.amazonaws.com", + }, + "ap-northeast-2": endpoint{ + Hostname: "api.elastic-inference.ap-northeast-2.amazonaws.com", + }, + "eu-west-1": endpoint{ + Hostname: "api.elastic-inference.eu-west-1.amazonaws.com", + }, + "us-east-1": endpoint{ + Hostname: "api.elastic-inference.us-east-1.amazonaws.com", + }, + "us-east-2": endpoint{ + Hostname: "api.elastic-inference.us-east-2.amazonaws.com", + }, + "us-west-2": endpoint{ + Hostname: "api.elastic-inference.us-west-2.amazonaws.com", + }, + }, + }, + "api.mediatailor": service{ Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, "eu-west-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, "us-west-2": endpoint{}, }, }, - "clouddirectory": service{ + "api.pricing": service{ + Defaults: endpoint{ + CredentialScope: credentialScope{ + Service: "pricing", + }, + }, + Endpoints: endpoints{ + "ap-south-1": endpoint{}, + "us-east-1": endpoint{}, + }, + }, + "api.sagemaker": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, + "us-east-1-fips": endpoint{ + Hostname: "api-fips.sagemaker.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "api-fips.sagemaker.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "api-fips.sagemaker.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "api-fips.sagemaker.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, }, }, - "cloudformation": service{ + "apigateway": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -465,9 +590,11 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, @@ -475,42 +602,37 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "cloudfront": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, - - Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "cloudfront.amazonaws.com", - Protocols: []string{"http", "https"}, - CredentialScope: credentialScope{ - Region: "us-east-1", - }, - }, + "application-autoscaling": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, }, - }, - "cloudhsm": service{ - Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "cloudhsmv2": service{ - Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "cloudhsm", - }, - }, + "appmesh": service{ + Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -522,30 +644,64 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "cloudsearch": service{ + "appstream2": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + CredentialScope: credentialScope{ + Service: "appstream", + }, + }, + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "fips": endpoint{ + Hostname: "appstream2-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "us-east-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "appsync": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, + "us-east-2": endpoint{}, "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "cloudtrail": service{ + "athena": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -557,6 +713,7 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, @@ -564,9 +721,13 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "codebuild": service{ - + "autoscaling": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -574,43 +735,161 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-1-fips": endpoint{ - Hostname: "codebuild-fips.us-east-1.amazonaws.com", + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "autoscaling-plans": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "backup": service{ + + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "batch": service{ + + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "fips.batch.us-east-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, - "us-east-2": endpoint{}, - "us-east-2-fips": endpoint{ - Hostname: "codebuild-fips.us-east-2.amazonaws.com", + "fips-us-east-2": endpoint{ + Hostname: "fips.batch.us-east-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-2", }, }, - "us-west-1": endpoint{}, - "us-west-1-fips": endpoint{ - Hostname: "codebuild-fips.us-west-1.amazonaws.com", + "fips-us-west-1": endpoint{ + Hostname: "fips.batch.us-west-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-1", }, }, - "us-west-2": endpoint{}, - "us-west-2-fips": endpoint{ - Hostname: "codebuild-fips.us-west-2.amazonaws.com", + "fips-us-west-2": endpoint{ + Hostname: "fips.batch.us-west-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-2", }, }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "codecommit": service{ + "budgets": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "budgets.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + }, + }, + "ce": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "ce.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + }, + }, + "chime": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + Defaults: endpoint{ + SSLCommonName: "service.chime.aws.amazon.com", + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "service.chime.aws.amazon.com", + Protocols: []string{"https"}, + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + }, + }, + "cloud9": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -618,25 +897,37 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "fips": endpoint{ - Hostname: "codecommit-fips.ca-central-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "ca-central-1", - }, - }, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "codedeploy": service{ + "clouddirectory": service{ + + Endpoints: endpoints{ + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "cloudformation": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -645,97 +936,118 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-1-fips": endpoint{ - Hostname: "codedeploy-fips.us-east-1.amazonaws.com", + Hostname: "cloudformation-fips.us-east-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, "us-east-2": endpoint{}, "us-east-2-fips": endpoint{ - Hostname: "codedeploy-fips.us-east-2.amazonaws.com", + Hostname: "cloudformation-fips.us-east-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-2", }, }, "us-west-1": endpoint{}, "us-west-1-fips": endpoint{ - Hostname: "codedeploy-fips.us-west-1.amazonaws.com", + Hostname: "cloudformation-fips.us-west-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-1", }, }, "us-west-2": endpoint{}, "us-west-2-fips": endpoint{ - Hostname: "codedeploy-fips.us-west-2.amazonaws.com", + Hostname: "cloudformation-fips.us-west-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-2", }, }, }, }, - "codepipeline": service{ + "cloudfront": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "cloudfront.amazonaws.com", + Protocols: []string{"http", "https"}, + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + }, + }, + "cloudhsm": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "codestar": service{ - + "cloudhsmv2": service{ + Defaults: endpoint{ + CredentialScope: credentialScope{ + Service: "cloudhsm", + }, + }, Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "cognito-identity": service{ + "cloudsearch": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, + "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, + "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "cognito-idp": service{ + "cloudtrail": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -743,45 +1055,62 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "cloudtrail-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "cloudtrail-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "cloudtrail-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "cloudtrail-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "cognito-sync": service{ + "codeartifact": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "comprehend": service{ - Defaults: endpoint{ - Protocols: []string{"https"}, - }, - Endpoints: endpoints{ - "ap-southeast-2": endpoint{}, - "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, "us-west-2": endpoint{}, }, }, - "config": service{ + "codebuild": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -793,68 +1122,42 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "cur": service{ - - Endpoints: endpoints{ - "us-east-1": endpoint{}, - }, - }, - "datapipeline": service{ - - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "datasync": service{ - - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "dax": service{ - - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "eu-west-1": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "devicefarm": service{ - - Endpoints: endpoints{ + "us-east-1-fips": endpoint{ + Hostname: "codebuild-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "codebuild-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "codebuild-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "codebuild-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, }, }, - "directconnect": service{ + "codecommit": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -866,22 +1169,25 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "discovery": service{ - - Endpoints: endpoints{ - "us-west-2": endpoint{}, + "fips": endpoint{ + Hostname: "codecommit-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "dms": service{ + "codedeploy": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -889,68 +1195,45 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "docdb": service{ - - Endpoints: endpoints{ - "eu-west-1": endpoint{ - Hostname: "rds.eu-west-1.amazonaws.com", + "us-east-1-fips": endpoint{ + Hostname: "codedeploy-fips.us-east-1.amazonaws.com", CredentialScope: credentialScope{ - Region: "eu-west-1", + Region: "us-east-1", }, }, - "us-east-1": endpoint{ - Hostname: "rds.us-east-1.amazonaws.com", + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "codedeploy-fips.us-east-2.amazonaws.com", CredentialScope: credentialScope{ - Region: "us-east-1", + Region: "us-east-2", }, }, - "us-east-2": endpoint{ - Hostname: "rds.us-east-2.amazonaws.com", + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "codedeploy-fips.us-west-1.amazonaws.com", CredentialScope: credentialScope{ - Region: "us-east-2", + Region: "us-west-1", }, }, - "us-west-2": endpoint{ - Hostname: "rds.us-west-2.amazonaws.com", + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "codedeploy-fips.us-west-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-2", }, }, }, }, - "ds": service{ + "codepipeline": service{ - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "dynamodb": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - }, Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, @@ -963,13 +1246,36 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "local": endpoint{ - Hostname: "localhost:8000", - Protocols: []string{"http"}, + "fips-ca-central-1": endpoint{ + Hostname: "codepipeline-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "codepipeline-fips.us-east-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, + "fips-us-east-2": endpoint{ + Hostname: "codepipeline-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "codepipeline-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "codepipeline-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, @@ -977,14 +1283,11 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "ec2": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - }, + "codestar": service{ + Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, @@ -992,26 +1295,13 @@ var awsPartition = partition{ "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "ec2metadata": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, - - Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "169.254.169.254/latest", - Protocols: []string{"http"}, - }, - }, - }, - "ecr": service{ + "codestar-connections": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, @@ -1032,7 +1322,7 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "ecs": service{ + "cognito-identity": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, @@ -1042,18 +1332,32 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "cognito-identity-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "cognito-identity-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "cognito-identity-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, }, }, - "elasticache": service{ + "cognito-idp": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, @@ -1063,24 +1367,32 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "fips": endpoint{ - Hostname: "elasticache-fips.us-west-1.amazonaws.com", + "fips-us-east-1": endpoint{ + Hostname: "cognito-idp-fips.us-east-1.amazonaws.com", CredentialScope: credentialScope{ - Region: "us-west-1", + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "cognito-idp-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "cognito-idp-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", }, }, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "elasticbeanstalk": service{ + "cognito-sync": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, @@ -1088,35 +1400,15 @@ var awsPartition = partition{ "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "elasticfilesystem": service{ - - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "elasticloadbalancing": service{ + "comprehend": service{ Defaults: endpoint{ Protocols: []string{"https"}, }, @@ -1128,79 +1420,65 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "elasticmapreduce": service{ - Defaults: endpoint{ - SSLCommonName: "{region}.{service}.{dnsSuffix}", - Protocols: []string{"https"}, - }, - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, - "eu-central-1": endpoint{ - SSLCommonName: "{service}.{region}.{dnsSuffix}", + "fips-us-east-1": endpoint{ + Hostname: "comprehend-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, }, - "eu-north-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{ - SSLCommonName: "{service}.{region}.{dnsSuffix}", + "fips-us-east-2": endpoint{ + Hostname: "comprehend-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "comprehend-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, }, + "us-east-1": endpoint{}, "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "elastictranscoder": service{ + "comprehendmedical": service{ Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "email": service{ - - Endpoints: endpoints{ - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "entitlement.marketplace": service{ - Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "aws-marketplace", + "eu-west-2": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "comprehendmedical-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "comprehendmedical-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "comprehendmedical-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, }, - }, - Endpoints: endpoints{ "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, }, }, - "es": service{ + "config": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1212,6 +1490,7 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, @@ -1219,73 +1498,69 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "events": service{ + "connect": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, - "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "firehose": service{ + "cur": service{ + + Endpoints: endpoints{ + "us-east-1": endpoint{}, + }, + }, + "data.mediastore": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "fms": service{ - Defaults: endpoint{ - Protocols: []string{"https"}, - }, + "dataexchange": service{ + Endpoints: endpoints{ "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "eu-central-1": endpoint{}, "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, + "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "fsx": service{ + "datapipeline": service{ Endpoints: endpoints{ - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "gamelift": service{ + "datasync": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1293,28 +1568,57 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-ca-central-1": endpoint{ + Hostname: "datasync-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "datasync-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "datasync-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "datasync-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "datasync-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "glacier": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - }, + "dax": service{ + Endpoints: endpoints{ "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, @@ -1325,9 +1629,17 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "glue": service{ + "devicefarm": service{ + + Endpoints: endpoints{ + "us-west-2": endpoint{}, + }, + }, + "directconnect": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1335,134 +1647,291 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "directconnect-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "directconnect-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "directconnect-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "directconnect-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "greengrass": service{ - IsRegionalized: boxedTrue, - Defaults: endpoint{ - Protocols: []string{"https"}, - }, + "discovery": service{ + Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, "us-east-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "guardduty": service{ - IsRegionalized: boxedTrue, - Defaults: endpoint{ - Protocols: []string{"https"}, - }, + "dms": service{ + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "health": service{ - - Endpoints: endpoints{ - "us-east-1": endpoint{}, - }, - }, - "iam": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, - - Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "iam.amazonaws.com", + "dms-fips": endpoint{ + Hostname: "dms-fips.us-west-1.amazonaws.com", CredentialScope: credentialScope{ - Region: "us-east-1", + Region: "us-west-1", }, }, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "importexport": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, + "docdb": service{ Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "importexport.amazonaws.com", - SignatureVersions: []string{"v2", "v4"}, + "ap-northeast-1": endpoint{ + Hostname: "rds.ap-northeast-1.amazonaws.com", CredentialScope: credentialScope{ - Region: "us-east-1", - Service: "IngestionService", + Region: "ap-northeast-1", + }, + }, + "ap-northeast-2": endpoint{ + Hostname: "rds.ap-northeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-2", + }, + }, + "ap-south-1": endpoint{ + Hostname: "rds.ap-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-south-1", + }, + }, + "ap-southeast-1": endpoint{ + Hostname: "rds.ap-southeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-1", + }, + }, + "ap-southeast-2": endpoint{ + Hostname: "rds.ap-southeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-2", + }, + }, + "ca-central-1": endpoint{ + Hostname: "rds.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "eu-central-1": endpoint{ + Hostname: "rds.eu-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-central-1", + }, + }, + "eu-west-1": endpoint{ + Hostname: "rds.eu-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-1", + }, + }, + "eu-west-2": endpoint{ + Hostname: "rds.eu-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-2", + }, + }, + "eu-west-3": endpoint{ + Hostname: "rds.eu-west-3.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-3", + }, + }, + "us-east-1": endpoint{ + Hostname: "rds.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{ + Hostname: "rds.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-2": endpoint{ + Hostname: "rds.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", }, }, }, }, - "inspector": service{ + "ds": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-ca-central-1": endpoint{ + Hostname: "ds-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "ds-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "ds-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "ds-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "ds-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "iot": service{ + "dynamodb": service{ Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "execute-api", - }, + Protocols: []string{"http", "https"}, }, Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, + "ca-central-1": endpoint{}, + "ca-central-1-fips": endpoint{ + Hostname: "dynamodb-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "local": endpoint{ + Hostname: "localhost:8000", + Protocols: []string{"http"}, + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-1-fips": endpoint{ + Hostname: "dynamodb-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "dynamodb-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "dynamodb-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "dynamodb-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, }, }, - "iotanalytics": service{ - - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, + "ec2": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, }, - }, - "kinesis": service{ - Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1471,39 +1940,64 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "kinesisanalytics": service{ - - Endpoints: endpoints{ - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, + "fips-ca-central-1": endpoint{ + Hostname: "ec2-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "ec2-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "ec2-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "ec2-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "ec2-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "kinesisvideo": service{ + "ec2metadata": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-west-2": endpoint{}, + "aws-global": endpoint{ + Hostname: "169.254.169.254/latest", + Protocols: []string{"http"}, + }, }, }, - "kms": service{ + "ecs": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1512,19 +2006,48 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "ecs-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "ecs-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "ecs-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "ecs-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "lambda": service{ - + "eks": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1536,16 +2059,36 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "fips.eks.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "fips.eks.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "fips.eks.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, }, }, - "lightsail": service{ + "elasticache": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1553,17 +2096,30 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-2": endpoint{}, + "fips": endpoint{ + Hostname: "elasticache-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "logs": service{ + "elasticbeanstalk": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1572,32 +2128,47 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "machinelearning": service{ - - Endpoints: endpoints{ - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - }, - }, - "marketplacecommerceanalytics": service{ - - Endpoints: endpoints{ - "us-east-1": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "elasticbeanstalk-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "elasticbeanstalk-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "elasticbeanstalk-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "elasticbeanstalk-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "mediaconvert": service{ + "elasticfilesystem": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1605,113 +2176,291 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-af-south-1": endpoint{ + Hostname: "elasticfilesystem-fips.af-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "af-south-1", + }, + }, + "fips-ap-east-1": endpoint{ + Hostname: "elasticfilesystem-fips.ap-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-east-1", + }, + }, + "fips-ap-northeast-1": endpoint{ + Hostname: "elasticfilesystem-fips.ap-northeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-1", + }, + }, + "fips-ap-northeast-2": endpoint{ + Hostname: "elasticfilesystem-fips.ap-northeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-2", + }, + }, + "fips-ap-south-1": endpoint{ + Hostname: "elasticfilesystem-fips.ap-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-south-1", + }, + }, + "fips-ap-southeast-1": endpoint{ + Hostname: "elasticfilesystem-fips.ap-southeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-1", + }, + }, + "fips-ap-southeast-2": endpoint{ + Hostname: "elasticfilesystem-fips.ap-southeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-2", + }, + }, + "fips-ca-central-1": endpoint{ + Hostname: "elasticfilesystem-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-eu-central-1": endpoint{ + Hostname: "elasticfilesystem-fips.eu-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-central-1", + }, + }, + "fips-eu-north-1": endpoint{ + Hostname: "elasticfilesystem-fips.eu-north-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-north-1", + }, + }, + "fips-eu-south-1": endpoint{ + Hostname: "elasticfilesystem-fips.eu-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-south-1", + }, + }, + "fips-eu-west-1": endpoint{ + Hostname: "elasticfilesystem-fips.eu-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-1", + }, + }, + "fips-eu-west-2": endpoint{ + Hostname: "elasticfilesystem-fips.eu-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-2", + }, + }, + "fips-eu-west-3": endpoint{ + Hostname: "elasticfilesystem-fips.eu-west-3.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-3", + }, + }, + "fips-me-south-1": endpoint{ + Hostname: "elasticfilesystem-fips.me-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "me-south-1", + }, + }, + "fips-sa-east-1": endpoint{ + Hostname: "elasticfilesystem-fips.sa-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "sa-east-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "elasticfilesystem-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "elasticfilesystem-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "elasticfilesystem-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "elasticfilesystem-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "medialive": service{ - + "elasticloadbalancing": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-west-2": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "elasticloadbalancing-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "elasticloadbalancing-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "elasticloadbalancing-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "elasticloadbalancing-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "mediapackage": service{ - + "elasticmapreduce": service{ + Defaults: endpoint{ + SSLCommonName: "{region}.{service}.{dnsSuffix}", + Protocols: []string{"https"}, + }, Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "mediastore": service{ - - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, - "ap-southeast-2": endpoint{}, - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "metering.marketplace": service{ - Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "aws-marketplace", + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{ + SSLCommonName: "{service}.{region}.{dnsSuffix}", + }, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-ca-central-1": endpoint{ + Hostname: "elasticmapreduce-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "elasticmapreduce-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "elasticmapreduce-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, }, + "fips-us-west-1": endpoint{ + Hostname: "elasticmapreduce-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "elasticmapreduce-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{ + SSLCommonName: "{service}.{region}.{dnsSuffix}", + }, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, + }, + "elastictranscoder": service{ + Endpoints: endpoints{ "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "mgh": service{ - - Endpoints: endpoints{ - "us-west-2": endpoint{}, - }, - }, - "mobileanalytics": service{ + "email": service{ Endpoints: endpoints{ - "us-east-1": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "models.lex": service{ + "entitlement.marketplace": service{ Defaults: endpoint{ CredentialScope: credentialScope{ - Service: "lex", + Service: "aws-marketplace", }, }, Endpoints: endpoints{ - "eu-west-1": endpoint{}, "us-east-1": endpoint{}, - "us-west-2": endpoint{}, }, }, - "monitoring": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - }, + "es": service{ + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1720,95 +2469,236 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "fips": endpoint{ + Hostname: "es-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "mq": service{ + "events": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "events-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "events-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "events-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "events-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "mturk-requester": service{ - IsRegionalized: boxedFalse, + "firehose": service{ Endpoints: endpoints{ - "sandbox": endpoint{ - Hostname: "mturk-requester-sandbox.us-east-1.amazonaws.com", + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "firehose-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, }, - "us-east-1": endpoint{}, + "fips-us-east-2": endpoint{ + Hostname: "firehose-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "firehose-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "firehose-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "neptune": service{ - + "fms": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, Endpoints: endpoints{ - "ap-southeast-1": endpoint{ - Hostname: "rds.ap-southeast-1.amazonaws.com", + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-ap-northeast-1": endpoint{ + Hostname: "fms-fips.ap-northeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-1", + }, + }, + "fips-ap-northeast-2": endpoint{ + Hostname: "fms-fips.ap-northeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-2", + }, + }, + "fips-ap-south-1": endpoint{ + Hostname: "fms-fips.ap-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-south-1", + }, + }, + "fips-ap-southeast-1": endpoint{ + Hostname: "fms-fips.ap-southeast-1.amazonaws.com", CredentialScope: credentialScope{ Region: "ap-southeast-1", }, }, - "ap-southeast-2": endpoint{ - Hostname: "rds.ap-southeast-2.amazonaws.com", + "fips-ap-southeast-2": endpoint{ + Hostname: "fms-fips.ap-southeast-2.amazonaws.com", CredentialScope: credentialScope{ Region: "ap-southeast-2", }, }, - "eu-central-1": endpoint{ - Hostname: "rds.eu-central-1.amazonaws.com", + "fips-ca-central-1": endpoint{ + Hostname: "fms-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-eu-central-1": endpoint{ + Hostname: "fms-fips.eu-central-1.amazonaws.com", CredentialScope: credentialScope{ Region: "eu-central-1", }, }, - "eu-west-1": endpoint{ - Hostname: "rds.eu-west-1.amazonaws.com", + "fips-eu-west-1": endpoint{ + Hostname: "fms-fips.eu-west-1.amazonaws.com", CredentialScope: credentialScope{ Region: "eu-west-1", }, }, - "eu-west-2": endpoint{ - Hostname: "rds.eu-west-2.amazonaws.com", + "fips-eu-west-2": endpoint{ + Hostname: "fms-fips.eu-west-2.amazonaws.com", CredentialScope: credentialScope{ Region: "eu-west-2", }, }, - "us-east-1": endpoint{ - Hostname: "rds.us-east-1.amazonaws.com", + "fips-eu-west-3": endpoint{ + Hostname: "fms-fips.eu-west-3.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-3", + }, + }, + "fips-sa-east-1": endpoint{ + Hostname: "fms-fips.sa-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "sa-east-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "fms-fips.us-east-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, - "us-east-2": endpoint{ - Hostname: "rds.us-east-2.amazonaws.com", + "fips-us-east-2": endpoint{ + Hostname: "fms-fips.us-east-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-2", }, }, - "us-west-2": endpoint{ - Hostname: "rds.us-west-2.amazonaws.com", + "fips-us-west-1": endpoint{ + Hostname: "fms-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "fms-fips.us-west-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-2", }, }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "opsworks": service{ + "forecast": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, @@ -1816,59 +2706,49 @@ var awsPartition = partition{ "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "opsworks-cm": service{ + "forecastquery": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "eu-central-1": endpoint{}, "eu-west-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "organizations": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, + "fsx": service{ Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "organizations.us-east-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-east-1", - }, - }, - }, - }, - "pinpoint": service{ - Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "mobiletargeting", - }, - }, - Endpoints: endpoints{ - "eu-central-1": endpoint{}, - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-west-2": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "polly": service{ + "gamelift": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, @@ -1880,7 +2760,6 @@ var awsPartition = partition{ "eu-central-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, @@ -1888,9 +2767,13 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "rds": service{ - + "glacier": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1899,21 +2782,52 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{ - SSLCommonName: "{service}.{dnsSuffix}", + "fips-ca-central-1": endpoint{ + Hostname: "glacier-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, }, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "glacier-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "glacier-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "glacier-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "glacier-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "redshift": service{ + "glue": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1925,27 +2839,75 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "glue-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "glue-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "glue-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "glue-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "greengrass": service{ + IsRegionalized: boxedTrue, + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "rekognition": service{ + "groundstation": service{ Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, "ap-southeast-2": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, + "me-south-1": endpoint{}, "us-east-2": endpoint{}, "us-west-2": endpoint{}, }, }, - "resource-groups": service{ - + "guardduty": service{ + IsRegionalized: boxedTrue, + Defaults: endpoint{ + Protocols: []string{"https"}, + }, Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -1957,316 +2919,286 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "us-east-1-fips": endpoint{ + Hostname: "guardduty-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "guardduty-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "guardduty-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "guardduty-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, }, }, - "robomaker": service{ + "health": service{ Endpoints: endpoints{ - "eu-west-1": endpoint{}, "us-east-1": endpoint{}, + }, + }, + "honeycode": service{ + + Endpoints: endpoints{ "us-west-2": endpoint{}, }, }, - "route53": service{ + "iam": service{ PartitionEndpoint: "aws-global", IsRegionalized: boxedFalse, Endpoints: endpoints{ "aws-global": endpoint{ - Hostname: "route53.amazonaws.com", + Hostname: "iam.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "iam-fips": endpoint{ + Hostname: "iam-fips.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, }, }, - "route53domains": service{ + "importexport": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "importexport.amazonaws.com", + SignatureVersions: []string{"v2", "v4"}, + CredentialScope: credentialScope{ + Region: "us-east-1", + Service: "IngestionService", + }, + }, + }, + }, + "inspector": service{ Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "inspector-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "inspector-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "inspector-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "inspector-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "route53resolver": service{ + "iot": service{ Defaults: endpoint{ - Protocols: []string{"http", "https"}, + CredentialScope: credentialScope{ + Service: "execute-api", + }, }, Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, + "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "runtime.lex": service{ - Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "lex", - }, - }, + "iotanalytics": service{ + Endpoints: endpoints{ - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-west-2": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, }, }, - "runtime.sagemaker": service{ + "iotevents": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "s3": service{ - PartitionEndpoint: "us-east-1", - IsRegionalized: boxedTrue, - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - SignatureVersions: []string{"s3v4"}, + "ioteventsdata": service{ - HasDualStack: boxedTrue, - DualStackHostname: "{service}.dualstack.{region}.{dnsSuffix}", - }, Endpoints: endpoints{ "ap-northeast-1": endpoint{ - Hostname: "s3.ap-northeast-1.amazonaws.com", - SignatureVersions: []string{"s3", "s3v4"}, - }, - "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{ - Hostname: "s3.ap-southeast-1.amazonaws.com", - SignatureVersions: []string{"s3", "s3v4"}, - }, - "ap-southeast-2": endpoint{ - Hostname: "s3.ap-southeast-2.amazonaws.com", - SignatureVersions: []string{"s3", "s3v4"}, - }, - "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, - "eu-west-1": endpoint{ - Hostname: "s3.eu-west-1.amazonaws.com", - SignatureVersions: []string{"s3", "s3v4"}, - }, - "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "s3-external-1": endpoint{ - Hostname: "s3-external-1.amazonaws.com", - SignatureVersions: []string{"s3", "s3v4"}, - CredentialScope: credentialScope{ - Region: "us-east-1", - }, - }, - "sa-east-1": endpoint{ - Hostname: "s3.sa-east-1.amazonaws.com", - SignatureVersions: []string{"s3", "s3v4"}, - }, - "us-east-1": endpoint{ - Hostname: "s3.amazonaws.com", - SignatureVersions: []string{"s3", "s3v4"}, - }, - "us-east-2": endpoint{}, - "us-west-1": endpoint{ - Hostname: "s3.us-west-1.amazonaws.com", - SignatureVersions: []string{"s3", "s3v4"}, - }, - "us-west-2": endpoint{ - Hostname: "s3.us-west-2.amazonaws.com", - SignatureVersions: []string{"s3", "s3v4"}, - }, - }, - }, - "s3-control": service{ - Defaults: endpoint{ - Protocols: []string{"https"}, - SignatureVersions: []string{"s3v4"}, - - HasDualStack: boxedTrue, - DualStackHostname: "{service}.dualstack.{region}.{dnsSuffix}", - }, - Endpoints: endpoints{ - "ap-northeast-1": endpoint{ - Hostname: "s3-control.ap-northeast-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, + Hostname: "data.iotevents.ap-northeast-1.amazonaws.com", CredentialScope: credentialScope{ Region: "ap-northeast-1", }, }, "ap-northeast-2": endpoint{ - Hostname: "s3-control.ap-northeast-2.amazonaws.com", - SignatureVersions: []string{"s3v4"}, + Hostname: "data.iotevents.ap-northeast-2.amazonaws.com", CredentialScope: credentialScope{ Region: "ap-northeast-2", }, }, - "ap-south-1": endpoint{ - Hostname: "s3-control.ap-south-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "ap-south-1", - }, - }, "ap-southeast-1": endpoint{ - Hostname: "s3-control.ap-southeast-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, + Hostname: "data.iotevents.ap-southeast-1.amazonaws.com", CredentialScope: credentialScope{ Region: "ap-southeast-1", }, }, "ap-southeast-2": endpoint{ - Hostname: "s3-control.ap-southeast-2.amazonaws.com", - SignatureVersions: []string{"s3v4"}, + Hostname: "data.iotevents.ap-southeast-2.amazonaws.com", CredentialScope: credentialScope{ Region: "ap-southeast-2", }, }, - "ca-central-1": endpoint{ - Hostname: "s3-control.ca-central-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "ca-central-1", - }, - }, "eu-central-1": endpoint{ - Hostname: "s3-control.eu-central-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, + Hostname: "data.iotevents.eu-central-1.amazonaws.com", CredentialScope: credentialScope{ Region: "eu-central-1", }, }, - "eu-north-1": endpoint{ - Hostname: "s3-control.eu-north-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "eu-north-1", - }, - }, "eu-west-1": endpoint{ - Hostname: "s3-control.eu-west-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, + Hostname: "data.iotevents.eu-west-1.amazonaws.com", CredentialScope: credentialScope{ Region: "eu-west-1", }, }, "eu-west-2": endpoint{ - Hostname: "s3-control.eu-west-2.amazonaws.com", - SignatureVersions: []string{"s3v4"}, + Hostname: "data.iotevents.eu-west-2.amazonaws.com", CredentialScope: credentialScope{ Region: "eu-west-2", }, }, - "eu-west-3": endpoint{ - Hostname: "s3-control.eu-west-3.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "eu-west-3", - }, - }, - "sa-east-1": endpoint{ - Hostname: "s3-control.sa-east-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "sa-east-1", - }, - }, "us-east-1": endpoint{ - Hostname: "s3-control.us-east-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "us-east-1", - }, - }, - "us-east-1-fips": endpoint{ - Hostname: "s3-control-fips.us-east-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, + Hostname: "data.iotevents.us-east-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, "us-east-2": endpoint{ - Hostname: "s3-control.us-east-2.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "us-east-2", - }, - }, - "us-east-2-fips": endpoint{ - Hostname: "s3-control-fips.us-east-2.amazonaws.com", - SignatureVersions: []string{"s3v4"}, + Hostname: "data.iotevents.us-east-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-2", }, }, - "us-west-1": endpoint{ - Hostname: "s3-control.us-west-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "us-west-1", - }, - }, - "us-west-1-fips": endpoint{ - Hostname: "s3-control-fips.us-west-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "us-west-1", - }, - }, "us-west-2": endpoint{ - Hostname: "s3-control.us-west-2.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "us-west-2", - }, - }, - "us-west-2-fips": endpoint{ - Hostname: "s3-control-fips.us-west-2.amazonaws.com", - SignatureVersions: []string{"s3v4"}, + Hostname: "data.iotevents.us-west-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-2", }, }, }, }, - "sdb": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - SignatureVersions: []string{"v2"}, - }, + "iotsecuredtunneling": service{ + Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, - "us-east-1": endpoint{ - Hostname: "sdb.amazonaws.com", + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "iotthingsgraph": service{ + Defaults: endpoint{ + CredentialScope: credentialScope{ + Service: "iotthingsgraph", }, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + }, + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "secretsmanager": service{ + "kafka": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -2274,92 +3206,93 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-1-fips": endpoint{ - Hostname: "secretsmanager-fips.us-east-1.amazonaws.com", + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "kinesis": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "kinesis-fips.us-east-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, - "us-east-2": endpoint{}, - "us-east-2-fips": endpoint{ - Hostname: "secretsmanager-fips.us-east-2.amazonaws.com", + "fips-us-east-2": endpoint{ + Hostname: "kinesis-fips.us-east-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-2", }, }, - "us-west-1": endpoint{}, - "us-west-1-fips": endpoint{ - Hostname: "secretsmanager-fips.us-west-1.amazonaws.com", + "fips-us-west-1": endpoint{ + Hostname: "kinesis-fips.us-west-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-1", }, }, - "us-west-2": endpoint{}, - "us-west-2-fips": endpoint{ - Hostname: "secretsmanager-fips.us-west-2.amazonaws.com", + "fips-us-west-2": endpoint{ + Hostname: "kinesis-fips.us-west-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-2", }, }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "serverlessrepo": service{ - Defaults: endpoint{ - Protocols: []string{"https"}, + "kinesisanalytics": service{ + + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, }, + }, + "kinesisvideo": service{ + Endpoints: endpoints{ - "ap-northeast-1": endpoint{ - Protocols: []string{"https"}, - }, - "ap-northeast-2": endpoint{ - Protocols: []string{"https"}, - }, - "ap-south-1": endpoint{ - Protocols: []string{"https"}, - }, - "ap-southeast-1": endpoint{ - Protocols: []string{"https"}, - }, - "ap-southeast-2": endpoint{ - Protocols: []string{"https"}, - }, - "ca-central-1": endpoint{ - Protocols: []string{"https"}, - }, - "eu-central-1": endpoint{ - Protocols: []string{"https"}, - }, - "eu-west-1": endpoint{ - Protocols: []string{"https"}, - }, - "eu-west-2": endpoint{ - Protocols: []string{"https"}, - }, - "sa-east-1": endpoint{ - Protocols: []string{"https"}, - }, - "us-east-1": endpoint{ - Protocols: []string{"https"}, - }, - "us-east-2": endpoint{ - Protocols: []string{"https"}, - }, - "us-west-1": endpoint{ - Protocols: []string{"https"}, - }, - "us-west-2": endpoint{ - Protocols: []string{"https"}, - }, - }, - }, - "servicecatalog": service{ - - Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -2372,38 +3305,15 @@ var awsPartition = partition{ "eu-west-3": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-1-fips": endpoint{ - Hostname: "servicecatalog-fips.us-east-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-east-1", - }, - }, - "us-east-2": endpoint{}, - "us-east-2-fips": endpoint{ - Hostname: "servicecatalog-fips.us-east-2.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-east-2", - }, - }, - "us-west-1": endpoint{}, - "us-west-1-fips": endpoint{ - Hostname: "servicecatalog-fips.us-west-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-west-1", - }, - }, - "us-west-2": endpoint{}, - "us-west-2-fips": endpoint{ - Hostname: "servicecatalog-fips.us-west-2.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-west-2", - }, - }, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, }, }, - "servicediscovery": service{ + "kms": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -2411,9 +3321,12 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, @@ -2421,17 +3334,7 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "shield": service{ - IsRegionalized: boxedFalse, - Defaults: endpoint{ - SSLCommonName: "shield.us-east-1.amazonaws.com", - Protocols: []string{"https"}, - }, - Endpoints: endpoints{ - "us-east-1": endpoint{}, - }, - }, - "sms": service{ + "lakeformation": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, @@ -2441,6 +3344,7 @@ var awsPartition = partition{ "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, @@ -2451,30 +3355,60 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "snowball": service{ + "lambda": service{ Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "lambda-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "lambda-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "lambda-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "lambda-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "sns": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - }, + "license-manager": service{ + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -2483,22 +3417,65 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "license-manager-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "license-manager-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "license-manager-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "license-manager-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "lightsail": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "sqs": service{ - Defaults: endpoint{ - SSLCommonName: "{region}.queue.{dnsSuffix}", - Protocols: []string{"http", "https"}, - }, + "logs": service{ + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -2507,45 +3484,72 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, "fips-us-east-1": endpoint{ - Hostname: "sqs-fips.us-east-1.amazonaws.com", + Hostname: "logs-fips.us-east-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, "fips-us-east-2": endpoint{ - Hostname: "sqs-fips.us-east-2.amazonaws.com", + Hostname: "logs-fips.us-east-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-2", }, }, "fips-us-west-1": endpoint{ - Hostname: "sqs-fips.us-west-1.amazonaws.com", + Hostname: "logs-fips.us-west-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-1", }, }, "fips-us-west-2": endpoint{ - Hostname: "sqs-fips.us-west-2.amazonaws.com", + Hostname: "logs-fips.us-west-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-2", }, }, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{ - SSLCommonName: "queue.{dnsSuffix}", + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "machinelearning": service{ + + Endpoints: endpoints{ + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + }, + }, + "macie": service{ + + Endpoints: endpoints{ + "fips-us-east-1": endpoint{ + Hostname: "macie-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, }, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, + "fips-us-west-2": endpoint{ + Hostname: "macie-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "us-east-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "ssm": service{ + "macie2": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -2557,43 +3561,62 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "macie2-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "macie2-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "macie2-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "macie2-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "states": service{ + "managedblockchain": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, - "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, }, }, - "storagegateway": service{ + "marketplacecommerceanalytics": service{ + + Endpoints: endpoints{ + "us-east-1": endpoint{}, + }, + }, + "mediaconnect": service{ Endpoints: endpoints{ + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, @@ -2606,13 +3629,8 @@ var awsPartition = partition{ "us-west-2": endpoint{}, }, }, - "streams.dynamodb": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - CredentialScope: credentialScope{ - Service: "dynamodb", - }, - }, + "mediaconvert": service{ + Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, @@ -2625,84 +3643,44 @@ var awsPartition = partition{ "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "local": endpoint{ - Hostname: "localhost:8000", - Protocols: []string{"http"}, + "fips-ca-central-1": endpoint{ + Hostname: "mediaconvert-fips.ca-central-1.amazonaws.com", CredentialScope: credentialScope{ - Region: "us-east-1", + Region: "ca-central-1", }, }, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, - }, - }, - "sts": service{ - PartitionEndpoint: "aws-global", - Defaults: endpoint{ - Hostname: "sts.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-east-1", - }, - }, - Endpoints: endpoints{ - "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{ - Hostname: "sts.ap-northeast-2.amazonaws.com", - CredentialScope: credentialScope{ - Region: "ap-northeast-2", - }, - }, - "ap-south-1": endpoint{}, - "ap-southeast-1": endpoint{}, - "ap-southeast-2": endpoint{}, - "aws-global": endpoint{}, - "ca-central-1": endpoint{}, - "eu-central-1": endpoint{}, - "eu-north-1": endpoint{}, - "eu-west-1": endpoint{}, - "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-1-fips": endpoint{ - Hostname: "sts-fips.us-east-1.amazonaws.com", + "fips-us-east-1": endpoint{ + Hostname: "mediaconvert-fips.us-east-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-1", }, }, - "us-east-2": endpoint{}, - "us-east-2-fips": endpoint{ - Hostname: "sts-fips.us-east-2.amazonaws.com", + "fips-us-east-2": endpoint{ + Hostname: "mediaconvert-fips.us-east-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-east-2", }, }, - "us-west-1": endpoint{}, - "us-west-1-fips": endpoint{ - Hostname: "sts-fips.us-west-1.amazonaws.com", + "fips-us-west-1": endpoint{ + Hostname: "mediaconvert-fips.us-west-1.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-1", }, }, - "us-west-2": endpoint{}, - "us-west-2-fips": endpoint{ - Hostname: "sts-fips.us-west-2.amazonaws.com", + "fips-us-west-2": endpoint{ + Hostname: "mediaconvert-fips.us-west-2.amazonaws.com", CredentialScope: credentialScope{ Region: "us-west-2", }, }, - }, - }, - "support": service{ - - Endpoints: endpoints{ + "sa-east-1": endpoint{}, "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - "swf": service{ + "medialive": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, @@ -2710,7 +3688,6 @@ var awsPartition = partition{ "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, @@ -2719,11 +3696,10 @@ var awsPartition = partition{ "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "tagging": service{ + "mediapackage": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, @@ -2731,7 +3707,6 @@ var awsPartition = partition{ "ap-south-1": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, @@ -2739,124 +3714,93 @@ var awsPartition = partition{ "eu-west-3": endpoint{}, "sa-east-1": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "transfer": service{ + "mediastore": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, - "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "eu-west-3": endpoint{}, "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "translate": service{ + "metering.marketplace": service{ Defaults: endpoint{ - Protocols: []string{"https"}, - }, - Endpoints: endpoints{ - "eu-west-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-1-fips": endpoint{ - Hostname: "translate-fips.us-east-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-east-1", - }, - }, - "us-east-2": endpoint{}, - "us-east-2-fips": endpoint{ - Hostname: "translate-fips.us-east-2.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-east-2", - }, - }, - "us-west-2": endpoint{}, - "us-west-2-fips": endpoint{ - Hostname: "translate-fips.us-west-2.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-west-2", - }, - }, - }, - }, - "waf": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, - - Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "waf.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-east-1", - }, + CredentialScope: credentialScope{ + Service: "aws-marketplace", }, }, - }, - "waf-regional": service{ - Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-east-2": endpoint{}, "us-west-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "workdocs": service{ + "mgh": service{ Endpoints: endpoints{ "ap-northeast-1": endpoint{}, - "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "eu-west-1": endpoint{}, + "eu-central-1": endpoint{}, "us-east-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "workmail": service{ - Defaults: endpoint{ - Protocols: []string{"https"}, - }, + "mobileanalytics": service{ + Endpoints: endpoints{ - "eu-west-1": endpoint{}, "us-east-1": endpoint{}, - "us-west-2": endpoint{}, }, }, - "workspaces": service{ - + "models.lex": service{ + Defaults: endpoint{ + CredentialScope: credentialScope{ + Service: "lex", + }, + }, Endpoints: endpoints{ "ap-northeast-1": endpoint{}, - "ap-northeast-2": endpoint{}, "ap-southeast-1": endpoint{}, "ap-southeast-2": endpoint{}, - "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, - "sa-east-1": endpoint{}, "us-east-1": endpoint{}, "us-west-2": endpoint{}, }, }, - "xray": service{ - + "monitoring": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, "ap-northeast-1": endpoint{}, "ap-northeast-2": endpoint{}, "ap-south-1": endpoint{}, @@ -2865,970 +3809,5480 @@ var awsPartition = partition{ "ca-central-1": endpoint{}, "eu-central-1": endpoint{}, "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, "eu-west-1": endpoint{}, "eu-west-2": endpoint{}, "eu-west-3": endpoint{}, - "sa-east-1": endpoint{}, - "us-east-1": endpoint{}, - "us-east-2": endpoint{}, - "us-west-1": endpoint{}, - "us-west-2": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "monitoring-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "monitoring-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "monitoring-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "monitoring-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, }, }, - }, -} - -// AwsCnPartition returns the Resolver for AWS China. -func AwsCnPartition() Partition { - return awscnPartition.Partition() -} + "mq": service{ -var awscnPartition = partition{ - ID: "aws-cn", - Name: "AWS China", - DNSSuffix: "amazonaws.com.cn", - RegionRegex: regionRegex{ - Regexp: func() *regexp.Regexp { - reg, _ := regexp.Compile("^cn\\-\\w+\\-\\d+$") - return reg - }(), - }, - Defaults: endpoint{ - Hostname: "{service}.{region}.{dnsSuffix}", - Protocols: []string{"https"}, - SignatureVersions: []string{"v4"}, - }, - Regions: regions{ - "cn-north-1": region{ - Description: "China (Beijing)", - }, - "cn-northwest-1": region{ - Description: "China (Ningxia)", + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "mq-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "mq-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "mq-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "mq-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, }, - }, - Services: services{ - "apigateway": service{ + "mturk-requester": service{ + IsRegionalized: boxedFalse, Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "sandbox": endpoint{ + Hostname: "mturk-requester-sandbox.us-east-1.amazonaws.com", + }, + "us-east-1": endpoint{}, }, }, - "application-autoscaling": service{ - Defaults: endpoint{ - Hostname: "autoscaling.{region}.amazonaws.com.cn", - Protocols: []string{"http", "https"}, + "neptune": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{ + Hostname: "rds.ap-northeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-1", + }, + }, + "ap-northeast-2": endpoint{ + Hostname: "rds.ap-northeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-2", + }, + }, + "ap-south-1": endpoint{ + Hostname: "rds.ap-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-south-1", + }, + }, + "ap-southeast-1": endpoint{ + Hostname: "rds.ap-southeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-1", + }, + }, + "ap-southeast-2": endpoint{ + Hostname: "rds.ap-southeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-2", + }, + }, + "ca-central-1": endpoint{ + Hostname: "rds.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "eu-central-1": endpoint{ + Hostname: "rds.eu-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-central-1", + }, + }, + "eu-north-1": endpoint{ + Hostname: "rds.eu-north-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-north-1", + }, + }, + "eu-west-1": endpoint{ + Hostname: "rds.eu-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-1", + }, + }, + "eu-west-2": endpoint{ + Hostname: "rds.eu-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-2", + }, + }, + "eu-west-3": endpoint{ + Hostname: "rds.eu-west-3.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-3", + }, + }, + "me-south-1": endpoint{ + Hostname: "rds.me-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "me-south-1", + }, + }, + "us-east-1": endpoint{ + Hostname: "rds.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{ + Hostname: "rds.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{ + Hostname: "rds.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{ + Hostname: "rds.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "oidc": service{ + + Endpoints: endpoints{ + "ap-southeast-1": endpoint{ + Hostname: "oidc.ap-southeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-1", + }, + }, + "ap-southeast-2": endpoint{ + Hostname: "oidc.ap-southeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-2", + }, + }, + "ca-central-1": endpoint{ + Hostname: "oidc.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "eu-central-1": endpoint{ + Hostname: "oidc.eu-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-central-1", + }, + }, + "eu-west-1": endpoint{ + Hostname: "oidc.eu-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-1", + }, + }, + "eu-west-2": endpoint{ + Hostname: "oidc.eu-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-2", + }, + }, + "us-east-1": endpoint{ + Hostname: "oidc.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{ + Hostname: "oidc.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-2": endpoint{ + Hostname: "oidc.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "opsworks": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "opsworks-cm": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "organizations": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "organizations.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-aws-global": endpoint{ + Hostname: "organizations-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + }, + }, + "outposts": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-ca-central-1": endpoint{ + Hostname: "outposts-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "outposts-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "outposts-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "outposts-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "outposts-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "pinpoint": service{ + Defaults: endpoint{ + CredentialScope: credentialScope{ + Service: "mobiletargeting", + }, + }, + Endpoints: endpoints{ + "ap-south-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "pinpoint-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "pinpoint-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "us-east-1": endpoint{ + Hostname: "pinpoint.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-west-2": endpoint{ + Hostname: "pinpoint.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "polly": service{ + + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "polly-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "polly-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "polly-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "polly-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "portal.sso": service{ + + Endpoints: endpoints{ + "ap-southeast-1": endpoint{ + Hostname: "portal.sso.ap-southeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-1", + }, + }, + "ap-southeast-2": endpoint{ + Hostname: "portal.sso.ap-southeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-2", + }, + }, + "ca-central-1": endpoint{ + Hostname: "portal.sso.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "eu-central-1": endpoint{ + Hostname: "portal.sso.eu-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-central-1", + }, + }, + "eu-west-1": endpoint{ + Hostname: "portal.sso.eu-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-1", + }, + }, + "eu-west-2": endpoint{ + Hostname: "portal.sso.eu-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-2", + }, + }, + "us-east-1": endpoint{ + Hostname: "portal.sso.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{ + Hostname: "portal.sso.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-2": endpoint{ + Hostname: "portal.sso.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "projects.iot1click": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "qldb": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "ram": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "rds": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "rds-fips.ca-central-1": endpoint{ + Hostname: "rds-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "rds-fips.us-east-1": endpoint{ + Hostname: "rds-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "rds-fips.us-east-2": endpoint{ + Hostname: "rds-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "rds-fips.us-west-1": endpoint{ + Hostname: "rds-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "rds-fips.us-west-2": endpoint{ + Hostname: "rds-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{ + SSLCommonName: "{service}.{dnsSuffix}", + }, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "redshift": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-ca-central-1": endpoint{ + Hostname: "redshift-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "redshift-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "redshift-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "redshift-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "redshift-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "rekognition": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "rekognition-fips.us-east-1": endpoint{ + Hostname: "rekognition-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "rekognition-fips.us-east-2": endpoint{ + Hostname: "rekognition-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "rekognition-fips.us-west-1": endpoint{ + Hostname: "rekognition-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "rekognition-fips.us-west-2": endpoint{ + Hostname: "rekognition-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "resource-groups": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "resource-groups-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "resource-groups-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "resource-groups-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "resource-groups-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "robomaker": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "route53": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "route53.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + }, + }, + "route53domains": service{ + + Endpoints: endpoints{ + "us-east-1": endpoint{}, + }, + }, + "route53resolver": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "runtime.lex": service{ + Defaults: endpoint{ + CredentialScope: credentialScope{ + Service: "lex", + }, + }, + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "us-east-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "runtime.sagemaker": service{ + + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-1-fips": endpoint{ + Hostname: "runtime-fips.sagemaker.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "runtime-fips.sagemaker.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "runtime-fips.sagemaker.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "runtime-fips.sagemaker.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "s3": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedTrue, + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + SignatureVersions: []string{"s3v4"}, + + HasDualStack: boxedTrue, + DualStackHostname: "{service}.dualstack.{region}.{dnsSuffix}", + }, + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{ + Hostname: "s3.ap-northeast-1.amazonaws.com", + SignatureVersions: []string{"s3", "s3v4"}, + }, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{ + Hostname: "s3.ap-southeast-1.amazonaws.com", + SignatureVersions: []string{"s3", "s3v4"}, + }, + "ap-southeast-2": endpoint{ + Hostname: "s3.ap-southeast-2.amazonaws.com", + SignatureVersions: []string{"s3", "s3v4"}, + }, + "aws-global": endpoint{ + Hostname: "s3.amazonaws.com", + SignatureVersions: []string{"s3", "s3v4"}, + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{ + Hostname: "s3.eu-west-1.amazonaws.com", + SignatureVersions: []string{"s3", "s3v4"}, + }, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "s3-external-1": endpoint{ + Hostname: "s3-external-1.amazonaws.com", + SignatureVersions: []string{"s3", "s3v4"}, + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "sa-east-1": endpoint{ + Hostname: "s3.sa-east-1.amazonaws.com", + SignatureVersions: []string{"s3", "s3v4"}, + }, + "us-east-1": endpoint{ + Hostname: "s3.us-east-1.amazonaws.com", + SignatureVersions: []string{"s3", "s3v4"}, + }, + "us-east-2": endpoint{}, + "us-west-1": endpoint{ + Hostname: "s3.us-west-1.amazonaws.com", + SignatureVersions: []string{"s3", "s3v4"}, + }, + "us-west-2": endpoint{ + Hostname: "s3.us-west-2.amazonaws.com", + SignatureVersions: []string{"s3", "s3v4"}, + }, + }, + }, + "s3-control": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + SignatureVersions: []string{"s3v4"}, + + HasDualStack: boxedTrue, + DualStackHostname: "{service}.dualstack.{region}.{dnsSuffix}", + }, + Endpoints: endpoints{ + "ap-northeast-1": endpoint{ + Hostname: "s3-control.ap-northeast-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "ap-northeast-1", + }, + }, + "ap-northeast-2": endpoint{ + Hostname: "s3-control.ap-northeast-2.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "ap-northeast-2", + }, + }, + "ap-south-1": endpoint{ + Hostname: "s3-control.ap-south-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "ap-south-1", + }, + }, + "ap-southeast-1": endpoint{ + Hostname: "s3-control.ap-southeast-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "ap-southeast-1", + }, + }, + "ap-southeast-2": endpoint{ + Hostname: "s3-control.ap-southeast-2.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "ap-southeast-2", + }, + }, + "ca-central-1": endpoint{ + Hostname: "s3-control.ca-central-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "eu-central-1": endpoint{ + Hostname: "s3-control.eu-central-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "eu-central-1", + }, + }, + "eu-north-1": endpoint{ + Hostname: "s3-control.eu-north-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "eu-north-1", + }, + }, + "eu-west-1": endpoint{ + Hostname: "s3-control.eu-west-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "eu-west-1", + }, + }, + "eu-west-2": endpoint{ + Hostname: "s3-control.eu-west-2.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "eu-west-2", + }, + }, + "eu-west-3": endpoint{ + Hostname: "s3-control.eu-west-3.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "eu-west-3", + }, + }, + "sa-east-1": endpoint{ + Hostname: "s3-control.sa-east-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "sa-east-1", + }, + }, + "us-east-1": endpoint{ + Hostname: "s3-control.us-east-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-1-fips": endpoint{ + Hostname: "s3-control-fips.us-east-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{ + Hostname: "s3-control.us-east-2.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-east-2-fips": endpoint{ + Hostname: "s3-control-fips.us-east-2.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{ + Hostname: "s3-control.us-west-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-1-fips": endpoint{ + Hostname: "s3-control-fips.us-west-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{ + Hostname: "s3-control.us-west-2.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "us-west-2-fips": endpoint{ + Hostname: "s3-control-fips.us-west-2.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "savingsplans": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "savingsplans.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + }, + }, + "schemas": service{ + + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "sdb": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + SignatureVersions: []string{"v2"}, + }, + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-west-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{ + Hostname: "sdb.amazonaws.com", + }, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "secretsmanager": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-1-fips": endpoint{ + Hostname: "secretsmanager-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "secretsmanager-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "secretsmanager-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "secretsmanager-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "securityhub": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "securityhub-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "securityhub-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "securityhub-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "securityhub-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "serverlessrepo": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "ap-east-1": endpoint{ + Protocols: []string{"https"}, + }, + "ap-northeast-1": endpoint{ + Protocols: []string{"https"}, + }, + "ap-northeast-2": endpoint{ + Protocols: []string{"https"}, + }, + "ap-south-1": endpoint{ + Protocols: []string{"https"}, + }, + "ap-southeast-1": endpoint{ + Protocols: []string{"https"}, + }, + "ap-southeast-2": endpoint{ + Protocols: []string{"https"}, + }, + "ca-central-1": endpoint{ + Protocols: []string{"https"}, + }, + "eu-central-1": endpoint{ + Protocols: []string{"https"}, + }, + "eu-north-1": endpoint{ + Protocols: []string{"https"}, + }, + "eu-west-1": endpoint{ + Protocols: []string{"https"}, + }, + "eu-west-2": endpoint{ + Protocols: []string{"https"}, + }, + "eu-west-3": endpoint{ + Protocols: []string{"https"}, + }, + "me-south-1": endpoint{ + Protocols: []string{"https"}, + }, + "sa-east-1": endpoint{ + Protocols: []string{"https"}, + }, + "us-east-1": endpoint{ + Protocols: []string{"https"}, + }, + "us-east-2": endpoint{ + Protocols: []string{"https"}, + }, + "us-west-1": endpoint{ + Protocols: []string{"https"}, + }, + "us-west-2": endpoint{ + Protocols: []string{"https"}, + }, + }, + }, + "servicecatalog": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-1-fips": endpoint{ + Hostname: "servicecatalog-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "servicecatalog-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "servicecatalog-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "servicecatalog-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "servicediscovery": service{ + + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "session.qldb": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "shield": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + Defaults: endpoint{ + SSLCommonName: "shield.us-east-1.amazonaws.com", + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "shield.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-aws-global": endpoint{ + Hostname: "shield-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + }, + }, + "sms": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "sms-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "sms-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "sms-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "sms-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "snowball": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-ap-northeast-1": endpoint{ + Hostname: "snowball-fips.ap-northeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-1", + }, + }, + "fips-ap-northeast-2": endpoint{ + Hostname: "snowball-fips.ap-northeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-2", + }, + }, + "fips-ap-south-1": endpoint{ + Hostname: "snowball-fips.ap-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-south-1", + }, + }, + "fips-ap-southeast-1": endpoint{ + Hostname: "snowball-fips.ap-southeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-1", + }, + }, + "fips-ap-southeast-2": endpoint{ + Hostname: "snowball-fips.ap-southeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-2", + }, + }, + "fips-ca-central-1": endpoint{ + Hostname: "snowball-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-eu-central-1": endpoint{ + Hostname: "snowball-fips.eu-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-central-1", + }, + }, + "fips-eu-west-1": endpoint{ + Hostname: "snowball-fips.eu-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-1", + }, + }, + "fips-eu-west-2": endpoint{ + Hostname: "snowball-fips.eu-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-2", + }, + }, + "fips-eu-west-3": endpoint{ + Hostname: "snowball-fips.eu-west-3.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-3", + }, + }, + "fips-sa-east-1": endpoint{ + Hostname: "snowball-fips.sa-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "sa-east-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "snowball-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "snowball-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "snowball-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "snowball-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "sns": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "sns-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "sns-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "sns-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "sns-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "sqs": service{ + Defaults: endpoint{ + SSLCommonName: "{region}.queue.{dnsSuffix}", + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "sqs-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "sqs-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "sqs-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "sqs-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{ + SSLCommonName: "queue.{dnsSuffix}", + }, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "ssm": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "ssm-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "ssm-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "ssm-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "ssm-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "ssm-facade-fips-us-east-1": endpoint{ + Hostname: "ssm-facade-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "ssm-facade-fips-us-east-2": endpoint{ + Hostname: "ssm-facade-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "ssm-facade-fips-us-west-1": endpoint{ + Hostname: "ssm-facade-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "ssm-facade-fips-us-west-2": endpoint{ + Hostname: "ssm-facade-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "states": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "states-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "states-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "states-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "states-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "storagegateway": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "streams.dynamodb": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + CredentialScope: credentialScope{ + Service: "dynamodb", + }, + }, + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "ca-central-1-fips": endpoint{ + Hostname: "dynamodb-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "local": endpoint{ + Hostname: "localhost:8000", + Protocols: []string{"http"}, + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-1-fips": endpoint{ + Hostname: "dynamodb-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "dynamodb-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "dynamodb-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "dynamodb-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "sts": service{ + PartitionEndpoint: "aws-global", + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "aws-global": endpoint{ + Hostname: "sts.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-1-fips": endpoint{ + Hostname: "sts-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "sts-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-1-fips": endpoint{ + Hostname: "sts-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "sts-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "support": service{ + PartitionEndpoint: "aws-global", + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "support.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + }, + }, + "swf": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "swf-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "swf-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "swf-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "swf-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "tagging": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "transcribe": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "fips.transcribe.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "fips.transcribe.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "fips.transcribe.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "fips.transcribe.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "transcribestreaming": service{ + + Endpoints: endpoints{ + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "transfer": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "translate": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "us-east-1": endpoint{}, + "us-east-1-fips": endpoint{ + Hostname: "translate-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{}, + "us-east-2-fips": endpoint{ + Hostname: "translate-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + "us-west-2-fips": endpoint{ + Hostname: "translate-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "waf": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-fips": endpoint{ + Hostname: "waf-fips.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "aws-global": endpoint{ + Hostname: "waf.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + }, + }, + "waf-regional": service{ + + Endpoints: endpoints{ + "ap-east-1": endpoint{ + Hostname: "waf-regional.ap-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-east-1", + }, + }, + "ap-northeast-1": endpoint{ + Hostname: "waf-regional.ap-northeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-1", + }, + }, + "ap-northeast-2": endpoint{ + Hostname: "waf-regional.ap-northeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-2", + }, + }, + "ap-south-1": endpoint{ + Hostname: "waf-regional.ap-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-south-1", + }, + }, + "ap-southeast-1": endpoint{ + Hostname: "waf-regional.ap-southeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-1", + }, + }, + "ap-southeast-2": endpoint{ + Hostname: "waf-regional.ap-southeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-2", + }, + }, + "ca-central-1": endpoint{ + Hostname: "waf-regional.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "eu-central-1": endpoint{ + Hostname: "waf-regional.eu-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-central-1", + }, + }, + "eu-north-1": endpoint{ + Hostname: "waf-regional.eu-north-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-north-1", + }, + }, + "eu-west-1": endpoint{ + Hostname: "waf-regional.eu-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-1", + }, + }, + "eu-west-2": endpoint{ + Hostname: "waf-regional.eu-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-2", + }, + }, + "eu-west-3": endpoint{ + Hostname: "waf-regional.eu-west-3.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-3", + }, + }, + "fips-ap-east-1": endpoint{ + Hostname: "waf-regional-fips.ap-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-east-1", + }, + }, + "fips-ap-northeast-1": endpoint{ + Hostname: "waf-regional-fips.ap-northeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-1", + }, + }, + "fips-ap-northeast-2": endpoint{ + Hostname: "waf-regional-fips.ap-northeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-northeast-2", + }, + }, + "fips-ap-south-1": endpoint{ + Hostname: "waf-regional-fips.ap-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-south-1", + }, + }, + "fips-ap-southeast-1": endpoint{ + Hostname: "waf-regional-fips.ap-southeast-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-1", + }, + }, + "fips-ap-southeast-2": endpoint{ + Hostname: "waf-regional-fips.ap-southeast-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ap-southeast-2", + }, + }, + "fips-ca-central-1": endpoint{ + Hostname: "waf-regional-fips.ca-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "ca-central-1", + }, + }, + "fips-eu-central-1": endpoint{ + Hostname: "waf-regional-fips.eu-central-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-central-1", + }, + }, + "fips-eu-north-1": endpoint{ + Hostname: "waf-regional-fips.eu-north-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-north-1", + }, + }, + "fips-eu-west-1": endpoint{ + Hostname: "waf-regional-fips.eu-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-1", + }, + }, + "fips-eu-west-2": endpoint{ + Hostname: "waf-regional-fips.eu-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-2", + }, + }, + "fips-eu-west-3": endpoint{ + Hostname: "waf-regional-fips.eu-west-3.amazonaws.com", + CredentialScope: credentialScope{ + Region: "eu-west-3", + }, + }, + "fips-me-south-1": endpoint{ + Hostname: "waf-regional-fips.me-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "me-south-1", + }, + }, + "fips-sa-east-1": endpoint{ + Hostname: "waf-regional-fips.sa-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "sa-east-1", + }, + }, + "fips-us-east-1": endpoint{ + Hostname: "waf-regional-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-east-2": endpoint{ + Hostname: "waf-regional-fips.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "fips-us-west-1": endpoint{ + Hostname: "waf-regional-fips.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "waf-regional-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "me-south-1": endpoint{ + Hostname: "waf-regional.me-south-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "me-south-1", + }, + }, + "sa-east-1": endpoint{ + Hostname: "waf-regional.sa-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "sa-east-1", + }, + }, + "us-east-1": endpoint{ + Hostname: "waf-regional.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "us-east-2": endpoint{ + Hostname: "waf-regional.us-east-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-2", + }, + }, + "us-west-1": endpoint{ + Hostname: "waf-regional.us-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-1", + }, + }, + "us-west-2": endpoint{ + Hostname: "waf-regional.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + }, + }, + "workdocs": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "eu-west-1": endpoint{}, + "fips-us-east-1": endpoint{ + Hostname: "workdocs-fips.us-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-east-1", + }, + }, + "fips-us-west-2": endpoint{ + Hostname: "workdocs-fips.us-west-2.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-west-2", + }, + }, + "us-east-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "workmail": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "eu-west-1": endpoint{}, + "us-east-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "workspaces": service{ + + Endpoints: endpoints{ + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + "xray": service{ + + Endpoints: endpoints{ + "af-south-1": endpoint{}, + "ap-east-1": endpoint{}, + "ap-northeast-1": endpoint{}, + "ap-northeast-2": endpoint{}, + "ap-south-1": endpoint{}, + "ap-southeast-1": endpoint{}, + "ap-southeast-2": endpoint{}, + "ca-central-1": endpoint{}, + "eu-central-1": endpoint{}, + "eu-north-1": endpoint{}, + "eu-south-1": endpoint{}, + "eu-west-1": endpoint{}, + "eu-west-2": endpoint{}, + "eu-west-3": endpoint{}, + "me-south-1": endpoint{}, + "sa-east-1": endpoint{}, + "us-east-1": endpoint{}, + "us-east-2": endpoint{}, + "us-west-1": endpoint{}, + "us-west-2": endpoint{}, + }, + }, + }, +} + +// AwsCnPartition returns the Resolver for AWS China. +func AwsCnPartition() Partition { + return awscnPartition.Partition() +} + +var awscnPartition = partition{ + ID: "aws-cn", + Name: "AWS China", + DNSSuffix: "amazonaws.com.cn", + RegionRegex: regionRegex{ + Regexp: func() *regexp.Regexp { + reg, _ := regexp.Compile("^cn\\-\\w+\\-\\d+$") + return reg + }(), + }, + Defaults: endpoint{ + Hostname: "{service}.{region}.{dnsSuffix}", + Protocols: []string{"https"}, + SignatureVersions: []string{"v4"}, + }, + Regions: regions{ + "cn-north-1": region{ + Description: "China (Beijing)", + }, + "cn-northwest-1": region{ + Description: "China (Ningxia)", + }, + }, + Services: services{ + "acm": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "api.ecr": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{ + Hostname: "api.ecr.cn-north-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-north-1", + }, + }, + "cn-northwest-1": endpoint{ + Hostname: "api.ecr.cn-northwest-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "api.sagemaker": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "apigateway": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "application-autoscaling": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "appsync": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + }, + }, + "athena": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "autoscaling": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "autoscaling-plans": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "backup": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "batch": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "budgets": service{ + PartitionEndpoint: "aws-cn-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-cn-global": endpoint{ + Hostname: "budgets.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "ce": service{ + PartitionEndpoint: "aws-cn-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-cn-global": endpoint{ + Hostname: "ce.cn-northwest-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "cloudformation": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "cloudfront": service{ + PartitionEndpoint: "aws-cn-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-cn-global": endpoint{ + Hostname: "cloudfront.cn-northwest-1.amazonaws.com.cn", + Protocols: []string{"http", "https"}, + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "cloudtrail": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "codebuild": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "codecommit": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "codedeploy": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "cognito-identity": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + }, + }, + "config": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "cur": service{ + + Endpoints: endpoints{ + "cn-northwest-1": endpoint{}, + }, + }, + "dax": service{ + + Endpoints: endpoints{ + "cn-northwest-1": endpoint{}, + }, + }, + "directconnect": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "dms": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "ds": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "dynamodb": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "ec2": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "ec2metadata": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "169.254.169.254/latest", + Protocols: []string{"http"}, + }, + }, + }, + "ecs": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "eks": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "elasticache": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "elasticbeanstalk": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "elasticfilesystem": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + "fips-cn-north-1": endpoint{ + Hostname: "elasticfilesystem-fips.cn-north-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-north-1", + }, + }, + "fips-cn-northwest-1": endpoint{ + Hostname: "elasticfilesystem-fips.cn-northwest-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "elasticloadbalancing": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "elasticmapreduce": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "es": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "events": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "firehose": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "gamelift": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + }, + }, + "glacier": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "glue": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "greengrass": service{ + IsRegionalized: boxedTrue, + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + }, + }, + "health": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "iam": service{ + PartitionEndpoint: "aws-cn-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-cn-global": endpoint{ + Hostname: "iam.cn-north-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-north-1", + }, + }, + }, + }, + "iot": service{ + Defaults: endpoint{ + CredentialScope: credentialScope{ + Service: "execute-api", + }, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "iotsecuredtunneling": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "kafka": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "kinesis": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "kinesisanalytics": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "kms": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "lambda": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "license-manager": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "logs": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "mediaconvert": service{ + + Endpoints: endpoints{ + "cn-northwest-1": endpoint{ + Hostname: "subscribe.mediaconvert.cn-northwest-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "monitoring": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "neptune": service{ + + Endpoints: endpoints{ + "cn-northwest-1": endpoint{ + Hostname: "rds.cn-northwest-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "organizations": service{ + PartitionEndpoint: "aws-cn-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-cn-global": endpoint{ + Hostname: "organizations.cn-northwest-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + "fips-aws-cn-global": endpoint{ + Hostname: "organizations.cn-northwest-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "polly": service{ + + Endpoints: endpoints{ + "cn-northwest-1": endpoint{}, + }, + }, + "rds": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "redshift": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "route53": service{ + PartitionEndpoint: "aws-cn-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-cn-global": endpoint{ + Hostname: "route53.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "runtime.sagemaker": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "s3": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + SignatureVersions: []string{"s3v4"}, + + HasDualStack: boxedTrue, + DualStackHostname: "{service}.dualstack.{region}.{dnsSuffix}", + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "s3-control": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + SignatureVersions: []string{"s3v4"}, + + HasDualStack: boxedTrue, + DualStackHostname: "{service}.dualstack.{region}.{dnsSuffix}", + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{ + Hostname: "s3-control.cn-north-1.amazonaws.com.cn", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "cn-north-1", + }, + }, + "cn-northwest-1": endpoint{ + Hostname: "s3-control.cn-northwest-1.amazonaws.com.cn", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "secretsmanager": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "serverlessrepo": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{ + Protocols: []string{"https"}, + }, + "cn-northwest-1": endpoint{ + Protocols: []string{"https"}, + }, + }, + }, + "sms": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "snowball": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + "fips-cn-north-1": endpoint{ + Hostname: "snowball-fips.cn-north-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-north-1", + }, + }, + "fips-cn-northwest-1": endpoint{ + Hostname: "snowball-fips.cn-northwest-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "sns": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "sqs": service{ + Defaults: endpoint{ + SSLCommonName: "{region}.queue.{dnsSuffix}", + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "ssm": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "states": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "storagegateway": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "streams.dynamodb": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + CredentialScope: credentialScope{ + Service: "dynamodb", + }, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "sts": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "support": service{ + PartitionEndpoint: "aws-cn-global", + + Endpoints: endpoints{ + "aws-cn-global": endpoint{ + Hostname: "support.cn-north-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-north-1", + }, + }, + }, + }, + "swf": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "tagging": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + "transcribe": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "cn-north-1": endpoint{ + Hostname: "cn.transcribe.cn-north-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-north-1", + }, + }, + "cn-northwest-1": endpoint{ + Hostname: "cn.transcribe.cn-northwest-1.amazonaws.com.cn", + CredentialScope: credentialScope{ + Region: "cn-northwest-1", + }, + }, + }, + }, + "workspaces": service{ + + Endpoints: endpoints{ + "cn-northwest-1": endpoint{}, + }, + }, + "xray": service{ + + Endpoints: endpoints{ + "cn-north-1": endpoint{}, + "cn-northwest-1": endpoint{}, + }, + }, + }, +} + +// AwsUsGovPartition returns the Resolver for AWS GovCloud (US). +func AwsUsGovPartition() Partition { + return awsusgovPartition.Partition() +} + +var awsusgovPartition = partition{ + ID: "aws-us-gov", + Name: "AWS GovCloud (US)", + DNSSuffix: "amazonaws.com", + RegionRegex: regionRegex{ + Regexp: func() *regexp.Regexp { + reg, _ := regexp.Compile("^us\\-gov\\-\\w+\\-\\d+$") + return reg + }(), + }, + Defaults: endpoint{ + Hostname: "{service}.{region}.{dnsSuffix}", + Protocols: []string{"https"}, + SignatureVersions: []string{"v4"}, + }, + Regions: regions{ + "us-gov-east-1": region{ + Description: "AWS GovCloud (US-East)", + }, + "us-gov-west-1": region{ + Description: "AWS GovCloud (US-West)", + }, + }, + Services: services{ + "access-analyzer": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "acm": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "acm-pca": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "acm-pca.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "acm-pca.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "api.ecr": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "ecr-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "ecr-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{ + Hostname: "api.ecr.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "api.ecr.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "api.sagemaker": service{ + + Endpoints: endpoints{ + "us-gov-west-1": endpoint{}, + "us-gov-west-1-fips": endpoint{ + Hostname: "api-fips.sagemaker.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1-fips-secondary": endpoint{ + Hostname: "api.sagemaker.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "apigateway": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "application-autoscaling": service{ + Defaults: endpoint{ + Hostname: "autoscaling.{region}.amazonaws.com", + Protocols: []string{"http", "https"}, + CredentialScope: credentialScope{ + Service: "application-autoscaling", + }, + }, + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "appstream2": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + CredentialScope: credentialScope{ + Service: "appstream", + }, + }, + Endpoints: endpoints{ + "fips": endpoint{ + Hostname: "appstream2-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1": endpoint{}, + }, + }, + "athena": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "athena-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "athena-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "autoscaling": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Protocols: []string{"http", "https"}, + }, + "us-gov-west-1": endpoint{ + Protocols: []string{"http", "https"}, + }, + }, + }, + "autoscaling-plans": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "backup": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "batch": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "batch.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "batch.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "clouddirectory": service{ + + Endpoints: endpoints{ + "us-gov-west-1": endpoint{}, + }, + }, + "cloudformation": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "cloudformation.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "cloudformation.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "cloudhsm": service{ + + Endpoints: endpoints{ + "us-gov-west-1": endpoint{}, + }, + }, + "cloudhsmv2": service{ + Defaults: endpoint{ + CredentialScope: credentialScope{ + Service: "cloudhsm", + }, + }, + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "cloudtrail": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "cloudtrail.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "cloudtrail.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "codebuild": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-east-1-fips": endpoint{ + Hostname: "codebuild-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{}, + "us-gov-west-1-fips": endpoint{ + Hostname: "codebuild-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "codecommit": service{ + + Endpoints: endpoints{ + "fips": endpoint{ + Hostname: "codecommit-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "codedeploy": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-east-1-fips": endpoint{ + Hostname: "codedeploy-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{}, + "us-gov-west-1-fips": endpoint{ + Hostname: "codedeploy-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "codepipeline": service{ + + Endpoints: endpoints{ + "fips-us-gov-west-1": endpoint{ + Hostname: "codepipeline-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1": endpoint{}, + }, + }, + "cognito-identity": service{ + + Endpoints: endpoints{ + "us-gov-west-1": endpoint{}, + }, + }, + "cognito-idp": service{ + + Endpoints: endpoints{ + "fips-us-gov-west-1": endpoint{ + Hostname: "cognito-idp-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1": endpoint{}, + }, + }, + "comprehend": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "fips-us-gov-west-1": endpoint{ + Hostname: "comprehend-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1": endpoint{}, + }, + }, + "comprehendmedical": service{ + + Endpoints: endpoints{ + "fips-us-gov-west-1": endpoint{ + Hostname: "comprehendmedical-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1": endpoint{}, + }, + }, + "config": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "datasync": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "datasync-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "datasync-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "directconnect": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "directconnect.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "directconnect.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "dms": service{ + + Endpoints: endpoints{ + "dms-fips": endpoint{ + Hostname: "dms.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "docdb": service{ + + Endpoints: endpoints{ + "us-gov-west-1": endpoint{ + Hostname: "rds.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "ds": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "ds-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "ds-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "dynamodb": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-east-1-fips": endpoint{ + Hostname: "dynamodb.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{}, + "us-gov-west-1-fips": endpoint{ + Hostname: "dynamodb.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "ec2": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "ec2.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "ec2.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "ec2metadata": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "169.254.169.254/latest", + Protocols: []string{"http"}, + }, + }, + }, + "ecs": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "ecs-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "ecs-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "eks": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "elasticache": service{ + + Endpoints: endpoints{ + "fips": endpoint{ + Hostname: "elasticache.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "elasticbeanstalk": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "elasticbeanstalk.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "elasticbeanstalk.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "elasticfilesystem": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "elasticfilesystem-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "elasticfilesystem-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "elasticloadbalancing": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "elasticloadbalancing-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "elasticloadbalancing-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{ + Protocols: []string{"http", "https"}, + }, + }, + }, + "elasticmapreduce": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "elasticmapreduce.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "elasticmapreduce.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{ + Protocols: []string{"https"}, + }, + }, + }, + "email": service{ + + Endpoints: endpoints{ + "fips-us-gov-west-1": endpoint{ + Hostname: "email-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1": endpoint{}, + }, + }, + "es": service{ + + Endpoints: endpoints{ + "fips": endpoint{ + Hostname: "es-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "events": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "events.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "events.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "firehose": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "firehose-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "firehose-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "glacier": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "glacier.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "glacier.us-gov-west-1.amazonaws.com", + Protocols: []string{"http", "https"}, + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "glue": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "glue-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "glue-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "greengrass": service{ + IsRegionalized: boxedTrue, + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "us-gov-west-1": endpoint{ + Hostname: "greengrass.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "guardduty": service{ + IsRegionalized: boxedTrue, + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "us-gov-west-1": endpoint{}, + "us-gov-west-1-fips": endpoint{ + Hostname: "guardduty.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "health": service{ + + Endpoints: endpoints{ + "us-gov-west-1": endpoint{}, + }, + }, + "iam": service{ + PartitionEndpoint: "aws-us-gov-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-us-gov-global": endpoint{ + Hostname: "iam.us-gov.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "iam-govcloud-fips": endpoint{ + Hostname: "iam.us-gov.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "inspector": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "inspector-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "inspector-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "iot": service{ + Defaults: endpoint{ + CredentialScope: credentialScope{ + Service: "execute-api", + }, + }, + Endpoints: endpoints{ + "us-gov-west-1": endpoint{}, + }, + }, + "iotsecuredtunneling": service{ + + Endpoints: endpoints{ + "us-gov-west-1": endpoint{}, + }, + }, + "kafka": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "kinesis": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "kinesis-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "kinesis-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "kinesisanalytics": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "kms": service{ + + Endpoints: endpoints{ + "ProdFips": endpoint{ + Hostname: "kms-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "lambda": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "lambda-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "lambda-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "license-manager": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "license-manager-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "license-manager-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "logs": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "logs.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "logs.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "mediaconvert": service{ + + Endpoints: endpoints{ + "us-gov-west-1": endpoint{ + Hostname: "mediaconvert.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "metering.marketplace": service{ + Defaults: endpoint{ CredentialScope: credentialScope{ - Service: "application-autoscaling", + Service: "aws-marketplace", + }, + }, + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "monitoring": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "monitoring.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "monitoring.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "neptune": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "rds.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "rds.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "organizations": service{ + PartitionEndpoint: "aws-us-gov-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-us-gov-global": endpoint{ + Hostname: "organizations.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "fips-aws-us-gov-global": endpoint{ + Hostname: "organizations.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "outposts": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "outposts.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "outposts.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "pinpoint": service{ + Defaults: endpoint{ + CredentialScope: credentialScope{ + Service: "mobiletargeting", + }, + }, + Endpoints: endpoints{ + "us-gov-west-1": endpoint{}, + }, + }, + "polly": service{ + + Endpoints: endpoints{ + "fips-us-gov-west-1": endpoint{ + Hostname: "polly-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1": endpoint{}, + }, + }, + "ram": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "rds": service{ + + Endpoints: endpoints{ + "rds.us-gov-east-1": endpoint{ + Hostname: "rds.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "rds.us-gov-west-1": endpoint{ + Hostname: "rds.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "redshift": service{ + + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "redshift.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "redshift.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "rekognition": service{ + + Endpoints: endpoints{ + "rekognition-fips.us-gov-west-1": endpoint{ + Hostname: "rekognition-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1": endpoint{}, + }, + }, + "resource-groups": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "resource-groups.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "resource-groups.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, + }, + }, + "route53": service{ + PartitionEndpoint: "aws-us-gov-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-us-gov-global": endpoint{ + Hostname: "route53.us-gov.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, }, }, + }, + "route53resolver": service{ + Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, }, - "autoscaling": service{ + "runtime.sagemaker": service{ + + Endpoints: endpoints{ + "us-gov-west-1": endpoint{}, + }, + }, + "s3": service{ Defaults: endpoint{ - Protocols: []string{"http", "https"}, + SignatureVersions: []string{"s3", "s3v4"}, + + HasDualStack: boxedTrue, + DualStackHostname: "{service}.dualstack.{region}.{dnsSuffix}", }, Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "fips-us-gov-west-1": endpoint{ + Hostname: "s3-fips-us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{ + Hostname: "s3.us-gov-east-1.amazonaws.com", + Protocols: []string{"http", "https"}, + }, + "us-gov-west-1": endpoint{ + Hostname: "s3.us-gov-west-1.amazonaws.com", + Protocols: []string{"http", "https"}, + }, }, }, - "cloudformation": service{ + "s3-control": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + SignatureVersions: []string{"s3v4"}, + HasDualStack: boxedTrue, + DualStackHostname: "{service}.dualstack.{region}.{dnsSuffix}", + }, Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-east-1": endpoint{ + Hostname: "s3-control.us-gov-east-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-east-1-fips": endpoint{ + Hostname: "s3-control-fips.us-gov-east-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "s3-control.us-gov-west-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1-fips": endpoint{ + Hostname: "s3-control-fips.us-gov-west-1.amazonaws.com", + SignatureVersions: []string{"s3v4"}, + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, }, }, - "cloudtrail": service{ + "secretsmanager": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-east-1": endpoint{}, + "us-gov-east-1-fips": endpoint{ + Hostname: "secretsmanager-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{}, + "us-gov-west-1-fips": endpoint{ + Hostname: "secretsmanager-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, }, }, - "codebuild": service{ + "securityhub": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "fips-us-gov-east-1": endpoint{ + Hostname: "securityhub-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "securityhub-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, }, - "codedeploy": service{ + "serverlessrepo": service{ + Defaults: endpoint{ + Protocols: []string{"https"}, + }, + Endpoints: endpoints{ + "us-gov-east-1": endpoint{ + Hostname: "serverlessrepo.us-gov-east-1.amazonaws.com", + Protocols: []string{"https"}, + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "serverlessrepo.us-gov-west-1.amazonaws.com", + Protocols: []string{"https"}, + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + }, + }, + "servicecatalog": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-east-1": endpoint{}, + "us-gov-east-1-fips": endpoint{ + Hostname: "servicecatalog-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{}, + "us-gov-west-1-fips": endpoint{ + Hostname: "servicecatalog-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, }, }, - "cognito-identity": service{ + "sms": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, + "fips-us-gov-east-1": endpoint{ + Hostname: "sms-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "sms-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, }, - "config": service{ + "snowball": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "fips-us-gov-east-1": endpoint{ + Hostname: "snowball-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "snowball-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, }, - "directconnect": service{ + "sns": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-east-1": endpoint{ + Hostname: "sns.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "sns.us-gov-west-1.amazonaws.com", + Protocols: []string{"http", "https"}, + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, }, }, - "dms": service{ + "sqs": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-east-1": endpoint{ + Hostname: "sqs.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "sqs.us-gov-west-1.amazonaws.com", + SSLCommonName: "{region}.queue.{dnsSuffix}", + Protocols: []string{"http", "https"}, + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, }, }, - "ds": service{ + "ssm": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "fips-us-gov-east-1": endpoint{ + Hostname: "ssm.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "ssm.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "ssm-facade-fips-us-gov-east-1": endpoint{ + Hostname: "ssm-facade.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "ssm-facade-fips-us-gov-west-1": endpoint{ + Hostname: "ssm-facade.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, }, - "dynamodb": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, + "states": service{ + + Endpoints: endpoints{ + "fips-us-gov-east-1": endpoint{ + Hostname: "states-fips.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "states.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, + }, + "storagegateway": service{ + Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "fips": endpoint{ + Hostname: "storagegateway-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, }, - "ec2": service{ + "streams.dynamodb": service{ Defaults: endpoint{ - Protocols: []string{"http", "https"}, - }, - Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + CredentialScope: credentialScope{ + Service: "dynamodb", + }, }, - }, - "ec2metadata": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, - Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "169.254.169.254/latest", - Protocols: []string{"http"}, + "us-gov-east-1": endpoint{}, + "us-gov-east-1-fips": endpoint{ + Hostname: "dynamodb.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{}, + "us-gov-west-1-fips": endpoint{ + Hostname: "dynamodb.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, }, }, }, - "ecr": service{ + "sts": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-east-1": endpoint{}, + "us-gov-east-1-fips": endpoint{ + Hostname: "sts.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{}, + "us-gov-west-1-fips": endpoint{ + Hostname: "sts.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, }, }, - "ecs": service{ + "support": service{ + PartitionEndpoint: "aws-us-gov-global", Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "aws-us-gov-global": endpoint{ + Hostname: "support.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "support.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, }, }, - "elasticache": service{ + "swf": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-east-1": endpoint{ + Hostname: "swf.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "swf.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, }, }, - "elasticbeanstalk": service{ + "tagging": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, }, - "elasticloadbalancing": service{ + "transcribe": service{ Defaults: endpoint{ Protocols: []string{"https"}, }, Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "fips-us-gov-east-1": endpoint{ + Hostname: "fips.transcribe.us-gov-east-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-east-1", + }, + }, + "fips-us-gov-west-1": endpoint{ + Hostname: "fips.transcribe.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, }, - "elasticmapreduce": service{ + "translate": service{ Defaults: endpoint{ Protocols: []string{"https"}, }, Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-west-1": endpoint{}, + "us-gov-west-1-fips": endpoint{ + Hostname: "translate-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, }, }, - "es": service{ + "waf-regional": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "fips-us-gov-west-1": endpoint{ + Hostname: "waf-regional-fips.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, + "us-gov-west-1": endpoint{ + Hostname: "waf-regional.us-gov-west-1.amazonaws.com", + CredentialScope: credentialScope{ + Region: "us-gov-west-1", + }, + }, }, }, - "events": service{ + "workspaces": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, }, - "glacier": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - }, + "xray": service{ + Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-gov-east-1": endpoint{}, + "us-gov-west-1": endpoint{}, }, }, - "iam": service{ - PartitionEndpoint: "aws-cn-global", - IsRegionalized: boxedFalse, + }, +} + +// AwsIsoPartition returns the Resolver for AWS ISO (US). +func AwsIsoPartition() Partition { + return awsisoPartition.Partition() +} + +var awsisoPartition = partition{ + ID: "aws-iso", + Name: "AWS ISO (US)", + DNSSuffix: "c2s.ic.gov", + RegionRegex: regionRegex{ + Regexp: func() *regexp.Regexp { + reg, _ := regexp.Compile("^us\\-iso\\-\\w+\\-\\d+$") + return reg + }(), + }, + Defaults: endpoint{ + Hostname: "{service}.{region}.{dnsSuffix}", + Protocols: []string{"https"}, + SignatureVersions: []string{"v4"}, + }, + Regions: regions{ + "us-iso-east-1": region{ + Description: "US ISO East", + }, + }, + Services: services{ + "api.ecr": service{ Endpoints: endpoints{ - "aws-cn-global": endpoint{ - Hostname: "iam.cn-north-1.amazonaws.com.cn", + "us-iso-east-1": endpoint{ + Hostname: "api.ecr.us-iso-east-1.c2s.ic.gov", CredentialScope: credentialScope{ - Region: "cn-north-1", + Region: "us-iso-east-1", }, }, }, }, - "iot": service{ - Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "execute-api", - }, - }, - Endpoints: endpoints{ - "cn-north-1": endpoint{}, - }, - }, - "kinesis": service{ - - Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, - }, - }, - "lambda": service{ + "api.sagemaker": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "logs": service{ + "apigateway": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "monitoring": service{ + "application-autoscaling": service{ Defaults: endpoint{ Protocols: []string{"http", "https"}, }, Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "polly": service{ + "autoscaling": service{ Endpoints: endpoints{ - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{ + Protocols: []string{"http", "https"}, + }, }, }, - "rds": service{ + "cloudformation": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "redshift": service{ + "cloudtrail": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "s3": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - SignatureVersions: []string{"s3v4"}, - }, + "codedeploy": service{ + Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "s3-control": service{ + "comprehend": service{ Defaults: endpoint{ - Protocols: []string{"https"}, - SignatureVersions: []string{"s3v4"}, + Protocols: []string{"https"}, }, Endpoints: endpoints{ - "cn-north-1": endpoint{ - Hostname: "s3-control.cn-north-1.amazonaws.com.cn", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "cn-north-1", - }, - }, - "cn-northwest-1": endpoint{ - Hostname: "s3-control.cn-northwest-1.amazonaws.com.cn", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "cn-northwest-1", - }, - }, + "us-iso-east-1": endpoint{}, }, }, - "sms": service{ + "config": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "snowball": service{ + "datapipeline": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "sns": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, + "directconnect": service{ + + Endpoints: endpoints{ + "us-iso-east-1": endpoint{}, }, + }, + "dms": service{ + Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "dms-fips": endpoint{ + Hostname: "dms.us-iso-east-1.c2s.ic.gov", + CredentialScope: credentialScope{ + Region: "us-iso-east-1", + }, + }, + "us-iso-east-1": endpoint{}, }, }, - "sqs": service{ - Defaults: endpoint{ - SSLCommonName: "{region}.queue.{dnsSuffix}", - Protocols: []string{"http", "https"}, - }, + "ds": service{ + Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "ssm": service{ + "dynamodb": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{ + Protocols: []string{"http", "https"}, + }, }, }, - "storagegateway": service{ + "ec2": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "streams.dynamodb": service{ - Defaults: endpoint{ - Protocols: []string{"http", "https"}, - CredentialScope: credentialScope{ - Service: "dynamodb", + "ec2metadata": service{ + PartitionEndpoint: "aws-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-global": endpoint{ + Hostname: "169.254.169.254/latest", + Protocols: []string{"http"}, }, }, + }, + "ecs": service{ + Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "sts": service{ + "elasticache": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "swf": service{ + "elasticloadbalancing": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{ + Protocols: []string{"http", "https"}, + }, }, }, - "tagging": service{ + "elasticmapreduce": service{ Endpoints: endpoints{ - "cn-north-1": endpoint{}, - "cn-northwest-1": endpoint{}, + "us-iso-east-1": endpoint{ + Protocols: []string{"https"}, + }, }, }, - }, -} - -// AwsUsGovPartition returns the Resolver for AWS GovCloud (US). -func AwsUsGovPartition() Partition { - return awsusgovPartition.Partition() -} + "es": service{ -var awsusgovPartition = partition{ - ID: "aws-us-gov", - Name: "AWS GovCloud (US)", - DNSSuffix: "amazonaws.com", - RegionRegex: regionRegex{ - Regexp: func() *regexp.Regexp { - reg, _ := regexp.Compile("^us\\-gov\\-\\w+\\-\\d+$") - return reg - }(), - }, - Defaults: endpoint{ - Hostname: "{service}.{region}.{dnsSuffix}", - Protocols: []string{"https"}, - SignatureVersions: []string{"v4"}, - }, - Regions: regions{ - "us-gov-east-1": region{ - Description: "AWS GovCloud (US-East)", - }, - "us-gov-west-1": region{ - Description: "AWS GovCloud (US)", + Endpoints: endpoints{ + "us-iso-east-1": endpoint{}, + }, }, - }, - Services: services{ - "acm": service{ + "events": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "api.sagemaker": service{ + "glacier": service{ Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{ + Protocols: []string{"http", "https"}, + }, }, }, - "apigateway": service{ + "health": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "application-autoscaling": service{ - Defaults: endpoint{ - Hostname: "autoscaling.{region}.amazonaws.com", - CredentialScope: credentialScope{ - Service: "application-autoscaling", + "iam": service{ + PartitionEndpoint: "aws-iso-global", + IsRegionalized: boxedFalse, + + Endpoints: endpoints{ + "aws-iso-global": endpoint{ + Hostname: "iam.us-iso-east-1.c2s.ic.gov", + CredentialScope: credentialScope{ + Region: "us-iso-east-1", + }, }, }, + }, + "kinesis": service{ + Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "autoscaling": service{ + "kms": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{ - Protocols: []string{"http", "https"}, + "ProdFips": endpoint{ + Hostname: "kms-fips.us-iso-east-1.c2s.ic.gov", + CredentialScope: credentialScope{ + Region: "us-iso-east-1", + }, }, + "us-iso-east-1": endpoint{}, }, }, - "clouddirectory": service{ + "lambda": service{ Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "cloudformation": service{ + "logs": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "cloudhsm": service{ + "monitoring": service{ Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "cloudhsmv2": service{ - Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "cloudhsm", - }, - }, + "rds": service{ + Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "cloudtrail": service{ + "redshift": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "codedeploy": service{ + "route53": service{ + PartitionEndpoint: "aws-iso-global", + IsRegionalized: boxedFalse, Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-east-1-fips": endpoint{ - Hostname: "codedeploy-fips.us-gov-east-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-gov-east-1", - }, - }, - "us-gov-west-1": endpoint{}, - "us-gov-west-1-fips": endpoint{ - Hostname: "codedeploy-fips.us-gov-west-1.amazonaws.com", + "aws-iso-global": endpoint{ + Hostname: "route53.c2s.ic.gov", CredentialScope: credentialScope{ - Region: "us-gov-west-1", + Region: "us-iso-east-1", }, }, }, }, - "config": service{ + "runtime.sagemaker": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "directconnect": service{ + "s3": service{ + Defaults: endpoint{ + SignatureVersions: []string{"s3v4"}, + }, + Endpoints: endpoints{ + "us-iso-east-1": endpoint{ + Protocols: []string{"http", "https"}, + SignatureVersions: []string{"s3v4"}, + }, + }, + }, + "snowball": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "dms": service{ + "sns": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{ + Protocols: []string{"http", "https"}, + }, }, }, - "ds": service{ + "sqs": service{ Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{ + Protocols: []string{"http", "https"}, + }, }, }, - "dynamodb": service{ + "states": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, - "us-gov-west-1-fips": endpoint{ - Hostname: "dynamodb.us-gov-west-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-gov-west-1", - }, + "us-iso-east-1": endpoint{}, + }, + }, + "streams.dynamodb": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + CredentialScope: credentialScope{ + Service: "dynamodb", + }, + }, + Endpoints: endpoints{ + "us-iso-east-1": endpoint{ + Protocols: []string{"http", "https"}, }, }, }, - "ec2": service{ + "sts": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "ec2metadata": service{ - PartitionEndpoint: "aws-global", - IsRegionalized: boxedFalse, + "support": service{ + PartitionEndpoint: "aws-iso-global", Endpoints: endpoints{ - "aws-global": endpoint{ - Hostname: "169.254.169.254/latest", - Protocols: []string{"http"}, + "aws-iso-global": endpoint{ + Hostname: "support.us-iso-east-1.c2s.ic.gov", + CredentialScope: credentialScope{ + Region: "us-iso-east-1", + }, }, }, }, - "ecr": service{ + "swf": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "ecs": service{ + "transcribestreaming": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "elasticache": service{ + "workspaces": service{ Endpoints: endpoints{ - "fips": endpoint{ - Hostname: "elasticache-fips.us-gov-west-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-gov-west-1", - }, - }, - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-iso-east-1": endpoint{}, }, }, - "elasticbeanstalk": service{ + }, +} +// AwsIsoBPartition returns the Resolver for AWS ISOB (US). +func AwsIsoBPartition() Partition { + return awsisobPartition.Partition() +} + +var awsisobPartition = partition{ + ID: "aws-iso-b", + Name: "AWS ISOB (US)", + DNSSuffix: "sc2s.sgov.gov", + RegionRegex: regionRegex{ + Regexp: func() *regexp.Regexp { + reg, _ := regexp.Compile("^us\\-isob\\-\\w+\\-\\d+$") + return reg + }(), + }, + Defaults: endpoint{ + Hostname: "{service}.{region}.{dnsSuffix}", + Protocols: []string{"https"}, + SignatureVersions: []string{"v4"}, + }, + Regions: regions{ + "us-isob-east-1": region{ + Description: "US ISOB East (Ohio)", + }, + }, + Services: services{ + "application-autoscaling": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "elasticfilesystem": service{ - + "autoscaling": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "elasticloadbalancing": service{ + "cloudformation": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{ - Protocols: []string{"http", "https"}, - }, + "us-isob-east-1": endpoint{}, }, }, - "elasticmapreduce": service{ + "cloudtrail": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{ - Protocols: []string{"https"}, - }, + "us-isob-east-1": endpoint{}, }, }, - "es": service{ + "config": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "events": service{ + "directconnect": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "glacier": service{ + "dms": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{ - Protocols: []string{"http", "https"}, + "dms-fips": endpoint{ + Hostname: "dms.us-isob-east-1.sc2s.sgov.gov", + CredentialScope: credentialScope{ + Region: "us-isob-east-1", + }, }, + "us-isob-east-1": endpoint{}, }, }, - "guardduty": service{ - IsRegionalized: boxedTrue, + "dynamodb": service{ Defaults: endpoint{ - Protocols: []string{"https"}, + Protocols: []string{"http", "https"}, }, Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "iam": service{ - PartitionEndpoint: "aws-us-gov-global", + "ec2": service{ + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, + Endpoints: endpoints{ + "us-isob-east-1": endpoint{}, + }, + }, + "ec2metadata": service{ + PartitionEndpoint: "aws-global", IsRegionalized: boxedFalse, Endpoints: endpoints{ - "aws-us-gov-global": endpoint{ - Hostname: "iam.us-gov.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-gov-west-1", - }, + "aws-global": endpoint{ + Hostname: "169.254.169.254/latest", + Protocols: []string{"http"}, }, }, }, - "inspector": service{ + "elasticache": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "iot": service{ - Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "execute-api", - }, - }, + "elasticloadbalancing": service{ + Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{ + Protocols: []string{"https"}, + }, }, }, - "kinesis": service{ + "elasticmapreduce": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "kms": service{ + "events": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "lambda": service{ + "glacier": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "logs": service{ + "health": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "mediaconvert": service{ + "iam": service{ + PartitionEndpoint: "aws-iso-b-global", + IsRegionalized: boxedFalse, Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, - }, - }, - "metering.marketplace": service{ - Defaults: endpoint{ - CredentialScope: credentialScope{ - Service: "aws-marketplace", + "aws-iso-b-global": endpoint{ + Hostname: "iam.us-isob-east-1.sc2s.sgov.gov", + CredentialScope: credentialScope{ + Region: "us-isob-east-1", + }, }, }, + }, + "kinesis": service{ + Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "monitoring": service{ + "kms": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "ProdFips": endpoint{ + Hostname: "kms-fips.us-isob-east-1.sc2s.sgov.gov", + CredentialScope: credentialScope{ + Region: "us-isob-east-1", + }, + }, + "us-isob-east-1": endpoint{}, }, }, - "polly": service{ + "lambda": service{ Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "rds": service{ + "license-manager": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "redshift": service{ + "logs": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "rekognition": service{ + "monitoring": service{ Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "runtime.sagemaker": service{ + "rds": service{ Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "s3": service{ - Defaults: endpoint{ - SignatureVersions: []string{"s3", "s3v4"}, - }, + "redshift": service{ + Endpoints: endpoints{ - "fips-us-gov-west-1": endpoint{ - Hostname: "s3-fips-us-gov-west-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-gov-west-1", - }, - }, - "us-gov-east-1": endpoint{ - Hostname: "s3.us-gov-east-1.amazonaws.com", - Protocols: []string{"http", "https"}, - }, - "us-gov-west-1": endpoint{ - Hostname: "s3.us-gov-west-1.amazonaws.com", - Protocols: []string{"http", "https"}, - }, + "us-isob-east-1": endpoint{}, }, }, - "s3-control": service{ + "s3": service{ Defaults: endpoint{ - Protocols: []string{"https"}, + Protocols: []string{"http", "https"}, SignatureVersions: []string{"s3v4"}, }, Endpoints: endpoints{ - "us-gov-east-1": endpoint{ - Hostname: "s3-control.us-gov-east-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "us-gov-east-1", - }, - }, - "us-gov-east-1-fips": endpoint{ - Hostname: "s3-control-fips.us-gov-east-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "us-gov-east-1", - }, - }, - "us-gov-west-1": endpoint{ - Hostname: "s3-control.us-gov-west-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "us-gov-west-1", - }, - }, - "us-gov-west-1-fips": endpoint{ - Hostname: "s3-control-fips.us-gov-west-1.amazonaws.com", - SignatureVersions: []string{"s3v4"}, - CredentialScope: credentialScope{ - Region: "us-gov-west-1", - }, - }, - }, - }, - "sms": service{ - - Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, "snowball": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, "sns": service{ - + Defaults: endpoint{ + Protocols: []string{"http", "https"}, + }, Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{ - Protocols: []string{"http", "https"}, - }, + "us-isob-east-1": endpoint{}, }, }, "sqs": service{ - + Defaults: endpoint{ + SSLCommonName: "{region}.queue.{dnsSuffix}", + Protocols: []string{"http", "https"}, + }, Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{ - SSLCommonName: "{region}.queue.{dnsSuffix}", - Protocols: []string{"http", "https"}, - }, + "us-isob-east-1": endpoint{}, }, }, "ssm": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, "states": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, - }, - }, - "storagegateway": service{ - - Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, "streams.dynamodb": service{ Defaults: endpoint{ + Protocols: []string{"http", "https"}, CredentialScope: credentialScope{ Service: "dynamodb", }, }, Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, - "us-gov-west-1-fips": endpoint{ - Hostname: "dynamodb.us-gov-west-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-gov-west-1", - }, - }, + "us-isob-east-1": endpoint{}, }, }, "sts": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "us-isob-east-1": endpoint{}, }, }, - "swf": service{ + "support": service{ + PartitionEndpoint: "aws-iso-b-global", Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, + "aws-iso-b-global": endpoint{ + Hostname: "support.us-isob-east-1.sc2s.sgov.gov", + CredentialScope: credentialScope{ + Region: "us-isob-east-1", + }, + }, }, }, - "tagging": service{ + "swf": service{ Endpoints: endpoints{ - "us-gov-east-1": endpoint{}, - "us-gov-west-1": endpoint{}, - }, - }, - "translate": service{ - Defaults: endpoint{ - Protocols: []string{"https"}, - }, - Endpoints: endpoints{ - "us-gov-west-1": endpoint{}, - "us-gov-west-1-fips": endpoint{ - Hostname: "translate-fips.us-gov-west-1.amazonaws.com", - CredentialScope: credentialScope{ - Region: "us-gov-west-1", - }, - }, + "us-isob-east-1": endpoint{}, }, }, }, diff --git a/github.com/aws/aws-sdk-go/aws/endpoints/dep_service_ids.go b/github.com/aws/aws-sdk-go/aws/endpoints/dep_service_ids.go index 000dd79eec..ca8fc828e1 100644 --- a/github.com/aws/aws-sdk-go/aws/endpoints/dep_service_ids.go +++ b/github.com/aws/aws-sdk-go/aws/endpoints/dep_service_ids.go @@ -2,7 +2,7 @@ package endpoints // Service identifiers // -// Deprecated: Use client package's EndpointID value instead of these +// Deprecated: Use client package's EndpointsID value instead of these // ServiceIDs. These IDs are not maintained, and are out of date. const ( A4bServiceID = "a4b" // A4b. diff --git a/github.com/aws/aws-sdk-go/aws/endpoints/endpoints.go b/github.com/aws/aws-sdk-go/aws/endpoints/endpoints.go index f82babf6f9..ca956e5f12 100644 --- a/github.com/aws/aws-sdk-go/aws/endpoints/endpoints.go +++ b/github.com/aws/aws-sdk-go/aws/endpoints/endpoints.go @@ -3,6 +3,7 @@ package endpoints import ( "fmt" "regexp" + "strings" "github.com/aws/aws-sdk-go/aws/awserr" ) @@ -46,6 +47,108 @@ type Options struct { // // This option is ignored if StrictMatching is enabled. ResolveUnknownService bool + + // STS Regional Endpoint flag helps with resolving the STS endpoint + STSRegionalEndpoint STSRegionalEndpoint + + // S3 Regional Endpoint flag helps with resolving the S3 endpoint + S3UsEast1RegionalEndpoint S3UsEast1RegionalEndpoint +} + +// STSRegionalEndpoint is an enum for the states of the STS Regional Endpoint +// options. +type STSRegionalEndpoint int + +func (e STSRegionalEndpoint) String() string { + switch e { + case LegacySTSEndpoint: + return "legacy" + case RegionalSTSEndpoint: + return "regional" + case UnsetSTSEndpoint: + return "" + default: + return "unknown" + } +} + +const ( + + // UnsetSTSEndpoint represents that STS Regional Endpoint flag is not specified. + UnsetSTSEndpoint STSRegionalEndpoint = iota + + // LegacySTSEndpoint represents when STS Regional Endpoint flag is specified + // to use legacy endpoints. + LegacySTSEndpoint + + // RegionalSTSEndpoint represents when STS Regional Endpoint flag is specified + // to use regional endpoints. + RegionalSTSEndpoint +) + +// GetSTSRegionalEndpoint function returns the STSRegionalEndpointFlag based +// on the input string provided in env config or shared config by the user. +// +// `legacy`, `regional` are the only case-insensitive valid strings for +// resolving the STS regional Endpoint flag. +func GetSTSRegionalEndpoint(s string) (STSRegionalEndpoint, error) { + switch { + case strings.EqualFold(s, "legacy"): + return LegacySTSEndpoint, nil + case strings.EqualFold(s, "regional"): + return RegionalSTSEndpoint, nil + default: + return UnsetSTSEndpoint, fmt.Errorf("unable to resolve the value of STSRegionalEndpoint for %v", s) + } +} + +// S3UsEast1RegionalEndpoint is an enum for the states of the S3 us-east-1 +// Regional Endpoint options. +type S3UsEast1RegionalEndpoint int + +func (e S3UsEast1RegionalEndpoint) String() string { + switch e { + case LegacyS3UsEast1Endpoint: + return "legacy" + case RegionalS3UsEast1Endpoint: + return "regional" + case UnsetS3UsEast1Endpoint: + return "" + default: + return "unknown" + } +} + +const ( + + // UnsetS3UsEast1Endpoint represents that S3 Regional Endpoint flag is not + // specified. + UnsetS3UsEast1Endpoint S3UsEast1RegionalEndpoint = iota + + // LegacyS3UsEast1Endpoint represents when S3 Regional Endpoint flag is + // specified to use legacy endpoints. + LegacyS3UsEast1Endpoint + + // RegionalS3UsEast1Endpoint represents when S3 Regional Endpoint flag is + // specified to use regional endpoints. + RegionalS3UsEast1Endpoint +) + +// GetS3UsEast1RegionalEndpoint function returns the S3UsEast1RegionalEndpointFlag based +// on the input string provided in env config or shared config by the user. +// +// `legacy`, `regional` are the only case-insensitive valid strings for +// resolving the S3 regional Endpoint flag. +func GetS3UsEast1RegionalEndpoint(s string) (S3UsEast1RegionalEndpoint, error) { + switch { + case strings.EqualFold(s, "legacy"): + return LegacyS3UsEast1Endpoint, nil + case strings.EqualFold(s, "regional"): + return RegionalS3UsEast1Endpoint, nil + default: + return UnsetS3UsEast1Endpoint, + fmt.Errorf("unable to resolve the value of S3UsEast1RegionalEndpoint for %v", s) + } } // Set combines all of the option functions together. @@ -79,6 +182,12 @@ func ResolveUnknownServiceOption(o *Options) { o.ResolveUnknownService = true } +// STSRegionalEndpointOption enables the STS endpoint resolver behavior to resolve +// STS endpoint to their regional endpoint, instead of the global endpoint. +func STSRegionalEndpointOption(o *Options) { + o.STSRegionalEndpoint = RegionalSTSEndpoint +} + // A Resolver provides the interface for functionality to resolve endpoints. // The build in Partition and DefaultResolver return value satisfy this interface. type Resolver interface { @@ -170,10 +279,13 @@ func PartitionForRegion(ps []Partition, regionID string) (Partition, bool) { // A Partition provides the ability to enumerate the partition's regions // and services. type Partition struct { - id string - p *partition + id, dnsSuffix string + p *partition } +// DNSSuffix returns the base domain name of the partition. +func (p Partition) DNSSuffix() string { return p.dnsSuffix } + // ID returns the identifier of the partition. func (p Partition) ID() string { return p.id } @@ -191,7 +303,7 @@ func (p Partition) ID() string { return p.id } // require the provided service and region to be known by the partition. // If the endpoint cannot be strictly resolved an error will be returned. This // mode is useful to ensure the endpoint resolved is valid. Without -// StrictMatching enabled the endpoint returned my look valid but may not work. +// StrictMatching enabled the endpoint returned may look valid but may not work. // StrictMatching requires the SDK to be updated if you want to take advantage // of new regions and services expansions. // @@ -205,7 +317,7 @@ func (p Partition) EndpointFor(service, region string, opts ...func(*Options)) ( // Regions returns a map of Regions indexed by their ID. This is useful for // enumerating over the regions in a partition. func (p Partition) Regions() map[string]Region { - rs := map[string]Region{} + rs := make(map[string]Region, len(p.p.Regions)) for id, r := range p.p.Regions { rs[id] = Region{ id: id, @@ -220,7 +332,7 @@ func (p Partition) Regions() map[string]Region { // Services returns a map of Service indexed by their ID. This is useful for // enumerating over the services in a partition. func (p Partition) Services() map[string]Service { - ss := map[string]Service{} + ss := make(map[string]Service, len(p.p.Services)) for id := range p.p.Services { ss[id] = Service{ id: id, @@ -307,7 +419,7 @@ func (s Service) Regions() map[string]Region { // A region is the AWS region the service exists in. Whereas a Endpoint is // an URL that can be resolved to a instance of a service. func (s Service) Endpoints() map[string]Endpoint { - es := map[string]Endpoint{} + es := make(map[string]Endpoint, len(s.p.Services[s.id].Endpoints)) for id := range s.p.Services[s.id].Endpoints { es[id] = Endpoint{ id: id, @@ -347,6 +459,9 @@ type ResolvedEndpoint struct { // The endpoint URL URL string + // The endpoint partition + PartitionID string + // The region that should be used for signing requests. SigningRegion string diff --git a/github.com/aws/aws-sdk-go/aws/endpoints/legacy_regions.go b/github.com/aws/aws-sdk-go/aws/endpoints/legacy_regions.go new file mode 100644 index 0000000000..df75e899ad --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/endpoints/legacy_regions.go @@ -0,0 +1,24 @@ +package endpoints + +var legacyGlobalRegions = map[string]map[string]struct{}{ + "sts": { + "ap-northeast-1": {}, + "ap-south-1": {}, + "ap-southeast-1": {}, + "ap-southeast-2": {}, + "ca-central-1": {}, + "eu-central-1": {}, + "eu-north-1": {}, + "eu-west-1": {}, + "eu-west-2": {}, + "eu-west-3": {}, + "sa-east-1": {}, + "us-east-1": {}, + "us-east-2": {}, + "us-west-1": {}, + "us-west-2": {}, + }, + "s3": { + "us-east-1": {}, + }, +} diff --git a/github.com/aws/aws-sdk-go/aws/endpoints/v3model.go b/github.com/aws/aws-sdk-go/aws/endpoints/v3model.go index ff6f76db6e..773613722f 100644 --- a/github.com/aws/aws-sdk-go/aws/endpoints/v3model.go +++ b/github.com/aws/aws-sdk-go/aws/endpoints/v3model.go @@ -7,6 +7,8 @@ import ( "strings" ) +var regionValidationRegex = regexp.MustCompile(`^[[:alnum:]]([[:alnum:]\-]*[[:alnum:]])?$`) + type partitions []partition func (ps partitions) EndpointFor(service, region string, opts ...func(*Options)) (ResolvedEndpoint, error) { @@ -54,8 +56,9 @@ type partition struct { func (p partition) Partition() Partition { return Partition{ - id: p.ID, - p: &p, + dnsSuffix: p.DNSSuffix, + id: p.ID, + p: &p, } } @@ -74,24 +77,56 @@ func (p partition) canResolveEndpoint(service, region string, strictMatch bool) return p.RegionRegex.MatchString(region) } +func allowLegacyEmptyRegion(service string) bool { + legacy := map[string]struct{}{ + "budgets": {}, + "ce": {}, + "chime": {}, + "cloudfront": {}, + "ec2metadata": {}, + "iam": {}, + "importexport": {}, + "organizations": {}, + "route53": {}, + "sts": {}, + "support": {}, + "waf": {}, + } + + _, allowed := legacy[service] + return allowed +} + func (p partition) EndpointFor(service, region string, opts ...func(*Options)) (resolved ResolvedEndpoint, err error) { var opt Options opt.Set(opts...) s, hasService := p.Services[service] - if !(hasService || opt.ResolveUnknownService) { + if len(service) == 0 || !(hasService || opt.ResolveUnknownService) { // Only return error if the resolver will not fallback to creating // endpoint based on service endpoint ID passed in. return resolved, NewUnknownServiceError(p.ID, service, serviceList(p.Services)) } + if len(region) == 0 && allowLegacyEmptyRegion(service) && len(s.PartitionEndpoint) != 0 { + region = s.PartitionEndpoint + } + + if (service == "sts" && opt.STSRegionalEndpoint != RegionalSTSEndpoint) || + (service == "s3" && opt.S3UsEast1RegionalEndpoint != RegionalS3UsEast1Endpoint) { + if _, ok := legacyGlobalRegions[service][region]; ok { + region = "aws-global" + } + } + e, hasEndpoint := s.endpointForRegion(region) - if !hasEndpoint && opt.StrictMatching { + if len(region) == 0 || (!hasEndpoint && opt.StrictMatching) { return resolved, NewUnknownEndpointError(p.ID, service, region, endpointList(s.Endpoints)) } defs := []endpoint{p.Defaults, s.Defaults} - return e.resolve(service, region, p.DNSSuffix, defs, opt), nil + + return e.resolve(service, p.ID, region, p.DNSSuffix, defs, opt) } func serviceList(ss services) []string { @@ -200,7 +235,7 @@ func getByPriority(s []string, p []string, def string) string { return s[0] } -func (e endpoint) resolve(service, region, dnsSuffix string, defs []endpoint, opts Options) ResolvedEndpoint { +func (e endpoint) resolve(service, partitionID, region, dnsSuffix string, defs []endpoint, opts Options) (ResolvedEndpoint, error) { var merged endpoint for _, def := range defs { merged.mergeIn(def) @@ -208,11 +243,27 @@ func (e endpoint) resolve(service, region, dnsSuffix string, defs []endpoint, op merged.mergeIn(e) e = merged - hostname := e.Hostname + signingRegion := e.CredentialScope.Region + if len(signingRegion) == 0 { + signingRegion = region + } + signingName := e.CredentialScope.Service + var signingNameDerived bool + if len(signingName) == 0 { + signingName = service + signingNameDerived = true + } + + hostname := e.Hostname // Offset the hostname for dualstack if enabled if opts.UseDualStack && e.HasDualStack == boxedTrue { hostname = e.DualStackHostname + region = signingRegion + } + + if !validateInputRegion(region) { + return ResolvedEndpoint{}, fmt.Errorf("invalid region identifier format provided") } u := strings.Replace(hostname, "{service}", service, 1) @@ -222,25 +273,14 @@ func (e endpoint) resolve(service, region, dnsSuffix string, defs []endpoint, op scheme := getEndpointScheme(e.Protocols, opts.DisableSSL) u = fmt.Sprintf("%s://%s", scheme, u) - signingRegion := e.CredentialScope.Region - if len(signingRegion) == 0 { - signingRegion = region - } - - signingName := e.CredentialScope.Service - var signingNameDerived bool - if len(signingName) == 0 { - signingName = service - signingNameDerived = true - } - return ResolvedEndpoint{ URL: u, + PartitionID: partitionID, SigningRegion: signingRegion, SigningName: signingName, SigningNameDerived: signingNameDerived, SigningMethod: getByPriority(e.SignatureVersions, signerPriority, defaultSigner), - } + }, nil } func getEndpointScheme(protocols []string, disableSSL bool) string { @@ -305,3 +345,7 @@ const ( boxedFalse boxedTrue ) + +func validateInputRegion(region string) bool { + return regionValidationRegex.MatchString(region) +} diff --git a/github.com/aws/aws-sdk-go/aws/request/connection_reset_error.go b/github.com/aws/aws-sdk-go/aws/request/connection_reset_error.go index 271da432ce..d9b37f4d32 100644 --- a/github.com/aws/aws-sdk-go/aws/request/connection_reset_error.go +++ b/github.com/aws/aws-sdk-go/aws/request/connection_reset_error.go @@ -1,18 +1,17 @@ -// +build !appengine,!plan9 - package request import ( - "net" - "os" - "syscall" + "strings" ) func isErrConnectionReset(err error) bool { - if opErr, ok := err.(*net.OpError); ok { - if sysErr, ok := opErr.Err.(*os.SyscallError); ok { - return sysErr.Err == syscall.ECONNRESET - } + if strings.Contains(err.Error(), "read: connection reset") { + return false + } + + if strings.Contains(err.Error(), "connection reset") || + strings.Contains(err.Error(), "broken pipe") { + return true } return false diff --git a/github.com/aws/aws-sdk-go/aws/request/connection_reset_error_other.go b/github.com/aws/aws-sdk-go/aws/request/connection_reset_error_other.go deleted file mode 100644 index daf9eca437..0000000000 --- a/github.com/aws/aws-sdk-go/aws/request/connection_reset_error_other.go +++ /dev/null @@ -1,11 +0,0 @@ -// +build appengine plan9 - -package request - -import ( - "strings" -) - -func isErrConnectionReset(err error) bool { - return strings.Contains(err.Error(), "connection reset") -} diff --git a/github.com/aws/aws-sdk-go/aws/request/handlers.go b/github.com/aws/aws-sdk-go/aws/request/handlers.go index 8ef8548a96..e819ab6c0e 100644 --- a/github.com/aws/aws-sdk-go/aws/request/handlers.go +++ b/github.com/aws/aws-sdk-go/aws/request/handlers.go @@ -10,6 +10,7 @@ import ( type Handlers struct { Validate HandlerList Build HandlerList + BuildStream HandlerList Sign HandlerList Send HandlerList ValidateResponse HandlerList @@ -23,11 +24,12 @@ type Handlers struct { Complete HandlerList } -// Copy returns of this handler's lists. +// Copy returns a copy of this handler's lists. func (h *Handlers) Copy() Handlers { return Handlers{ Validate: h.Validate.copy(), Build: h.Build.copy(), + BuildStream: h.BuildStream.copy(), Sign: h.Sign.copy(), Send: h.Send.copy(), ValidateResponse: h.ValidateResponse.copy(), @@ -42,10 +44,11 @@ func (h *Handlers) Copy() Handlers { } } -// Clear removes callback functions for all handlers +// Clear removes callback functions for all handlers. func (h *Handlers) Clear() { h.Validate.Clear() h.Build.Clear() + h.BuildStream.Clear() h.Send.Clear() h.Sign.Clear() h.Unmarshal.Clear() @@ -59,6 +62,54 @@ func (h *Handlers) Clear() { h.Complete.Clear() } +// IsEmpty returns if there are no handlers in any of the handlerlists. +func (h *Handlers) IsEmpty() bool { + if h.Validate.Len() != 0 { + return false + } + if h.Build.Len() != 0 { + return false + } + if h.BuildStream.Len() != 0 { + return false + } + if h.Send.Len() != 0 { + return false + } + if h.Sign.Len() != 0 { + return false + } + if h.Unmarshal.Len() != 0 { + return false + } + if h.UnmarshalStream.Len() != 0 { + return false + } + if h.UnmarshalMeta.Len() != 0 { + return false + } + if h.UnmarshalError.Len() != 0 { + return false + } + if h.ValidateResponse.Len() != 0 { + return false + } + if h.Retry.Len() != 0 { + return false + } + if h.AfterRetry.Len() != 0 { + return false + } + if h.CompleteAttempt.Len() != 0 { + return false + } + if h.Complete.Len() != 0 { + return false + } + + return true +} + // A HandlerListRunItem represents an entry in the HandlerList which // is being run. type HandlerListRunItem struct { @@ -275,3 +326,18 @@ func MakeAddToUserAgentFreeFormHandler(s string) func(*Request) { AddToUserAgent(r, s) } } + +// WithSetRequestHeaders updates the operation request's HTTP header to contain +// the header key value pairs provided. If the header key already exists in the +// request's HTTP header set, the existing value(s) will be replaced. +func WithSetRequestHeaders(h map[string]string) Option { + return withRequestHeader(h).SetRequestHeaders +} + +type withRequestHeader map[string]string + +func (h withRequestHeader) SetRequestHeaders(r *Request) { + for k, v := range h { + r.HTTPRequest.Header[k] = []string{v} + } +} diff --git a/github.com/aws/aws-sdk-go/aws/request/offset_reader.go b/github.com/aws/aws-sdk-go/aws/request/offset_reader.go index b0c2ef4fe6..9370fa50c3 100644 --- a/github.com/aws/aws-sdk-go/aws/request/offset_reader.go +++ b/github.com/aws/aws-sdk-go/aws/request/offset_reader.go @@ -15,12 +15,15 @@ type offsetReader struct { closed bool } -func newOffsetReader(buf io.ReadSeeker, offset int64) *offsetReader { +func newOffsetReader(buf io.ReadSeeker, offset int64) (*offsetReader, error) { reader := &offsetReader{} - buf.Seek(offset, sdkio.SeekStart) + _, err := buf.Seek(offset, sdkio.SeekStart) + if err != nil { + return nil, err + } reader.buf = buf - return reader + return reader, nil } // Close will close the instance of the offset reader's access to @@ -54,7 +57,9 @@ func (o *offsetReader) Seek(offset int64, whence int) (int64, error) { // CloseAndCopy will return a new offsetReader with a copy of the old buffer // and close the old buffer. -func (o *offsetReader) CloseAndCopy(offset int64) *offsetReader { - o.Close() +func (o *offsetReader) CloseAndCopy(offset int64) (*offsetReader, error) { + if err := o.Close(); err != nil { + return nil, err + } return newOffsetReader(o.buf, offset) } diff --git a/github.com/aws/aws-sdk-go/aws/request/request.go b/github.com/aws/aws-sdk-go/aws/request/request.go index b716684650..d597c6ead5 100644 --- a/github.com/aws/aws-sdk-go/aws/request/request.go +++ b/github.com/aws/aws-sdk-go/aws/request/request.go @@ -4,7 +4,6 @@ import ( "bytes" "fmt" "io" - "net" "net/http" "net/url" "reflect" @@ -37,6 +36,10 @@ const ( // API request that was canceled. Requests given a aws.Context may // return this error when canceled. CanceledErrorCode = "RequestCanceled" + + // ErrCodeRequestError is an error preventing the SDK from continuing to + // process the request. + ErrCodeRequestError = "RequestError" ) // A Request is the service request to be made. @@ -52,6 +55,7 @@ type Request struct { HTTPRequest *http.Request HTTPResponse *http.Response Body io.ReadSeeker + streamingBody io.ReadCloser BodyStart int64 // offset from beginning of Body that the request body starts Params interface{} Error error @@ -65,6 +69,15 @@ type Request struct { LastSignedAt time.Time DisableFollowRedirects bool + // Additional API error codes that should be retried. IsErrorRetryable + // will consider these codes in addition to its built in cases. + RetryErrorCodes []string + + // Additional API error codes that should be retried with throttle backoff + // delay. IsErrorThrottle will consider these codes in addition to its + // built in cases. + ThrottleErrorCodes []string + // A value greater than 0 instructs the request to be signed as Presigned URL // You should not set this field directly. Instead use Request's // Presign or PresignRequest methods. @@ -91,8 +104,12 @@ type Operation struct { BeforePresignFn func(r *Request) error } -// New returns a new Request pointer for the service API -// operation and parameters. +// New returns a new Request pointer for the service API operation and +// parameters. +// +// A Retryer should be provided to direct how the request is retried. If +// Retryer is nil, a default no retry value will be used. You can use +// NoOpRetryer in the Client package to disable retry behavior directly. // // Params is any value of input parameters to be the request payload. // Data is pointer value to an object which the request's response @@ -100,6 +117,10 @@ type Operation struct { func New(cfg aws.Config, clientInfo metadata.ClientInfo, handlers Handlers, retryer Retryer, operation *Operation, params interface{}, data interface{}) *Request { + if retryer == nil { + retryer = noOpRetryer{} + } + method := operation.HTTPMethod if method == "" { method = "POST" @@ -114,8 +135,6 @@ func New(cfg aws.Config, clientInfo metadata.ClientInfo, handlers Handlers, err = awserr.New("InvalidEndpointURL", "invalid endpoint uri", err) } - SanitizeHostForHeader(httpReq) - r := &Request{ Config: cfg, ClientInfo: clientInfo, @@ -232,6 +251,10 @@ func (r *Request) WillRetry() bool { return r.Error != nil && aws.BoolValue(r.Retryable) && r.RetryCount < r.MaxRetries() } +func fmtAttemptCount(retryCount, maxRetries int) string { + return fmt.Sprintf("attempt %v/%v", retryCount, maxRetries) +} + // ParamsFilled returns if the request's parameters have been populated // and the parameters are valid. False is returned if no parameters are // provided or invalid. @@ -260,10 +283,28 @@ func (r *Request) SetStringBody(s string) { // SetReaderBody will set the request's body reader. func (r *Request) SetReaderBody(reader io.ReadSeeker) { r.Body = reader - r.BodyStart, _ = reader.Seek(0, sdkio.SeekCurrent) // Get the Bodies current offset. + + if aws.IsReaderSeekable(reader) { + var err error + // Get the Bodies current offset so retries will start from the same + // initial position. + r.BodyStart, err = reader.Seek(0, sdkio.SeekCurrent) + if err != nil { + r.Error = awserr.New(ErrCodeSerialization, + "failed to determine start of request body", err) + return + } + } r.ResetBody() } +// SetStreamingBody set the reader to be used for the request that will stream +// bytes to the server. Request's Body must not be set to any reader. +func (r *Request) SetStreamingBody(reader io.ReadCloser) { + r.streamingBody = reader + r.SetReaderBody(aws.ReadSeekCloser(reader)) +} + // Presign returns the request's signed URL. Error will be returned // if the signing fails. The expire parameter is only used for presigned Amazon // S3 API requests. All other AWS services will use a fixed expiration @@ -331,16 +372,15 @@ func getPresignedURL(r *Request, expire time.Duration) (string, http.Header, err return r.HTTPRequest.URL.String(), r.SignedHeaderVals, nil } -func debugLogReqError(r *Request, stage string, retrying bool, err error) { +const ( + notRetrying = "not retrying" +) + +func debugLogReqError(r *Request, stage, retryStr string, err error) { if !r.Config.LogLevel.Matches(aws.LogDebugWithRequestErrors) { return } - retryStr := "not retrying" - if retrying { - retryStr = "will retry" - } - r.Config.Logger.Log(fmt.Sprintf("DEBUG: %s %s/%s failed, %s, error %v", stage, r.ClientInfo.ServiceName, r.Operation.Name, retryStr, err)) } @@ -359,12 +399,12 @@ func (r *Request) Build() error { if !r.built { r.Handlers.Validate.Run(r) if r.Error != nil { - debugLogReqError(r, "Validate Request", false, r.Error) + debugLogReqError(r, "Validate Request", notRetrying, r.Error) return r.Error } r.Handlers.Build.Run(r) if r.Error != nil { - debugLogReqError(r, "Build Request", false, r.Error) + debugLogReqError(r, "Build Request", notRetrying, r.Error) return r.Error } r.built = true @@ -380,20 +420,30 @@ func (r *Request) Build() error { func (r *Request) Sign() error { r.Build() if r.Error != nil { - debugLogReqError(r, "Build Request", false, r.Error) + debugLogReqError(r, "Build Request", notRetrying, r.Error) return r.Error } + SanitizeHostForHeader(r.HTTPRequest) + r.Handlers.Sign.Run(r) return r.Error } -func (r *Request) getNextRequestBody() (io.ReadCloser, error) { +func (r *Request) getNextRequestBody() (body io.ReadCloser, err error) { + if r.streamingBody != nil { + return r.streamingBody, nil + } + if r.safeBody != nil { r.safeBody.Close() } - r.safeBody = newOffsetReader(r.Body, r.BodyStart) + r.safeBody, err = newOffsetReader(r.Body, r.BodyStart) + if err != nil { + return nil, awserr.New(ErrCodeSerialization, + "failed to get next request body reader", err) + } // Go 1.8 tightened and clarified the rules code needs to use when building // requests with the http package. Go 1.8 removed the automatic detection @@ -410,10 +460,10 @@ func (r *Request) getNextRequestBody() (io.ReadCloser, error) { // Related golang/go#18257 l, err := aws.SeekerLen(r.Body) if err != nil { - return nil, awserr.New(ErrCodeSerialization, "failed to compute request body size", err) + return nil, awserr.New(ErrCodeSerialization, + "failed to compute request body size", err) } - var body io.ReadCloser if l == 0 { body = NoBody } else if l > 0 { @@ -474,29 +524,28 @@ func (r *Request) Send() error { r.AttemptTime = time.Now() if err := r.Sign(); err != nil { - debugLogReqError(r, "Sign Request", false, err) + debugLogReqError(r, "Sign Request", notRetrying, err) return err } if err := r.sendRequest(); err == nil { return nil - } else if !shouldRetryCancel(r) { - return err - } else { - r.Handlers.Retry.Run(r) - r.Handlers.AfterRetry.Run(r) + } + r.Handlers.Retry.Run(r) + r.Handlers.AfterRetry.Run(r) - if r.Error != nil || !aws.BoolValue(r.Retryable) { - return r.Error - } + if r.Error != nil || !aws.BoolValue(r.Retryable) { + return r.Error + } - r.prepareRetry() - continue + if err := r.prepareRetry(); err != nil { + r.Error = err + return err } } } -func (r *Request) prepareRetry() { +func (r *Request) prepareRetry() error { if r.Config.LogLevel.Matches(aws.LogDebugWithRequestRetries) { r.Config.Logger.Log(fmt.Sprintf("DEBUG: Retrying Request %s/%s, attempt %d", r.ClientInfo.ServiceName, r.Operation.Name, r.RetryCount)) @@ -507,12 +556,19 @@ func (r *Request) prepareRetry() { // the request's body even though the Client's Do returned. r.HTTPRequest = copyHTTPRequest(r.HTTPRequest, nil) r.ResetBody() + if err := r.Error; err != nil { + return awserr.New(ErrCodeSerialization, + "failed to prepare body for retry", err) + + } // Closing response body to ensure that no response body is leaked // between retry attempts. if r.HTTPResponse != nil && r.HTTPResponse.Body != nil { r.HTTPResponse.Body.Close() } + + return nil } func (r *Request) sendRequest() (sendErr error) { @@ -521,7 +577,9 @@ func (r *Request) sendRequest() (sendErr error) { r.Retryable = nil r.Handlers.Send.Run(r) if r.Error != nil { - debugLogReqError(r, "Send Request", r.WillRetry(), r.Error) + debugLogReqError(r, "Send Request", + fmtAttemptCount(r.RetryCount, r.MaxRetries()), + r.Error) return r.Error } @@ -529,13 +587,17 @@ func (r *Request) sendRequest() (sendErr error) { r.Handlers.ValidateResponse.Run(r) if r.Error != nil { r.Handlers.UnmarshalError.Run(r) - debugLogReqError(r, "Validate Response", r.WillRetry(), r.Error) + debugLogReqError(r, "Validate Response", + fmtAttemptCount(r.RetryCount, r.MaxRetries()), + r.Error) return r.Error } r.Handlers.Unmarshal.Run(r) if r.Error != nil { - debugLogReqError(r, "Unmarshal Response", r.WillRetry(), r.Error) + debugLogReqError(r, "Unmarshal Response", + fmtAttemptCount(r.RetryCount, r.MaxRetries()), + r.Error) return r.Error } @@ -562,32 +624,6 @@ func AddToUserAgent(r *Request, s string) { r.HTTPRequest.Header.Set("User-Agent", s) } -func shouldRetryCancel(r *Request) bool { - awsErr, ok := r.Error.(awserr.Error) - timeoutErr := false - errStr := r.Error.Error() - if ok { - if awsErr.Code() == CanceledErrorCode { - return false - } - err := awsErr.OrigErr() - netErr, netOK := err.(net.Error) - timeoutErr = netOK && netErr.Temporary() - if urlErr, ok := err.(*url.Error); !timeoutErr && ok { - errStr = urlErr.Err.Error() - } - } - - // There can be two types of canceled errors here. - // The first being a net.Error and the other being an error. - // If the request was timed out, we want to continue the retry - // process. Otherwise, return the canceled error. - return timeoutErr || - (errStr != "net/http: request canceled" && - errStr != "net/http: request canceled while waiting for connection") - -} - // SanitizeHostForHeader removes default port from host and updates request.Host func SanitizeHostForHeader(r *http.Request) { host := getHost(r) @@ -603,6 +639,10 @@ func getHost(r *http.Request) string { return r.Host } + if r.URL == nil { + return "" + } + return r.URL.Host } diff --git a/github.com/aws/aws-sdk-go/aws/request/request_1_8.go b/github.com/aws/aws-sdk-go/aws/request/request_1_8.go index 7c6a8000f6..de1292f45a 100644 --- a/github.com/aws/aws-sdk-go/aws/request/request_1_8.go +++ b/github.com/aws/aws-sdk-go/aws/request/request_1_8.go @@ -4,6 +4,8 @@ package request import ( "net/http" + + "github.com/aws/aws-sdk-go/aws/awserr" ) // NoBody is a http.NoBody reader instructing Go HTTP client to not include @@ -24,7 +26,8 @@ var NoBody = http.NoBody func (r *Request) ResetBody() { body, err := r.getNextRequestBody() if err != nil { - r.Error = err + r.Error = awserr.New(ErrCodeSerialization, + "failed to reset request body", err) return } diff --git a/github.com/aws/aws-sdk-go/aws/request/request_pagination.go b/github.com/aws/aws-sdk-go/aws/request/request_pagination.go index a633ed5acf..64784e16f3 100644 --- a/github.com/aws/aws-sdk-go/aws/request/request_pagination.go +++ b/github.com/aws/aws-sdk-go/aws/request/request_pagination.go @@ -17,11 +17,13 @@ import ( // does the pagination between API operations, and Paginator defines the // configuration that will be used per page request. // -// cont := true -// for p.Next() && cont { +// for p.Next() { // data := p.Page().(*s3.ListObjectsOutput) // // process the page's data +// // ... +// // break out of loop to stop fetching additional pages // } +// // return p.Err() // // See service client API operation Pages methods for examples how the SDK will @@ -146,7 +148,7 @@ func (r *Request) nextPageTokens() []interface{} { return nil } case bool: - if v == false { + if !v { return nil } } diff --git a/github.com/aws/aws-sdk-go/aws/request/retryer.go b/github.com/aws/aws-sdk-go/aws/request/retryer.go index 7bc5da7826..752ae47f84 100644 --- a/github.com/aws/aws-sdk-go/aws/request/retryer.go +++ b/github.com/aws/aws-sdk-go/aws/request/retryer.go @@ -1,32 +1,81 @@ package request import ( + "net" + "net/url" + "strings" "time" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/awserr" ) -// Retryer is an interface to control retry logic for a given service. -// The default implementation used by most services is the client.DefaultRetryer -// structure, which contains basic retry logic using exponential backoff. +// Retryer provides the interface drive the SDK's request retry behavior. The +// Retryer implementation is responsible for implementing exponential backoff, +// and determine if a request API error should be retried. +// +// client.DefaultRetryer is the SDK's default implementation of the Retryer. It +// uses the which uses the Request.IsErrorRetryable and Request.IsErrorThrottle +// methods to determine if the request is retried. type Retryer interface { + // RetryRules return the retry delay that should be used by the SDK before + // making another request attempt for the failed request. RetryRules(*Request) time.Duration + + // ShouldRetry returns if the failed request is retryable. + // + // Implementations may consider request attempt count when determining if a + // request is retryable, but the SDK will use MaxRetries to limit the + // number of attempts a request are made. ShouldRetry(*Request) bool + + // MaxRetries is the number of times a request may be retried before + // failing. MaxRetries() int } -// WithRetryer sets a config Retryer value to the given Config returning it -// for chaining. +// WithRetryer sets a Retryer value to the given Config returning the Config +// value for chaining. The value must not be nil. func WithRetryer(cfg *aws.Config, retryer Retryer) *aws.Config { + if retryer == nil { + if cfg.Logger != nil { + cfg.Logger.Log("ERROR: Request.WithRetryer called with nil retryer. Replacing with retry disabled Retryer.") + } + retryer = noOpRetryer{} + } cfg.Retryer = retryer return cfg + +} + +// noOpRetryer is a internal no op retryer used when a request is created +// without a retryer. +// +// Provides a retryer that performs no retries. +// It should be used when we do not want retries to be performed. +type noOpRetryer struct{} + +// MaxRetries returns the number of maximum returns the service will use to make +// an individual API; For NoOpRetryer the MaxRetries will always be zero. +func (d noOpRetryer) MaxRetries() int { + return 0 +} + +// ShouldRetry will always return false for NoOpRetryer, as it should never retry. +func (d noOpRetryer) ShouldRetry(_ *Request) bool { + return false +} + +// RetryRules returns the delay duration before retrying this request again; +// since NoOpRetryer does not retry, RetryRules always returns 0. +func (d noOpRetryer) RetryRules(_ *Request) time.Duration { + return 0 } // retryableCodes is a collection of service response codes which are retry-able // without any further action. var retryableCodes = map[string]struct{}{ - "RequestError": {}, + ErrCodeRequestError: {}, "RequestTimeout": {}, ErrCodeResponseTimeout: {}, "RequestTimeoutException": {}, // Glacier's flavor of RequestTimeout @@ -34,13 +83,16 @@ var retryableCodes = map[string]struct{}{ var throttleCodes = map[string]struct{}{ "ProvisionedThroughputExceededException": {}, + "ThrottledException": {}, // SNS, XRay, ResourceGroupsTagging API "Throttling": {}, "ThrottlingException": {}, "RequestLimitExceeded": {}, "RequestThrottled": {}, + "RequestThrottledException": {}, "TooManyRequestsException": {}, // Lambda functions "PriorRequestNotComplete": {}, // Route53 "TransactionInProgressException": {}, + "EC2ThrottledException": {}, // EC2 } // credsExpiredCodes is a collection of error codes which signify the credentials @@ -75,10 +127,6 @@ var validParentCodes = map[string]struct{}{ ErrCodeRead: {}, } -type temporaryError interface { - Temporary() bool -} - func isNestedErrorRetryable(parentErr awserr.Error) bool { if parentErr == nil { return false @@ -97,7 +145,7 @@ func isNestedErrorRetryable(parentErr awserr.Error) bool { return isCodeRetryable(aerr.Code()) } - if t, ok := err.(temporaryError); ok { + if t, ok := err.(temporary); ok { return t.Temporary() || isErrConnectionReset(err) } @@ -107,32 +155,90 @@ func isNestedErrorRetryable(parentErr awserr.Error) bool { // IsErrorRetryable returns whether the error is retryable, based on its Code. // Returns false if error is nil. func IsErrorRetryable(err error) bool { - if err != nil { - if aerr, ok := err.(awserr.Error); ok { - return isCodeRetryable(aerr.Code()) || isNestedErrorRetryable(aerr) + if err == nil { + return false + } + return shouldRetryError(err) +} + +type temporary interface { + Temporary() bool +} + +func shouldRetryError(origErr error) bool { + switch err := origErr.(type) { + case awserr.Error: + if err.Code() == CanceledErrorCode { + return false } + if isNestedErrorRetryable(err) { + return true + } + + origErr := err.OrigErr() + var shouldRetry bool + if origErr != nil { + shouldRetry = shouldRetryError(origErr) + if err.Code() == ErrCodeRequestError && !shouldRetry { + return false + } + } + if isCodeRetryable(err.Code()) { + return true + } + return shouldRetry + + case *url.Error: + if strings.Contains(err.Error(), "connection refused") { + // Refused connections should be retried as the service may not yet + // be running on the port. Go TCP dial considers refused + // connections as not temporary. + return true + } + // *url.Error only implements Temporary after golang 1.6 but since + // url.Error only wraps the error: + return shouldRetryError(err.Err) + + case temporary: + if netErr, ok := err.(*net.OpError); ok && netErr.Op == "dial" { + return true + } + // If the error is temporary, we want to allow continuation of the + // retry process + return err.Temporary() || isErrConnectionReset(origErr) + + case nil: + // `awserr.Error.OrigErr()` can be nil, meaning there was an error but + // because we don't know the cause, it is marked as retryable. See + // TestRequest4xxUnretryable for an example. + return true + + default: + switch err.Error() { + case "net/http: request canceled", + "net/http: request canceled while waiting for connection": + // known 1.5 error case when an http request is cancelled + return false + } + // here we don't know the error; so we allow a retry. + return true } - return false } // IsErrorThrottle returns whether the error is to be throttled based on its code. // Returns false if error is nil. func IsErrorThrottle(err error) bool { - if err != nil { - if aerr, ok := err.(awserr.Error); ok { - return isCodeThrottle(aerr.Code()) - } + if aerr, ok := err.(awserr.Error); ok && aerr != nil { + return isCodeThrottle(aerr.Code()) } return false } -// IsErrorExpiredCreds returns whether the error code is a credential expiry error. -// Returns false if error is nil. +// IsErrorExpiredCreds returns whether the error code is a credential expiry +// error. Returns false if error is nil. func IsErrorExpiredCreds(err error) bool { - if err != nil { - if aerr, ok := err.(awserr.Error); ok { - return isCodeExpiredCreds(aerr.Code()) - } + if aerr, ok := err.(awserr.Error); ok && aerr != nil { + return isCodeExpiredCreds(aerr.Code()) } return false } @@ -142,17 +248,58 @@ func IsErrorExpiredCreds(err error) bool { // // Alias for the utility function IsErrorRetryable func (r *Request) IsErrorRetryable() bool { + if isErrCode(r.Error, r.RetryErrorCodes) { + return true + } + + // HTTP response status code 501 should not be retried. + // 501 represents Not Implemented which means the request method is not + // supported by the server and cannot be handled. + if r.HTTPResponse != nil { + // HTTP response status code 500 represents internal server error and + // should be retried without any throttle. + if r.HTTPResponse.StatusCode == 500 { + return true + } + } return IsErrorRetryable(r.Error) } -// IsErrorThrottle returns whether the error is to be throttled based on its code. -// Returns false if the request has no Error set +// IsErrorThrottle returns whether the error is to be throttled based on its +// code. Returns false if the request has no Error set. // // Alias for the utility function IsErrorThrottle func (r *Request) IsErrorThrottle() bool { + if isErrCode(r.Error, r.ThrottleErrorCodes) { + return true + } + + if r.HTTPResponse != nil { + switch r.HTTPResponse.StatusCode { + case + 429, // error caused due to too many requests + 502, // Bad Gateway error should be throttled + 503, // caused when service is unavailable + 504: // error occurred due to gateway timeout + return true + } + } + return IsErrorThrottle(r.Error) } +func isErrCode(err error, codes []string) bool { + if aerr, ok := err.(awserr.Error); ok && aerr != nil { + for _, code := range codes { + if code == aerr.Code() { + return true + } + } + } + + return false +} + // IsErrorExpired returns whether the error code is a credential expiry error. // Returns false if the request has no Error set. // diff --git a/github.com/aws/aws-sdk-go/aws/session/cabundle_transport.go b/github.com/aws/aws-sdk-go/aws/session/cabundle_transport.go new file mode 100644 index 0000000000..ea9ebb6f6a --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/session/cabundle_transport.go @@ -0,0 +1,26 @@ +// +build go1.7 + +package session + +import ( + "net" + "net/http" + "time" +) + +// Transport that should be used when a custom CA bundle is specified with the +// SDK. +func getCABundleTransport() *http.Transport { + return &http.Transport{ + Proxy: http.ProxyFromEnvironment, + DialContext: (&net.Dialer{ + Timeout: 30 * time.Second, + KeepAlive: 30 * time.Second, + DualStack: true, + }).DialContext, + MaxIdleConns: 100, + IdleConnTimeout: 90 * time.Second, + TLSHandshakeTimeout: 10 * time.Second, + ExpectContinueTimeout: 1 * time.Second, + } +} diff --git a/github.com/aws/aws-sdk-go/aws/session/cabundle_transport_1_5.go b/github.com/aws/aws-sdk-go/aws/session/cabundle_transport_1_5.go new file mode 100644 index 0000000000..fec39dfc12 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/session/cabundle_transport_1_5.go @@ -0,0 +1,22 @@ +// +build !go1.6,go1.5 + +package session + +import ( + "net" + "net/http" + "time" +) + +// Transport that should be used when a custom CA bundle is specified with the +// SDK. +func getCABundleTransport() *http.Transport { + return &http.Transport{ + Proxy: http.ProxyFromEnvironment, + Dial: (&net.Dialer{ + Timeout: 30 * time.Second, + KeepAlive: 30 * time.Second, + }).Dial, + TLSHandshakeTimeout: 10 * time.Second, + } +} diff --git a/github.com/aws/aws-sdk-go/aws/session/cabundle_transport_1_6.go b/github.com/aws/aws-sdk-go/aws/session/cabundle_transport_1_6.go new file mode 100644 index 0000000000..1c5a5391e6 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/session/cabundle_transport_1_6.go @@ -0,0 +1,23 @@ +// +build !go1.7,go1.6 + +package session + +import ( + "net" + "net/http" + "time" +) + +// Transport that should be used when a custom CA bundle is specified with the +// SDK. +func getCABundleTransport() *http.Transport { + return &http.Transport{ + Proxy: http.ProxyFromEnvironment, + Dial: (&net.Dialer{ + Timeout: 30 * time.Second, + KeepAlive: 30 * time.Second, + }).Dial, + TLSHandshakeTimeout: 10 * time.Second, + ExpectContinueTimeout: 1 * time.Second, + } +} diff --git a/github.com/aws/aws-sdk-go/aws/session/credentials.go b/github.com/aws/aws-sdk-go/aws/session/credentials.go new file mode 100644 index 0000000000..fe6dac1f47 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/session/credentials.go @@ -0,0 +1,267 @@ +package session + +import ( + "fmt" + "os" + "time" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/aws/credentials" + "github.com/aws/aws-sdk-go/aws/credentials/processcreds" + "github.com/aws/aws-sdk-go/aws/credentials/stscreds" + "github.com/aws/aws-sdk-go/aws/defaults" + "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/internal/shareddefaults" +) + +func resolveCredentials(cfg *aws.Config, + envCfg envConfig, sharedCfg sharedConfig, + handlers request.Handlers, + sessOpts Options, +) (*credentials.Credentials, error) { + + switch { + case len(sessOpts.Profile) != 0: + // User explicitly provided an Profile in the session's configuration + // so load that profile from shared config first. + // Github(aws/aws-sdk-go#2727) + return resolveCredsFromProfile(cfg, envCfg, sharedCfg, handlers, sessOpts) + + case envCfg.Creds.HasKeys(): + // Environment credentials + return credentials.NewStaticCredentialsFromCreds(envCfg.Creds), nil + + case len(envCfg.WebIdentityTokenFilePath) != 0: + // Web identity token from environment, RoleARN required to also be + // set. + return assumeWebIdentity(cfg, handlers, + envCfg.WebIdentityTokenFilePath, + envCfg.RoleARN, + envCfg.RoleSessionName, + ) + + default: + // Fallback to the "default" credential resolution chain. + return resolveCredsFromProfile(cfg, envCfg, sharedCfg, handlers, sessOpts) + } +} + +// WebIdentityEmptyRoleARNErr will occur if 'AWS_WEB_IDENTITY_TOKEN_FILE' was set but +// 'AWS_ROLE_ARN' was not set. +var WebIdentityEmptyRoleARNErr = awserr.New(stscreds.ErrCodeWebIdentity, "role ARN is not set", nil) + +// WebIdentityEmptyTokenFilePathErr will occur if 'AWS_ROLE_ARN' was set but +// 'AWS_WEB_IDENTITY_TOKEN_FILE' was not set. +var WebIdentityEmptyTokenFilePathErr = awserr.New(stscreds.ErrCodeWebIdentity, "token file path is not set", nil) + +func assumeWebIdentity(cfg *aws.Config, handlers request.Handlers, + filepath string, + roleARN, sessionName string, +) (*credentials.Credentials, error) { + + if len(filepath) == 0 { + return nil, WebIdentityEmptyTokenFilePathErr + } + + if len(roleARN) == 0 { + return nil, WebIdentityEmptyRoleARNErr + } + + creds := stscreds.NewWebIdentityCredentials( + &Session{ + Config: cfg, + Handlers: handlers.Copy(), + }, + roleARN, + sessionName, + filepath, + ) + + return creds, nil +} + +func resolveCredsFromProfile(cfg *aws.Config, + envCfg envConfig, sharedCfg sharedConfig, + handlers request.Handlers, + sessOpts Options, +) (creds *credentials.Credentials, err error) { + + switch { + case sharedCfg.SourceProfile != nil: + // Assume IAM role with credentials source from a different profile. + creds, err = resolveCredsFromProfile(cfg, envCfg, + *sharedCfg.SourceProfile, handlers, sessOpts, + ) + + case sharedCfg.Creds.HasKeys(): + // Static Credentials from Shared Config/Credentials file. + creds = credentials.NewStaticCredentialsFromCreds( + sharedCfg.Creds, + ) + + case len(sharedCfg.CredentialProcess) != 0: + // Get credentials from CredentialProcess + creds = processcreds.NewCredentials(sharedCfg.CredentialProcess) + + case len(sharedCfg.CredentialSource) != 0: + creds, err = resolveCredsFromSource(cfg, envCfg, + sharedCfg, handlers, sessOpts, + ) + + case len(sharedCfg.WebIdentityTokenFile) != 0: + // Credentials from Assume Web Identity token require an IAM Role, and + // that roll will be assumed. May be wrapped with another assume role + // via SourceProfile. + return assumeWebIdentity(cfg, handlers, + sharedCfg.WebIdentityTokenFile, + sharedCfg.RoleARN, + sharedCfg.RoleSessionName, + ) + + default: + // Fallback to default credentials provider, include mock errors for + // the credential chain so user can identify why credentials failed to + // be retrieved. + creds = credentials.NewCredentials(&credentials.ChainProvider{ + VerboseErrors: aws.BoolValue(cfg.CredentialsChainVerboseErrors), + Providers: []credentials.Provider{ + &credProviderError{ + Err: awserr.New("EnvAccessKeyNotFound", + "failed to find credentials in the environment.", nil), + }, + &credProviderError{ + Err: awserr.New("SharedCredsLoad", + fmt.Sprintf("failed to load profile, %s.", envCfg.Profile), nil), + }, + defaults.RemoteCredProvider(*cfg, handlers), + }, + }) + } + if err != nil { + return nil, err + } + + if len(sharedCfg.RoleARN) > 0 { + cfgCp := *cfg + cfgCp.Credentials = creds + return credsFromAssumeRole(cfgCp, handlers, sharedCfg, sessOpts) + } + + return creds, nil +} + +// valid credential source values +const ( + credSourceEc2Metadata = "Ec2InstanceMetadata" + credSourceEnvironment = "Environment" + credSourceECSContainer = "EcsContainer" +) + +func resolveCredsFromSource(cfg *aws.Config, + envCfg envConfig, sharedCfg sharedConfig, + handlers request.Handlers, + sessOpts Options, +) (creds *credentials.Credentials, err error) { + + switch sharedCfg.CredentialSource { + case credSourceEc2Metadata: + p := defaults.RemoteCredProvider(*cfg, handlers) + creds = credentials.NewCredentials(p) + + case credSourceEnvironment: + creds = credentials.NewStaticCredentialsFromCreds(envCfg.Creds) + + case credSourceECSContainer: + if len(os.Getenv(shareddefaults.ECSCredsProviderEnvVar)) == 0 { + return nil, ErrSharedConfigECSContainerEnvVarEmpty + } + + p := defaults.RemoteCredProvider(*cfg, handlers) + creds = credentials.NewCredentials(p) + + default: + return nil, ErrSharedConfigInvalidCredSource + } + + return creds, nil +} + +func credsFromAssumeRole(cfg aws.Config, + handlers request.Handlers, + sharedCfg sharedConfig, + sessOpts Options, +) (*credentials.Credentials, error) { + + if len(sharedCfg.MFASerial) != 0 && sessOpts.AssumeRoleTokenProvider == nil { + // AssumeRole Token provider is required if doing Assume Role + // with MFA. + return nil, AssumeRoleTokenProviderNotSetError{} + } + + return stscreds.NewCredentials( + &Session{ + Config: &cfg, + Handlers: handlers.Copy(), + }, + sharedCfg.RoleARN, + func(opt *stscreds.AssumeRoleProvider) { + opt.RoleSessionName = sharedCfg.RoleSessionName + + if sessOpts.AssumeRoleDuration == 0 && + sharedCfg.AssumeRoleDuration != nil && + *sharedCfg.AssumeRoleDuration/time.Minute > 15 { + opt.Duration = *sharedCfg.AssumeRoleDuration + } else if sessOpts.AssumeRoleDuration != 0 { + opt.Duration = sessOpts.AssumeRoleDuration + } + + // Assume role with external ID + if len(sharedCfg.ExternalID) > 0 { + opt.ExternalID = aws.String(sharedCfg.ExternalID) + } + + // Assume role with MFA + if len(sharedCfg.MFASerial) > 0 { + opt.SerialNumber = aws.String(sharedCfg.MFASerial) + opt.TokenProvider = sessOpts.AssumeRoleTokenProvider + } + }, + ), nil +} + +// AssumeRoleTokenProviderNotSetError is an error returned when creating a +// session when the MFAToken option is not set when shared config is configured +// load assume a role with an MFA token. +type AssumeRoleTokenProviderNotSetError struct{} + +// Code is the short id of the error. +func (e AssumeRoleTokenProviderNotSetError) Code() string { + return "AssumeRoleTokenProviderNotSetError" +} + +// Message is the description of the error +func (e AssumeRoleTokenProviderNotSetError) Message() string { + return fmt.Sprintf("assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.") +} + +// OrigErr is the underlying error that caused the failure. +func (e AssumeRoleTokenProviderNotSetError) OrigErr() error { + return nil +} + +// Error satisfies the error interface. +func (e AssumeRoleTokenProviderNotSetError) Error() string { + return awserr.SprintError(e.Code(), e.Message(), "", nil) +} + +type credProviderError struct { + Err error +} + +func (c credProviderError) Retrieve() (credentials.Value, error) { + return credentials.Value{}, c.Err +} +func (c credProviderError) IsExpired() bool { + return true +} diff --git a/github.com/aws/aws-sdk-go/aws/session/doc.go b/github.com/aws/aws-sdk-go/aws/session/doc.go index 38a7b05a62..7ec66e7e58 100644 --- a/github.com/aws/aws-sdk-go/aws/session/doc.go +++ b/github.com/aws/aws-sdk-go/aws/session/doc.go @@ -1,97 +1,93 @@ /* -Package session provides configuration for the SDK's service clients. - -Sessions can be shared across all service clients that share the same base -configuration. The Session is built from the SDK's default configuration and -request handlers. - -Sessions should be cached when possible, because creating a new Session will -load all configuration values from the environment, and config files each time -the Session is created. Sharing the Session value across all of your service -clients will ensure the configuration is loaded the fewest number of times possible. - -Concurrency +Package session provides configuration for the SDK's service clients. Sessions +can be shared across service clients that share the same base configuration. Sessions are safe to use concurrently as long as the Session is not being -modified. The SDK will not modify the Session once the Session has been created. -Creating service clients concurrently from a shared Session is safe. - -Sessions from Shared Config - -Sessions can be created using the method above that will only load the -additional config if the AWS_SDK_LOAD_CONFIG environment variable is set. -Alternatively you can explicitly create a Session with shared config enabled. -To do this you can use NewSessionWithOptions to configure how the Session will -be created. Using the NewSessionWithOptions with SharedConfigState set to -SharedConfigEnable will create the session as if the AWS_SDK_LOAD_CONFIG -environment variable was set. +modified. Sessions should be cached when possible, because creating a new +Session will load all configuration values from the environment, and config +files each time the Session is created. Sharing the Session value across all of +your service clients will ensure the configuration is loaded the fewest number +of times possible. -Creating Sessions - -When creating Sessions optional aws.Config values can be passed in that will -override the default, or loaded config values the Session is being created -with. This allows you to provide additional, or case based, configuration -as needed. +Sessions options from Shared Config By default NewSession will only load credentials from the shared credentials file (~/.aws/credentials). If the AWS_SDK_LOAD_CONFIG environment variable is set to a truthy value the Session will be created from the configuration values from the shared config (~/.aws/config) and shared credentials -(~/.aws/credentials) files. See the section Sessions from Shared Config for -more information. +(~/.aws/credentials) files. Using the NewSessionWithOptions with +SharedConfigState set to SharedConfigEnable will create the session as if the +AWS_SDK_LOAD_CONFIG environment variable was set. -Create a Session with the default config and request handlers. With credentials -region, and profile loaded from the environment and shared config automatically. -Requires the AWS_PROFILE to be set, or "default" is used. +Credential and config loading order - // Create Session - sess := session.Must(session.NewSession()) +The Session will attempt to load configuration and credentials from the +environment, configuration files, and other credential sources. The order +configuration is loaded in is: - // Create a Session with a custom region - sess := session.Must(session.NewSession(&aws.Config{ - Region: aws.String("us-east-1"), - })) + * Environment Variables + * Shared Credentials file + * Shared Configuration file (if SharedConfig is enabled) + * EC2 Instance Metadata (credentials only) - // Create a S3 client instance from a session - sess := session.Must(session.NewSession()) +The Environment variables for credentials will have precedence over shared +config even if SharedConfig is enabled. To override this behavior, and use +shared config credentials instead specify the session.Options.Profile, (e.g. +when using credential_source=Environment to assume a role). + + sess, err := session.NewSessionWithOptions(session.Options{ + Profile: "myProfile", + }) - svc := s3.New(sess) +Creating Sessions -Create Session With Option Overrides +Creating a Session without additional options will load credentials region, and +profile loaded from the environment and shared config automatically. See, +"Environment Variables" section for information on environment variables used +by Session. -In addition to NewSession, Sessions can be created using NewSessionWithOptions. -This func allows you to control and override how the Session will be created -through code instead of being driven by environment variables only. + // Create Session + sess, err := session.NewSession() -Use NewSessionWithOptions when you want to provide the config profile, or -override the shared config state (AWS_SDK_LOAD_CONFIG). + +When creating Sessions optional aws.Config values can be passed in that will +override the default, or loaded, config values the Session is being created +with. This allows you to provide additional, or case based, configuration +as needed. + + // Create a Session with a custom region + sess, err := session.NewSession(&aws.Config{ + Region: aws.String("us-west-2"), + }) + +Use NewSessionWithOptions to provide additional configuration driving how the +Session's configuration will be loaded. Such as, specifying shared config +profile, or override the shared config state, (AWS_SDK_LOAD_CONFIG). // Equivalent to session.NewSession() - sess := session.Must(session.NewSessionWithOptions(session.Options{ + sess, err := session.NewSessionWithOptions(session.Options{ // Options - })) + }) - // Specify profile to load for the session's config - sess := session.Must(session.NewSessionWithOptions(session.Options{ - Profile: "profile_name", - })) + sess, err := session.NewSessionWithOptions(session.Options{ + // Specify profile to load for the session's config + Profile: "profile_name", - // Specify profile for config and region for requests - sess := session.Must(session.NewSessionWithOptions(session.Options{ - Config: aws.Config{Region: aws.String("us-east-1")}, - Profile: "profile_name", - })) + // Provide SDK Config options, such as Region. + Config: aws.Config{ + Region: aws.String("us-west-2"), + }, - // Force enable Shared Config support - sess := session.Must(session.NewSessionWithOptions(session.Options{ + // Force enable Shared Config support SharedConfigState: session.SharedConfigEnable, - })) + }) Adding Handlers -You can add handlers to a session for processing HTTP requests. All service -clients that use the session inherit the handlers. For example, the following -handler logs every request and its payload made by a service client: +You can add handlers to a session to decorate API operation, (e.g. adding HTTP +headers). All clients that use the Session receive a copy of the Session's +handlers. For example, the following request handler added to the Session logs +every requests made. // Create a session, and add additional handlers for all service // clients created with the Session to inherit. Adds logging handler. @@ -99,22 +95,15 @@ handler logs every request and its payload made by a service client: sess.Handlers.Send.PushFront(func(r *request.Request) { // Log every request made and its payload - logger.Printf("Request: %s/%s, Payload: %s", + logger.Printf("Request: %s/%s, Params: %s", r.ClientInfo.ServiceName, r.Operation, r.Params) }) -Deprecated "New" function - -The New session function has been deprecated because it does not provide good -way to return errors that occur when loading the configuration files and values. -Because of this, NewSession was created so errors can be retrieved when -creating a session fails. - Shared Config Fields -By default the SDK will only load the shared credentials file's (~/.aws/credentials) -credentials values, and all other config is provided by the environment variables, -SDK defaults, and user provided aws.Config values. +By default the SDK will only load the shared credentials file's +(~/.aws/credentials) credentials values, and all other config is provided by +the environment variables, SDK defaults, and user provided aws.Config values. If the AWS_SDK_LOAD_CONFIG environment variable is set, or SharedConfigEnable option is used to create the Session the full shared config values will be @@ -125,24 +114,31 @@ files have the same format. If both config files are present the configuration from both files will be read. The Session will be created from configuration values from the shared -credentials file (~/.aws/credentials) over those in the shared config file (~/.aws/config). +credentials file (~/.aws/credentials) over those in the shared config file +(~/.aws/config). -Credentials are the values the SDK should use for authenticating requests with -AWS Services. They are from a configuration file will need to include both -aws_access_key_id and aws_secret_access_key must be provided together in the -same file to be considered valid. The values will be ignored if not a complete -group. aws_session_token is an optional field that can be provided if both of -the other two fields are also provided. +Credentials are the values the SDK uses to authenticating requests with AWS +Services. When specified in a file, both aws_access_key_id and +aws_secret_access_key must be provided together in the same file to be +considered valid. They will be ignored if both are not present. +aws_session_token is an optional field that can be provided in addition to the +other two fields. aws_access_key_id = AKID aws_secret_access_key = SECRET aws_session_token = TOKEN -Assume Role values allow you to configure the SDK to assume an IAM role using -a set of credentials provided in a config file via the source_profile field. -Both "role_arn" and "source_profile" are required. The SDK supports assuming -a role with MFA token if the session option AssumeRoleTokenProvider -is set. + ; region only supported if SharedConfigEnabled. + region = us-east-1 + +Assume Role configuration + +The role_arn field allows you to configure the SDK to assume an IAM role using +a set of credentials from another source. Such as when paired with static +credentials, "profile_source", "credential_process", or "credential_source" +fields. If "role_arn" is provided, a source of credentials must also be +specified, such as "source_profile", "credential_source", or +"credential_process". role_arn = arn:aws:iam:::role/ source_profile = profile_with_creds @@ -150,40 +146,16 @@ is set. mfa_serial = role_session_name = session_name -Region is the region the SDK should use for looking up AWS service endpoints -and signing requests. - - region = us-east-1 - -Assume Role with MFA token -To create a session with support for assuming an IAM role with MFA set the -session option AssumeRoleTokenProvider to a function that will prompt for the -MFA token code when the SDK assumes the role and refreshes the role's credentials. -This allows you to configure the SDK via the shared config to assumea role -with MFA tokens. - -In order for the SDK to assume a role with MFA the SharedConfigState -session option must be set to SharedConfigEnable, or AWS_SDK_LOAD_CONFIG -environment variable set. - -The shared configuration instructs the SDK to assume an IAM role with MFA -when the mfa_serial configuration field is set in the shared config -(~/.aws/config) or shared credentials (~/.aws/credentials) file. - -If mfa_serial is set in the configuration, the SDK will assume the role, and -the AssumeRoleTokenProvider session option is not set an an error will -be returned when creating the session. +The SDK supports assuming a role with MFA token. If "mfa_serial" is set, you +must also set the Session Option.AssumeRoleTokenProvider. The Session will fail +to load if the AssumeRoleTokenProvider is not specified. sess := session.Must(session.NewSessionWithOptions(session.Options{ AssumeRoleTokenProvider: stscreds.StdinTokenProvider, })) - // Create service client value configured for credentials - // from assumed role. - svc := s3.New(sess) - -To setup assume role outside of a session see the stscreds.AssumeRoleProvider +To setup Assume Role outside of a session see the stscreds.AssumeRoleProvider documentation. Environment Variables diff --git a/github.com/aws/aws-sdk-go/aws/session/env_config.go b/github.com/aws/aws-sdk-go/aws/session/env_config.go index e3959b959e..c1e0e9c954 100644 --- a/github.com/aws/aws-sdk-go/aws/session/env_config.go +++ b/github.com/aws/aws-sdk-go/aws/session/env_config.go @@ -1,12 +1,15 @@ package session import ( + "fmt" "os" "strconv" + "strings" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/defaults" + "github.com/aws/aws-sdk-go/aws/endpoints" ) // EnvProviderName provides a name of the provider when config is loaded from environment. @@ -99,21 +102,61 @@ type envConfig struct { CustomCABundle string csmEnabled string - CSMEnabled bool + CSMEnabled *bool CSMPort string + CSMHost string CSMClientID string - enableEndpointDiscovery string // Enables endpoint discovery via environment variables. // // AWS_ENABLE_ENDPOINT_DISCOVERY=true EnableEndpointDiscovery *bool + enableEndpointDiscovery string + + // Specifies the WebIdentity token the SDK should use to assume a role + // with. + // + // AWS_WEB_IDENTITY_TOKEN_FILE=file_path + WebIdentityTokenFilePath string + + // Specifies the IAM role arn to use when assuming an role. + // + // AWS_ROLE_ARN=role_arn + RoleARN string + + // Specifies the IAM role session name to use when assuming a role. + // + // AWS_ROLE_SESSION_NAME=session_name + RoleSessionName string + + // Specifies the STS Regional Endpoint flag for the SDK to resolve the endpoint + // for a service. + // + // AWS_STS_REGIONAL_ENDPOINTS=regional + // This can take value as `regional` or `legacy` + STSRegionalEndpoint endpoints.STSRegionalEndpoint + + // Specifies the S3 Regional Endpoint flag for the SDK to resolve the + // endpoint for a service. + // + // AWS_S3_US_EAST_1_REGIONAL_ENDPOINT=regional + // This can take value as `regional` or `legacy` + S3UsEast1RegionalEndpoint endpoints.S3UsEast1RegionalEndpoint + + // Specifies if the S3 service should allow ARNs to direct the region + // the client's requests are sent to. + // + // AWS_S3_USE_ARN_REGION=true + S3UseARNRegion bool } var ( csmEnabledEnvKey = []string{ "AWS_CSM_ENABLED", } + csmHostEnvKey = []string{ + "AWS_CSM_HOST", + } csmPortEnvKey = []string{ "AWS_CSM_PORT", } @@ -150,6 +193,24 @@ var ( sharedConfigFileEnvKey = []string{ "AWS_CONFIG_FILE", } + webIdentityTokenFilePathEnvKey = []string{ + "AWS_WEB_IDENTITY_TOKEN_FILE", + } + roleARNEnvKey = []string{ + "AWS_ROLE_ARN", + } + roleSessionNameEnvKey = []string{ + "AWS_ROLE_SESSION_NAME", + } + stsRegionalEndpointKey = []string{ + "AWS_STS_REGIONAL_ENDPOINTS", + } + s3UsEast1RegionalEndpoint = []string{ + "AWS_S3_US_EAST_1_REGIONAL_ENDPOINT", + } + s3UseARNRegionEnvKey = []string{ + "AWS_S3_USE_ARN_REGION", + } ) // loadEnvConfig retrieves the SDK's environment configuration. @@ -158,7 +219,7 @@ var ( // If the environment variable `AWS_SDK_LOAD_CONFIG` is set to a truthy value // the shared SDK config will be loaded in addition to the SDK's specific // configuration values. -func loadEnvConfig() envConfig { +func loadEnvConfig() (envConfig, error) { enableSharedConfig, _ := strconv.ParseBool(os.Getenv("AWS_SDK_LOAD_CONFIG")) return envConfigLoad(enableSharedConfig) } @@ -169,30 +230,42 @@ func loadEnvConfig() envConfig { // Loads the shared configuration in addition to the SDK's specific configuration. // This will load the same values as `loadEnvConfig` if the `AWS_SDK_LOAD_CONFIG` // environment variable is set. -func loadSharedEnvConfig() envConfig { +func loadSharedEnvConfig() (envConfig, error) { return envConfigLoad(true) } -func envConfigLoad(enableSharedConfig bool) envConfig { +func envConfigLoad(enableSharedConfig bool) (envConfig, error) { cfg := envConfig{} cfg.EnableSharedConfig = enableSharedConfig - setFromEnvVal(&cfg.Creds.AccessKeyID, credAccessEnvKey) - setFromEnvVal(&cfg.Creds.SecretAccessKey, credSecretEnvKey) - setFromEnvVal(&cfg.Creds.SessionToken, credSessionEnvKey) + // Static environment credentials + var creds credentials.Value + setFromEnvVal(&creds.AccessKeyID, credAccessEnvKey) + setFromEnvVal(&creds.SecretAccessKey, credSecretEnvKey) + setFromEnvVal(&creds.SessionToken, credSessionEnvKey) + if creds.HasKeys() { + // Require logical grouping of credentials + creds.ProviderName = EnvProviderName + cfg.Creds = creds + } + + // Role Metadata + setFromEnvVal(&cfg.RoleARN, roleARNEnvKey) + setFromEnvVal(&cfg.RoleSessionName, roleSessionNameEnvKey) + + // Web identity environment variables + setFromEnvVal(&cfg.WebIdentityTokenFilePath, webIdentityTokenFilePathEnvKey) // CSM environment variables setFromEnvVal(&cfg.csmEnabled, csmEnabledEnvKey) + setFromEnvVal(&cfg.CSMHost, csmHostEnvKey) setFromEnvVal(&cfg.CSMPort, csmPortEnvKey) setFromEnvVal(&cfg.CSMClientID, csmClientIDEnvKey) - cfg.CSMEnabled = len(cfg.csmEnabled) > 0 - // Require logical grouping of credentials - if len(cfg.Creds.AccessKeyID) == 0 || len(cfg.Creds.SecretAccessKey) == 0 { - cfg.Creds = credentials.Value{} - } else { - cfg.Creds.ProviderName = EnvProviderName + if len(cfg.csmEnabled) != 0 { + v, _ := strconv.ParseBool(cfg.csmEnabled) + cfg.CSMEnabled = &v } regionKeys := regionEnvKeys @@ -223,12 +296,48 @@ func envConfigLoad(enableSharedConfig bool) envConfig { cfg.CustomCABundle = os.Getenv("AWS_CA_BUNDLE") - return cfg + var err error + // STS Regional Endpoint variable + for _, k := range stsRegionalEndpointKey { + if v := os.Getenv(k); len(v) != 0 { + cfg.STSRegionalEndpoint, err = endpoints.GetSTSRegionalEndpoint(v) + if err != nil { + return cfg, fmt.Errorf("failed to load, %v from env config, %v", k, err) + } + } + } + + // S3 Regional Endpoint variable + for _, k := range s3UsEast1RegionalEndpoint { + if v := os.Getenv(k); len(v) != 0 { + cfg.S3UsEast1RegionalEndpoint, err = endpoints.GetS3UsEast1RegionalEndpoint(v) + if err != nil { + return cfg, fmt.Errorf("failed to load, %v from env config, %v", k, err) + } + } + } + + var s3UseARNRegion string + setFromEnvVal(&s3UseARNRegion, s3UseARNRegionEnvKey) + if len(s3UseARNRegion) != 0 { + switch { + case strings.EqualFold(s3UseARNRegion, "false"): + cfg.S3UseARNRegion = false + case strings.EqualFold(s3UseARNRegion, "true"): + cfg.S3UseARNRegion = true + default: + return envConfig{}, fmt.Errorf( + "invalid value for environment variable, %s=%s, need true or false", + s3UseARNRegionEnvKey[0], s3UseARNRegion) + } + } + + return cfg, nil } func setFromEnvVal(dst *string, keys []string) { for _, k := range keys { - if v := os.Getenv(k); len(v) > 0 { + if v := os.Getenv(k); len(v) != 0 { *dst = v break } diff --git a/github.com/aws/aws-sdk-go/aws/session/session.go b/github.com/aws/aws-sdk-go/aws/session/session.go index 9bdbafd65c..0ff4996051 100644 --- a/github.com/aws/aws-sdk-go/aws/session/session.go +++ b/github.com/aws/aws-sdk-go/aws/session/session.go @@ -8,19 +8,17 @@ import ( "io/ioutil" "net/http" "os" + "time" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/client" "github.com/aws/aws-sdk-go/aws/corehandlers" "github.com/aws/aws-sdk-go/aws/credentials" - "github.com/aws/aws-sdk-go/aws/credentials/processcreds" - "github.com/aws/aws-sdk-go/aws/credentials/stscreds" "github.com/aws/aws-sdk-go/aws/csm" "github.com/aws/aws-sdk-go/aws/defaults" "github.com/aws/aws-sdk-go/aws/endpoints" "github.com/aws/aws-sdk-go/aws/request" - "github.com/aws/aws-sdk-go/internal/shareddefaults" ) const ( @@ -75,7 +73,7 @@ type Session struct { // func is called instead of waiting to receive an error until a request is made. func New(cfgs ...*aws.Config) *Session { // load initial config from environment - envCfg := loadEnvConfig() + envCfg, envErr := loadEnvConfig() if envCfg.EnableSharedConfig { var cfg aws.Config @@ -95,19 +93,28 @@ func New(cfgs ...*aws.Config) *Session { // Session creation failed, need to report the error and prevent // any requests from succeeding. s = &Session{Config: defaults.Config()} - s.Config.MergeIn(cfgs...) - s.Config.Logger.Log("ERROR:", msg, "Error:", err) - s.Handlers.Validate.PushBack(func(r *request.Request) { - r.Error = err - }) + s.logDeprecatedNewSessionError(msg, err, cfgs) } return s } s := deprecatedNewSession(cfgs...) - if envCfg.CSMEnabled { - enableCSM(&s.Handlers, envCfg.CSMClientID, envCfg.CSMPort, s.Config.Logger) + if envErr != nil { + msg := "failed to load env config" + s.logDeprecatedNewSessionError(msg, envErr, cfgs) + } + + if csmCfg, err := loadCSMConfig(envCfg, []string{}); err != nil { + if l := s.Config.Logger; l != nil { + l.Log(fmt.Sprintf("ERROR: failed to load CSM configuration, %v", err)) + } + } else if csmCfg.Enabled { + err := enableCSM(&s.Handlers, csmCfg, s.Config.Logger) + if err != nil { + msg := "failed to enable CSM" + s.logDeprecatedNewSessionError(msg, err, cfgs) + } } return s @@ -126,7 +133,7 @@ func New(cfgs ...*aws.Config) *Session { // to be built with retrieving credentials with AssumeRole set in the config. // // See the NewSessionWithOptions func for information on how to override or -// control through code how the Session will be created. Such as specifying the +// control through code how the Session will be created, such as specifying the // config profile, and controlling if shared config is enabled or not. func NewSession(cfgs ...*aws.Config) (*Session, error) { opts := Options{} @@ -210,6 +217,12 @@ type Options struct { // the config enables assume role wit MFA via the mfa_serial field. AssumeRoleTokenProvider func() (string, error) + // When the SDK's shared config is configured to assume a role this option + // may be provided to set the expiry duration of the STS credentials. + // Defaults to 15 minutes if not set as documented in the + // stscreds.AssumeRoleProvider. + AssumeRoleDuration time.Duration + // Reader for a custom Credentials Authority (CA) bundle in PEM format that // the SDK will use instead of the default system's root CA bundle. Use this // only if you want to replace the CA bundle the SDK uses for TLS requests. @@ -224,6 +237,12 @@ type Options struct { // to also enable this feature. CustomCABundle session option field has priority // over the AWS_CA_BUNDLE environment variable, and will be used if both are set. CustomCABundle io.Reader + + // The handlers that the session and all API clients will be created with. + // This must be a complete set of handlers. Use the defaults.Handlers() + // function to initialize this value before changing the handlers to be + // used by the SDK. + Handlers request.Handlers } // NewSessionWithOptions returns a new Session created from SDK defaults, config files, @@ -257,13 +276,20 @@ type Options struct { // })) func NewSessionWithOptions(opts Options) (*Session, error) { var envCfg envConfig + var err error if opts.SharedConfigState == SharedConfigEnable { - envCfg = loadSharedEnvConfig() + envCfg, err = loadSharedEnvConfig() + if err != nil { + return nil, fmt.Errorf("failed to load shared config, %v", err) + } } else { - envCfg = loadEnvConfig() + envCfg, err = loadEnvConfig() + if err != nil { + return nil, fmt.Errorf("failed to load environment config, %v", err) + } } - if len(opts.Profile) > 0 { + if len(opts.Profile) != 0 { envCfg.Profile = opts.Profile } @@ -329,27 +355,33 @@ func deprecatedNewSession(cfgs ...*aws.Config) *Session { return s } -func enableCSM(handlers *request.Handlers, clientID string, port string, logger aws.Logger) { - logger.Log("Enabling CSM") - if len(port) == 0 { - port = csm.DefaultPort +func enableCSM(handlers *request.Handlers, cfg csmConfig, logger aws.Logger) error { + if logger != nil { + logger.Log("Enabling CSM") } - r, err := csm.Start(clientID, "127.0.0.1:"+port) + r, err := csm.Start(cfg.ClientID, csm.AddressWithDefaults(cfg.Host, cfg.Port)) if err != nil { - return + return err } r.InjectHandlers(handlers) + + return nil } func newSession(opts Options, envCfg envConfig, cfgs ...*aws.Config) (*Session, error) { cfg := defaults.Config() - handlers := defaults.Handlers() + + handlers := opts.Handlers + if handlers.IsEmpty() { + handlers = defaults.Handlers() + } // Get a merged version of the user provided config to determine if // credentials were. userCfg := &aws.Config{} userCfg.MergeIn(cfgs...) + cfg.MergeIn(userCfg) // Ordered config files will be loaded in with later files overwriting // previous config file values. @@ -366,9 +398,17 @@ func newSession(opts Options, envCfg envConfig, cfgs ...*aws.Config) (*Session, } // Load additional config from file(s) - sharedCfg, err := loadSharedConfig(envCfg.Profile, cfgFiles) + sharedCfg, err := loadSharedConfig(envCfg.Profile, cfgFiles, envCfg.EnableSharedConfig) if err != nil { - return nil, err + if len(envCfg.Profile) == 0 && !envCfg.EnableSharedConfig && (envCfg.Creds.HasKeys() || userCfg.Credentials != nil) { + // Special case where the user has not explicitly specified an AWS_PROFILE, + // or session.Options.profile, shared config is not enabled, and the + // environment has credentials, allow the shared config file to fail to + // load since the user has already provided credentials, and nothing else + // is required to be read file. Github(aws/aws-sdk-go#2455) + } else if _, ok := err.(SharedConfigProfileNotExistsError); !ok { + return nil, err + } } if err := mergeConfigSrcs(cfg, userCfg, envCfg, sharedCfg, handlers, opts); err != nil { @@ -381,8 +421,16 @@ func newSession(opts Options, envCfg envConfig, cfgs ...*aws.Config) (*Session, } initHandlers(s) - if envCfg.CSMEnabled { - enableCSM(&s.Handlers, envCfg.CSMClientID, envCfg.CSMPort, s.Config.Logger) + + if csmCfg, err := loadCSMConfig(envCfg, cfgFiles); err != nil { + if l := s.Config.Logger; l != nil { + l.Log(fmt.Sprintf("ERROR: failed to load CSM configuration, %v", err)) + } + } else if csmCfg.Enabled { + err = enableCSM(&s.Handlers, csmCfg, s.Config.Logger) + if err != nil { + return nil, err + } } // Setup HTTP client with custom cert bundle if enabled @@ -395,6 +443,46 @@ func newSession(opts Options, envCfg envConfig, cfgs ...*aws.Config) (*Session, return s, nil } +type csmConfig struct { + Enabled bool + Host string + Port string + ClientID string +} + +var csmProfileName = "aws_csm" + +func loadCSMConfig(envCfg envConfig, cfgFiles []string) (csmConfig, error) { + if envCfg.CSMEnabled != nil { + if *envCfg.CSMEnabled { + return csmConfig{ + Enabled: true, + ClientID: envCfg.CSMClientID, + Host: envCfg.CSMHost, + Port: envCfg.CSMPort, + }, nil + } + return csmConfig{}, nil + } + + sharedCfg, err := loadSharedConfig(csmProfileName, cfgFiles, false) + if err != nil { + if _, ok := err.(SharedConfigProfileNotExistsError); !ok { + return csmConfig{}, err + } + } + if sharedCfg.CSMEnabled != nil && *sharedCfg.CSMEnabled == true { + return csmConfig{ + Enabled: true, + ClientID: sharedCfg.CSMClientID, + Host: sharedCfg.CSMHost, + Port: sharedCfg.CSMPort, + }, nil + } + + return csmConfig{}, nil +} + func loadCustomCABundle(s *Session, bundle io.Reader) error { var t *http.Transport switch v := s.Config.HTTPClient.Transport.(type) { @@ -407,7 +495,10 @@ func loadCustomCABundle(s *Session, bundle io.Reader) error { } } if t == nil { - t = &http.Transport{} + // Nil transport implies `http.DefaultTransport` should be used. Since + // the SDK cannot modify, nor copy the `DefaultTransport` specifying + // the values the next closest behavior. + t = getCABundleTransport() } p, err := loadCertPool(bundle) @@ -440,9 +531,11 @@ func loadCertPool(r io.Reader) (*x509.CertPool, error) { return p, nil } -func mergeConfigSrcs(cfg, userCfg *aws.Config, envCfg envConfig, sharedCfg sharedConfig, handlers request.Handlers, sessOpts Options) error { - // Merge in user provided configuration - cfg.MergeIn(userCfg) +func mergeConfigSrcs(cfg, userCfg *aws.Config, + envCfg envConfig, sharedCfg sharedConfig, + handlers request.Handlers, + sessOpts Options, +) error { // Region if not already set by user if len(aws.StringValue(cfg.Region)) == 0 { @@ -461,162 +554,59 @@ func mergeConfigSrcs(cfg, userCfg *aws.Config, envCfg envConfig, sharedCfg share } } - // Configure credentials if not already set + // Regional Endpoint flag for STS endpoint resolving + mergeSTSRegionalEndpointConfig(cfg, []endpoints.STSRegionalEndpoint{ + userCfg.STSRegionalEndpoint, + envCfg.STSRegionalEndpoint, + sharedCfg.STSRegionalEndpoint, + endpoints.LegacySTSEndpoint, + }) + + // Regional Endpoint flag for S3 endpoint resolving + mergeS3UsEast1RegionalEndpointConfig(cfg, []endpoints.S3UsEast1RegionalEndpoint{ + userCfg.S3UsEast1RegionalEndpoint, + envCfg.S3UsEast1RegionalEndpoint, + sharedCfg.S3UsEast1RegionalEndpoint, + endpoints.LegacyS3UsEast1Endpoint, + }) + + // Configure credentials if not already set by the user when creating the + // Session. if cfg.Credentials == credentials.AnonymousCredentials && userCfg.Credentials == nil { - - // inspect the profile to see if a credential source has been specified. - if envCfg.EnableSharedConfig && len(sharedCfg.AssumeRole.CredentialSource) > 0 { - - // if both credential_source and source_profile have been set, return an error - // as this is undefined behavior. - if len(sharedCfg.AssumeRole.SourceProfile) > 0 { - return ErrSharedConfigSourceCollision - } - - // valid credential source values - const ( - credSourceEc2Metadata = "Ec2InstanceMetadata" - credSourceEnvironment = "Environment" - credSourceECSContainer = "EcsContainer" - ) - - switch sharedCfg.AssumeRole.CredentialSource { - case credSourceEc2Metadata: - cfgCp := *cfg - p := defaults.RemoteCredProvider(cfgCp, handlers) - cfgCp.Credentials = credentials.NewCredentials(p) - - if len(sharedCfg.AssumeRole.MFASerial) > 0 && sessOpts.AssumeRoleTokenProvider == nil { - // AssumeRole Token provider is required if doing Assume Role - // with MFA. - return AssumeRoleTokenProviderNotSetError{} - } - - cfg.Credentials = assumeRoleCredentials(cfgCp, handlers, sharedCfg, sessOpts) - case credSourceEnvironment: - cfg.Credentials = credentials.NewStaticCredentialsFromCreds( - envCfg.Creds, - ) - case credSourceECSContainer: - if len(os.Getenv(shareddefaults.ECSCredsProviderEnvVar)) == 0 { - return ErrSharedConfigECSContainerEnvVarEmpty - } - - cfgCp := *cfg - p := defaults.RemoteCredProvider(cfgCp, handlers) - creds := credentials.NewCredentials(p) - - cfg.Credentials = creds - default: - return ErrSharedConfigInvalidCredSource - } - - return nil - } - - if len(envCfg.Creds.AccessKeyID) > 0 { - cfg.Credentials = credentials.NewStaticCredentialsFromCreds( - envCfg.Creds, - ) - } else if envCfg.EnableSharedConfig && len(sharedCfg.AssumeRole.RoleARN) > 0 && sharedCfg.AssumeRoleSource != nil { - cfgCp := *cfg - cfgCp.Credentials = credentials.NewStaticCredentialsFromCreds( - sharedCfg.AssumeRoleSource.Creds, - ) - - if len(sharedCfg.AssumeRole.MFASerial) > 0 && sessOpts.AssumeRoleTokenProvider == nil { - // AssumeRole Token provider is required if doing Assume Role - // with MFA. - return AssumeRoleTokenProviderNotSetError{} - } - - cfg.Credentials = assumeRoleCredentials(cfgCp, handlers, sharedCfg, sessOpts) - } else if len(sharedCfg.Creds.AccessKeyID) > 0 { - cfg.Credentials = credentials.NewStaticCredentialsFromCreds( - sharedCfg.Creds, - ) - } else if len(sharedCfg.CredentialProcess) > 0 { - cfg.Credentials = processcreds.NewCredentials( - sharedCfg.CredentialProcess, - ) - } else { - // Fallback to default credentials provider, include mock errors - // for the credential chain so user can identify why credentials - // failed to be retrieved. - cfg.Credentials = credentials.NewCredentials(&credentials.ChainProvider{ - VerboseErrors: aws.BoolValue(cfg.CredentialsChainVerboseErrors), - Providers: []credentials.Provider{ - &credProviderError{Err: awserr.New("EnvAccessKeyNotFound", "failed to find credentials in the environment.", nil)}, - &credProviderError{Err: awserr.New("SharedCredsLoad", fmt.Sprintf("failed to load profile, %s.", envCfg.Profile), nil)}, - defaults.RemoteCredProvider(*cfg, handlers), - }, - }) + creds, err := resolveCredentials(cfg, envCfg, sharedCfg, handlers, sessOpts) + if err != nil { + return err } + cfg.Credentials = creds } - return nil -} - -func assumeRoleCredentials(cfg aws.Config, handlers request.Handlers, sharedCfg sharedConfig, sessOpts Options) *credentials.Credentials { - return stscreds.NewCredentials( - &Session{ - Config: &cfg, - Handlers: handlers.Copy(), - }, - sharedCfg.AssumeRole.RoleARN, - func(opt *stscreds.AssumeRoleProvider) { - opt.RoleSessionName = sharedCfg.AssumeRole.RoleSessionName - - // Assume role with external ID - if len(sharedCfg.AssumeRole.ExternalID) > 0 { - opt.ExternalID = aws.String(sharedCfg.AssumeRole.ExternalID) - } - - // Assume role with MFA - if len(sharedCfg.AssumeRole.MFASerial) > 0 { - opt.SerialNumber = aws.String(sharedCfg.AssumeRole.MFASerial) - opt.TokenProvider = sessOpts.AssumeRoleTokenProvider - } - }, - ) -} - -// AssumeRoleTokenProviderNotSetError is an error returned when creating a session when the -// MFAToken option is not set when shared config is configured load assume a -// role with an MFA token. -type AssumeRoleTokenProviderNotSetError struct{} - -// Code is the short id of the error. -func (e AssumeRoleTokenProviderNotSetError) Code() string { - return "AssumeRoleTokenProviderNotSetError" -} - -// Message is the description of the error -func (e AssumeRoleTokenProviderNotSetError) Message() string { - return fmt.Sprintf("assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.") -} + cfg.S3UseARNRegion = userCfg.S3UseARNRegion + if cfg.S3UseARNRegion == nil { + cfg.S3UseARNRegion = &envCfg.S3UseARNRegion + } + if cfg.S3UseARNRegion == nil { + cfg.S3UseARNRegion = &sharedCfg.S3UseARNRegion + } -// OrigErr is the underlying error that caused the failure. -func (e AssumeRoleTokenProviderNotSetError) OrigErr() error { return nil } -// Error satisfies the error interface. -func (e AssumeRoleTokenProviderNotSetError) Error() string { - return awserr.SprintError(e.Code(), e.Message(), "", nil) -} - -type credProviderError struct { - Err error +func mergeSTSRegionalEndpointConfig(cfg *aws.Config, values []endpoints.STSRegionalEndpoint) { + for _, v := range values { + if v != endpoints.UnsetSTSEndpoint { + cfg.STSRegionalEndpoint = v + break + } + } } -var emptyCreds = credentials.Value{} - -func (c credProviderError) Retrieve() (credentials.Value, error) { - return credentials.Value{}, c.Err -} -func (c credProviderError) IsExpired() bool { - return true +func mergeS3UsEast1RegionalEndpointConfig(cfg *aws.Config, values []endpoints.S3UsEast1RegionalEndpoint) { + for _, v := range values { + if v != endpoints.UnsetS3UsEast1Endpoint { + cfg.S3UsEast1RegionalEndpoint = v + break + } + } } func initHandlers(s *Session) { @@ -627,7 +617,7 @@ func initHandlers(s *Session) { } } -// Copy creates and returns a copy of the current Session, coping the config +// Copy creates and returns a copy of the current Session, copying the config // and handlers. If any additional configs are provided they will be merged // on top of the Session's copied config. // @@ -647,47 +637,67 @@ func (s *Session) Copy(cfgs ...*aws.Config) *Session { // ClientConfig satisfies the client.ConfigProvider interface and is used to // configure the service client instances. Passing the Session to the service // client's constructor (New) will use this method to configure the client. -func (s *Session) ClientConfig(serviceName string, cfgs ...*aws.Config) client.Config { - // Backwards compatibility, the error will be eaten if user calls ClientConfig - // directly. All SDK services will use ClientconfigWithError. - cfg, _ := s.clientConfigWithErr(serviceName, cfgs...) - - return cfg -} - -func (s *Session) clientConfigWithErr(serviceName string, cfgs ...*aws.Config) (client.Config, error) { +func (s *Session) ClientConfig(service string, cfgs ...*aws.Config) client.Config { s = s.Copy(cfgs...) - var resolved endpoints.ResolvedEndpoint - var err error - region := aws.StringValue(s.Config.Region) - - if endpoint := aws.StringValue(s.Config.Endpoint); len(endpoint) != 0 { - resolved.URL = endpoints.AddScheme(endpoint, aws.BoolValue(s.Config.DisableSSL)) - resolved.SigningRegion = region - } else { - resolved, err = s.Config.EndpointResolver.EndpointFor( - serviceName, region, - func(opt *endpoints.Options) { - opt.DisableSSL = aws.BoolValue(s.Config.DisableSSL) - opt.UseDualStack = aws.BoolValue(s.Config.UseDualStack) - - // Support the condition where the service is modeled but its - // endpoint metadata is not available. - opt.ResolveUnknownService = true - }, - ) + resolved, err := s.resolveEndpoint(service, region, s.Config) + if err != nil { + s.Handlers.Validate.PushBack(func(r *request.Request) { + if len(r.ClientInfo.Endpoint) != 0 { + // Error occurred while resolving endpoint, but the request + // being invoked has had an endpoint specified after the client + // was created. + return + } + r.Error = err + }) } return client.Config{ Config: s.Config, Handlers: s.Handlers, + PartitionID: resolved.PartitionID, Endpoint: resolved.URL, SigningRegion: resolved.SigningRegion, SigningNameDerived: resolved.SigningNameDerived, SigningName: resolved.SigningName, - }, err + } +} + +func (s *Session) resolveEndpoint(service, region string, cfg *aws.Config) (endpoints.ResolvedEndpoint, error) { + + if ep := aws.StringValue(cfg.Endpoint); len(ep) != 0 { + return endpoints.ResolvedEndpoint{ + URL: endpoints.AddScheme(ep, aws.BoolValue(cfg.DisableSSL)), + SigningRegion: region, + }, nil + } + + resolved, err := cfg.EndpointResolver.EndpointFor(service, region, + func(opt *endpoints.Options) { + opt.DisableSSL = aws.BoolValue(cfg.DisableSSL) + opt.UseDualStack = aws.BoolValue(cfg.UseDualStack) + // Support for STSRegionalEndpoint where the STSRegionalEndpoint is + // provided in envConfig or sharedConfig with envConfig getting + // precedence. + opt.STSRegionalEndpoint = cfg.STSRegionalEndpoint + + // Support for S3UsEast1RegionalEndpoint where the S3UsEast1RegionalEndpoint is + // provided in envConfig or sharedConfig with envConfig getting + // precedence. + opt.S3UsEast1RegionalEndpoint = cfg.S3UsEast1RegionalEndpoint + + // Support the condition where the service is modeled but its + // endpoint metadata is not available. + opt.ResolveUnknownService = true + }, + ) + if err != nil { + return endpoints.ResolvedEndpoint{}, err + } + + return resolved, nil } // ClientConfigNoResolveEndpoint is the same as ClientConfig with the exception @@ -697,12 +707,9 @@ func (s *Session) ClientConfigNoResolveEndpoint(cfgs ...*aws.Config) client.Conf s = s.Copy(cfgs...) var resolved endpoints.ResolvedEndpoint - - region := aws.StringValue(s.Config.Region) - if ep := aws.StringValue(s.Config.Endpoint); len(ep) > 0 { resolved.URL = endpoints.AddScheme(ep, aws.BoolValue(s.Config.DisableSSL)) - resolved.SigningRegion = region + resolved.SigningRegion = aws.StringValue(s.Config.Region) } return client.Config{ @@ -714,3 +721,14 @@ func (s *Session) ClientConfigNoResolveEndpoint(cfgs ...*aws.Config) client.Conf SigningName: resolved.SigningName, } } + +// logDeprecatedNewSessionError function enables error handling for session +func (s *Session) logDeprecatedNewSessionError(msg string, err error, cfgs []*aws.Config) { + // Session creation failed, need to report the error and prevent + // any requests from succeeding. + s.Config.MergeIn(cfgs...) + s.Config.Logger.Log("ERROR:", msg, "Error:", err) + s.Handlers.Validate.PushBack(func(r *request.Request) { + r.Error = err + }) +} diff --git a/github.com/aws/aws-sdk-go/aws/session/shared_config.go b/github.com/aws/aws-sdk-go/aws/session/shared_config.go index 7cb44021b3..680805a38a 100644 --- a/github.com/aws/aws-sdk-go/aws/session/shared_config.go +++ b/github.com/aws/aws-sdk-go/aws/session/shared_config.go @@ -2,10 +2,11 @@ package session import ( "fmt" + "time" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/credentials" - + "github.com/aws/aws-sdk-go/aws/endpoints" "github.com/aws/aws-sdk-go/internal/ini" ) @@ -16,57 +17,75 @@ const ( sessionTokenKey = `aws_session_token` // optional // Assume Role Credentials group - roleArnKey = `role_arn` // group required - sourceProfileKey = `source_profile` // group required (or credential_source) - credentialSourceKey = `credential_source` // group required (or source_profile) - externalIDKey = `external_id` // optional - mfaSerialKey = `mfa_serial` // optional - roleSessionNameKey = `role_session_name` // optional + roleArnKey = `role_arn` // group required + sourceProfileKey = `source_profile` // group required (or credential_source) + credentialSourceKey = `credential_source` // group required (or source_profile) + externalIDKey = `external_id` // optional + mfaSerialKey = `mfa_serial` // optional + roleSessionNameKey = `role_session_name` // optional + roleDurationSecondsKey = "duration_seconds" // optional + + // CSM options + csmEnabledKey = `csm_enabled` + csmHostKey = `csm_host` + csmPortKey = `csm_port` + csmClientIDKey = `csm_client_id` // Additional Config fields regionKey = `region` // endpoint discovery group enableEndpointDiscoveryKey = `endpoint_discovery_enabled` // optional + // External Credential Process - credentialProcessKey = `credential_process` + credentialProcessKey = `credential_process` // optional + + // Web Identity Token File + webIdentityTokenFileKey = `web_identity_token_file` // optional + + // Additional config fields for regional or legacy endpoints + stsRegionalEndpointSharedKey = `sts_regional_endpoints` + + // Additional config fields for regional or legacy endpoints + s3UsEast1RegionalSharedKey = `s3_us_east_1_regional_endpoint` // DefaultSharedConfigProfile is the default profile to be used when // loading configuration from the config files if another profile name // is not provided. DefaultSharedConfigProfile = `default` -) -type assumeRoleConfig struct { - RoleARN string - SourceProfile string - CredentialSource string - ExternalID string - MFASerial string - RoleSessionName string -} + // S3 ARN Region Usage + s3UseARNRegionKey = "s3_use_arn_region" +) // sharedConfig represents the configuration fields of the SDK config files. type sharedConfig struct { - // Credentials values from the config file. Both aws_access_key_id - // and aws_secret_access_key must be provided together in the same file - // to be considered valid. The values will be ignored if not a complete group. - // aws_session_token is an optional field that can be provided if both of the - // other two fields are also provided. + // Credentials values from the config file. Both aws_access_key_id and + // aws_secret_access_key must be provided together in the same file to be + // considered valid. The values will be ignored if not a complete group. + // aws_session_token is an optional field that can be provided if both of + // the other two fields are also provided. // // aws_access_key_id // aws_secret_access_key // aws_session_token Creds credentials.Value - AssumeRole assumeRoleConfig - AssumeRoleSource *sharedConfig + CredentialSource string + CredentialProcess string + WebIdentityTokenFile string + + RoleARN string + RoleSessionName string + ExternalID string + MFASerial string + AssumeRoleDuration *time.Duration - // An external process to request credentials - CredentialProcess string + SourceProfileName string + SourceProfile *sharedConfig - // Region is the region the SDK should use for looking up AWS service endpoints - // and signing requests. + // Region is the region the SDK should use for looking up AWS service + // endpoints and signing requests. // // region Region string @@ -76,6 +95,30 @@ type sharedConfig struct { // // endpoint_discovery_enabled = true EnableEndpointDiscovery *bool + + // CSM Options + CSMEnabled *bool + CSMHost string + CSMPort string + CSMClientID string + + // Specifies the Regional Endpoint flag for the SDK to resolve the endpoint for a service + // + // sts_regional_endpoints = regional + // This can take value as `LegacySTSEndpoint` or `RegionalSTSEndpoint` + STSRegionalEndpoint endpoints.STSRegionalEndpoint + + // Specifies the Regional Endpoint flag for the SDK to resolve the endpoint for a service + // + // s3_us_east_1_regional_endpoint = regional + // This can take value as `LegacyS3UsEast1Endpoint` or `RegionalS3UsEast1Endpoint` + S3UsEast1RegionalEndpoint endpoints.S3UsEast1RegionalEndpoint + + // Specifies if the S3 service should allow ARNs to direct the region + // the client's requests are sent to. + // + // s3_use_arn_region=true + S3UseARNRegion bool } type sharedConfigFile struct { @@ -83,17 +126,18 @@ type sharedConfigFile struct { IniData ini.Sections } -// loadSharedConfig retrieves the configuration from the list of files -// using the profile provided. The order the files are listed will determine +// loadSharedConfig retrieves the configuration from the list of files using +// the profile provided. The order the files are listed will determine // precedence. Values in subsequent files will overwrite values defined in // earlier files. // // For example, given two files A and B. Both define credentials. If the order -// of the files are A then B, B's credential values will be used instead of A's. +// of the files are A then B, B's credential values will be used instead of +// A's. // // See sharedConfig.setFromFile for information how the config files // will be loaded. -func loadSharedConfig(profile string, filenames []string) (sharedConfig, error) { +func loadSharedConfig(profile string, filenames []string, exOpts bool) (sharedConfig, error) { if len(profile) == 0 { profile = DefaultSharedConfigProfile } @@ -104,16 +148,11 @@ func loadSharedConfig(profile string, filenames []string) (sharedConfig, error) } cfg := sharedConfig{} - if err = cfg.setFromIniFiles(profile, files); err != nil { + profiles := map[string]struct{}{} + if err = cfg.setFromIniFiles(profiles, profile, files, exOpts); err != nil { return sharedConfig{}, err } - if len(cfg.AssumeRole.SourceProfile) > 0 { - if err := cfg.setAssumeRoleSource(profile, files); err != nil { - return sharedConfig{}, err - } - } - return cfg, nil } @@ -137,60 +176,88 @@ func loadSharedConfigIniFiles(filenames []string) ([]sharedConfigFile, error) { return files, nil } -func (cfg *sharedConfig) setAssumeRoleSource(origProfile string, files []sharedConfigFile) error { - var assumeRoleSrc sharedConfig - - if len(cfg.AssumeRole.CredentialSource) > 0 { - // setAssumeRoleSource is only called when source_profile is found. - // If both source_profile and credential_source are set, then - // ErrSharedConfigSourceCollision will be returned - return ErrSharedConfigSourceCollision +func (cfg *sharedConfig) setFromIniFiles(profiles map[string]struct{}, profile string, files []sharedConfigFile, exOpts bool) error { + // Trim files from the list that don't exist. + var skippedFiles int + var profileNotFoundErr error + for _, f := range files { + if err := cfg.setFromIniFile(profile, f, exOpts); err != nil { + if _, ok := err.(SharedConfigProfileNotExistsError); ok { + // Ignore profiles not defined in individual files. + profileNotFoundErr = err + skippedFiles++ + continue + } + return err + } + } + if skippedFiles == len(files) { + // If all files were skipped because the profile is not found, return + // the original profile not found error. + return profileNotFoundErr } - // Multiple level assume role chains are not support - if cfg.AssumeRole.SourceProfile == origProfile { - assumeRoleSrc = *cfg - assumeRoleSrc.AssumeRole = assumeRoleConfig{} + if _, ok := profiles[profile]; ok { + // if this is the second instance of the profile the Assume Role + // options must be cleared because they are only valid for the + // first reference of a profile. The self linked instance of the + // profile only have credential provider options. + cfg.clearAssumeRoleOptions() } else { - err := assumeRoleSrc.setFromIniFiles(cfg.AssumeRole.SourceProfile, files) - if err != nil { + // First time a profile has been seen, It must either be a assume role + // or credentials. Assert if the credential type requires a role ARN, + // the ARN is also set. + if err := cfg.validateCredentialsRequireARN(profile); err != nil { return err } } + profiles[profile] = struct{}{} - if len(assumeRoleSrc.Creds.AccessKeyID) == 0 { - return SharedConfigAssumeRoleError{RoleARN: cfg.AssumeRole.RoleARN} + if err := cfg.validateCredentialType(); err != nil { + return err } - cfg.AssumeRoleSource = &assumeRoleSrc + // Link source profiles for assume roles + if len(cfg.SourceProfileName) != 0 { + // Linked profile via source_profile ignore credential provider + // options, the source profile must provide the credentials. + cfg.clearCredentialOptions() - return nil -} - -func (cfg *sharedConfig) setFromIniFiles(profile string, files []sharedConfigFile) error { - // Trim files from the list that don't exist. - for _, f := range files { - if err := cfg.setFromIniFile(profile, f); err != nil { + srcCfg := &sharedConfig{} + err := srcCfg.setFromIniFiles(profiles, cfg.SourceProfileName, files, exOpts) + if err != nil { + // SourceProfile that doesn't exist is an error in configuration. if _, ok := err.(SharedConfigProfileNotExistsError); ok { - // Ignore proviles missings - continue + err = SharedConfigAssumeRoleError{ + RoleARN: cfg.RoleARN, + SourceProfile: cfg.SourceProfileName, + } } return err } + + if !srcCfg.hasCredentials() { + return SharedConfigAssumeRoleError{ + RoleARN: cfg.RoleARN, + SourceProfile: cfg.SourceProfileName, + } + } + + cfg.SourceProfile = srcCfg } return nil } -// setFromFile loads the configuration from the file using -// the profile provided. A sharedConfig pointer type value is used so that -// multiple config file loadings can be chained. +// setFromFile loads the configuration from the file using the profile +// provided. A sharedConfig pointer type value is used so that multiple config +// file loadings can be chained. // // Only loads complete logically grouped values, and will not set fields in cfg -// for incomplete grouped values in the config. Such as credentials. For example -// if a config file only includes aws_access_key_id but no aws_secret_access_key -// the aws_access_key_id will be ignored. -func (cfg *sharedConfig) setFromIniFile(profile string, file sharedConfigFile) error { +// for incomplete grouped values in the config. Such as credentials. For +// example if a config file only includes aws_access_key_id but no +// aws_secret_access_key the aws_access_key_id will be ignored. +func (cfg *sharedConfig) setFromIniFile(profile string, file sharedConfigFile, exOpts bool) error { section, ok := file.IniData.GetSection(profile) if !ok { // Fallback to to alternate profile name: profile @@ -200,53 +267,176 @@ func (cfg *sharedConfig) setFromIniFile(profile string, file sharedConfigFile) e } } - // Shared Credentials - akid := section.String(accessKeyIDKey) - secret := section.String(secretAccessKey) - if len(akid) > 0 && len(secret) > 0 { - cfg.Creds = credentials.Value{ - AccessKeyID: akid, - SecretAccessKey: secret, - SessionToken: section.String(sessionTokenKey), - ProviderName: fmt.Sprintf("SharedConfigCredentials: %s", file.Filename), + if exOpts { + // Assume Role Parameters + updateString(&cfg.RoleARN, section, roleArnKey) + updateString(&cfg.ExternalID, section, externalIDKey) + updateString(&cfg.MFASerial, section, mfaSerialKey) + updateString(&cfg.RoleSessionName, section, roleSessionNameKey) + updateString(&cfg.SourceProfileName, section, sourceProfileKey) + updateString(&cfg.CredentialSource, section, credentialSourceKey) + updateString(&cfg.Region, section, regionKey) + + if section.Has(roleDurationSecondsKey) { + d := time.Duration(section.Int(roleDurationSecondsKey)) * time.Second + cfg.AssumeRoleDuration = &d } - } - // Assume Role - roleArn := section.String(roleArnKey) - srcProfile := section.String(sourceProfileKey) - credentialSource := section.String(credentialSourceKey) - hasSource := len(srcProfile) > 0 || len(credentialSource) > 0 - if len(roleArn) > 0 && hasSource { - cfg.AssumeRole = assumeRoleConfig{ - RoleARN: roleArn, - SourceProfile: srcProfile, - CredentialSource: credentialSource, - ExternalID: section.String(externalIDKey), - MFASerial: section.String(mfaSerialKey), - RoleSessionName: section.String(roleSessionNameKey), + if v := section.String(stsRegionalEndpointSharedKey); len(v) != 0 { + sre, err := endpoints.GetSTSRegionalEndpoint(v) + if err != nil { + return fmt.Errorf("failed to load %s from shared config, %s, %v", + stsRegionalEndpointSharedKey, file.Filename, err) + } + cfg.STSRegionalEndpoint = sre } - } - // `credential_process` - if credProc := section.String(credentialProcessKey); len(credProc) > 0 { - cfg.CredentialProcess = credProc + if v := section.String(s3UsEast1RegionalSharedKey); len(v) != 0 { + sre, err := endpoints.GetS3UsEast1RegionalEndpoint(v) + if err != nil { + return fmt.Errorf("failed to load %s from shared config, %s, %v", + s3UsEast1RegionalSharedKey, file.Filename, err) + } + cfg.S3UsEast1RegionalEndpoint = sre + } } - // Region - if v := section.String(regionKey); len(v) > 0 { - cfg.Region = v + updateString(&cfg.CredentialProcess, section, credentialProcessKey) + updateString(&cfg.WebIdentityTokenFile, section, webIdentityTokenFileKey) + + // Shared Credentials + creds := credentials.Value{ + AccessKeyID: section.String(accessKeyIDKey), + SecretAccessKey: section.String(secretAccessKey), + SessionToken: section.String(sessionTokenKey), + ProviderName: fmt.Sprintf("SharedConfigCredentials: %s", file.Filename), + } + if creds.HasKeys() { + cfg.Creds = creds } // Endpoint discovery - if section.Has(enableEndpointDiscoveryKey) { - v := section.Bool(enableEndpointDiscoveryKey) - cfg.EnableEndpointDiscovery = &v + updateBoolPtr(&cfg.EnableEndpointDiscovery, section, enableEndpointDiscoveryKey) + + // CSM options + updateBoolPtr(&cfg.CSMEnabled, section, csmEnabledKey) + updateString(&cfg.CSMHost, section, csmHostKey) + updateString(&cfg.CSMPort, section, csmPortKey) + updateString(&cfg.CSMClientID, section, csmClientIDKey) + + updateBool(&cfg.S3UseARNRegion, section, s3UseARNRegionKey) + + return nil +} + +func (cfg *sharedConfig) validateCredentialsRequireARN(profile string) error { + var credSource string + + switch { + case len(cfg.SourceProfileName) != 0: + credSource = sourceProfileKey + case len(cfg.CredentialSource) != 0: + credSource = credentialSourceKey + case len(cfg.WebIdentityTokenFile) != 0: + credSource = webIdentityTokenFileKey + } + + if len(credSource) != 0 && len(cfg.RoleARN) == 0 { + return CredentialRequiresARNError{ + Type: credSource, + Profile: profile, + } + } + + return nil +} + +func (cfg *sharedConfig) validateCredentialType() error { + // Only one or no credential type can be defined. + if !oneOrNone( + len(cfg.SourceProfileName) != 0, + len(cfg.CredentialSource) != 0, + len(cfg.CredentialProcess) != 0, + len(cfg.WebIdentityTokenFile) != 0, + ) { + return ErrSharedConfigSourceCollision } return nil } +func (cfg *sharedConfig) hasCredentials() bool { + switch { + case len(cfg.SourceProfileName) != 0: + case len(cfg.CredentialSource) != 0: + case len(cfg.CredentialProcess) != 0: + case len(cfg.WebIdentityTokenFile) != 0: + case cfg.Creds.HasKeys(): + default: + return false + } + + return true +} + +func (cfg *sharedConfig) clearCredentialOptions() { + cfg.CredentialSource = "" + cfg.CredentialProcess = "" + cfg.WebIdentityTokenFile = "" + cfg.Creds = credentials.Value{} +} + +func (cfg *sharedConfig) clearAssumeRoleOptions() { + cfg.RoleARN = "" + cfg.ExternalID = "" + cfg.MFASerial = "" + cfg.RoleSessionName = "" + cfg.SourceProfileName = "" +} + +func oneOrNone(bs ...bool) bool { + var count int + + for _, b := range bs { + if b { + count++ + if count > 1 { + return false + } + } + } + + return true +} + +// updateString will only update the dst with the value in the section key, key +// is present in the section. +func updateString(dst *string, section ini.Section, key string) { + if !section.Has(key) { + return + } + *dst = section.String(key) +} + +// updateBool will only update the dst with the value in the section key, key +// is present in the section. +func updateBool(dst *bool, section ini.Section, key string) { + if !section.Has(key) { + return + } + *dst = section.Bool(key) +} + +// updateBoolPtr will only update the dst with the value in the section key, +// key is present in the section. +func updateBoolPtr(dst **bool, section ini.Section, key string) { + if !section.Has(key) { + return + } + *dst = new(bool) + **dst = section.Bool(key) +} + // SharedConfigLoadError is an error for the shared config file failed to load. type SharedConfigLoadError struct { Filename string @@ -304,7 +494,8 @@ func (e SharedConfigProfileNotExistsError) Error() string { // profile contains assume role information, but that information is invalid // or not complete. type SharedConfigAssumeRoleError struct { - RoleARN string + RoleARN string + SourceProfile string } // Code is the short id of the error. @@ -314,8 +505,10 @@ func (e SharedConfigAssumeRoleError) Code() string { // Message is the description of the error func (e SharedConfigAssumeRoleError) Message() string { - return fmt.Sprintf("failed to load assume role for %s, source profile has no shared credentials", - e.RoleARN) + return fmt.Sprintf( + "failed to load assume role for %s, source profile %s has no shared credentials", + e.RoleARN, e.SourceProfile, + ) } // OrigErr is the underlying error that caused the failure. @@ -327,3 +520,36 @@ func (e SharedConfigAssumeRoleError) OrigErr() error { func (e SharedConfigAssumeRoleError) Error() string { return awserr.SprintError(e.Code(), e.Message(), "", nil) } + +// CredentialRequiresARNError provides the error for shared config credentials +// that are incorrectly configured in the shared config or credentials file. +type CredentialRequiresARNError struct { + // type of credentials that were configured. + Type string + + // Profile name the credentials were in. + Profile string +} + +// Code is the short id of the error. +func (e CredentialRequiresARNError) Code() string { + return "CredentialRequiresARNError" +} + +// Message is the description of the error +func (e CredentialRequiresARNError) Message() string { + return fmt.Sprintf( + "credential type %s requires role_arn, profile %s", + e.Type, e.Profile, + ) +} + +// OrigErr is the underlying error that caused the failure. +func (e CredentialRequiresARNError) OrigErr() error { + return nil +} + +// Error satisfies the error interface. +func (e CredentialRequiresARNError) Error() string { + return awserr.SprintError(e.Code(), e.Message(), "", nil) +} diff --git a/github.com/aws/aws-sdk-go/aws/signer/v4/header_rules.go b/github.com/aws/aws-sdk-go/aws/signer/v4/header_rules.go index 244c86da05..07ea799fbd 100644 --- a/github.com/aws/aws-sdk-go/aws/signer/v4/header_rules.go +++ b/github.com/aws/aws-sdk-go/aws/signer/v4/header_rules.go @@ -1,8 +1,7 @@ package v4 import ( - "net/http" - "strings" + "github.com/aws/aws-sdk-go/internal/strings" ) // validator houses a set of rule needed for validation of a @@ -61,7 +60,7 @@ type patterns []string // been found func (p patterns) IsValid(value string) bool { for _, pattern := range p { - if strings.HasPrefix(http.CanonicalHeaderKey(value), pattern) { + if strings.HasPrefixFold(value, pattern) { return true } } diff --git a/github.com/aws/aws-sdk-go/aws/signer/v4/request_context_go1.5.go b/github.com/aws/aws-sdk-go/aws/signer/v4/request_context_go1.5.go new file mode 100644 index 0000000000..f35fc860b3 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/signer/v4/request_context_go1.5.go @@ -0,0 +1,13 @@ +// +build !go1.7 + +package v4 + +import ( + "net/http" + + "github.com/aws/aws-sdk-go/aws" +) + +func requestContext(r *http.Request) aws.Context { + return aws.BackgroundContext() +} diff --git a/github.com/aws/aws-sdk-go/aws/signer/v4/request_context_go1.7.go b/github.com/aws/aws-sdk-go/aws/signer/v4/request_context_go1.7.go new file mode 100644 index 0000000000..fed5c859ca --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/signer/v4/request_context_go1.7.go @@ -0,0 +1,13 @@ +// +build go1.7 + +package v4 + +import ( + "net/http" + + "github.com/aws/aws-sdk-go/aws" +) + +func requestContext(r *http.Request) aws.Context { + return r.Context() +} diff --git a/github.com/aws/aws-sdk-go/aws/signer/v4/stream.go b/github.com/aws/aws-sdk-go/aws/signer/v4/stream.go new file mode 100644 index 0000000000..02cbd97e23 --- /dev/null +++ b/github.com/aws/aws-sdk-go/aws/signer/v4/stream.go @@ -0,0 +1,63 @@ +package v4 + +import ( + "encoding/hex" + "strings" + "time" + + "github.com/aws/aws-sdk-go/aws/credentials" +) + +type credentialValueProvider interface { + Get() (credentials.Value, error) +} + +// StreamSigner implements signing of event stream encoded payloads +type StreamSigner struct { + region string + service string + + credentials credentialValueProvider + + prevSig []byte +} + +// NewStreamSigner creates a SigV4 signer used to sign Event Stream encoded messages +func NewStreamSigner(region, service string, seedSignature []byte, credentials *credentials.Credentials) *StreamSigner { + return &StreamSigner{ + region: region, + service: service, + credentials: credentials, + prevSig: seedSignature, + } +} + +// GetSignature takes an event stream encoded headers and payload and returns a signature +func (s *StreamSigner) GetSignature(headers, payload []byte, date time.Time) ([]byte, error) { + credValue, err := s.credentials.Get() + if err != nil { + return nil, err + } + + sigKey := deriveSigningKey(s.region, s.service, credValue.SecretAccessKey, date) + + keyPath := buildSigningScope(s.region, s.service, date) + + stringToSign := buildEventStreamStringToSign(headers, payload, s.prevSig, keyPath, date) + + signature := hmacSHA256(sigKey, []byte(stringToSign)) + s.prevSig = signature + + return signature, nil +} + +func buildEventStreamStringToSign(headers, payload, prevSig []byte, scope string, date time.Time) string { + return strings.Join([]string{ + "AWS4-HMAC-SHA256-PAYLOAD", + formatTime(date), + scope, + hex.EncodeToString(prevSig), + hex.EncodeToString(hashSHA256(headers)), + hex.EncodeToString(hashSHA256(payload)), + }, "\n") +} diff --git a/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go b/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go index 523db79f8d..d71f7b3f4f 100644 --- a/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go +++ b/github.com/aws/aws-sdk-go/aws/signer/v4/v4.go @@ -76,9 +76,14 @@ import ( ) const ( + authorizationHeader = "Authorization" + authHeaderSignatureElem = "Signature=" + signatureQueryKey = "X-Amz-Signature" + authHeaderPrefix = "AWS4-HMAC-SHA256" timeFormat = "20060102T150405Z" shortTimeFormat = "20060102" + awsV4Request = "aws4_request" // emptyStringSHA256 is a SHA256 of an empty string emptyStringSHA256 = `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` @@ -87,9 +92,9 @@ const ( var ignoredHeaders = rules{ blacklist{ mapRule{ - "Authorization": struct{}{}, - "User-Agent": struct{}{}, - "X-Amzn-Trace-Id": struct{}{}, + authorizationHeader: struct{}{}, + "User-Agent": struct{}{}, + "X-Amzn-Trace-Id": struct{}{}, }, }, } @@ -229,11 +234,9 @@ type signingCtx struct { DisableURIPathEscaping bool - credValues credentials.Value - isPresign bool - formattedTime string - formattedShortTime string - unsignedPayload bool + credValues credentials.Value + isPresign bool + unsignedPayload bool bodyDigest string signedHeaders string @@ -337,7 +340,7 @@ func (v4 Signer) signWithBody(r *http.Request, body io.ReadSeeker, service, regi } var err error - ctx.credValues, err = v4.Credentials.Get() + ctx.credValues, err = v4.Credentials.GetWithContext(requestContext(r)) if err != nil { return http.Header{}, err } @@ -532,39 +535,56 @@ func (ctx *signingCtx) build(disableHeaderHoisting bool) error { ctx.buildSignature() // depends on string to sign if ctx.isPresign { - ctx.Request.URL.RawQuery += "&X-Amz-Signature=" + ctx.signature + ctx.Request.URL.RawQuery += "&" + signatureQueryKey + "=" + ctx.signature } else { parts := []string{ authHeaderPrefix + " Credential=" + ctx.credValues.AccessKeyID + "/" + ctx.credentialString, "SignedHeaders=" + ctx.signedHeaders, - "Signature=" + ctx.signature, + authHeaderSignatureElem + ctx.signature, } - ctx.Request.Header.Set("Authorization", strings.Join(parts, ", ")) + ctx.Request.Header.Set(authorizationHeader, strings.Join(parts, ", ")) } return nil } -func (ctx *signingCtx) buildTime() { - ctx.formattedTime = ctx.Time.UTC().Format(timeFormat) - ctx.formattedShortTime = ctx.Time.UTC().Format(shortTimeFormat) +// GetSignedRequestSignature attempts to extract the signature of the request. +// Returning an error if the request is unsigned, or unable to extract the +// signature. +func GetSignedRequestSignature(r *http.Request) ([]byte, error) { + + if auth := r.Header.Get(authorizationHeader); len(auth) != 0 { + ps := strings.Split(auth, ", ") + for _, p := range ps { + if idx := strings.Index(p, authHeaderSignatureElem); idx >= 0 { + sig := p[len(authHeaderSignatureElem):] + if len(sig) == 0 { + return nil, fmt.Errorf("invalid request signature authorization header") + } + return hex.DecodeString(sig) + } + } + } + if sig := r.URL.Query().Get("X-Amz-Signature"); len(sig) != 0 { + return hex.DecodeString(sig) + } + + return nil, fmt.Errorf("request not signed") +} + +func (ctx *signingCtx) buildTime() { if ctx.isPresign { duration := int64(ctx.ExpireTime / time.Second) - ctx.Query.Set("X-Amz-Date", ctx.formattedTime) + ctx.Query.Set("X-Amz-Date", formatTime(ctx.Time)) ctx.Query.Set("X-Amz-Expires", strconv.FormatInt(duration, 10)) } else { - ctx.Request.Header.Set("X-Amz-Date", ctx.formattedTime) + ctx.Request.Header.Set("X-Amz-Date", formatTime(ctx.Time)) } } func (ctx *signingCtx) buildCredentialString() { - ctx.credentialString = strings.Join([]string{ - ctx.formattedShortTime, - ctx.Region, - ctx.ServiceName, - "aws4_request", - }, "/") + ctx.credentialString = buildSigningScope(ctx.Region, ctx.ServiceName, ctx.Time) if ctx.isPresign { ctx.Query.Set("X-Amz-Credential", ctx.credValues.AccessKeyID+"/"+ctx.credentialString) @@ -588,8 +608,7 @@ func (ctx *signingCtx) buildCanonicalHeaders(r rule, header http.Header) { var headers []string headers = append(headers, "host") for k, v := range header { - canonicalKey := http.CanonicalHeaderKey(k) - if !r.IsValid(canonicalKey) { + if !r.IsValid(k) { continue // ignored header } if ctx.SignedHeaderVals == nil { @@ -653,19 +672,15 @@ func (ctx *signingCtx) buildCanonicalString() { func (ctx *signingCtx) buildStringToSign() { ctx.stringToSign = strings.Join([]string{ authHeaderPrefix, - ctx.formattedTime, + formatTime(ctx.Time), ctx.credentialString, - hex.EncodeToString(makeSha256([]byte(ctx.canonicalString))), + hex.EncodeToString(hashSHA256([]byte(ctx.canonicalString))), }, "\n") } func (ctx *signingCtx) buildSignature() { - secret := ctx.credValues.SecretAccessKey - date := makeHmac([]byte("AWS4"+secret), []byte(ctx.formattedShortTime)) - region := makeHmac(date, []byte(ctx.Region)) - service := makeHmac(region, []byte(ctx.ServiceName)) - credentials := makeHmac(service, []byte("aws4_request")) - signature := makeHmac(credentials, []byte(ctx.stringToSign)) + creds := deriveSigningKey(ctx.Region, ctx.ServiceName, ctx.credValues.SecretAccessKey, ctx.Time) + signature := hmacSHA256(creds, []byte(ctx.stringToSign)) ctx.signature = hex.EncodeToString(signature) } @@ -687,7 +702,11 @@ func (ctx *signingCtx) buildBodyDigest() error { if !aws.IsReaderSeekable(ctx.Body) { return fmt.Errorf("cannot use unseekable request body %T, for signed request with body", ctx.Body) } - hash = hex.EncodeToString(makeSha256Reader(ctx.Body)) + hashBytes, err := makeSha256Reader(ctx.Body) + if err != nil { + return err + } + hash = hex.EncodeToString(hashBytes) } if includeSHA256Header { @@ -722,22 +741,28 @@ func (ctx *signingCtx) removePresign() { ctx.Query.Del("X-Amz-SignedHeaders") } -func makeHmac(key []byte, data []byte) []byte { +func hmacSHA256(key []byte, data []byte) []byte { hash := hmac.New(sha256.New, key) hash.Write(data) return hash.Sum(nil) } -func makeSha256(data []byte) []byte { +func hashSHA256(data []byte) []byte { hash := sha256.New() hash.Write(data) return hash.Sum(nil) } -func makeSha256Reader(reader io.ReadSeeker) []byte { +func makeSha256Reader(reader io.ReadSeeker) (hashBytes []byte, err error) { hash := sha256.New() - start, _ := reader.Seek(0, sdkio.SeekCurrent) - defer reader.Seek(start, sdkio.SeekStart) + start, err := reader.Seek(0, sdkio.SeekCurrent) + if err != nil { + return nil, err + } + defer func() { + // ensure error is return if unable to seek back to start of payload. + _, err = reader.Seek(start, sdkio.SeekStart) + }() // Use CopyN to avoid allocating the 32KB buffer in io.Copy for bodies // smaller than 32KB. Fall back to io.Copy if we fail to determine the size. @@ -748,7 +773,7 @@ func makeSha256Reader(reader io.ReadSeeker) []byte { io.CopyN(hash, reader, size) } - return hash.Sum(nil) + return hash.Sum(nil), nil } const doubleSpace = " " @@ -794,3 +819,28 @@ func stripExcessSpaces(vals []string) { vals[i] = string(buf[:m]) } } + +func buildSigningScope(region, service string, dt time.Time) string { + return strings.Join([]string{ + formatShortTime(dt), + region, + service, + awsV4Request, + }, "/") +} + +func deriveSigningKey(region, service, secretKey string, dt time.Time) []byte { + kDate := hmacSHA256([]byte("AWS4"+secretKey), []byte(formatShortTime(dt))) + kRegion := hmacSHA256(kDate, []byte(region)) + kService := hmacSHA256(kRegion, []byte(service)) + signingKey := hmacSHA256(kService, []byte(awsV4Request)) + return signingKey +} + +func formatShortTime(dt time.Time) string { + return dt.UTC().Format(shortTimeFormat) +} + +func formatTime(dt time.Time) string { + return dt.UTC().Format(timeFormat) +} diff --git a/github.com/aws/aws-sdk-go/aws/types.go b/github.com/aws/aws-sdk-go/aws/types.go index 8b6f23425a..98751ee84f 100644 --- a/github.com/aws/aws-sdk-go/aws/types.go +++ b/github.com/aws/aws-sdk-go/aws/types.go @@ -2,18 +2,24 @@ package aws import ( "io" + "strings" "sync" "github.com/aws/aws-sdk-go/internal/sdkio" ) -// ReadSeekCloser wraps a io.Reader returning a ReaderSeekerCloser. Should -// only be used with an io.Reader that is also an io.Seeker. Doing so may -// cause request signature errors, or request body's not sent for GET, HEAD -// and DELETE HTTP methods. +// ReadSeekCloser wraps a io.Reader returning a ReaderSeekerCloser. Allows the +// SDK to accept an io.Reader that is not also an io.Seeker for unsigned +// streaming payload API operations. // -// Deprecated: Should only be used with io.ReadSeeker. If using for -// S3 PutObject to stream content use s3manager.Uploader instead. +// A ReadSeekCloser wrapping an nonseekable io.Reader used in an API +// operation's input will prevent that operation being retried in the case of +// network errors, and cause operation requests to fail if the operation +// requires payload signing. +// +// Note: If using With S3 PutObject to stream an object upload The SDK's S3 +// Upload manager (s3manager.Uploader) provides support for streaming with the +// ability to retry network errors. func ReadSeekCloser(r io.Reader) ReaderSeekerCloser { return ReaderSeekerCloser{r} } @@ -43,7 +49,8 @@ func IsReaderSeekable(r io.Reader) bool { // Read reads from the reader up to size of p. The number of bytes read, and // error if it occurred will be returned. // -// If the reader is not an io.Reader zero bytes read, and nil error will be returned. +// If the reader is not an io.Reader zero bytes read, and nil error will be +// returned. // // Performs the same functionality as io.Reader Read func (r ReaderSeekerCloser) Read(p []byte) (int, error) { @@ -199,3 +206,59 @@ func (b *WriteAtBuffer) Bytes() []byte { defer b.m.Unlock() return b.buf } + +// MultiCloser is a utility to close multiple io.Closers within a single +// statement. +type MultiCloser []io.Closer + +// Close closes all of the io.Closers making up the MultiClosers. Any +// errors that occur while closing will be returned in the order they +// occur. +func (m MultiCloser) Close() error { + var errs errors + for _, c := range m { + err := c.Close() + if err != nil { + errs = append(errs, err) + } + } + if len(errs) != 0 { + return errs + } + + return nil +} + +type errors []error + +func (es errors) Error() string { + var parts []string + for _, e := range es { + parts = append(parts, e.Error()) + } + + return strings.Join(parts, "\n") +} + +// CopySeekableBody copies the seekable body to an io.Writer +func CopySeekableBody(dst io.Writer, src io.ReadSeeker) (int64, error) { + curPos, err := src.Seek(0, sdkio.SeekCurrent) + if err != nil { + return 0, err + } + + // copy errors may be assumed to be from the body. + n, err := io.Copy(dst, src) + if err != nil { + return n, err + } + + // seek back to the first position after reading to reset + // the body for transmission. + _, err = src.Seek(curPos, sdkio.SeekStart) + if err != nil { + return n, err + } + + return n, nil +} diff --git a/github.com/aws/aws-sdk-go/aws/version.go b/github.com/aws/aws-sdk-go/aws/version.go index f8869e4378..ab2e5269e0 100644 --- a/github.com/aws/aws-sdk-go/aws/version.go +++ b/github.com/aws/aws-sdk-go/aws/version.go @@ -5,4 +5,4 @@ package aws const SDKName = "aws-sdk-go" // SDKVersion is the version of this SDK -const SDKVersion = "1.16.19" +const SDKVersion = "1.33.8" diff --git a/github.com/aws/aws-sdk-go/aws/context_1_6.go b/github.com/aws/aws-sdk-go/internal/context/background_go1.5.go similarity index 86% rename from github.com/aws/aws-sdk-go/aws/context_1_6.go rename to github.com/aws/aws-sdk-go/internal/context/background_go1.5.go index 8fdda53033..876dcb3fde 100644 --- a/github.com/aws/aws-sdk-go/aws/context_1_6.go +++ b/github.com/aws/aws-sdk-go/internal/context/background_go1.5.go @@ -1,6 +1,6 @@ // +build !go1.7 -package aws +package context import "time" @@ -30,12 +30,11 @@ func (*emptyCtx) Value(key interface{}) interface{} { func (e *emptyCtx) String() string { switch e { - case backgroundCtx: + case BackgroundCtx: return "aws.BackgroundContext" } return "unknown empty Context" } -var ( - backgroundCtx = new(emptyCtx) -) +// BackgroundCtx is the common base context. +var BackgroundCtx = new(emptyCtx) diff --git a/github.com/aws/aws-sdk-go/internal/ini/ini_parser.go b/github.com/aws/aws-sdk-go/internal/ini/ini_parser.go index f99703372c..cf9fad81e7 100644 --- a/github.com/aws/aws-sdk-go/internal/ini/ini_parser.go +++ b/github.com/aws/aws-sdk-go/internal/ini/ini_parser.go @@ -162,7 +162,7 @@ loop: if len(tokens) == 0 { break loop } - + // if should skip is true, we skip the tokens until should skip is set to false. step = SkipTokenState } @@ -218,7 +218,7 @@ loop: // S -> equal_expr' expr_stmt' switch k.Kind { case ASTKindEqualExpr: - // assiging a value to some key + // assigning a value to some key k.AppendChild(newExpression(tok)) stack.Push(newExprStatement(k)) case ASTKindExpr: @@ -250,6 +250,13 @@ loop: if !runeCompare(tok.Raw(), openBrace) { return nil, NewParseError("expected '['") } + // If OpenScopeState is not at the start, we must mark the previous ast as complete + // + // for example: if previous ast was a skip statement; + // we should mark it as complete before we create a new statement + if k.Kind != ASTKindStart { + stack.MarkComplete(k) + } stmt := newStatement() stack.Push(stmt) @@ -304,7 +311,9 @@ loop: stmt := newCommentStatement(tok) stack.Push(stmt) default: - return nil, NewParseError(fmt.Sprintf("invalid state with ASTKind %v and TokenType %v", k, tok)) + return nil, NewParseError( + fmt.Sprintf("invalid state with ASTKind %v and TokenType %v", + k, tok.Type())) } if len(tokens) > 0 { @@ -314,7 +323,7 @@ loop: // this occurs when a statement has not been completed if stack.top > 1 { - return nil, NewParseError(fmt.Sprintf("incomplete expression: %v", stack.container)) + return nil, NewParseError(fmt.Sprintf("incomplete ini expression")) } // returns a sublist which excludes the start symbol diff --git a/github.com/aws/aws-sdk-go/internal/ini/skipper.go b/github.com/aws/aws-sdk-go/internal/ini/skipper.go index 6bb6964475..da7a4049cf 100644 --- a/github.com/aws/aws-sdk-go/internal/ini/skipper.go +++ b/github.com/aws/aws-sdk-go/internal/ini/skipper.go @@ -22,24 +22,24 @@ func newSkipper() skipper { } func (s *skipper) ShouldSkip(tok Token) bool { + // should skip state will be modified only if previous token was new line (NL); + // and the current token is not WhiteSpace (WS). if s.shouldSkip && s.prevTok.Type() == TokenNL && tok.Type() != TokenWS { - s.Continue() return false } s.prevTok = tok - return s.shouldSkip } func (s *skipper) Skip() { s.shouldSkip = true - s.prevTok = emptyToken } func (s *skipper) Continue() { s.shouldSkip = false + // empty token is assigned as we return to default state, when should skip is false s.prevTok = emptyToken } diff --git a/github.com/aws/aws-sdk-go/internal/sdkio/byte.go b/github.com/aws/aws-sdk-go/internal/sdkio/byte.go new file mode 100644 index 0000000000..6c443988bb --- /dev/null +++ b/github.com/aws/aws-sdk-go/internal/sdkio/byte.go @@ -0,0 +1,12 @@ +package sdkio + +const ( + // Byte is 8 bits + Byte int64 = 1 + // KibiByte (KiB) is 1024 Bytes + KibiByte = Byte * 1024 + // MebiByte (MiB) is 1024 KiB + MebiByte = KibiByte * 1024 + // GibiByte (GiB) is 1024 MiB + GibiByte = MebiByte * 1024 +) diff --git a/github.com/aws/aws-sdk-go/internal/sdkmath/floor.go b/github.com/aws/aws-sdk-go/internal/sdkmath/floor.go new file mode 100644 index 0000000000..44898eed0f --- /dev/null +++ b/github.com/aws/aws-sdk-go/internal/sdkmath/floor.go @@ -0,0 +1,15 @@ +// +build go1.10 + +package sdkmath + +import "math" + +// Round returns the nearest integer, rounding half away from zero. +// +// Special cases are: +// Round(±0) = ±0 +// Round(±Inf) = ±Inf +// Round(NaN) = NaN +func Round(x float64) float64 { + return math.Round(x) +} diff --git a/github.com/aws/aws-sdk-go/internal/sdkmath/floor_go1.9.go b/github.com/aws/aws-sdk-go/internal/sdkmath/floor_go1.9.go new file mode 100644 index 0000000000..810ec7f08b --- /dev/null +++ b/github.com/aws/aws-sdk-go/internal/sdkmath/floor_go1.9.go @@ -0,0 +1,56 @@ +// +build !go1.10 + +package sdkmath + +import "math" + +// Copied from the Go standard library's (Go 1.12) math/floor.go for use in +// Go version prior to Go 1.10. +const ( + uvone = 0x3FF0000000000000 + mask = 0x7FF + shift = 64 - 11 - 1 + bias = 1023 + signMask = 1 << 63 + fracMask = 1<= 0.5 { + // return t + Copysign(1, x) + // } + // return t + // } + bits := math.Float64bits(x) + e := uint(bits>>shift) & mask + if e < bias { + // Round abs(x) < 1 including denormals. + bits &= signMask // +-0 + if e == bias-1 { + bits |= uvone // +-1 + } + } else if e < bias+shift { + // Round any abs(x) >= 1 containing a fractional component [0,1). + // + // Numbers with larger exponents are returned unchanged since they + // must be either an integer, infinity, or NaN. + const half = 1 << (shift - 1) + e -= bias + bits += half >> e + bits &^= fracMask >> e + } + return math.Float64frombits(bits) +} diff --git a/github.com/aws/aws-sdk-go/internal/sdkrand/read.go b/github.com/aws/aws-sdk-go/internal/sdkrand/read.go new file mode 100644 index 0000000000..f4651da2da --- /dev/null +++ b/github.com/aws/aws-sdk-go/internal/sdkrand/read.go @@ -0,0 +1,11 @@ +// +build go1.6 + +package sdkrand + +import "math/rand" + +// Read provides the stub for math.Rand.Read method support for go version's +// 1.6 and greater. +func Read(r *rand.Rand, p []byte) (int, error) { + return r.Read(p) +} diff --git a/github.com/aws/aws-sdk-go/internal/sdkrand/read_1_5.go b/github.com/aws/aws-sdk-go/internal/sdkrand/read_1_5.go new file mode 100644 index 0000000000..b1d93a33d4 --- /dev/null +++ b/github.com/aws/aws-sdk-go/internal/sdkrand/read_1_5.go @@ -0,0 +1,24 @@ +// +build !go1.6 + +package sdkrand + +import "math/rand" + +// Read backfills Go 1.6's math.Rand.Reader for Go 1.5 +func Read(r *rand.Rand, p []byte) (n int, err error) { + // Copy of Go standard libraries math package's read function not added to + // standard library until Go 1.6. + var pos int8 + var val int64 + for n = 0; n < len(p); n++ { + if pos == 0 { + val = r.Int63() + pos = 7 + } + p[n] = byte(val) + val >>= 8 + pos-- + } + + return n, err +} diff --git a/github.com/aws/aws-sdk-go/internal/strings/strings.go b/github.com/aws/aws-sdk-go/internal/strings/strings.go new file mode 100644 index 0000000000..d008ae27cb --- /dev/null +++ b/github.com/aws/aws-sdk-go/internal/strings/strings.go @@ -0,0 +1,11 @@ +package strings + +import ( + "strings" +) + +// HasPrefixFold tests whether the string s begins with prefix, interpreted as UTF-8 strings, +// under Unicode case-folding. +func HasPrefixFold(s, prefix string) bool { + return len(s) >= len(prefix) && strings.EqualFold(s[0:len(prefix)], prefix) +} diff --git a/github.com/aws/aws-sdk-go/internal/sync/singleflight/LICENSE b/github.com/aws/aws-sdk-go/internal/sync/singleflight/LICENSE new file mode 100644 index 0000000000..6a66aea5ea --- /dev/null +++ b/github.com/aws/aws-sdk-go/internal/sync/singleflight/LICENSE @@ -0,0 +1,27 @@ +Copyright (c) 2009 The Go Authors. All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + + * Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. + * Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/github.com/aws/aws-sdk-go/internal/sync/singleflight/singleflight.go b/github.com/aws/aws-sdk-go/internal/sync/singleflight/singleflight.go new file mode 100644 index 0000000000..14ad0c5891 --- /dev/null +++ b/github.com/aws/aws-sdk-go/internal/sync/singleflight/singleflight.go @@ -0,0 +1,120 @@ +// Copyright 2013 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// Package singleflight provides a duplicate function call suppression +// mechanism. +package singleflight + +import "sync" + +// call is an in-flight or completed singleflight.Do call +type call struct { + wg sync.WaitGroup + + // These fields are written once before the WaitGroup is done + // and are only read after the WaitGroup is done. + val interface{} + err error + + // forgotten indicates whether Forget was called with this call's key + // while the call was still in flight. + forgotten bool + + // These fields are read and written with the singleflight + // mutex held before the WaitGroup is done, and are read but + // not written after the WaitGroup is done. + dups int + chans []chan<- Result +} + +// Group represents a class of work and forms a namespace in +// which units of work can be executed with duplicate suppression. +type Group struct { + mu sync.Mutex // protects m + m map[string]*call // lazily initialized +} + +// Result holds the results of Do, so they can be passed +// on a channel. +type Result struct { + Val interface{} + Err error + Shared bool +} + +// Do executes and returns the results of the given function, making +// sure that only one execution is in-flight for a given key at a +// time. If a duplicate comes in, the duplicate caller waits for the +// original to complete and receives the same results. +// The return value shared indicates whether v was given to multiple callers. +func (g *Group) Do(key string, fn func() (interface{}, error)) (v interface{}, err error, shared bool) { + g.mu.Lock() + if g.m == nil { + g.m = make(map[string]*call) + } + if c, ok := g.m[key]; ok { + c.dups++ + g.mu.Unlock() + c.wg.Wait() + return c.val, c.err, true + } + c := new(call) + c.wg.Add(1) + g.m[key] = c + g.mu.Unlock() + + g.doCall(c, key, fn) + return c.val, c.err, c.dups > 0 +} + +// DoChan is like Do but returns a channel that will receive the +// results when they are ready. +func (g *Group) DoChan(key string, fn func() (interface{}, error)) <-chan Result { + ch := make(chan Result, 1) + g.mu.Lock() + if g.m == nil { + g.m = make(map[string]*call) + } + if c, ok := g.m[key]; ok { + c.dups++ + c.chans = append(c.chans, ch) + g.mu.Unlock() + return ch + } + c := &call{chans: []chan<- Result{ch}} + c.wg.Add(1) + g.m[key] = c + g.mu.Unlock() + + go g.doCall(c, key, fn) + + return ch +} + +// doCall handles the single call for a key. +func (g *Group) doCall(c *call, key string, fn func() (interface{}, error)) { + c.val, c.err = fn() + c.wg.Done() + + g.mu.Lock() + if !c.forgotten { + delete(g.m, key) + } + for _, ch := range c.chans { + ch <- Result{c.val, c.err, c.dups > 0} + } + g.mu.Unlock() +} + +// Forget tells the singleflight to forget about a key. Future calls +// to Do for this key will call the function rather than waiting for +// an earlier call to complete. +func (g *Group) Forget(key string) { + g.mu.Lock() + if c, ok := g.m[key]; ok { + c.forgotten = true + } + delete(g.m, key) + g.mu.Unlock() +} diff --git a/github.com/aws/aws-sdk-go/private/checksum/content_md5.go b/github.com/aws/aws-sdk-go/private/checksum/content_md5.go new file mode 100644 index 0000000000..e045f38d83 --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/checksum/content_md5.go @@ -0,0 +1,53 @@ +package checksum + +import ( + "crypto/md5" + "encoding/base64" + "fmt" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/aws/request" +) + +const contentMD5Header = "Content-Md5" + +// AddBodyContentMD5Handler computes and sets the HTTP Content-MD5 header for requests that +// require it. +func AddBodyContentMD5Handler(r *request.Request) { + // if Content-MD5 header is already present, return + if v := r.HTTPRequest.Header.Get(contentMD5Header); len(v) != 0 { + return + } + + // if S3DisableContentMD5Validation flag is set, return + if aws.BoolValue(r.Config.S3DisableContentMD5Validation) { + return + } + + // if request is presigned, return + if r.IsPresigned() { + return + } + + // if body is not seekable, return + if !aws.IsReaderSeekable(r.Body) { + if r.Config.Logger != nil { + r.Config.Logger.Log(fmt.Sprintf( + "Unable to compute Content-MD5 for unseekable body, S3.%s", + r.Operation.Name)) + } + return + } + + h := md5.New() + + if _, err := aws.CopySeekableBody(h, r.Body); err != nil { + r.Error = awserr.New("ContentMD5", "failed to compute body MD5", err) + return + } + + // encode the md5 checksum in base64 and set the request header. + v := base64.StdEncoding.EncodeToString(h.Sum(nil)) + r.HTTPRequest.Header.Set(contentMD5Header, v) +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/debug.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/debug.go index ecc7bf82fa..151054971a 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/eventstream/debug.go +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/debug.go @@ -101,7 +101,7 @@ func (hs *decodedHeaders) UnmarshalJSON(b []byte) error { } headers.Set(h.Name, value) } - (*hs) = decodedHeaders(headers) + *hs = decodedHeaders(headers) return nil } diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/decode.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/decode.go index 4b972b2d66..4743393918 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/eventstream/decode.go +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/decode.go @@ -21,10 +21,24 @@ type Decoder struct { // NewDecoder initializes and returns a Decoder for decoding event // stream messages from the reader provided. -func NewDecoder(r io.Reader) *Decoder { - return &Decoder{ +func NewDecoder(r io.Reader, opts ...func(*Decoder)) *Decoder { + d := &Decoder{ r: r, } + + for _, opt := range opts { + opt(d) + } + + return d +} + +// DecodeWithLogger adds a logger to be used by the decoder when decoding +// stream events. +func DecodeWithLogger(logger aws.Logger) func(*Decoder) { + return func(d *Decoder) { + d.logger = logger + } } // Decode attempts to decode a single message from the event stream reader. @@ -40,6 +54,15 @@ func (d *Decoder) Decode(payloadBuf []byte) (m Message, err error) { }() } + m, err = Decode(reader, payloadBuf) + + return m, err +} + +// Decode attempts to decode a single message from the event stream reader. +// Will return the event stream message, or error if Decode fails to read +// the message from the reader. +func Decode(reader io.Reader, payloadBuf []byte) (m Message, err error) { crc := crc32.New(crc32IEEETable) hashReader := io.TeeReader(reader, crc) @@ -72,12 +95,6 @@ func (d *Decoder) Decode(payloadBuf []byte) (m Message, err error) { return m, nil } -// UseLogger specifies the Logger that that the decoder should use to log the -// message decode to. -func (d *Decoder) UseLogger(logger aws.Logger) { - d.logger = logger -} - func logMessageDecode(logger aws.Logger, msgBuf *bytes.Buffer, msg Message, decodeErr error) { w := bytes.NewBuffer(nil) defer func() { logger.Log(w.String()) }() diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/encode.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/encode.go index 150a60981d..ffade3bc0c 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/eventstream/encode.go +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/encode.go @@ -3,61 +3,107 @@ package eventstream import ( "bytes" "encoding/binary" + "encoding/hex" + "encoding/json" + "fmt" "hash" "hash/crc32" "io" + + "github.com/aws/aws-sdk-go/aws" ) // Encoder provides EventStream message encoding. type Encoder struct { - w io.Writer + w io.Writer + logger aws.Logger headersBuf *bytes.Buffer } // NewEncoder initializes and returns an Encoder to encode Event Stream // messages to an io.Writer. -func NewEncoder(w io.Writer) *Encoder { - return &Encoder{ +func NewEncoder(w io.Writer, opts ...func(*Encoder)) *Encoder { + e := &Encoder{ w: w, headersBuf: bytes.NewBuffer(nil), } + + for _, opt := range opts { + opt(e) + } + + return e +} + +// EncodeWithLogger adds a logger to be used by the encode when decoding +// stream events. +func EncodeWithLogger(logger aws.Logger) func(*Encoder) { + return func(d *Encoder) { + d.logger = logger + } } // Encode encodes a single EventStream message to the io.Writer the Encoder // was created with. An error is returned if writing the message fails. -func (e *Encoder) Encode(msg Message) error { +func (e *Encoder) Encode(msg Message) (err error) { e.headersBuf.Reset() - err := encodeHeaders(e.headersBuf, msg.Headers) - if err != nil { + writer := e.w + if e.logger != nil { + encodeMsgBuf := bytes.NewBuffer(nil) + writer = io.MultiWriter(writer, encodeMsgBuf) + defer func() { + logMessageEncode(e.logger, encodeMsgBuf, msg, err) + }() + } + + if err = EncodeHeaders(e.headersBuf, msg.Headers); err != nil { return err } crc := crc32.New(crc32IEEETable) - hashWriter := io.MultiWriter(e.w, crc) + hashWriter := io.MultiWriter(writer, crc) headersLen := uint32(e.headersBuf.Len()) payloadLen := uint32(len(msg.Payload)) - if err := encodePrelude(hashWriter, crc, headersLen, payloadLen); err != nil { + if err = encodePrelude(hashWriter, crc, headersLen, payloadLen); err != nil { return err } if headersLen > 0 { - if _, err := io.Copy(hashWriter, e.headersBuf); err != nil { + if _, err = io.Copy(hashWriter, e.headersBuf); err != nil { return err } } if payloadLen > 0 { - if _, err := hashWriter.Write(msg.Payload); err != nil { + if _, err = hashWriter.Write(msg.Payload); err != nil { return err } } msgCRC := crc.Sum32() - return binary.Write(e.w, binary.BigEndian, msgCRC) + return binary.Write(writer, binary.BigEndian, msgCRC) +} + +func logMessageEncode(logger aws.Logger, msgBuf *bytes.Buffer, msg Message, encodeErr error) { + w := bytes.NewBuffer(nil) + defer func() { logger.Log(w.String()) }() + + fmt.Fprintf(w, "Message to encode:\n") + encoder := json.NewEncoder(w) + if err := encoder.Encode(msg); err != nil { + fmt.Fprintf(w, "Failed to get encoded message, %v\n", err) + } + + if encodeErr != nil { + fmt.Fprintf(w, "Encode error: %v\n", encodeErr) + return + } + + fmt.Fprintf(w, "Raw message:\n%s\n", hex.Dump(msgBuf.Bytes())) } func encodePrelude(w io.Writer, crc hash.Hash32, headersLen, payloadLen uint32) error { @@ -86,7 +132,9 @@ func encodePrelude(w io.Writer, crc hash.Hash32, headersLen, payloadLen uint32) return nil } -func encodeHeaders(w io.Writer, headers Headers) error { +// EncodeHeaders writes the header values to the writer encoded in the event +// stream format. Returns an error if a header fails to encode. +func EncodeHeaders(w io.Writer, headers Headers) error { for _, h := range headers { hn := headerName{ Len: uint8(len(h.Name)), diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/error.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/error.go index 5ea5a988b6..34c2e89d53 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/error.go +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/error.go @@ -1,6 +1,9 @@ package eventstreamapi -import "fmt" +import ( + "fmt" + "sync" +) type messageError struct { code string @@ -22,3 +25,53 @@ func (e messageError) Error() string { func (e messageError) OrigErr() error { return nil } + +// OnceError wraps the behavior of recording an error +// once and signal on a channel when this has occurred. +// Signaling is done by closing of the channel. +// +// Type is safe for concurrent usage. +type OnceError struct { + mu sync.RWMutex + err error + ch chan struct{} +} + +// NewOnceError return a new OnceError +func NewOnceError() *OnceError { + return &OnceError{ + ch: make(chan struct{}, 1), + } +} + +// Err acquires a read-lock and returns an +// error if one has been set. +func (e *OnceError) Err() error { + e.mu.RLock() + err := e.err + e.mu.RUnlock() + + return err +} + +// SetError acquires a write-lock and will set +// the underlying error value if one has not been set. +func (e *OnceError) SetError(err error) { + if err == nil { + return + } + + e.mu.Lock() + if e.err == nil { + e.err = err + close(e.ch) + } + e.mu.Unlock() +} + +// ErrorSet returns a channel that will be used to signal +// that an error has been set. This channel will be closed +// when the error value has been set for OnceError. +func (e *OnceError) ErrorSet() <-chan struct{} { + return e.ch +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/api.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/reader.go similarity index 76% rename from github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/api.go rename to github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/reader.go index 97937c8e59..0e4aa42f3e 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/api.go +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/reader.go @@ -2,9 +2,7 @@ package eventstreamapi import ( "fmt" - "io" - "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/private/protocol" "github.com/aws/aws-sdk-go/private/protocol/eventstream" ) @@ -15,27 +13,8 @@ type Unmarshaler interface { UnmarshalEvent(protocol.PayloadUnmarshaler, eventstream.Message) error } -// EventStream headers with specific meaning to async API functionality. -const ( - MessageTypeHeader = `:message-type` // Identifies type of message. - EventMessageType = `event` - ErrorMessageType = `error` - ExceptionMessageType = `exception` - - // Message Events - EventTypeHeader = `:event-type` // Identifies message event type e.g. "Stats". - - // Message Error - ErrorCodeHeader = `:error-code` - ErrorMessageHeader = `:error-message` - - // Message Exception - ExceptionTypeHeader = `:exception-type` -) - // EventReader provides reading from the EventStream of an reader. type EventReader struct { - reader io.ReadCloser decoder *eventstream.Decoder unmarshalerForEventType func(string) (Unmarshaler, error) @@ -47,27 +26,18 @@ type EventReader struct { // NewEventReader returns a EventReader built from the reader and unmarshaler // provided. Use ReadStream method to start reading from the EventStream. func NewEventReader( - reader io.ReadCloser, + decoder *eventstream.Decoder, payloadUnmarshaler protocol.PayloadUnmarshaler, unmarshalerForEventType func(string) (Unmarshaler, error), ) *EventReader { return &EventReader{ - reader: reader, - decoder: eventstream.NewDecoder(reader), + decoder: decoder, payloadUnmarshaler: payloadUnmarshaler, unmarshalerForEventType: unmarshalerForEventType, payloadBuf: make([]byte, 10*1024), } } -// UseLogger instructs the EventReader to use the logger and log level -// specified. -func (r *EventReader) UseLogger(logger aws.Logger, logLevel aws.LogLevelType) { - if logger != nil && logLevel.Matches(aws.LogDebugWithEventStreamBody) { - r.decoder.UseLogger(logger) - } -} - // ReadEvent attempts to read a message from the EventStream and return the // unmarshaled event value that the message is for. // @@ -95,15 +65,27 @@ func (r *EventReader) ReadEvent() (event interface{}, err error) { case EventMessageType: return r.unmarshalEventMessage(msg) case ExceptionMessageType: - err = r.unmarshalEventException(msg) - return nil, err + return nil, r.unmarshalEventException(msg) case ErrorMessageType: return nil, r.unmarshalErrorMessage(msg) default: - return nil, fmt.Errorf("unknown eventstream message type, %v", typ) + return nil, &UnknownMessageTypeError{ + Type: typ, Message: msg.Clone(), + } } } +// UnknownMessageTypeError provides an error when a message is received from +// the stream, but the reader is unable to determine what kind of message it is. +type UnknownMessageTypeError struct { + Type string + Message eventstream.Message +} + +func (e *UnknownMessageTypeError) Error() string { + return "unknown eventstream message type, " + e.Type +} + func (r *EventReader) unmarshalEventMessage( msg eventstream.Message, ) (event interface{}, err error) { @@ -174,11 +156,6 @@ func (r *EventReader) unmarshalErrorMessage(msg eventstream.Message) (err error) return msgErr } -// Close closes the EventReader's EventStream reader. -func (r *EventReader) Close() error { - return r.reader.Close() -} - // GetHeaderString returns the value of the header as a string. If the header // is not set or the value is not a string an error will be returned. func GetHeaderString(msg eventstream.Message, headerName string) (string, error) { diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/shared.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/shared.go new file mode 100644 index 0000000000..e46b8acc20 --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/shared.go @@ -0,0 +1,23 @@ +package eventstreamapi + +// EventStream headers with specific meaning to async API functionality. +const ( + ChunkSignatureHeader = `:chunk-signature` // chunk signature for message + DateHeader = `:date` // Date header for signature + + // Message header and values + MessageTypeHeader = `:message-type` // Identifies type of message. + EventMessageType = `event` + ErrorMessageType = `error` + ExceptionMessageType = `exception` + + // Message Events + EventTypeHeader = `:event-type` // Identifies message event type e.g. "Stats". + + // Message Error + ErrorCodeHeader = `:error-code` + ErrorMessageHeader = `:error-message` + + // Message Exception + ExceptionTypeHeader = `:exception-type` +) diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/signer.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/signer.go new file mode 100644 index 0000000000..3a7ba5cd57 --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/signer.go @@ -0,0 +1,123 @@ +package eventstreamapi + +import ( + "bytes" + "strings" + "time" + + "github.com/aws/aws-sdk-go/private/protocol/eventstream" +) + +var timeNow = time.Now + +// StreamSigner defines an interface for the implementation of signing of event stream payloads +type StreamSigner interface { + GetSignature(headers, payload []byte, date time.Time) ([]byte, error) +} + +// SignEncoder envelopes event stream messages +// into an event stream message payload with included +// signature headers using the provided signer and encoder. +type SignEncoder struct { + signer StreamSigner + encoder Encoder + bufEncoder *BufferEncoder + + closeErr error + closed bool +} + +// NewSignEncoder returns a new SignEncoder using the provided stream signer and +// event stream encoder. +func NewSignEncoder(signer StreamSigner, encoder Encoder) *SignEncoder { + // TODO: Need to pass down logging + + return &SignEncoder{ + signer: signer, + encoder: encoder, + bufEncoder: NewBufferEncoder(), + } +} + +// Close encodes a final event stream signing envelope with an empty event stream +// payload. This final end-frame is used to mark the conclusion of the stream. +func (s *SignEncoder) Close() error { + if s.closed { + return s.closeErr + } + + if err := s.encode([]byte{}); err != nil { + if strings.Contains(err.Error(), "on closed pipe") { + return nil + } + + s.closeErr = err + s.closed = true + return s.closeErr + } + + return nil +} + +// Encode takes the provided message and add envelopes the message +// with the required signature. +func (s *SignEncoder) Encode(msg eventstream.Message) error { + payload, err := s.bufEncoder.Encode(msg) + if err != nil { + return err + } + + return s.encode(payload) +} + +func (s SignEncoder) encode(payload []byte) error { + date := timeNow() + + var msg eventstream.Message + msg.Headers.Set(DateHeader, eventstream.TimestampValue(date)) + msg.Payload = payload + + var headers bytes.Buffer + if err := eventstream.EncodeHeaders(&headers, msg.Headers); err != nil { + return err + } + + sig, err := s.signer.GetSignature(headers.Bytes(), msg.Payload, date) + if err != nil { + return err + } + + msg.Headers.Set(ChunkSignatureHeader, eventstream.BytesValue(sig)) + + return s.encoder.Encode(msg) +} + +// BufferEncoder is a utility that provides a buffered +// event stream encoder +type BufferEncoder struct { + encoder Encoder + buffer *bytes.Buffer +} + +// NewBufferEncoder returns a new BufferEncoder initialized +// with a 1024 byte buffer. +func NewBufferEncoder() *BufferEncoder { + buf := bytes.NewBuffer(make([]byte, 1024)) + return &BufferEncoder{ + encoder: eventstream.NewEncoder(buf), + buffer: buf, + } +} + +// Encode returns the encoded message as a byte slice. +// The returned byte slice will be modified on the next encode call +// and should not be held onto. +func (e *BufferEncoder) Encode(msg eventstream.Message) ([]byte, error) { + e.buffer.Reset() + + if err := e.encoder.Encode(msg); err != nil { + return nil, err + } + + return e.buffer.Bytes(), nil +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/stream_writer.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/stream_writer.go new file mode 100644 index 0000000000..433bb1630a --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/stream_writer.go @@ -0,0 +1,129 @@ +package eventstreamapi + +import ( + "fmt" + "io" + "sync" + + "github.com/aws/aws-sdk-go/aws" +) + +// StreamWriter provides concurrent safe writing to an event stream. +type StreamWriter struct { + eventWriter *EventWriter + stream chan eventWriteAsyncReport + + done chan struct{} + closeOnce sync.Once + err *OnceError + + streamCloser io.Closer +} + +// NewStreamWriter returns a StreamWriter for the event writer, and stream +// closer provided. +func NewStreamWriter(eventWriter *EventWriter, streamCloser io.Closer) *StreamWriter { + w := &StreamWriter{ + eventWriter: eventWriter, + streamCloser: streamCloser, + stream: make(chan eventWriteAsyncReport), + done: make(chan struct{}), + err: NewOnceError(), + } + go w.writeStream() + + return w +} + +// Close terminates the writers ability to write new events to the stream. Any +// future call to Send will fail with an error. +func (w *StreamWriter) Close() error { + w.closeOnce.Do(w.safeClose) + return w.Err() +} + +func (w *StreamWriter) safeClose() { + close(w.done) +} + +// ErrorSet returns a channel which will be closed +// if an error occurs. +func (w *StreamWriter) ErrorSet() <-chan struct{} { + return w.err.ErrorSet() +} + +// Err returns any error that occurred while attempting to write an event to the +// stream. +func (w *StreamWriter) Err() error { + return w.err.Err() +} + +// Send writes a single event to the stream returning an error if the write +// failed. +// +// Send may be called concurrently. Events will be written to the stream +// safely. +func (w *StreamWriter) Send(ctx aws.Context, event Marshaler) error { + if err := w.Err(); err != nil { + return err + } + + resultCh := make(chan error) + wrapped := eventWriteAsyncReport{ + Event: event, + Result: resultCh, + } + + select { + case w.stream <- wrapped: + case <-ctx.Done(): + return ctx.Err() + case <-w.done: + return fmt.Errorf("stream closed, unable to send event") + } + + select { + case err := <-resultCh: + return err + case <-ctx.Done(): + return ctx.Err() + case <-w.done: + return fmt.Errorf("stream closed, unable to send event") + } +} + +func (w *StreamWriter) writeStream() { + defer w.Close() + + for { + select { + case wrapper := <-w.stream: + err := w.eventWriter.WriteEvent(wrapper.Event) + wrapper.ReportResult(w.done, err) + if err != nil { + w.err.SetError(err) + return + } + + case <-w.done: + if err := w.streamCloser.Close(); err != nil { + w.err.SetError(err) + } + return + } + } +} + +type eventWriteAsyncReport struct { + Event Marshaler + Result chan<- error +} + +func (e eventWriteAsyncReport) ReportResult(cancel <-chan struct{}, err error) bool { + select { + case e.Result <- err: + return true + case <-cancel: + return false + } +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/writer.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/writer.go new file mode 100644 index 0000000000..10a3823dfa --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi/writer.go @@ -0,0 +1,109 @@ +package eventstreamapi + +import ( + "github.com/aws/aws-sdk-go/private/protocol" + "github.com/aws/aws-sdk-go/private/protocol/eventstream" +) + +// Marshaler provides a marshaling interface for event types to event stream +// messages. +type Marshaler interface { + MarshalEvent(protocol.PayloadMarshaler) (eventstream.Message, error) +} + +// Encoder is an stream encoder that will encode an event stream message for +// the transport. +type Encoder interface { + Encode(eventstream.Message) error +} + +// EventWriter provides a wrapper around the underlying event stream encoder +// for an io.WriteCloser. +type EventWriter struct { + encoder Encoder + payloadMarshaler protocol.PayloadMarshaler + eventTypeFor func(Marshaler) (string, error) +} + +// NewEventWriter returns a new event stream writer, that will write to the +// writer provided. Use the WriteEvent method to write an event to the stream. +func NewEventWriter(encoder Encoder, pm protocol.PayloadMarshaler, eventTypeFor func(Marshaler) (string, error), +) *EventWriter { + return &EventWriter{ + encoder: encoder, + payloadMarshaler: pm, + eventTypeFor: eventTypeFor, + } +} + +// WriteEvent writes an event to the stream. Returns an error if the event +// fails to marshal into a message, or writing to the underlying writer fails. +func (w *EventWriter) WriteEvent(event Marshaler) error { + msg, err := w.marshal(event) + if err != nil { + return err + } + + return w.encoder.Encode(msg) +} + +func (w *EventWriter) marshal(event Marshaler) (eventstream.Message, error) { + eventType, err := w.eventTypeFor(event) + if err != nil { + return eventstream.Message{}, err + } + + msg, err := event.MarshalEvent(w.payloadMarshaler) + if err != nil { + return eventstream.Message{}, err + } + + msg.Headers.Set(EventTypeHeader, eventstream.StringValue(eventType)) + return msg, nil +} + +//type EventEncoder struct { +// encoder Encoder +// ppayloadMarshaler protocol.PayloadMarshaler +// eventTypeFor func(Marshaler) (string, error) +//} +// +//func (e EventEncoder) Encode(event Marshaler) error { +// msg, err := e.marshal(event) +// if err != nil { +// return err +// } +// +// return w.encoder.Encode(msg) +//} +// +//func (e EventEncoder) marshal(event Marshaler) (eventstream.Message, error) { +// eventType, err := w.eventTypeFor(event) +// if err != nil { +// return eventstream.Message{}, err +// } +// +// msg, err := event.MarshalEvent(w.payloadMarshaler) +// if err != nil { +// return eventstream.Message{}, err +// } +// +// msg.Headers.Set(EventTypeHeader, eventstream.StringValue(eventType)) +// return msg, nil +//} +// +//func (w *EventWriter) marshal(event Marshaler) (eventstream.Message, error) { +// eventType, err := w.eventTypeFor(event) +// if err != nil { +// return eventstream.Message{}, err +// } +// +// msg, err := event.MarshalEvent(w.payloadMarshaler) +// if err != nil { +// return eventstream.Message{}, err +// } +// +// msg.Headers.Set(EventTypeHeader, eventstream.StringValue(eventType)) +// return msg, nil +//} +// diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/header.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/header.go index 3b44dde2f3..f6f8c5674e 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/eventstream/header.go +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/header.go @@ -52,6 +52,15 @@ func (hs *Headers) Del(name string) { } } +// Clone returns a deep copy of the headers +func (hs Headers) Clone() Headers { + o := make(Headers, 0, len(hs)) + for _, h := range hs { + o.Set(h.Name, h.Value) + } + return o +} + func decodeHeaders(r io.Reader) (Headers, error) { hs := Headers{} diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/header_value.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/header_value.go index e3fc0766a9..9f509d8f6d 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/eventstream/header_value.go +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/header_value.go @@ -461,6 +461,11 @@ func (v *TimestampValue) decode(r io.Reader) error { return nil } +// MarshalJSON implements the json.Marshaler interface +func (v TimestampValue) MarshalJSON() ([]byte, error) { + return []byte(v.String()), nil +} + func timeFromEpochMilli(t int64) time.Time { secs := t / 1e3 msec := t % 1e3 diff --git a/github.com/aws/aws-sdk-go/private/protocol/eventstream/message.go b/github.com/aws/aws-sdk-go/private/protocol/eventstream/message.go index 2dc012a66e..f7427da039 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/eventstream/message.go +++ b/github.com/aws/aws-sdk-go/private/protocol/eventstream/message.go @@ -27,7 +27,7 @@ func (m *Message) rawMessage() (rawMessage, error) { if len(m.Headers) > 0 { var headers bytes.Buffer - if err := encodeHeaders(&headers, m.Headers); err != nil { + if err := EncodeHeaders(&headers, m.Headers); err != nil { return rawMessage{}, err } raw.Headers = headers.Bytes() @@ -57,6 +57,20 @@ func (m *Message) rawMessage() (rawMessage, error) { return raw, nil } +// Clone returns a deep copy of the message. +func (m Message) Clone() Message { + var payload []byte + if m.Payload != nil { + payload = make([]byte, len(m.Payload)) + copy(payload, m.Payload) + } + + return Message{ + Headers: m.Headers.Clone(), + Payload: payload, + } +} + type messagePrelude struct { Length uint32 HeadersLen uint32 diff --git a/github.com/aws/aws-sdk-go/private/protocol/json/jsonutil/build.go b/github.com/aws/aws-sdk-go/private/protocol/json/jsonutil/build.go new file mode 100644 index 0000000000..864fb6704b --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/json/jsonutil/build.go @@ -0,0 +1,296 @@ +// Package jsonutil provides JSON serialization of AWS requests and responses. +package jsonutil + +import ( + "bytes" + "encoding/base64" + "encoding/json" + "fmt" + "math" + "reflect" + "sort" + "strconv" + "time" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/private/protocol" +) + +var timeType = reflect.ValueOf(time.Time{}).Type() +var byteSliceType = reflect.ValueOf([]byte{}).Type() + +// BuildJSON builds a JSON string for a given object v. +func BuildJSON(v interface{}) ([]byte, error) { + var buf bytes.Buffer + + err := buildAny(reflect.ValueOf(v), &buf, "") + return buf.Bytes(), err +} + +func buildAny(value reflect.Value, buf *bytes.Buffer, tag reflect.StructTag) error { + origVal := value + value = reflect.Indirect(value) + if !value.IsValid() { + return nil + } + + vtype := value.Type() + + t := tag.Get("type") + if t == "" { + switch vtype.Kind() { + case reflect.Struct: + // also it can't be a time object + if value.Type() != timeType { + t = "structure" + } + case reflect.Slice: + // also it can't be a byte slice + if _, ok := value.Interface().([]byte); !ok { + t = "list" + } + case reflect.Map: + // cannot be a JSONValue map + if _, ok := value.Interface().(aws.JSONValue); !ok { + t = "map" + } + } + } + + switch t { + case "structure": + if field, ok := vtype.FieldByName("_"); ok { + tag = field.Tag + } + return buildStruct(value, buf, tag) + case "list": + return buildList(value, buf, tag) + case "map": + return buildMap(value, buf, tag) + default: + return buildScalar(origVal, buf, tag) + } +} + +func buildStruct(value reflect.Value, buf *bytes.Buffer, tag reflect.StructTag) error { + if !value.IsValid() { + return nil + } + + // unwrap payloads + if payload := tag.Get("payload"); payload != "" { + field, _ := value.Type().FieldByName(payload) + tag = field.Tag + value = elemOf(value.FieldByName(payload)) + + if !value.IsValid() { + return nil + } + } + + buf.WriteByte('{') + + t := value.Type() + first := true + for i := 0; i < t.NumField(); i++ { + member := value.Field(i) + + // This allocates the most memory. + // Additionally, we cannot skip nil fields due to + // idempotency auto filling. + field := t.Field(i) + + if field.PkgPath != "" { + continue // ignore unexported fields + } + if field.Tag.Get("json") == "-" { + continue + } + if field.Tag.Get("location") != "" { + continue // ignore non-body elements + } + if field.Tag.Get("ignore") != "" { + continue + } + + if protocol.CanSetIdempotencyToken(member, field) { + token := protocol.GetIdempotencyToken() + member = reflect.ValueOf(&token) + } + + if (member.Kind() == reflect.Ptr || member.Kind() == reflect.Slice || member.Kind() == reflect.Map) && member.IsNil() { + continue // ignore unset fields + } + + if first { + first = false + } else { + buf.WriteByte(',') + } + + // figure out what this field is called + name := field.Name + if locName := field.Tag.Get("locationName"); locName != "" { + name = locName + } + + writeString(name, buf) + buf.WriteString(`:`) + + err := buildAny(member, buf, field.Tag) + if err != nil { + return err + } + + } + + buf.WriteString("}") + + return nil +} + +func buildList(value reflect.Value, buf *bytes.Buffer, tag reflect.StructTag) error { + buf.WriteString("[") + + for i := 0; i < value.Len(); i++ { + buildAny(value.Index(i), buf, "") + + if i < value.Len()-1 { + buf.WriteString(",") + } + } + + buf.WriteString("]") + + return nil +} + +type sortedValues []reflect.Value + +func (sv sortedValues) Len() int { return len(sv) } +func (sv sortedValues) Swap(i, j int) { sv[i], sv[j] = sv[j], sv[i] } +func (sv sortedValues) Less(i, j int) bool { return sv[i].String() < sv[j].String() } + +func buildMap(value reflect.Value, buf *bytes.Buffer, tag reflect.StructTag) error { + buf.WriteString("{") + + sv := sortedValues(value.MapKeys()) + sort.Sort(sv) + + for i, k := range sv { + if i > 0 { + buf.WriteByte(',') + } + + writeString(k.String(), buf) + buf.WriteString(`:`) + + buildAny(value.MapIndex(k), buf, "") + } + + buf.WriteString("}") + + return nil +} + +func buildScalar(v reflect.Value, buf *bytes.Buffer, tag reflect.StructTag) error { + // prevents allocation on the heap. + scratch := [64]byte{} + switch value := reflect.Indirect(v); value.Kind() { + case reflect.String: + writeString(value.String(), buf) + case reflect.Bool: + if value.Bool() { + buf.WriteString("true") + } else { + buf.WriteString("false") + } + case reflect.Int64: + buf.Write(strconv.AppendInt(scratch[:0], value.Int(), 10)) + case reflect.Float64: + f := value.Float() + if math.IsInf(f, 0) || math.IsNaN(f) { + return &json.UnsupportedValueError{Value: v, Str: strconv.FormatFloat(f, 'f', -1, 64)} + } + buf.Write(strconv.AppendFloat(scratch[:0], f, 'f', -1, 64)) + default: + switch converted := value.Interface().(type) { + case time.Time: + format := tag.Get("timestampFormat") + if len(format) == 0 { + format = protocol.UnixTimeFormatName + } + + ts := protocol.FormatTime(format, converted) + if format != protocol.UnixTimeFormatName { + ts = `"` + ts + `"` + } + + buf.WriteString(ts) + case []byte: + if !value.IsNil() { + buf.WriteByte('"') + if len(converted) < 1024 { + // for small buffers, using Encode directly is much faster. + dst := make([]byte, base64.StdEncoding.EncodedLen(len(converted))) + base64.StdEncoding.Encode(dst, converted) + buf.Write(dst) + } else { + // for large buffers, avoid unnecessary extra temporary + // buffer space. + enc := base64.NewEncoder(base64.StdEncoding, buf) + enc.Write(converted) + enc.Close() + } + buf.WriteByte('"') + } + case aws.JSONValue: + str, err := protocol.EncodeJSONValue(converted, protocol.QuotedEscape) + if err != nil { + return fmt.Errorf("unable to encode JSONValue, %v", err) + } + buf.WriteString(str) + default: + return fmt.Errorf("unsupported JSON value %v (%s)", value.Interface(), value.Type()) + } + } + return nil +} + +var hex = "0123456789abcdef" + +func writeString(s string, buf *bytes.Buffer) { + buf.WriteByte('"') + for i := 0; i < len(s); i++ { + if s[i] == '"' { + buf.WriteString(`\"`) + } else if s[i] == '\\' { + buf.WriteString(`\\`) + } else if s[i] == '\b' { + buf.WriteString(`\b`) + } else if s[i] == '\f' { + buf.WriteString(`\f`) + } else if s[i] == '\r' { + buf.WriteString(`\r`) + } else if s[i] == '\t' { + buf.WriteString(`\t`) + } else if s[i] == '\n' { + buf.WriteString(`\n`) + } else if s[i] < 32 { + buf.WriteString("\\u00") + buf.WriteByte(hex[s[i]>>4]) + buf.WriteByte(hex[s[i]&0xF]) + } else { + buf.WriteByte(s[i]) + } + } + buf.WriteByte('"') +} + +// Returns the reflection element of a value, if it is a pointer. +func elemOf(value reflect.Value) reflect.Value { + for value.Kind() == reflect.Ptr { + value = value.Elem() + } + return value +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/json/jsonutil/unmarshal.go b/github.com/aws/aws-sdk-go/private/protocol/json/jsonutil/unmarshal.go new file mode 100644 index 0000000000..5e9499699b --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/json/jsonutil/unmarshal.go @@ -0,0 +1,282 @@ +package jsonutil + +import ( + "bytes" + "encoding/base64" + "encoding/json" + "fmt" + "io" + "reflect" + "strings" + "time" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/private/protocol" +) + +// UnmarshalJSONError unmarshal's the reader's JSON document into the passed in +// type. The value to unmarshal the json document into must be a pointer to the +// type. +func UnmarshalJSONError(v interface{}, stream io.Reader) error { + var errBuf bytes.Buffer + body := io.TeeReader(stream, &errBuf) + + err := json.NewDecoder(body).Decode(v) + if err != nil { + msg := "failed decoding error message" + if err == io.EOF { + msg = "error message missing" + err = nil + } + return awserr.NewUnmarshalError(err, msg, errBuf.Bytes()) + } + + return nil +} + +// UnmarshalJSON reads a stream and unmarshals the results in object v. +func UnmarshalJSON(v interface{}, stream io.Reader) error { + var out interface{} + + err := json.NewDecoder(stream).Decode(&out) + if err == io.EOF { + return nil + } else if err != nil { + return err + } + + return unmarshaler{}.unmarshalAny(reflect.ValueOf(v), out, "") +} + +// UnmarshalJSONCaseInsensitive reads a stream and unmarshals the result into the +// object v. Ignores casing for structure members. +func UnmarshalJSONCaseInsensitive(v interface{}, stream io.Reader) error { + var out interface{} + + err := json.NewDecoder(stream).Decode(&out) + if err == io.EOF { + return nil + } else if err != nil { + return err + } + + return unmarshaler{ + caseInsensitive: true, + }.unmarshalAny(reflect.ValueOf(v), out, "") +} + +type unmarshaler struct { + caseInsensitive bool +} + +func (u unmarshaler) unmarshalAny(value reflect.Value, data interface{}, tag reflect.StructTag) error { + vtype := value.Type() + if vtype.Kind() == reflect.Ptr { + vtype = vtype.Elem() // check kind of actual element type + } + + t := tag.Get("type") + if t == "" { + switch vtype.Kind() { + case reflect.Struct: + // also it can't be a time object + if _, ok := value.Interface().(*time.Time); !ok { + t = "structure" + } + case reflect.Slice: + // also it can't be a byte slice + if _, ok := value.Interface().([]byte); !ok { + t = "list" + } + case reflect.Map: + // cannot be a JSONValue map + if _, ok := value.Interface().(aws.JSONValue); !ok { + t = "map" + } + } + } + + switch t { + case "structure": + if field, ok := vtype.FieldByName("_"); ok { + tag = field.Tag + } + return u.unmarshalStruct(value, data, tag) + case "list": + return u.unmarshalList(value, data, tag) + case "map": + return u.unmarshalMap(value, data, tag) + default: + return u.unmarshalScalar(value, data, tag) + } +} + +func (u unmarshaler) unmarshalStruct(value reflect.Value, data interface{}, tag reflect.StructTag) error { + if data == nil { + return nil + } + mapData, ok := data.(map[string]interface{}) + if !ok { + return fmt.Errorf("JSON value is not a structure (%#v)", data) + } + + t := value.Type() + if value.Kind() == reflect.Ptr { + if value.IsNil() { // create the structure if it's nil + s := reflect.New(value.Type().Elem()) + value.Set(s) + value = s + } + + value = value.Elem() + t = t.Elem() + } + + // unwrap any payloads + if payload := tag.Get("payload"); payload != "" { + field, _ := t.FieldByName(payload) + return u.unmarshalAny(value.FieldByName(payload), data, field.Tag) + } + + for i := 0; i < t.NumField(); i++ { + field := t.Field(i) + if field.PkgPath != "" { + continue // ignore unexported fields + } + + // figure out what this field is called + name := field.Name + if locName := field.Tag.Get("locationName"); locName != "" { + name = locName + } + if u.caseInsensitive { + if _, ok := mapData[name]; !ok { + // Fallback to uncased name search if the exact name didn't match. + for kn, v := range mapData { + if strings.EqualFold(kn, name) { + mapData[name] = v + } + } + } + } + + member := value.FieldByIndex(field.Index) + err := u.unmarshalAny(member, mapData[name], field.Tag) + if err != nil { + return err + } + } + return nil +} + +func (u unmarshaler) unmarshalList(value reflect.Value, data interface{}, tag reflect.StructTag) error { + if data == nil { + return nil + } + listData, ok := data.([]interface{}) + if !ok { + return fmt.Errorf("JSON value is not a list (%#v)", data) + } + + if value.IsNil() { + l := len(listData) + value.Set(reflect.MakeSlice(value.Type(), l, l)) + } + + for i, c := range listData { + err := u.unmarshalAny(value.Index(i), c, "") + if err != nil { + return err + } + } + + return nil +} + +func (u unmarshaler) unmarshalMap(value reflect.Value, data interface{}, tag reflect.StructTag) error { + if data == nil { + return nil + } + mapData, ok := data.(map[string]interface{}) + if !ok { + return fmt.Errorf("JSON value is not a map (%#v)", data) + } + + if value.IsNil() { + value.Set(reflect.MakeMap(value.Type())) + } + + for k, v := range mapData { + kvalue := reflect.ValueOf(k) + vvalue := reflect.New(value.Type().Elem()).Elem() + + u.unmarshalAny(vvalue, v, "") + value.SetMapIndex(kvalue, vvalue) + } + + return nil +} + +func (u unmarshaler) unmarshalScalar(value reflect.Value, data interface{}, tag reflect.StructTag) error { + + switch d := data.(type) { + case nil: + return nil // nothing to do here + case string: + switch value.Interface().(type) { + case *string: + value.Set(reflect.ValueOf(&d)) + case []byte: + b, err := base64.StdEncoding.DecodeString(d) + if err != nil { + return err + } + value.Set(reflect.ValueOf(b)) + case *time.Time: + format := tag.Get("timestampFormat") + if len(format) == 0 { + format = protocol.ISO8601TimeFormatName + } + + t, err := protocol.ParseTime(format, d) + if err != nil { + return err + } + value.Set(reflect.ValueOf(&t)) + case aws.JSONValue: + // No need to use escaping as the value is a non-quoted string. + v, err := protocol.DecodeJSONValue(d, protocol.NoEscape) + if err != nil { + return err + } + value.Set(reflect.ValueOf(v)) + default: + return fmt.Errorf("unsupported value: %v (%s)", value.Interface(), value.Type()) + } + case float64: + switch value.Interface().(type) { + case *int64: + di := int64(d) + value.Set(reflect.ValueOf(&di)) + case *float64: + value.Set(reflect.ValueOf(&d)) + case *time.Time: + // Time unmarshaled from a float64 can only be epoch seconds + t := time.Unix(int64(d), 0).UTC() + value.Set(reflect.ValueOf(&t)) + default: + return fmt.Errorf("unsupported value: %v (%s)", value.Interface(), value.Type()) + } + case bool: + switch value.Interface().(type) { + case *bool: + value.Set(reflect.ValueOf(&d)) + default: + return fmt.Errorf("unsupported value: %v (%s)", value.Interface(), value.Type()) + } + default: + return fmt.Errorf("unsupported JSON value (%v)", data) + } + return nil +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/jsonrpc/jsonrpc.go b/github.com/aws/aws-sdk-go/private/protocol/jsonrpc/jsonrpc.go new file mode 100644 index 0000000000..a029217e4c --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/jsonrpc/jsonrpc.go @@ -0,0 +1,88 @@ +// Package jsonrpc provides JSON RPC utilities for serialization of AWS +// requests and responses. +package jsonrpc + +//go:generate go run -tags codegen ../../../private/model/cli/gen-protocol-tests ../../../models/protocol_tests/input/json.json build_test.go +//go:generate go run -tags codegen ../../../private/model/cli/gen-protocol-tests ../../../models/protocol_tests/output/json.json unmarshal_test.go + +import ( + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/private/protocol/json/jsonutil" + "github.com/aws/aws-sdk-go/private/protocol/rest" +) + +var emptyJSON = []byte("{}") + +// BuildHandler is a named request handler for building jsonrpc protocol +// requests +var BuildHandler = request.NamedHandler{ + Name: "awssdk.jsonrpc.Build", + Fn: Build, +} + +// UnmarshalHandler is a named request handler for unmarshaling jsonrpc +// protocol requests +var UnmarshalHandler = request.NamedHandler{ + Name: "awssdk.jsonrpc.Unmarshal", + Fn: Unmarshal, +} + +// UnmarshalMetaHandler is a named request handler for unmarshaling jsonrpc +// protocol request metadata +var UnmarshalMetaHandler = request.NamedHandler{ + Name: "awssdk.jsonrpc.UnmarshalMeta", + Fn: UnmarshalMeta, +} + +// Build builds a JSON payload for a JSON RPC request. +func Build(req *request.Request) { + var buf []byte + var err error + if req.ParamsFilled() { + buf, err = jsonutil.BuildJSON(req.Params) + if err != nil { + req.Error = awserr.New(request.ErrCodeSerialization, "failed encoding JSON RPC request", err) + return + } + } else { + buf = emptyJSON + } + + if req.ClientInfo.TargetPrefix != "" || string(buf) != "{}" { + req.SetBufferBody(buf) + } + + if req.ClientInfo.TargetPrefix != "" { + target := req.ClientInfo.TargetPrefix + "." + req.Operation.Name + req.HTTPRequest.Header.Add("X-Amz-Target", target) + } + + // Only set the content type if one is not already specified and an + // JSONVersion is specified. + if ct, v := req.HTTPRequest.Header.Get("Content-Type"), req.ClientInfo.JSONVersion; len(ct) == 0 && len(v) != 0 { + jsonVersion := req.ClientInfo.JSONVersion + req.HTTPRequest.Header.Set("Content-Type", "application/x-amz-json-"+jsonVersion) + } +} + +// Unmarshal unmarshals a response for a JSON RPC service. +func Unmarshal(req *request.Request) { + defer req.HTTPResponse.Body.Close() + if req.DataFilled() { + err := jsonutil.UnmarshalJSON(req.Data, req.HTTPResponse.Body) + if err != nil { + req.Error = awserr.NewRequestFailure( + awserr.New(request.ErrCodeSerialization, "failed decoding JSON RPC response", err), + req.HTTPResponse.StatusCode, + req.RequestID, + ) + } + } + return +} + +// UnmarshalMeta unmarshals headers from a response for a JSON RPC service. +func UnmarshalMeta(req *request.Request) { + rest.UnmarshalMeta(req) +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/jsonrpc/unmarshal_error.go b/github.com/aws/aws-sdk-go/private/protocol/jsonrpc/unmarshal_error.go new file mode 100644 index 0000000000..c0c52e2db0 --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/jsonrpc/unmarshal_error.go @@ -0,0 +1,107 @@ +package jsonrpc + +import ( + "bytes" + "io" + "io/ioutil" + "net/http" + "strings" + + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/private/protocol" + "github.com/aws/aws-sdk-go/private/protocol/json/jsonutil" +) + +// UnmarshalTypedError provides unmarshaling errors API response errors +// for both typed and untyped errors. +type UnmarshalTypedError struct { + exceptions map[string]func(protocol.ResponseMetadata) error +} + +// NewUnmarshalTypedError returns an UnmarshalTypedError initialized for the +// set of exception names to the error unmarshalers +func NewUnmarshalTypedError(exceptions map[string]func(protocol.ResponseMetadata) error) *UnmarshalTypedError { + return &UnmarshalTypedError{ + exceptions: exceptions, + } +} + +// UnmarshalError attempts to unmarshal the HTTP response error as a known +// error type. If unable to unmarshal the error type, the generic SDK error +// type will be used. +func (u *UnmarshalTypedError) UnmarshalError( + resp *http.Response, + respMeta protocol.ResponseMetadata, +) (error, error) { + + var buf bytes.Buffer + var jsonErr jsonErrorResponse + teeReader := io.TeeReader(resp.Body, &buf) + err := jsonutil.UnmarshalJSONError(&jsonErr, teeReader) + if err != nil { + return nil, err + } + body := ioutil.NopCloser(&buf) + + // Code may be separated by hash(#), with the last element being the code + // used by the SDK. + codeParts := strings.SplitN(jsonErr.Code, "#", 2) + code := codeParts[len(codeParts)-1] + msg := jsonErr.Message + + if fn, ok := u.exceptions[code]; ok { + // If exception code is know, use associated constructor to get a value + // for the exception that the JSON body can be unmarshaled into. + v := fn(respMeta) + err := jsonutil.UnmarshalJSONCaseInsensitive(v, body) + if err != nil { + return nil, err + } + + return v, nil + } + + // fallback to unmodeled generic exceptions + return awserr.NewRequestFailure( + awserr.New(code, msg, nil), + respMeta.StatusCode, + respMeta.RequestID, + ), nil +} + +// UnmarshalErrorHandler is a named request handler for unmarshaling jsonrpc +// protocol request errors +var UnmarshalErrorHandler = request.NamedHandler{ + Name: "awssdk.jsonrpc.UnmarshalError", + Fn: UnmarshalError, +} + +// UnmarshalError unmarshals an error response for a JSON RPC service. +func UnmarshalError(req *request.Request) { + defer req.HTTPResponse.Body.Close() + + var jsonErr jsonErrorResponse + err := jsonutil.UnmarshalJSONError(&jsonErr, req.HTTPResponse.Body) + if err != nil { + req.Error = awserr.NewRequestFailure( + awserr.New(request.ErrCodeSerialization, + "failed to unmarshal error message", err), + req.HTTPResponse.StatusCode, + req.RequestID, + ) + return + } + + codes := strings.SplitN(jsonErr.Code, "#", 2) + req.Error = awserr.NewRequestFailure( + awserr.New(codes[len(codes)-1], jsonErr.Message, nil), + req.HTTPResponse.StatusCode, + req.RequestID, + ) +} + +type jsonErrorResponse struct { + Code string `json:"__type"` + Message string `json:"message"` +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/payload.go b/github.com/aws/aws-sdk-go/private/protocol/payload.go index e21614a125..0ea0647a57 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/payload.go +++ b/github.com/aws/aws-sdk-go/private/protocol/payload.go @@ -64,7 +64,7 @@ func (h HandlerPayloadMarshal) MarshalPayload(w io.Writer, v interface{}) error metadata.ClientInfo{}, request.Handlers{}, nil, - &request.Operation{HTTPMethod: "GET"}, + &request.Operation{HTTPMethod: "PUT"}, v, nil, ) diff --git a/github.com/aws/aws-sdk-go/private/protocol/protocol.go b/github.com/aws/aws-sdk-go/private/protocol/protocol.go new file mode 100644 index 0000000000..9d521dcb95 --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/protocol.go @@ -0,0 +1,49 @@ +package protocol + +import ( + "fmt" + "strings" + + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/aws/request" +) + +// RequireHTTPMinProtocol request handler is used to enforce that +// the target endpoint supports the given major and minor HTTP protocol version. +type RequireHTTPMinProtocol struct { + Major, Minor int +} + +// Handler will mark the request.Request with an error if the +// target endpoint did not connect with the required HTTP protocol +// major and minor version. +func (p RequireHTTPMinProtocol) Handler(r *request.Request) { + if r.Error != nil || r.HTTPResponse == nil { + return + } + + if !strings.HasPrefix(r.HTTPResponse.Proto, "HTTP") { + r.Error = newMinHTTPProtoError(p.Major, p.Minor, r) + } + + if r.HTTPResponse.ProtoMajor < p.Major || r.HTTPResponse.ProtoMinor < p.Minor { + r.Error = newMinHTTPProtoError(p.Major, p.Minor, r) + } +} + +// ErrCodeMinimumHTTPProtocolError error code is returned when the target endpoint +// did not match the required HTTP major and minor protocol version. +const ErrCodeMinimumHTTPProtocolError = "MinimumHTTPProtocolError" + +func newMinHTTPProtoError(major, minor int, r *request.Request) error { + return awserr.NewRequestFailure( + awserr.New("MinimumHTTPProtocolError", + fmt.Sprintf( + "operation requires minimum HTTP protocol of HTTP/%d.%d, but was %s", + major, minor, r.HTTPResponse.Proto, + ), + nil, + ), + r.HTTPResponse.StatusCode, r.RequestID, + ) +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/query/build.go b/github.com/aws/aws-sdk-go/private/protocol/query/build.go index 60e5b09d54..d40346a779 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/query/build.go +++ b/github.com/aws/aws-sdk-go/private/protocol/query/build.go @@ -1,7 +1,7 @@ // Package query provides serialization of AWS query requests, and responses. package query -//go:generate go run -tags codegen ../../../models/protocol_tests/generate.go ../../../models/protocol_tests/input/query.json build_test.go +//go:generate go run -tags codegen ../../../private/model/cli/gen-protocol-tests ../../../models/protocol_tests/input/query.json build_test.go import ( "net/url" @@ -21,7 +21,7 @@ func Build(r *request.Request) { "Version": {r.ClientInfo.APIVersion}, } if err := queryutil.Parse(body, r.Params, false); err != nil { - r.Error = awserr.New("SerializationError", "failed encoding Query request", err) + r.Error = awserr.New(request.ErrCodeSerialization, "failed encoding Query request", err) return } diff --git a/github.com/aws/aws-sdk-go/private/protocol/query/unmarshal.go b/github.com/aws/aws-sdk-go/private/protocol/query/unmarshal.go index 3495c73070..9231e95d16 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/query/unmarshal.go +++ b/github.com/aws/aws-sdk-go/private/protocol/query/unmarshal.go @@ -1,6 +1,6 @@ package query -//go:generate go run -tags codegen ../../../models/protocol_tests/generate.go ../../../models/protocol_tests/output/query.json unmarshal_test.go +//go:generate go run -tags codegen ../../../private/model/cli/gen-protocol-tests ../../../models/protocol_tests/output/query.json unmarshal_test.go import ( "encoding/xml" @@ -24,7 +24,7 @@ func Unmarshal(r *request.Request) { err := xmlutil.UnmarshalXML(r.Data, decoder, r.Operation.Name+"Result") if err != nil { r.Error = awserr.NewRequestFailure( - awserr.New("SerializationError", "failed decoding Query response", err), + awserr.New(request.ErrCodeSerialization, "failed decoding Query response", err), r.HTTPResponse.StatusCode, r.RequestID, ) diff --git a/github.com/aws/aws-sdk-go/private/protocol/query/unmarshal_error.go b/github.com/aws/aws-sdk-go/private/protocol/query/unmarshal_error.go index 46d354e826..831b0110c5 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/query/unmarshal_error.go +++ b/github.com/aws/aws-sdk-go/private/protocol/query/unmarshal_error.go @@ -2,73 +2,68 @@ package query import ( "encoding/xml" - "io/ioutil" + "fmt" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil" ) +// UnmarshalErrorHandler is a name request handler to unmarshal request errors +var UnmarshalErrorHandler = request.NamedHandler{Name: "awssdk.query.UnmarshalError", Fn: UnmarshalError} + type xmlErrorResponse struct { - XMLName xml.Name `xml:"ErrorResponse"` - Code string `xml:"Error>Code"` - Message string `xml:"Error>Message"` - RequestID string `xml:"RequestId"` + Code string `xml:"Error>Code"` + Message string `xml:"Error>Message"` + RequestID string `xml:"RequestId"` } -type xmlServiceUnavailableResponse struct { - XMLName xml.Name `xml:"ServiceUnavailableException"` +type xmlResponseError struct { + xmlErrorResponse } -// UnmarshalErrorHandler is a name request handler to unmarshal request errors -var UnmarshalErrorHandler = request.NamedHandler{Name: "awssdk.query.UnmarshalError", Fn: UnmarshalError} +func (e *xmlResponseError) UnmarshalXML(d *xml.Decoder, start xml.StartElement) error { + const svcUnavailableTagName = "ServiceUnavailableException" + const errorResponseTagName = "ErrorResponse" + + switch start.Name.Local { + case svcUnavailableTagName: + e.Code = svcUnavailableTagName + e.Message = "service is unavailable" + return d.Skip() + + case errorResponseTagName: + return d.DecodeElement(&e.xmlErrorResponse, &start) + + default: + return fmt.Errorf("unknown error response tag, %v", start) + } +} // UnmarshalError unmarshals an error response for an AWS Query service. func UnmarshalError(r *request.Request) { defer r.HTTPResponse.Body.Close() - bodyBytes, err := ioutil.ReadAll(r.HTTPResponse.Body) + var respErr xmlResponseError + err := xmlutil.UnmarshalXMLError(&respErr, r.HTTPResponse.Body) if err != nil { r.Error = awserr.NewRequestFailure( - awserr.New("SerializationError", "failed to read from query HTTP response body", err), + awserr.New(request.ErrCodeSerialization, + "failed to unmarshal error message", err), r.HTTPResponse.StatusCode, r.RequestID, ) return } - // First check for specific error - resp := xmlErrorResponse{} - decodeErr := xml.Unmarshal(bodyBytes, &resp) - if decodeErr == nil { - reqID := resp.RequestID - if reqID == "" { - reqID = r.RequestID - } - r.Error = awserr.NewRequestFailure( - awserr.New(resp.Code, resp.Message, nil), - r.HTTPResponse.StatusCode, - reqID, - ) - return - } - - // Check for unhandled error - servUnavailResp := xmlServiceUnavailableResponse{} - unavailErr := xml.Unmarshal(bodyBytes, &servUnavailResp) - if unavailErr == nil { - r.Error = awserr.NewRequestFailure( - awserr.New("ServiceUnavailableException", "service is unavailable", nil), - r.HTTPResponse.StatusCode, - r.RequestID, - ) - return + reqID := respErr.RequestID + if len(reqID) == 0 { + reqID = r.RequestID } - // Failed to retrieve any error message from the response body r.Error = awserr.NewRequestFailure( - awserr.New("SerializationError", - "failed to decode query XML error response", decodeErr), + awserr.New(respErr.Code, respErr.Message, nil), r.HTTPResponse.StatusCode, - r.RequestID, + reqID, ) } diff --git a/github.com/aws/aws-sdk-go/private/protocol/rest/build.go b/github.com/aws/aws-sdk-go/private/protocol/rest/build.go index b34f5258a4..1301b149d3 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/rest/build.go +++ b/github.com/aws/aws-sdk-go/private/protocol/rest/build.go @@ -25,6 +25,8 @@ var noEscape [256]bool var errValueNotSet = fmt.Errorf("value not set") +var byteSliceType = reflect.TypeOf([]byte{}) + func init() { for i := 0; i < len(noEscape); i++ { // AWS expects every character except these to be escaped @@ -94,6 +96,14 @@ func buildLocationElements(r *request.Request, v reflect.Value, buildGETQuery bo continue } + // Support the ability to customize values to be marshaled as a + // blob even though they were modeled as a string. Required for S3 + // API operations like SSECustomerKey is modeled as stirng but + // required to be base64 encoded in request. + if field.Tag.Get("marshal-as") == "blob" { + m = m.Convert(byteSliceType) + } + var err error switch field.Tag.Get("location") { case "headers": // header maps @@ -137,7 +147,7 @@ func buildBody(r *request.Request, v reflect.Value) { case string: r.SetStringBody(reader) default: - r.Error = awserr.New("SerializationError", + r.Error = awserr.New(request.ErrCodeSerialization, "failed to encode REST request", fmt.Errorf("unknown payload type %s", payload.Type())) } @@ -152,9 +162,12 @@ func buildHeader(header *http.Header, v reflect.Value, name string, tag reflect. if err == errValueNotSet { return nil } else if err != nil { - return awserr.New("SerializationError", "failed to encode REST request", err) + return awserr.New(request.ErrCodeSerialization, "failed to encode REST request", err) } + name = strings.TrimSpace(name) + str = strings.TrimSpace(str) + header.Add(name, str) return nil @@ -167,11 +180,13 @@ func buildHeaderMap(header *http.Header, v reflect.Value, tag reflect.StructTag) if err == errValueNotSet { continue } else if err != nil { - return awserr.New("SerializationError", "failed to encode REST request", err) + return awserr.New(request.ErrCodeSerialization, "failed to encode REST request", err) } + keyStr := strings.TrimSpace(key.String()) + str = strings.TrimSpace(str) - header.Add(prefix+key.String(), str) + header.Add(prefix+keyStr, str) } return nil } @@ -181,7 +196,7 @@ func buildURI(u *url.URL, v reflect.Value, name string, tag reflect.StructTag) e if err == errValueNotSet { return nil } else if err != nil { - return awserr.New("SerializationError", "failed to encode REST request", err) + return awserr.New(request.ErrCodeSerialization, "failed to encode REST request", err) } u.Path = strings.Replace(u.Path, "{"+name+"}", value, -1) @@ -214,7 +229,7 @@ func buildQueryString(query url.Values, v reflect.Value, name string, tag reflec if err == errValueNotSet { return nil } else if err != nil { - return awserr.New("SerializationError", "failed to encode REST request", err) + return awserr.New(request.ErrCodeSerialization, "failed to encode REST request", err) } query.Set(name, str) } diff --git a/github.com/aws/aws-sdk-go/private/protocol/rest/unmarshal.go b/github.com/aws/aws-sdk-go/private/protocol/rest/unmarshal.go index 33fd53b126..92f8b4d9a4 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/rest/unmarshal.go +++ b/github.com/aws/aws-sdk-go/private/protocol/rest/unmarshal.go @@ -15,6 +15,7 @@ import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/request" + awsStrings "github.com/aws/aws-sdk-go/internal/strings" "github.com/aws/aws-sdk-go/private/protocol" ) @@ -28,7 +29,9 @@ var UnmarshalMetaHandler = request.NamedHandler{Name: "awssdk.rest.UnmarshalMeta func Unmarshal(r *request.Request) { if r.DataFilled() { v := reflect.Indirect(reflect.ValueOf(r.Data)) - unmarshalBody(r, v) + if err := unmarshalBody(r, v); err != nil { + r.Error = err + } } } @@ -40,12 +43,21 @@ func UnmarshalMeta(r *request.Request) { r.RequestID = r.HTTPResponse.Header.Get("X-Amz-Request-Id") } if r.DataFilled() { - v := reflect.Indirect(reflect.ValueOf(r.Data)) - unmarshalLocationElements(r, v) + if err := UnmarshalResponse(r.HTTPResponse, r.Data, aws.BoolValue(r.Config.LowerCaseHeaderMaps)); err != nil { + r.Error = err + } } } -func unmarshalBody(r *request.Request, v reflect.Value) { +// UnmarshalResponse attempts to unmarshal the REST response headers to +// the data type passed in. The type must be a pointer. An error is returned +// with any error unmarshaling the response into the target datatype. +func UnmarshalResponse(resp *http.Response, data interface{}, lowerCaseHeaderMaps bool) error { + v := reflect.Indirect(reflect.ValueOf(data)) + return unmarshalLocationElements(resp, v, lowerCaseHeaderMaps) +} + +func unmarshalBody(r *request.Request, v reflect.Value) error { if field, ok := v.Type().FieldByName("_"); ok { if payloadName := field.Tag.Get("payload"); payloadName != "" { pfield, _ := v.Type().FieldByName(payloadName) @@ -57,35 +69,38 @@ func unmarshalBody(r *request.Request, v reflect.Value) { defer r.HTTPResponse.Body.Close() b, err := ioutil.ReadAll(r.HTTPResponse.Body) if err != nil { - r.Error = awserr.New("SerializationError", "failed to decode REST response", err) - } else { - payload.Set(reflect.ValueOf(b)) + return awserr.New(request.ErrCodeSerialization, "failed to decode REST response", err) } + + payload.Set(reflect.ValueOf(b)) + case *string: defer r.HTTPResponse.Body.Close() b, err := ioutil.ReadAll(r.HTTPResponse.Body) if err != nil { - r.Error = awserr.New("SerializationError", "failed to decode REST response", err) - } else { - str := string(b) - payload.Set(reflect.ValueOf(&str)) + return awserr.New(request.ErrCodeSerialization, "failed to decode REST response", err) } + + str := string(b) + payload.Set(reflect.ValueOf(&str)) + default: switch payload.Type().String() { case "io.ReadCloser": payload.Set(reflect.ValueOf(r.HTTPResponse.Body)) + case "io.ReadSeeker": b, err := ioutil.ReadAll(r.HTTPResponse.Body) if err != nil { - r.Error = awserr.New("SerializationError", + return awserr.New(request.ErrCodeSerialization, "failed to read response body", err) - return } payload.Set(reflect.ValueOf(ioutil.NopCloser(bytes.NewReader(b)))) + default: io.Copy(ioutil.Discard, r.HTTPResponse.Body) - defer r.HTTPResponse.Body.Close() - r.Error = awserr.New("SerializationError", + r.HTTPResponse.Body.Close() + return awserr.New(request.ErrCodeSerialization, "failed to decode REST response", fmt.Errorf("unknown payload type %s", payload.Type())) } @@ -94,9 +109,11 @@ func unmarshalBody(r *request.Request, v reflect.Value) { } } } + + return nil } -func unmarshalLocationElements(r *request.Request, v reflect.Value) { +func unmarshalLocationElements(resp *http.Response, v reflect.Value, lowerCaseHeaderMaps bool) error { for i := 0; i < v.NumField(); i++ { m, field := v.Field(i), v.Type().Field(i) if n := field.Name; n[0:1] == strings.ToLower(n[0:1]) { @@ -111,26 +128,25 @@ func unmarshalLocationElements(r *request.Request, v reflect.Value) { switch field.Tag.Get("location") { case "statusCode": - unmarshalStatusCode(m, r.HTTPResponse.StatusCode) + unmarshalStatusCode(m, resp.StatusCode) + case "header": - err := unmarshalHeader(m, r.HTTPResponse.Header.Get(name), field.Tag) + err := unmarshalHeader(m, resp.Header.Get(name), field.Tag) if err != nil { - r.Error = awserr.New("SerializationError", "failed to decode REST response", err) - break + return awserr.New(request.ErrCodeSerialization, "failed to decode REST response", err) } + case "headers": prefix := field.Tag.Get("locationName") - err := unmarshalHeaderMap(m, r.HTTPResponse.Header, prefix) + err := unmarshalHeaderMap(m, resp.Header, prefix, lowerCaseHeaderMaps) if err != nil { - r.Error = awserr.New("SerializationError", "failed to decode REST response", err) - break + awserr.New(request.ErrCodeSerialization, "failed to decode REST response", err) } } } - if r.Error != nil { - return - } } + + return nil } func unmarshalStatusCode(v reflect.Value, statusCode int) { @@ -145,29 +161,45 @@ func unmarshalStatusCode(v reflect.Value, statusCode int) { } } -func unmarshalHeaderMap(r reflect.Value, headers http.Header, prefix string) error { +func unmarshalHeaderMap(r reflect.Value, headers http.Header, prefix string, normalize bool) error { + if len(headers) == 0 { + return nil + } switch r.Interface().(type) { case map[string]*string: // we only support string map value types out := map[string]*string{} for k, v := range headers { - k = http.CanonicalHeaderKey(k) - if strings.HasPrefix(strings.ToLower(k), strings.ToLower(prefix)) { + if awsStrings.HasPrefixFold(k, prefix) { + if normalize == true { + k = strings.ToLower(k) + } else { + k = http.CanonicalHeaderKey(k) + } out[k[len(prefix):]] = &v[0] } } - r.Set(reflect.ValueOf(out)) + if len(out) != 0 { + r.Set(reflect.ValueOf(out)) + } + } return nil } func unmarshalHeader(v reflect.Value, header string, tag reflect.StructTag) error { - isJSONValue := tag.Get("type") == "jsonvalue" - if isJSONValue { + switch tag.Get("type") { + case "jsonvalue": if len(header) == 0 { return nil } - } else if !v.IsValid() || (header == "" && v.Elem().Kind() != reflect.String) { - return nil + case "blob": + if len(header) == 0 { + return nil + } + default: + if !v.IsValid() || (header == "" && v.Elem().Kind() != reflect.String) { + return nil + } } switch v.Interface().(type) { @@ -178,7 +210,7 @@ func unmarshalHeader(v reflect.Value, header string, tag reflect.StructTag) erro if err != nil { return err } - v.Set(reflect.ValueOf(&b)) + v.Set(reflect.ValueOf(b)) case *bool: b, err := strconv.ParseBool(header) if err != nil { diff --git a/github.com/aws/aws-sdk-go/private/protocol/restxml/restxml.go b/github.com/aws/aws-sdk-go/private/protocol/restxml/restxml.go index b0f4e24566..b1ae364871 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/restxml/restxml.go +++ b/github.com/aws/aws-sdk-go/private/protocol/restxml/restxml.go @@ -2,8 +2,8 @@ // requests and responses. package restxml -//go:generate go run -tags codegen ../../../models/protocol_tests/generate.go ../../../models/protocol_tests/input/rest-xml.json build_test.go -//go:generate go run -tags codegen ../../../models/protocol_tests/generate.go ../../../models/protocol_tests/output/rest-xml.json unmarshal_test.go +//go:generate go run -tags codegen ../../../private/model/cli/gen-protocol-tests ../../../models/protocol_tests/input/rest-xml.json build_test.go +//go:generate go run -tags codegen ../../../private/model/cli/gen-protocol-tests ../../../models/protocol_tests/output/rest-xml.json unmarshal_test.go import ( "bytes" @@ -37,8 +37,9 @@ func Build(r *request.Request) { err := xmlutil.BuildXML(r.Params, xml.NewEncoder(&buf)) if err != nil { r.Error = awserr.NewRequestFailure( - awserr.New("SerializationError", "failed to encode rest XML request", err), - r.HTTPResponse.StatusCode, + awserr.New(request.ErrCodeSerialization, + "failed to encode rest XML request", err), + 0, r.RequestID, ) return @@ -55,7 +56,8 @@ func Unmarshal(r *request.Request) { err := xmlutil.UnmarshalXML(r.Data, decoder, "") if err != nil { r.Error = awserr.NewRequestFailure( - awserr.New("SerializationError", "failed to decode REST XML response", err), + awserr.New(request.ErrCodeSerialization, + "failed to decode REST XML response", err), r.HTTPResponse.StatusCode, r.RequestID, ) diff --git a/github.com/aws/aws-sdk-go/private/protocol/timestamp.go b/github.com/aws/aws-sdk-go/private/protocol/timestamp.go index b7ed6c6f81..d2f6dae532 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/timestamp.go +++ b/github.com/aws/aws-sdk-go/private/protocol/timestamp.go @@ -1,8 +1,11 @@ package protocol import ( + "math" "strconv" "time" + + "github.com/aws/aws-sdk-go/internal/sdkmath" ) // Names of time formats supported by the SDK @@ -13,12 +16,19 @@ const ( ) // Time formats supported by the SDK +// Output time is intended to not contain decimals const ( // RFC 7231#section-7.1.1.1 timetamp format. e.g Tue, 29 Apr 2014 18:30:38 GMT RFC822TimeFormat = "Mon, 2 Jan 2006 15:04:05 GMT" + // This format is used for output time without seconds precision + RFC822OutputTimeFormat = "Mon, 02 Jan 2006 15:04:05 GMT" + // RFC3339 a subset of the ISO8601 timestamp format. e.g 2014-04-29T18:30:38Z - ISO8601TimeFormat = "2006-01-02T15:04:05Z" + ISO8601TimeFormat = "2006-01-02T15:04:05.999999999Z" + + // This format is used for output time without seconds precision + ISO8601OutputTimeFormat = "2006-01-02T15:04:05Z" ) // IsKnownTimestampFormat returns if the timestamp format name @@ -42,11 +52,12 @@ func FormatTime(name string, t time.Time) string { switch name { case RFC822TimeFormatName: - return t.Format(RFC822TimeFormat) + return t.Format(RFC822OutputTimeFormat) case ISO8601TimeFormatName: - return t.Format(ISO8601TimeFormat) + return t.Format(ISO8601OutputTimeFormat) case UnixTimeFormatName: - return strconv.FormatInt(t.Unix(), 10) + ms := t.UnixNano() / int64(time.Millisecond) + return strconv.FormatFloat(float64(ms)/1e3, 'f', -1, 64) default: panic("unknown timestamp format name, " + name) } @@ -62,10 +73,12 @@ func ParseTime(formatName, value string) (time.Time, error) { return time.Parse(ISO8601TimeFormat, value) case UnixTimeFormatName: v, err := strconv.ParseFloat(value, 64) + _, dec := math.Modf(v) + dec = sdkmath.Round(dec*1e3) / 1e3 //Rounds 0.1229999 to 0.123 if err != nil { return time.Time{}, err } - return time.Unix(int64(v), 0), nil + return time.Unix(int64(v), int64(dec*(1e9))), nil default: panic("unknown timestamp format name, " + formatName) } diff --git a/github.com/aws/aws-sdk-go/private/protocol/unmarshal.go b/github.com/aws/aws-sdk-go/private/protocol/unmarshal.go index da1a68111d..f614ef898b 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/unmarshal.go +++ b/github.com/aws/aws-sdk-go/private/protocol/unmarshal.go @@ -19,3 +19,9 @@ func UnmarshalDiscardBody(r *request.Request) { io.Copy(ioutil.Discard, r.HTTPResponse.Body) r.HTTPResponse.Body.Close() } + +// ResponseMetadata provides the SDK response metadata attributes. +type ResponseMetadata struct { + StatusCode int + RequestID string +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/unmarshal_error.go b/github.com/aws/aws-sdk-go/private/protocol/unmarshal_error.go new file mode 100644 index 0000000000..cc857f136c --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/unmarshal_error.go @@ -0,0 +1,65 @@ +package protocol + +import ( + "net/http" + + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/aws/request" +) + +// UnmarshalErrorHandler provides unmarshaling errors API response errors for +// both typed and untyped errors. +type UnmarshalErrorHandler struct { + unmarshaler ErrorUnmarshaler +} + +// ErrorUnmarshaler is an abstract interface for concrete implementations to +// unmarshal protocol specific response errors. +type ErrorUnmarshaler interface { + UnmarshalError(*http.Response, ResponseMetadata) (error, error) +} + +// NewUnmarshalErrorHandler returns an UnmarshalErrorHandler +// initialized for the set of exception names to the error unmarshalers +func NewUnmarshalErrorHandler(unmarshaler ErrorUnmarshaler) *UnmarshalErrorHandler { + return &UnmarshalErrorHandler{ + unmarshaler: unmarshaler, + } +} + +// UnmarshalErrorHandlerName is the name of the named handler. +const UnmarshalErrorHandlerName = "awssdk.protocol.UnmarshalError" + +// NamedHandler returns a NamedHandler for the unmarshaler using the set of +// errors the unmarshaler was initialized for. +func (u *UnmarshalErrorHandler) NamedHandler() request.NamedHandler { + return request.NamedHandler{ + Name: UnmarshalErrorHandlerName, + Fn: u.UnmarshalError, + } +} + +// UnmarshalError will attempt to unmarshal the API response's error message +// into either a generic SDK error type, or a typed error corresponding to the +// errors exception name. +func (u *UnmarshalErrorHandler) UnmarshalError(r *request.Request) { + defer r.HTTPResponse.Body.Close() + + respMeta := ResponseMetadata{ + StatusCode: r.HTTPResponse.StatusCode, + RequestID: r.RequestID, + } + + v, err := u.unmarshaler.UnmarshalError(r.HTTPResponse, respMeta) + if err != nil { + r.Error = awserr.NewRequestFailure( + awserr.New(request.ErrCodeSerialization, + "failed to unmarshal response error", err), + respMeta.StatusCode, + respMeta.RequestID, + ) + return + } + + r.Error = v +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/build.go b/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/build.go index cf981fe951..09ad951595 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/build.go +++ b/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/build.go @@ -8,6 +8,7 @@ import ( "reflect" "sort" "strconv" + "strings" "time" "github.com/aws/aws-sdk-go/private/protocol" @@ -60,6 +61,14 @@ func (b *xmlBuilder) buildValue(value reflect.Value, current *XMLNode, tag refle return nil } + xml := tag.Get("xml") + if len(xml) != 0 { + name := strings.SplitAfterN(xml, ",", 2)[0] + if name == "-" { + return nil + } + } + t := tag.Get("type") if t == "" { switch value.Kind() { diff --git a/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/sort.go b/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/sort.go new file mode 100644 index 0000000000..c1a511851f --- /dev/null +++ b/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/sort.go @@ -0,0 +1,32 @@ +package xmlutil + +import ( + "encoding/xml" + "strings" +) + +type xmlAttrSlice []xml.Attr + +func (x xmlAttrSlice) Len() int { + return len(x) +} + +func (x xmlAttrSlice) Less(i, j int) bool { + spaceI, spaceJ := x[i].Name.Space, x[j].Name.Space + localI, localJ := x[i].Name.Local, x[j].Name.Local + valueI, valueJ := x[i].Value, x[j].Value + + spaceCmp := strings.Compare(spaceI, spaceJ) + localCmp := strings.Compare(localI, localJ) + valueCmp := strings.Compare(valueI, valueJ) + + if spaceCmp == -1 || (spaceCmp == 0 && (localCmp == -1 || (localCmp == 0 && valueCmp == -1))) { + return true + } + + return false +} + +func (x xmlAttrSlice) Swap(i, j int) { + x[i], x[j] = x[j], x[i] +} diff --git a/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/unmarshal.go b/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/unmarshal.go index ff1ef6830b..107c053f8a 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/unmarshal.go +++ b/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/unmarshal.go @@ -1,6 +1,7 @@ package xmlutil import ( + "bytes" "encoding/base64" "encoding/xml" "fmt" @@ -10,9 +11,27 @@ import ( "strings" "time" + "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/private/protocol" ) +// UnmarshalXMLError unmarshals the XML error from the stream into the value +// type specified. The value must be a pointer. If the message fails to +// unmarshal, the message content will be included in the returned error as a +// awserr.UnmarshalError. +func UnmarshalXMLError(v interface{}, stream io.Reader) error { + var errBuf bytes.Buffer + body := io.TeeReader(stream, &errBuf) + + err := xml.NewDecoder(body).Decode(v) + if err != nil && err != io.EOF { + return awserr.NewUnmarshalError(err, + "failed to unmarshal error message", errBuf.Bytes()) + } + + return nil +} + // UnmarshalXML deserializes an xml.Decoder into the container v. V // needs to match the shape of the XML expected to be decoded. // If the shape doesn't match unmarshaling will fail. @@ -45,6 +64,14 @@ func UnmarshalXML(v interface{}, d *xml.Decoder, wrapper string) error { // parse deserializes any value from the XMLNode. The type tag is used to infer the type, or reflect // will be used to determine the type from r. func parse(r reflect.Value, node *XMLNode, tag reflect.StructTag) error { + xml := tag.Get("xml") + if len(xml) != 0 { + name := strings.SplitAfterN(xml, ",", 2)[0] + if name == "-" { + return nil + } + } + rtype := r.Type() if rtype.Kind() == reflect.Ptr { rtype = rtype.Elem() // check kind of actual element type diff --git a/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/xml_to_struct.go b/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/xml_to_struct.go index 515ce15215..42f71648ee 100644 --- a/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/xml_to_struct.go +++ b/github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil/xml_to_struct.go @@ -119,7 +119,18 @@ func (n *XMLNode) findElem(name string) (string, bool) { // StructToXML writes an XMLNode to a xml.Encoder as tokens. func StructToXML(e *xml.Encoder, node *XMLNode, sorted bool) error { - e.EncodeToken(xml.StartElement{Name: node.Name, Attr: node.Attr}) + // Sort Attributes + attrs := node.Attr + if sorted { + sortedAttrs := make([]xml.Attr, len(attrs)) + for _, k := range node.Attr { + sortedAttrs = append(sortedAttrs, k) + } + sort.Sort(xmlAttrSlice(sortedAttrs)) + attrs = sortedAttrs + } + + e.EncodeToken(xml.StartElement{Name: node.Name, Attr: attrs}) if node.Text != "" { e.EncodeToken(xml.CharData([]byte(node.Text))) diff --git a/github.com/aws/aws-sdk-go/service/kms/api.go b/github.com/aws/aws-sdk-go/service/kms/api.go new file mode 100644 index 0000000000..8793c2d808 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/kms/api.go @@ -0,0 +1,14793 @@ +// Code generated by private/model/cli/gen-api/main.go. DO NOT EDIT. + +package kms + +import ( + "fmt" + "time" + + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/awsutil" + "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/private/protocol" + "github.com/aws/aws-sdk-go/private/protocol/jsonrpc" +) + +const opCancelKeyDeletion = "CancelKeyDeletion" + +// CancelKeyDeletionRequest generates a "aws/request.Request" representing the +// client's request for the CancelKeyDeletion operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See CancelKeyDeletion for more information on using the CancelKeyDeletion +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the CancelKeyDeletionRequest method. +// req, resp := client.CancelKeyDeletionRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/CancelKeyDeletion +func (c *KMS) CancelKeyDeletionRequest(input *CancelKeyDeletionInput) (req *request.Request, output *CancelKeyDeletionOutput) { + op := &request.Operation{ + Name: opCancelKeyDeletion, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &CancelKeyDeletionInput{} + } + + output = &CancelKeyDeletionOutput{} + req = c.newRequest(op, input, output) + return +} + +// CancelKeyDeletion API operation for AWS Key Management Service. +// +// Cancels the deletion of a customer master key (CMK). When this operation +// succeeds, the key state of the CMK is Disabled. To enable the CMK, use EnableKey. +// You cannot perform this operation on a CMK in a different AWS account. +// +// For more information about scheduling and canceling deletion of a CMK, see +// Deleting Customer Master Keys (https://docs.aws.amazon.com/kms/latest/developerguide/deleting-keys.html) +// in the AWS Key Management Service Developer Guide. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation CancelKeyDeletion for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/CancelKeyDeletion +func (c *KMS) CancelKeyDeletion(input *CancelKeyDeletionInput) (*CancelKeyDeletionOutput, error) { + req, out := c.CancelKeyDeletionRequest(input) + return out, req.Send() +} + +// CancelKeyDeletionWithContext is the same as CancelKeyDeletion with the addition of +// the ability to pass a context and additional request options. +// +// See CancelKeyDeletion for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) CancelKeyDeletionWithContext(ctx aws.Context, input *CancelKeyDeletionInput, opts ...request.Option) (*CancelKeyDeletionOutput, error) { + req, out := c.CancelKeyDeletionRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opConnectCustomKeyStore = "ConnectCustomKeyStore" + +// ConnectCustomKeyStoreRequest generates a "aws/request.Request" representing the +// client's request for the ConnectCustomKeyStore operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See ConnectCustomKeyStore for more information on using the ConnectCustomKeyStore +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the ConnectCustomKeyStoreRequest method. +// req, resp := client.ConnectCustomKeyStoreRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ConnectCustomKeyStore +func (c *KMS) ConnectCustomKeyStoreRequest(input *ConnectCustomKeyStoreInput) (req *request.Request, output *ConnectCustomKeyStoreOutput) { + op := &request.Operation{ + Name: opConnectCustomKeyStore, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &ConnectCustomKeyStoreInput{} + } + + output = &ConnectCustomKeyStoreOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// ConnectCustomKeyStore API operation for AWS Key Management Service. +// +// Connects or reconnects a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) +// to its associated AWS CloudHSM cluster. +// +// The custom key store must be connected before you can create customer master +// keys (CMKs) in the key store or use the CMKs it contains. You can disconnect +// and reconnect a custom key store at any time. +// +// To connect a custom key store, its associated AWS CloudHSM cluster must have +// at least one active HSM. To get the number of active HSMs in a cluster, use +// the DescribeClusters (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_DescribeClusters.html) +// operation. To add HSMs to the cluster, use the CreateHsm (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_CreateHsm.html) +// operation. Also, the kmsuser crypto user (https://docs.aws.amazon.com/kms/latest/developerguide/key-store-concepts.html#concept-kmsuser) +// (CU) must not be logged into the cluster. This prevents AWS KMS from using +// this account to log in. +// +// The connection process can take an extended amount of time to complete; up +// to 20 minutes. This operation starts the connection process, but it does +// not wait for it to complete. When it succeeds, this operation quickly returns +// an HTTP 200 response and a JSON object with no properties. However, this +// response does not indicate that the custom key store is connected. To get +// the connection state of the custom key store, use the DescribeCustomKeyStores +// operation. +// +// During the connection process, AWS KMS finds the AWS CloudHSM cluster that +// is associated with the custom key store, creates the connection infrastructure, +// connects to the cluster, logs into the AWS CloudHSM client as the kmsuser +// CU, and rotates its password. +// +// The ConnectCustomKeyStore operation might fail for various reasons. To find +// the reason, use the DescribeCustomKeyStores operation and see the ConnectionErrorCode +// in the response. For help interpreting the ConnectionErrorCode, see CustomKeyStoresListEntry. +// +// To fix the failure, use the DisconnectCustomKeyStore operation to disconnect +// the custom key store, correct the error, use the UpdateCustomKeyStore operation +// if necessary, and then use ConnectCustomKeyStore again. +// +// If you are having trouble connecting or disconnecting a custom key store, +// see Troubleshooting a Custom Key Store (https://docs.aws.amazon.com/kms/latest/developerguide/fix-keystore.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation ConnectCustomKeyStore for usage and error information. +// +// Returned Error Types: +// * CloudHsmClusterNotActiveException +// The request was rejected because the AWS CloudHSM cluster that is associated +// with the custom key store is not active. Initialize and activate the cluster +// and try the command again. For detailed instructions, see Getting Started +// (https://docs.aws.amazon.com/cloudhsm/latest/userguide/getting-started.html) +// in the AWS CloudHSM User Guide. +// +// * CustomKeyStoreInvalidStateException +// The request was rejected because of the ConnectionState of the custom key +// store. To get the ConnectionState of a custom key store, use the DescribeCustomKeyStores +// operation. +// +// This exception is thrown under the following conditions: +// +// * You requested the CreateKey or GenerateRandom operation in a custom +// key store that is not connected. These operations are valid only when +// the custom key store ConnectionState is CONNECTED. +// +// * You requested the UpdateCustomKeyStore or DeleteCustomKeyStore operation +// on a custom key store that is not disconnected. This operation is valid +// only when the custom key store ConnectionState is DISCONNECTED. +// +// * You requested the ConnectCustomKeyStore operation on a custom key store +// with a ConnectionState of DISCONNECTING or FAILED. This operation is valid +// for all other ConnectionState values. +// +// * CustomKeyStoreNotFoundException +// The request was rejected because AWS KMS cannot find a custom key store with +// the specified key store name or ID. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * CloudHsmClusterInvalidConfigurationException +// The request was rejected because the associated AWS CloudHSM cluster did +// not meet the configuration requirements for a custom key store. +// +// * The cluster must be configured with private subnets in at least two +// different Availability Zones in the Region. +// +// * The security group for the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) +// (cloudhsm-cluster--sg) must include inbound rules and outbound +// rules that allow TCP traffic on ports 2223-2225. The Source in the inbound +// rules and the Destination in the outbound rules must match the security +// group ID. These rules are set by default when you create the cluster. +// Do not delete or change them. To get information about a particular security +// group, use the DescribeSecurityGroups (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSecurityGroups.html) +// operation. +// +// * The cluster must contain at least as many HSMs as the operation requires. +// To add HSMs, use the AWS CloudHSM CreateHsm (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_CreateHsm.html) +// operation. For the CreateCustomKeyStore, UpdateCustomKeyStore, and CreateKey +// operations, the AWS CloudHSM cluster must have at least two active HSMs, +// each in a different Availability Zone. For the ConnectCustomKeyStore operation, +// the AWS CloudHSM must contain at least one active HSM. +// +// For information about the requirements for an AWS CloudHSM cluster that is +// associated with a custom key store, see Assemble the Prerequisites (https://docs.aws.amazon.com/kms/latest/developerguide/create-keystore.html#before-keystore) +// in the AWS Key Management Service Developer Guide. For information about +// creating a private subnet for an AWS CloudHSM cluster, see Create a Private +// Subnet (https://docs.aws.amazon.com/cloudhsm/latest/userguide/create-subnets.html) +// in the AWS CloudHSM User Guide. For information about cluster security groups, +// see Configure a Default Security Group (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) +// in the AWS CloudHSM User Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ConnectCustomKeyStore +func (c *KMS) ConnectCustomKeyStore(input *ConnectCustomKeyStoreInput) (*ConnectCustomKeyStoreOutput, error) { + req, out := c.ConnectCustomKeyStoreRequest(input) + return out, req.Send() +} + +// ConnectCustomKeyStoreWithContext is the same as ConnectCustomKeyStore with the addition of +// the ability to pass a context and additional request options. +// +// See ConnectCustomKeyStore for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ConnectCustomKeyStoreWithContext(ctx aws.Context, input *ConnectCustomKeyStoreInput, opts ...request.Option) (*ConnectCustomKeyStoreOutput, error) { + req, out := c.ConnectCustomKeyStoreRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opCreateAlias = "CreateAlias" + +// CreateAliasRequest generates a "aws/request.Request" representing the +// client's request for the CreateAlias operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See CreateAlias for more information on using the CreateAlias +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the CreateAliasRequest method. +// req, resp := client.CreateAliasRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/CreateAlias +func (c *KMS) CreateAliasRequest(input *CreateAliasInput) (req *request.Request, output *CreateAliasOutput) { + op := &request.Operation{ + Name: opCreateAlias, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &CreateAliasInput{} + } + + output = &CreateAliasOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// CreateAlias API operation for AWS Key Management Service. +// +// Creates a display name for a customer managed customer master key (CMK). +// You can use an alias to identify a CMK in cryptographic operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations), +// such as Encrypt and GenerateDataKey. You can change the CMK associated with +// the alias at any time. +// +// Aliases are easier to remember than key IDs. They can also help to simplify +// your applications. For example, if you use an alias in your code, you can +// change the CMK your code uses by associating a given alias with a different +// CMK. +// +// To run the same code in multiple AWS regions, use an alias in your code, +// such as alias/ApplicationKey. Then, in each AWS Region, create an alias/ApplicationKey +// alias that is associated with a CMK in that Region. When you run your code, +// it uses the alias/ApplicationKey CMK for that AWS Region without any Region-specific +// code. +// +// This operation does not return a response. To get the alias that you created, +// use the ListAliases operation. +// +// To use aliases successfully, be aware of the following information. +// +// * Each alias points to only one CMK at a time, although a single CMK can +// have multiple aliases. The alias and its associated CMK must be in the +// same AWS account and Region. +// +// * You can associate an alias with any customer managed CMK in the same +// AWS account and Region. However, you do not have permission to associate +// an alias with an AWS managed CMK (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-managed-cmk) +// or an AWS owned CMK (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-owned-cmk). +// +// * To change the CMK associated with an alias, use the UpdateAlias operation. +// The current CMK and the new CMK must be the same type (both symmetric +// or both asymmetric) and they must have the same key usage (ENCRYPT_DECRYPT +// or SIGN_VERIFY). This restriction prevents cryptographic errors in code +// that uses aliases. +// +// * The alias name must begin with alias/ followed by a name, such as alias/ExampleAlias. +// It can contain only alphanumeric characters, forward slashes (/), underscores +// (_), and dashes (-). The alias name cannot begin with alias/aws/. The +// alias/aws/ prefix is reserved for AWS managed CMKs (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-managed-cmk). +// +// * The alias name must be unique within an AWS Region. However, you can +// use the same alias name in multiple Regions of the same AWS account. Each +// instance of the alias is associated with a CMK in its Region. +// +// * After you create an alias, you cannot change its alias name. However, +// you can use the DeleteAlias operation to delete the alias and then create +// a new alias with the desired name. +// +// * You can use an alias name or alias ARN to identify a CMK in AWS KMS +// cryptographic operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) +// and in the DescribeKey operation. However, you cannot use alias names +// or alias ARNs in API operations that manage CMKs, such as DisableKey or +// GetKeyPolicy. For information about the valid CMK identifiers for each +// AWS KMS API operation, see the descriptions of the KeyId parameter in +// the API operation documentation. +// +// Because an alias is not a property of a CMK, you can delete and change the +// aliases of a CMK without affecting the CMK. Also, aliases do not appear in +// the response from the DescribeKey operation. To get the aliases and alias +// ARNs of CMKs in each AWS account and Region, use the ListAliases operation. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation CreateAlias for usage and error information. +// +// Returned Error Types: +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * AlreadyExistsException +// The request was rejected because it attempted to create a resource that already +// exists. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidAliasNameException +// The request was rejected because the specified alias name is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * LimitExceededException +// The request was rejected because a quota was exceeded. For more information, +// see Quotas (https://docs.aws.amazon.com/kms/latest/developerguide/limits.html) +// in the AWS Key Management Service Developer Guide. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/CreateAlias +func (c *KMS) CreateAlias(input *CreateAliasInput) (*CreateAliasOutput, error) { + req, out := c.CreateAliasRequest(input) + return out, req.Send() +} + +// CreateAliasWithContext is the same as CreateAlias with the addition of +// the ability to pass a context and additional request options. +// +// See CreateAlias for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) CreateAliasWithContext(ctx aws.Context, input *CreateAliasInput, opts ...request.Option) (*CreateAliasOutput, error) { + req, out := c.CreateAliasRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opCreateCustomKeyStore = "CreateCustomKeyStore" + +// CreateCustomKeyStoreRequest generates a "aws/request.Request" representing the +// client's request for the CreateCustomKeyStore operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See CreateCustomKeyStore for more information on using the CreateCustomKeyStore +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the CreateCustomKeyStoreRequest method. +// req, resp := client.CreateCustomKeyStoreRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/CreateCustomKeyStore +func (c *KMS) CreateCustomKeyStoreRequest(input *CreateCustomKeyStoreInput) (req *request.Request, output *CreateCustomKeyStoreOutput) { + op := &request.Operation{ + Name: opCreateCustomKeyStore, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &CreateCustomKeyStoreInput{} + } + + output = &CreateCustomKeyStoreOutput{} + req = c.newRequest(op, input, output) + return +} + +// CreateCustomKeyStore API operation for AWS Key Management Service. +// +// Creates a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) +// that is associated with an AWS CloudHSM cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/clusters.html) +// that you own and manage. +// +// This operation is part of the Custom Key Store feature (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) +// feature in AWS KMS, which combines the convenience and extensive integration +// of AWS KMS with the isolation and control of a single-tenant key store. +// +// Before you create the custom key store, you must assemble the required elements, +// including an AWS CloudHSM cluster that fulfills the requirements for a custom +// key store. For details about the required elements, see Assemble the Prerequisites +// (https://docs.aws.amazon.com/kms/latest/developerguide/create-keystore.html#before-keystore) +// in the AWS Key Management Service Developer Guide. +// +// When the operation completes successfully, it returns the ID of the new custom +// key store. Before you can use your new custom key store, you need to use +// the ConnectCustomKeyStore operation to connect the new key store to its AWS +// CloudHSM cluster. Even if you are not going to use your custom key store +// immediately, you might want to connect it to verify that all settings are +// correct and then disconnect it until you are ready to use it. +// +// For help with failures, see Troubleshooting a Custom Key Store (https://docs.aws.amazon.com/kms/latest/developerguide/fix-keystore.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation CreateCustomKeyStore for usage and error information. +// +// Returned Error Types: +// * CloudHsmClusterInUseException +// The request was rejected because the specified AWS CloudHSM cluster is already +// associated with a custom key store or it shares a backup history with a cluster +// that is associated with a custom key store. Each custom key store must be +// associated with a different AWS CloudHSM cluster. +// +// Clusters that share a backup history have the same cluster certificate. To +// view the cluster certificate of a cluster, use the DescribeClusters (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_DescribeClusters.html) +// operation. +// +// * CustomKeyStoreNameInUseException +// The request was rejected because the specified custom key store name is already +// assigned to another custom key store in the account. Try again with a custom +// key store name that is unique in the account. +// +// * CloudHsmClusterNotFoundException +// The request was rejected because AWS KMS cannot find the AWS CloudHSM cluster +// with the specified cluster ID. Retry the request with a different cluster +// ID. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * CloudHsmClusterNotActiveException +// The request was rejected because the AWS CloudHSM cluster that is associated +// with the custom key store is not active. Initialize and activate the cluster +// and try the command again. For detailed instructions, see Getting Started +// (https://docs.aws.amazon.com/cloudhsm/latest/userguide/getting-started.html) +// in the AWS CloudHSM User Guide. +// +// * IncorrectTrustAnchorException +// The request was rejected because the trust anchor certificate in the request +// is not the trust anchor certificate for the specified AWS CloudHSM cluster. +// +// When you initialize the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/initialize-cluster.html#sign-csr), +// you create the trust anchor certificate and save it in the customerCA.crt +// file. +// +// * CloudHsmClusterInvalidConfigurationException +// The request was rejected because the associated AWS CloudHSM cluster did +// not meet the configuration requirements for a custom key store. +// +// * The cluster must be configured with private subnets in at least two +// different Availability Zones in the Region. +// +// * The security group for the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) +// (cloudhsm-cluster--sg) must include inbound rules and outbound +// rules that allow TCP traffic on ports 2223-2225. The Source in the inbound +// rules and the Destination in the outbound rules must match the security +// group ID. These rules are set by default when you create the cluster. +// Do not delete or change them. To get information about a particular security +// group, use the DescribeSecurityGroups (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSecurityGroups.html) +// operation. +// +// * The cluster must contain at least as many HSMs as the operation requires. +// To add HSMs, use the AWS CloudHSM CreateHsm (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_CreateHsm.html) +// operation. For the CreateCustomKeyStore, UpdateCustomKeyStore, and CreateKey +// operations, the AWS CloudHSM cluster must have at least two active HSMs, +// each in a different Availability Zone. For the ConnectCustomKeyStore operation, +// the AWS CloudHSM must contain at least one active HSM. +// +// For information about the requirements for an AWS CloudHSM cluster that is +// associated with a custom key store, see Assemble the Prerequisites (https://docs.aws.amazon.com/kms/latest/developerguide/create-keystore.html#before-keystore) +// in the AWS Key Management Service Developer Guide. For information about +// creating a private subnet for an AWS CloudHSM cluster, see Create a Private +// Subnet (https://docs.aws.amazon.com/cloudhsm/latest/userguide/create-subnets.html) +// in the AWS CloudHSM User Guide. For information about cluster security groups, +// see Configure a Default Security Group (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) +// in the AWS CloudHSM User Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/CreateCustomKeyStore +func (c *KMS) CreateCustomKeyStore(input *CreateCustomKeyStoreInput) (*CreateCustomKeyStoreOutput, error) { + req, out := c.CreateCustomKeyStoreRequest(input) + return out, req.Send() +} + +// CreateCustomKeyStoreWithContext is the same as CreateCustomKeyStore with the addition of +// the ability to pass a context and additional request options. +// +// See CreateCustomKeyStore for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) CreateCustomKeyStoreWithContext(ctx aws.Context, input *CreateCustomKeyStoreInput, opts ...request.Option) (*CreateCustomKeyStoreOutput, error) { + req, out := c.CreateCustomKeyStoreRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opCreateGrant = "CreateGrant" + +// CreateGrantRequest generates a "aws/request.Request" representing the +// client's request for the CreateGrant operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See CreateGrant for more information on using the CreateGrant +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the CreateGrantRequest method. +// req, resp := client.CreateGrantRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/CreateGrant +func (c *KMS) CreateGrantRequest(input *CreateGrantInput) (req *request.Request, output *CreateGrantOutput) { + op := &request.Operation{ + Name: opCreateGrant, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &CreateGrantInput{} + } + + output = &CreateGrantOutput{} + req = c.newRequest(op, input, output) + return +} + +// CreateGrant API operation for AWS Key Management Service. +// +// Adds a grant to a customer master key (CMK). The grant allows the grantee +// principal to use the CMK when the conditions specified in the grant are met. +// When setting permissions, grants are an alternative to key policies. +// +// To create a grant that allows a cryptographic operation (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) +// only when the request includes a particular encryption context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context), +// use the Constraints parameter. For details, see GrantConstraints. +// +// You can create grants on symmetric and asymmetric CMKs. However, if the grant +// allows an operation that the CMK does not support, CreateGrant fails with +// a ValidationException. +// +// * Grants for symmetric CMKs cannot allow operations that are not supported +// for symmetric CMKs, including Sign, Verify, and GetPublicKey. (There are +// limited exceptions to this rule for legacy operations, but you should +// not create a grant for an operation that AWS KMS does not support.) +// +// * Grants for asymmetric CMKs cannot allow operations that are not supported +// for asymmetric CMKs, including operations that generate data keys (https://docs.aws.amazon.com/kms/latest/APIReference/API_GenerateDataKey) +// or data key pairs (https://docs.aws.amazon.com/kms/latest/APIReference/API_GenerateDataKeyPair), +// or operations related to automatic key rotation (https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html), +// imported key material (https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys.html), +// or CMKs in custom key stores (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html). +// +// * Grants for asymmetric CMKs with a KeyUsage of ENCRYPT_DECRYPT cannot +// allow the Sign or Verify operations. Grants for asymmetric CMKs with a +// KeyUsage of SIGN_VERIFY cannot allow the Encrypt or Decrypt operations. +// +// * Grants for asymmetric CMKs cannot include an encryption context grant +// constraint. An encryption context is not supported on asymmetric CMKs. +// +// For information about symmetric and asymmetric CMKs, see Using Symmetric +// and Asymmetric CMKs (https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html) +// in the AWS Key Management Service Developer Guide. +// +// To perform this operation on a CMK in a different AWS account, specify the +// key ARN in the value of the KeyId parameter. For more information about grants, +// see Grants (https://docs.aws.amazon.com/kms/latest/developerguide/grants.html) +// in the AWS Key Management Service Developer Guide . +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation CreateGrant for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * LimitExceededException +// The request was rejected because a quota was exceeded. For more information, +// see Quotas (https://docs.aws.amazon.com/kms/latest/developerguide/limits.html) +// in the AWS Key Management Service Developer Guide. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/CreateGrant +func (c *KMS) CreateGrant(input *CreateGrantInput) (*CreateGrantOutput, error) { + req, out := c.CreateGrantRequest(input) + return out, req.Send() +} + +// CreateGrantWithContext is the same as CreateGrant with the addition of +// the ability to pass a context and additional request options. +// +// See CreateGrant for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) CreateGrantWithContext(ctx aws.Context, input *CreateGrantInput, opts ...request.Option) (*CreateGrantOutput, error) { + req, out := c.CreateGrantRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opCreateKey = "CreateKey" + +// CreateKeyRequest generates a "aws/request.Request" representing the +// client's request for the CreateKey operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See CreateKey for more information on using the CreateKey +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the CreateKeyRequest method. +// req, resp := client.CreateKeyRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/CreateKey +func (c *KMS) CreateKeyRequest(input *CreateKeyInput) (req *request.Request, output *CreateKeyOutput) { + op := &request.Operation{ + Name: opCreateKey, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &CreateKeyInput{} + } + + output = &CreateKeyOutput{} + req = c.newRequest(op, input, output) + return +} + +// CreateKey API operation for AWS Key Management Service. +// +// Creates a unique customer managed customer master key (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#master-keys) +// (CMK) in your AWS account and Region. You cannot use this operation to create +// a CMK in a different AWS account. +// +// You can use the CreateKey operation to create symmetric or asymmetric CMKs. +// +// * Symmetric CMKs contain a 256-bit symmetric key that never leaves AWS +// KMS unencrypted. To use the CMK, you must call AWS KMS. You can use a +// symmetric CMK to encrypt and decrypt small amounts of data, but they are +// typically used to generate data keys (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#data-keys) +// and data keys pairs (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#data-key-pairs). +// For details, see GenerateDataKey and GenerateDataKeyPair. +// +// * Asymmetric CMKs can contain an RSA key pair or an Elliptic Curve (ECC) +// key pair. The private key in an asymmetric CMK never leaves AWS KMS unencrypted. +// However, you can use the GetPublicKey operation to download the public +// key so it can be used outside of AWS KMS. CMKs with RSA key pairs can +// be used to encrypt or decrypt data or sign and verify messages (but not +// both). CMKs with ECC key pairs can be used only to sign and verify messages. +// +// For information about symmetric and asymmetric CMKs, see Using Symmetric +// and Asymmetric CMKs (https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html) +// in the AWS Key Management Service Developer Guide. +// +// To create different types of CMKs, use the following guidance: +// +// Asymmetric CMKs +// +// To create an asymmetric CMK, use the CustomerMasterKeySpec parameter to specify +// the type of key material in the CMK. Then, use the KeyUsage parameter to +// determine whether the CMK will be used to encrypt and decrypt or sign and +// verify. You can't change these properties after the CMK is created. +// +// Symmetric CMKs +// +// When creating a symmetric CMK, you don't need to specify the CustomerMasterKeySpec +// or KeyUsage parameters. The default value for CustomerMasterKeySpec, SYMMETRIC_DEFAULT, +// and the default value for KeyUsage, ENCRYPT_DECRYPT, are the only valid values +// for symmetric CMKs. +// +// Imported Key Material +// +// To import your own key material, begin by creating a symmetric CMK with no +// key material. To do this, use the Origin parameter of CreateKey with a value +// of EXTERNAL. Next, use GetParametersForImport operation to get a public key +// and import token, and use the public key to encrypt your key material. Then, +// use ImportKeyMaterial with your import token to import the key material. +// For step-by-step instructions, see Importing Key Material (https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys.html) +// in the AWS Key Management Service Developer Guide . You cannot import the +// key material into an asymmetric CMK. +// +// Custom Key Stores +// +// To create a symmetric CMK in a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html), +// use the CustomKeyStoreId parameter to specify the custom key store. You must +// also use the Origin parameter with a value of AWS_CLOUDHSM. The AWS CloudHSM +// cluster that is associated with the custom key store must have at least two +// active HSMs in different Availability Zones in the AWS Region. +// +// You cannot create an asymmetric CMK in a custom key store. For information +// about custom key stores in AWS KMS see Using Custom Key Stores (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) +// in the AWS Key Management Service Developer Guide . +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation CreateKey for usage and error information. +// +// Returned Error Types: +// * MalformedPolicyDocumentException +// The request was rejected because the specified policy is not syntactically +// or semantically correct. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * LimitExceededException +// The request was rejected because a quota was exceeded. For more information, +// see Quotas (https://docs.aws.amazon.com/kms/latest/developerguide/limits.html) +// in the AWS Key Management Service Developer Guide. +// +// * TagException +// The request was rejected because one or more tags are not valid. +// +// * CustomKeyStoreNotFoundException +// The request was rejected because AWS KMS cannot find a custom key store with +// the specified key store name or ID. +// +// * CustomKeyStoreInvalidStateException +// The request was rejected because of the ConnectionState of the custom key +// store. To get the ConnectionState of a custom key store, use the DescribeCustomKeyStores +// operation. +// +// This exception is thrown under the following conditions: +// +// * You requested the CreateKey or GenerateRandom operation in a custom +// key store that is not connected. These operations are valid only when +// the custom key store ConnectionState is CONNECTED. +// +// * You requested the UpdateCustomKeyStore or DeleteCustomKeyStore operation +// on a custom key store that is not disconnected. This operation is valid +// only when the custom key store ConnectionState is DISCONNECTED. +// +// * You requested the ConnectCustomKeyStore operation on a custom key store +// with a ConnectionState of DISCONNECTING or FAILED. This operation is valid +// for all other ConnectionState values. +// +// * CloudHsmClusterInvalidConfigurationException +// The request was rejected because the associated AWS CloudHSM cluster did +// not meet the configuration requirements for a custom key store. +// +// * The cluster must be configured with private subnets in at least two +// different Availability Zones in the Region. +// +// * The security group for the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) +// (cloudhsm-cluster--sg) must include inbound rules and outbound +// rules that allow TCP traffic on ports 2223-2225. The Source in the inbound +// rules and the Destination in the outbound rules must match the security +// group ID. These rules are set by default when you create the cluster. +// Do not delete or change them. To get information about a particular security +// group, use the DescribeSecurityGroups (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSecurityGroups.html) +// operation. +// +// * The cluster must contain at least as many HSMs as the operation requires. +// To add HSMs, use the AWS CloudHSM CreateHsm (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_CreateHsm.html) +// operation. For the CreateCustomKeyStore, UpdateCustomKeyStore, and CreateKey +// operations, the AWS CloudHSM cluster must have at least two active HSMs, +// each in a different Availability Zone. For the ConnectCustomKeyStore operation, +// the AWS CloudHSM must contain at least one active HSM. +// +// For information about the requirements for an AWS CloudHSM cluster that is +// associated with a custom key store, see Assemble the Prerequisites (https://docs.aws.amazon.com/kms/latest/developerguide/create-keystore.html#before-keystore) +// in the AWS Key Management Service Developer Guide. For information about +// creating a private subnet for an AWS CloudHSM cluster, see Create a Private +// Subnet (https://docs.aws.amazon.com/cloudhsm/latest/userguide/create-subnets.html) +// in the AWS CloudHSM User Guide. For information about cluster security groups, +// see Configure a Default Security Group (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) +// in the AWS CloudHSM User Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/CreateKey +func (c *KMS) CreateKey(input *CreateKeyInput) (*CreateKeyOutput, error) { + req, out := c.CreateKeyRequest(input) + return out, req.Send() +} + +// CreateKeyWithContext is the same as CreateKey with the addition of +// the ability to pass a context and additional request options. +// +// See CreateKey for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) CreateKeyWithContext(ctx aws.Context, input *CreateKeyInput, opts ...request.Option) (*CreateKeyOutput, error) { + req, out := c.CreateKeyRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opDecrypt = "Decrypt" + +// DecryptRequest generates a "aws/request.Request" representing the +// client's request for the Decrypt operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See Decrypt for more information on using the Decrypt +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the DecryptRequest method. +// req, resp := client.DecryptRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/Decrypt +func (c *KMS) DecryptRequest(input *DecryptInput) (req *request.Request, output *DecryptOutput) { + op := &request.Operation{ + Name: opDecrypt, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &DecryptInput{} + } + + output = &DecryptOutput{} + req = c.newRequest(op, input, output) + return +} + +// Decrypt API operation for AWS Key Management Service. +// +// Decrypts ciphertext that was encrypted by a AWS KMS customer master key (CMK) +// using any of the following operations: +// +// * Encrypt +// +// * GenerateDataKey +// +// * GenerateDataKeyPair +// +// * GenerateDataKeyWithoutPlaintext +// +// * GenerateDataKeyPairWithoutPlaintext +// +// You can use this operation to decrypt ciphertext that was encrypted under +// a symmetric or asymmetric CMK. When the CMK is asymmetric, you must specify +// the CMK and the encryption algorithm that was used to encrypt the ciphertext. +// For information about symmetric and asymmetric CMKs, see Using Symmetric +// and Asymmetric CMKs (https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html) +// in the AWS Key Management Service Developer Guide. +// +// The Decrypt operation also decrypts ciphertext that was encrypted outside +// of AWS KMS by the public key in an AWS KMS asymmetric CMK. However, it cannot +// decrypt ciphertext produced by other libraries, such as the AWS Encryption +// SDK (https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/) +// or Amazon S3 client-side encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html). +// These libraries return a ciphertext format that is incompatible with AWS +// KMS. +// +// If the ciphertext was encrypted under a symmetric CMK, you do not need to +// specify the CMK or the encryption algorithm. AWS KMS can get this information +// from metadata that it adds to the symmetric ciphertext blob. However, if +// you prefer, you can specify the KeyId to ensure that a particular CMK is +// used to decrypt the ciphertext. If you specify a different CMK than the one +// used to encrypt the ciphertext, the Decrypt operation fails. +// +// Whenever possible, use key policies to give users permission to call the +// Decrypt operation on a particular CMK, instead of using IAM policies. Otherwise, +// you might create an IAM user policy that gives the user Decrypt permission +// on all CMKs. This user could decrypt ciphertext that was encrypted by CMKs +// in other accounts if the key policy for the cross-account CMK permits it. +// If you must use an IAM policy for Decrypt permissions, limit the user to +// particular CMKs or particular trusted accounts. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation Decrypt for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * InvalidCiphertextException +// From the Decrypt or ReEncrypt operation, the request was rejected because +// the specified ciphertext, or additional authenticated data incorporated into +// the ciphertext, such as the encryption context, is corrupted, missing, or +// otherwise invalid. +// +// From the ImportKeyMaterial operation, the request was rejected because AWS +// KMS could not decrypt the encrypted (wrapped) key material. +// +// * KeyUnavailableException +// The request was rejected because the specified CMK was not available. You +// can retry the request. +// +// * IncorrectKeyException +// The request was rejected because the specified CMK cannot decrypt the data. +// The KeyId in a Decrypt request and the SourceKeyId in a ReEncrypt request +// must identify the same CMK that was used to encrypt the ciphertext. +// +// * InvalidKeyUsageException +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/Decrypt +func (c *KMS) Decrypt(input *DecryptInput) (*DecryptOutput, error) { + req, out := c.DecryptRequest(input) + return out, req.Send() +} + +// DecryptWithContext is the same as Decrypt with the addition of +// the ability to pass a context and additional request options. +// +// See Decrypt for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) DecryptWithContext(ctx aws.Context, input *DecryptInput, opts ...request.Option) (*DecryptOutput, error) { + req, out := c.DecryptRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opDeleteAlias = "DeleteAlias" + +// DeleteAliasRequest generates a "aws/request.Request" representing the +// client's request for the DeleteAlias operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See DeleteAlias for more information on using the DeleteAlias +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the DeleteAliasRequest method. +// req, resp := client.DeleteAliasRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DeleteAlias +func (c *KMS) DeleteAliasRequest(input *DeleteAliasInput) (req *request.Request, output *DeleteAliasOutput) { + op := &request.Operation{ + Name: opDeleteAlias, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &DeleteAliasInput{} + } + + output = &DeleteAliasOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// DeleteAlias API operation for AWS Key Management Service. +// +// Deletes the specified alias. You cannot perform this operation on an alias +// in a different AWS account. +// +// Because an alias is not a property of a CMK, you can delete and change the +// aliases of a CMK without affecting the CMK. Also, aliases do not appear in +// the response from the DescribeKey operation. To get the aliases of all CMKs, +// use the ListAliases operation. +// +// Each CMK can have multiple aliases. To change the alias of a CMK, use DeleteAlias +// to delete the current alias and CreateAlias to create a new alias. To associate +// an existing alias with a different customer master key (CMK), call UpdateAlias. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation DeleteAlias for usage and error information. +// +// Returned Error Types: +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DeleteAlias +func (c *KMS) DeleteAlias(input *DeleteAliasInput) (*DeleteAliasOutput, error) { + req, out := c.DeleteAliasRequest(input) + return out, req.Send() +} + +// DeleteAliasWithContext is the same as DeleteAlias with the addition of +// the ability to pass a context and additional request options. +// +// See DeleteAlias for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) DeleteAliasWithContext(ctx aws.Context, input *DeleteAliasInput, opts ...request.Option) (*DeleteAliasOutput, error) { + req, out := c.DeleteAliasRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opDeleteCustomKeyStore = "DeleteCustomKeyStore" + +// DeleteCustomKeyStoreRequest generates a "aws/request.Request" representing the +// client's request for the DeleteCustomKeyStore operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See DeleteCustomKeyStore for more information on using the DeleteCustomKeyStore +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the DeleteCustomKeyStoreRequest method. +// req, resp := client.DeleteCustomKeyStoreRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DeleteCustomKeyStore +func (c *KMS) DeleteCustomKeyStoreRequest(input *DeleteCustomKeyStoreInput) (req *request.Request, output *DeleteCustomKeyStoreOutput) { + op := &request.Operation{ + Name: opDeleteCustomKeyStore, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &DeleteCustomKeyStoreInput{} + } + + output = &DeleteCustomKeyStoreOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// DeleteCustomKeyStore API operation for AWS Key Management Service. +// +// Deletes a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html). +// This operation does not delete the AWS CloudHSM cluster that is associated +// with the custom key store, or affect any users or keys in the cluster. +// +// The custom key store that you delete cannot contain any AWS KMS customer +// master keys (CMKs) (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#master_keys). +// Before deleting the key store, verify that you will never need to use any +// of the CMKs in the key store for any cryptographic operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations). +// Then, use ScheduleKeyDeletion to delete the AWS KMS customer master keys +// (CMKs) from the key store. When the scheduled waiting period expires, the +// ScheduleKeyDeletion operation deletes the CMKs. Then it makes a best effort +// to delete the key material from the associated cluster. However, you might +// need to manually delete the orphaned key material (https://docs.aws.amazon.com/kms/latest/developerguide/fix-keystore.html#fix-keystore-orphaned-key) +// from the cluster and its backups. +// +// After all CMKs are deleted from AWS KMS, use DisconnectCustomKeyStore to +// disconnect the key store from AWS KMS. Then, you can delete the custom key +// store. +// +// Instead of deleting the custom key store, consider using DisconnectCustomKeyStore +// to disconnect it from AWS KMS. While the key store is disconnected, you cannot +// create or use the CMKs in the key store. But, you do not need to delete CMKs +// and you can reconnect a disconnected custom key store at any time. +// +// If the operation succeeds, it returns a JSON object with no properties. +// +// This operation is part of the Custom Key Store feature (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) +// feature in AWS KMS, which combines the convenience and extensive integration +// of AWS KMS with the isolation and control of a single-tenant key store. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation DeleteCustomKeyStore for usage and error information. +// +// Returned Error Types: +// * CustomKeyStoreHasCMKsException +// The request was rejected because the custom key store contains AWS KMS customer +// master keys (CMKs). After verifying that you do not need to use the CMKs, +// use the ScheduleKeyDeletion operation to delete the CMKs. After they are +// deleted, you can delete the custom key store. +// +// * CustomKeyStoreInvalidStateException +// The request was rejected because of the ConnectionState of the custom key +// store. To get the ConnectionState of a custom key store, use the DescribeCustomKeyStores +// operation. +// +// This exception is thrown under the following conditions: +// +// * You requested the CreateKey or GenerateRandom operation in a custom +// key store that is not connected. These operations are valid only when +// the custom key store ConnectionState is CONNECTED. +// +// * You requested the UpdateCustomKeyStore or DeleteCustomKeyStore operation +// on a custom key store that is not disconnected. This operation is valid +// only when the custom key store ConnectionState is DISCONNECTED. +// +// * You requested the ConnectCustomKeyStore operation on a custom key store +// with a ConnectionState of DISCONNECTING or FAILED. This operation is valid +// for all other ConnectionState values. +// +// * CustomKeyStoreNotFoundException +// The request was rejected because AWS KMS cannot find a custom key store with +// the specified key store name or ID. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DeleteCustomKeyStore +func (c *KMS) DeleteCustomKeyStore(input *DeleteCustomKeyStoreInput) (*DeleteCustomKeyStoreOutput, error) { + req, out := c.DeleteCustomKeyStoreRequest(input) + return out, req.Send() +} + +// DeleteCustomKeyStoreWithContext is the same as DeleteCustomKeyStore with the addition of +// the ability to pass a context and additional request options. +// +// See DeleteCustomKeyStore for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) DeleteCustomKeyStoreWithContext(ctx aws.Context, input *DeleteCustomKeyStoreInput, opts ...request.Option) (*DeleteCustomKeyStoreOutput, error) { + req, out := c.DeleteCustomKeyStoreRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opDeleteImportedKeyMaterial = "DeleteImportedKeyMaterial" + +// DeleteImportedKeyMaterialRequest generates a "aws/request.Request" representing the +// client's request for the DeleteImportedKeyMaterial operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See DeleteImportedKeyMaterial for more information on using the DeleteImportedKeyMaterial +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the DeleteImportedKeyMaterialRequest method. +// req, resp := client.DeleteImportedKeyMaterialRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DeleteImportedKeyMaterial +func (c *KMS) DeleteImportedKeyMaterialRequest(input *DeleteImportedKeyMaterialInput) (req *request.Request, output *DeleteImportedKeyMaterialOutput) { + op := &request.Operation{ + Name: opDeleteImportedKeyMaterial, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &DeleteImportedKeyMaterialInput{} + } + + output = &DeleteImportedKeyMaterialOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// DeleteImportedKeyMaterial API operation for AWS Key Management Service. +// +// Deletes key material that you previously imported. This operation makes the +// specified customer master key (CMK) unusable. For more information about +// importing key material into AWS KMS, see Importing Key Material (https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys.html) +// in the AWS Key Management Service Developer Guide. You cannot perform this +// operation on a CMK in a different AWS account. +// +// When the specified CMK is in the PendingDeletion state, this operation does +// not change the CMK's state. Otherwise, it changes the CMK's state to PendingImport. +// +// After you delete key material, you can use ImportKeyMaterial to reimport +// the same key material into the CMK. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation DeleteImportedKeyMaterial for usage and error information. +// +// Returned Error Types: +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DeleteImportedKeyMaterial +func (c *KMS) DeleteImportedKeyMaterial(input *DeleteImportedKeyMaterialInput) (*DeleteImportedKeyMaterialOutput, error) { + req, out := c.DeleteImportedKeyMaterialRequest(input) + return out, req.Send() +} + +// DeleteImportedKeyMaterialWithContext is the same as DeleteImportedKeyMaterial with the addition of +// the ability to pass a context and additional request options. +// +// See DeleteImportedKeyMaterial for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) DeleteImportedKeyMaterialWithContext(ctx aws.Context, input *DeleteImportedKeyMaterialInput, opts ...request.Option) (*DeleteImportedKeyMaterialOutput, error) { + req, out := c.DeleteImportedKeyMaterialRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opDescribeCustomKeyStores = "DescribeCustomKeyStores" + +// DescribeCustomKeyStoresRequest generates a "aws/request.Request" representing the +// client's request for the DescribeCustomKeyStores operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See DescribeCustomKeyStores for more information on using the DescribeCustomKeyStores +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the DescribeCustomKeyStoresRequest method. +// req, resp := client.DescribeCustomKeyStoresRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DescribeCustomKeyStores +func (c *KMS) DescribeCustomKeyStoresRequest(input *DescribeCustomKeyStoresInput) (req *request.Request, output *DescribeCustomKeyStoresOutput) { + op := &request.Operation{ + Name: opDescribeCustomKeyStores, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &DescribeCustomKeyStoresInput{} + } + + output = &DescribeCustomKeyStoresOutput{} + req = c.newRequest(op, input, output) + return +} + +// DescribeCustomKeyStores API operation for AWS Key Management Service. +// +// Gets information about custom key stores (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) +// in the account and region. +// +// This operation is part of the Custom Key Store feature (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) +// feature in AWS KMS, which combines the convenience and extensive integration +// of AWS KMS with the isolation and control of a single-tenant key store. +// +// By default, this operation returns information about all custom key stores +// in the account and region. To get only information about a particular custom +// key store, use either the CustomKeyStoreName or CustomKeyStoreId parameter +// (but not both). +// +// To determine whether the custom key store is connected to its AWS CloudHSM +// cluster, use the ConnectionState element in the response. If an attempt to +// connect the custom key store failed, the ConnectionState value is FAILED +// and the ConnectionErrorCode element in the response indicates the cause of +// the failure. For help interpreting the ConnectionErrorCode, see CustomKeyStoresListEntry. +// +// Custom key stores have a DISCONNECTED connection state if the key store has +// never been connected or you use the DisconnectCustomKeyStore operation to +// disconnect it. If your custom key store state is CONNECTED but you are having +// trouble using it, make sure that its associated AWS CloudHSM cluster is active +// and contains the minimum number of HSMs required for the operation, if any. +// +// For help repairing your custom key store, see the Troubleshooting Custom +// Key Stores (https://docs.aws.amazon.com/kms/latest/developerguide/fix-keystore.html) +// topic in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation DescribeCustomKeyStores for usage and error information. +// +// Returned Error Types: +// * CustomKeyStoreNotFoundException +// The request was rejected because AWS KMS cannot find a custom key store with +// the specified key store name or ID. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DescribeCustomKeyStores +func (c *KMS) DescribeCustomKeyStores(input *DescribeCustomKeyStoresInput) (*DescribeCustomKeyStoresOutput, error) { + req, out := c.DescribeCustomKeyStoresRequest(input) + return out, req.Send() +} + +// DescribeCustomKeyStoresWithContext is the same as DescribeCustomKeyStores with the addition of +// the ability to pass a context and additional request options. +// +// See DescribeCustomKeyStores for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) DescribeCustomKeyStoresWithContext(ctx aws.Context, input *DescribeCustomKeyStoresInput, opts ...request.Option) (*DescribeCustomKeyStoresOutput, error) { + req, out := c.DescribeCustomKeyStoresRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opDescribeKey = "DescribeKey" + +// DescribeKeyRequest generates a "aws/request.Request" representing the +// client's request for the DescribeKey operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See DescribeKey for more information on using the DescribeKey +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the DescribeKeyRequest method. +// req, resp := client.DescribeKeyRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DescribeKey +func (c *KMS) DescribeKeyRequest(input *DescribeKeyInput) (req *request.Request, output *DescribeKeyOutput) { + op := &request.Operation{ + Name: opDescribeKey, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &DescribeKeyInput{} + } + + output = &DescribeKeyOutput{} + req = c.newRequest(op, input, output) + return +} + +// DescribeKey API operation for AWS Key Management Service. +// +// Provides detailed information about a customer master key (CMK). You can +// run DescribeKey on a customer managed CMK (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#customer-cmk) +// or an AWS managed CMK (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#aws-managed-cmk). +// +// This detailed information includes the key ARN, creation date (and deletion +// date, if applicable), the key state, and the origin and expiration date (if +// any) of the key material. For CMKs in custom key stores, it includes information +// about the custom key store, such as the key store ID and the AWS CloudHSM +// cluster ID. It includes fields, like KeySpec, that help you distinguish symmetric +// from asymmetric CMKs. It also provides information that is particularly important +// to asymmetric CMKs, such as the key usage (encryption or signing) and the +// encryption algorithms or signing algorithms that the CMK supports. +// +// DescribeKey does not return the following information: +// +// * Aliases associated with the CMK. To get this information, use ListAliases. +// +// * Whether automatic key rotation is enabled on the CMK. To get this information, +// use GetKeyRotationStatus. Also, some key states prevent a CMK from being +// automatically rotated. For details, see How Automatic Key Rotation Works +// (https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html#rotate-keys-how-it-works) +// in AWS Key Management Service Developer Guide. +// +// * Tags on the CMK. To get this information, use ListResourceTags. +// +// * Key policies and grants on the CMK. To get this information, use GetKeyPolicy +// and ListGrants. +// +// If you call the DescribeKey operation on a predefined AWS alias, that is, +// an AWS alias with no key ID, AWS KMS creates an AWS managed CMK (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#master_keys). +// Then, it associates the alias with the new CMK, and returns the KeyId and +// Arn of the new CMK in the response. +// +// To perform this operation on a CMK in a different AWS account, specify the +// key ARN or alias ARN in the value of the KeyId parameter. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation DescribeKey for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DescribeKey +func (c *KMS) DescribeKey(input *DescribeKeyInput) (*DescribeKeyOutput, error) { + req, out := c.DescribeKeyRequest(input) + return out, req.Send() +} + +// DescribeKeyWithContext is the same as DescribeKey with the addition of +// the ability to pass a context and additional request options. +// +// See DescribeKey for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) DescribeKeyWithContext(ctx aws.Context, input *DescribeKeyInput, opts ...request.Option) (*DescribeKeyOutput, error) { + req, out := c.DescribeKeyRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opDisableKey = "DisableKey" + +// DisableKeyRequest generates a "aws/request.Request" representing the +// client's request for the DisableKey operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See DisableKey for more information on using the DisableKey +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the DisableKeyRequest method. +// req, resp := client.DisableKeyRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DisableKey +func (c *KMS) DisableKeyRequest(input *DisableKeyInput) (req *request.Request, output *DisableKeyOutput) { + op := &request.Operation{ + Name: opDisableKey, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &DisableKeyInput{} + } + + output = &DisableKeyOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// DisableKey API operation for AWS Key Management Service. +// +// Sets the state of a customer master key (CMK) to disabled, thereby preventing +// its use for cryptographic operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations). +// You cannot perform this operation on a CMK in a different AWS account. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects the Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation DisableKey for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DisableKey +func (c *KMS) DisableKey(input *DisableKeyInput) (*DisableKeyOutput, error) { + req, out := c.DisableKeyRequest(input) + return out, req.Send() +} + +// DisableKeyWithContext is the same as DisableKey with the addition of +// the ability to pass a context and additional request options. +// +// See DisableKey for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) DisableKeyWithContext(ctx aws.Context, input *DisableKeyInput, opts ...request.Option) (*DisableKeyOutput, error) { + req, out := c.DisableKeyRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opDisableKeyRotation = "DisableKeyRotation" + +// DisableKeyRotationRequest generates a "aws/request.Request" representing the +// client's request for the DisableKeyRotation operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See DisableKeyRotation for more information on using the DisableKeyRotation +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the DisableKeyRotationRequest method. +// req, resp := client.DisableKeyRotationRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DisableKeyRotation +func (c *KMS) DisableKeyRotationRequest(input *DisableKeyRotationInput) (req *request.Request, output *DisableKeyRotationOutput) { + op := &request.Operation{ + Name: opDisableKeyRotation, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &DisableKeyRotationInput{} + } + + output = &DisableKeyRotationOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// DisableKeyRotation API operation for AWS Key Management Service. +// +// Disables automatic rotation of the key material (https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html) +// for the specified symmetric customer master key (CMK). +// +// You cannot enable automatic rotation of asymmetric CMKs, CMKs with imported +// key material, or CMKs in a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html). +// You cannot perform this operation on a CMK in a different AWS account. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation DisableKeyRotation for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DisableKeyRotation +func (c *KMS) DisableKeyRotation(input *DisableKeyRotationInput) (*DisableKeyRotationOutput, error) { + req, out := c.DisableKeyRotationRequest(input) + return out, req.Send() +} + +// DisableKeyRotationWithContext is the same as DisableKeyRotation with the addition of +// the ability to pass a context and additional request options. +// +// See DisableKeyRotation for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) DisableKeyRotationWithContext(ctx aws.Context, input *DisableKeyRotationInput, opts ...request.Option) (*DisableKeyRotationOutput, error) { + req, out := c.DisableKeyRotationRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opDisconnectCustomKeyStore = "DisconnectCustomKeyStore" + +// DisconnectCustomKeyStoreRequest generates a "aws/request.Request" representing the +// client's request for the DisconnectCustomKeyStore operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See DisconnectCustomKeyStore for more information on using the DisconnectCustomKeyStore +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the DisconnectCustomKeyStoreRequest method. +// req, resp := client.DisconnectCustomKeyStoreRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DisconnectCustomKeyStore +func (c *KMS) DisconnectCustomKeyStoreRequest(input *DisconnectCustomKeyStoreInput) (req *request.Request, output *DisconnectCustomKeyStoreOutput) { + op := &request.Operation{ + Name: opDisconnectCustomKeyStore, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &DisconnectCustomKeyStoreInput{} + } + + output = &DisconnectCustomKeyStoreOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// DisconnectCustomKeyStore API operation for AWS Key Management Service. +// +// Disconnects the custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) +// from its associated AWS CloudHSM cluster. While a custom key store is disconnected, +// you can manage the custom key store and its customer master keys (CMKs), +// but you cannot create or use CMKs in the custom key store. You can reconnect +// the custom key store at any time. +// +// While a custom key store is disconnected, all attempts to create customer +// master keys (CMKs) in the custom key store or to use existing CMKs in cryptographic +// operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) +// will fail. This action can prevent users from storing and accessing sensitive +// data. +// +// To find the connection state of a custom key store, use the DescribeCustomKeyStores +// operation. To reconnect a custom key store, use the ConnectCustomKeyStore +// operation. +// +// If the operation succeeds, it returns a JSON object with no properties. +// +// This operation is part of the Custom Key Store feature (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) +// feature in AWS KMS, which combines the convenience and extensive integration +// of AWS KMS with the isolation and control of a single-tenant key store. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation DisconnectCustomKeyStore for usage and error information. +// +// Returned Error Types: +// * CustomKeyStoreInvalidStateException +// The request was rejected because of the ConnectionState of the custom key +// store. To get the ConnectionState of a custom key store, use the DescribeCustomKeyStores +// operation. +// +// This exception is thrown under the following conditions: +// +// * You requested the CreateKey or GenerateRandom operation in a custom +// key store that is not connected. These operations are valid only when +// the custom key store ConnectionState is CONNECTED. +// +// * You requested the UpdateCustomKeyStore or DeleteCustomKeyStore operation +// on a custom key store that is not disconnected. This operation is valid +// only when the custom key store ConnectionState is DISCONNECTED. +// +// * You requested the ConnectCustomKeyStore operation on a custom key store +// with a ConnectionState of DISCONNECTING or FAILED. This operation is valid +// for all other ConnectionState values. +// +// * CustomKeyStoreNotFoundException +// The request was rejected because AWS KMS cannot find a custom key store with +// the specified key store name or ID. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/DisconnectCustomKeyStore +func (c *KMS) DisconnectCustomKeyStore(input *DisconnectCustomKeyStoreInput) (*DisconnectCustomKeyStoreOutput, error) { + req, out := c.DisconnectCustomKeyStoreRequest(input) + return out, req.Send() +} + +// DisconnectCustomKeyStoreWithContext is the same as DisconnectCustomKeyStore with the addition of +// the ability to pass a context and additional request options. +// +// See DisconnectCustomKeyStore for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) DisconnectCustomKeyStoreWithContext(ctx aws.Context, input *DisconnectCustomKeyStoreInput, opts ...request.Option) (*DisconnectCustomKeyStoreOutput, error) { + req, out := c.DisconnectCustomKeyStoreRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opEnableKey = "EnableKey" + +// EnableKeyRequest generates a "aws/request.Request" representing the +// client's request for the EnableKey operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See EnableKey for more information on using the EnableKey +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the EnableKeyRequest method. +// req, resp := client.EnableKeyRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/EnableKey +func (c *KMS) EnableKeyRequest(input *EnableKeyInput) (req *request.Request, output *EnableKeyOutput) { + op := &request.Operation{ + Name: opEnableKey, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &EnableKeyInput{} + } + + output = &EnableKeyOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// EnableKey API operation for AWS Key Management Service. +// +// Sets the key state of a customer master key (CMK) to enabled. This allows +// you to use the CMK for cryptographic operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations). +// You cannot perform this operation on a CMK in a different AWS account. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation EnableKey for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * LimitExceededException +// The request was rejected because a quota was exceeded. For more information, +// see Quotas (https://docs.aws.amazon.com/kms/latest/developerguide/limits.html) +// in the AWS Key Management Service Developer Guide. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/EnableKey +func (c *KMS) EnableKey(input *EnableKeyInput) (*EnableKeyOutput, error) { + req, out := c.EnableKeyRequest(input) + return out, req.Send() +} + +// EnableKeyWithContext is the same as EnableKey with the addition of +// the ability to pass a context and additional request options. +// +// See EnableKey for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) EnableKeyWithContext(ctx aws.Context, input *EnableKeyInput, opts ...request.Option) (*EnableKeyOutput, error) { + req, out := c.EnableKeyRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opEnableKeyRotation = "EnableKeyRotation" + +// EnableKeyRotationRequest generates a "aws/request.Request" representing the +// client's request for the EnableKeyRotation operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See EnableKeyRotation for more information on using the EnableKeyRotation +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the EnableKeyRotationRequest method. +// req, resp := client.EnableKeyRotationRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/EnableKeyRotation +func (c *KMS) EnableKeyRotationRequest(input *EnableKeyRotationInput) (req *request.Request, output *EnableKeyRotationOutput) { + op := &request.Operation{ + Name: opEnableKeyRotation, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &EnableKeyRotationInput{} + } + + output = &EnableKeyRotationOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// EnableKeyRotation API operation for AWS Key Management Service. +// +// Enables automatic rotation of the key material (https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html) +// for the specified symmetric customer master key (CMK). You cannot perform +// this operation on a CMK in a different AWS account. +// +// You cannot enable automatic rotation of asymmetric CMKs, CMKs with imported +// key material, or CMKs in a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html). +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation EnableKeyRotation for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/EnableKeyRotation +func (c *KMS) EnableKeyRotation(input *EnableKeyRotationInput) (*EnableKeyRotationOutput, error) { + req, out := c.EnableKeyRotationRequest(input) + return out, req.Send() +} + +// EnableKeyRotationWithContext is the same as EnableKeyRotation with the addition of +// the ability to pass a context and additional request options. +// +// See EnableKeyRotation for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) EnableKeyRotationWithContext(ctx aws.Context, input *EnableKeyRotationInput, opts ...request.Option) (*EnableKeyRotationOutput, error) { + req, out := c.EnableKeyRotationRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opEncrypt = "Encrypt" + +// EncryptRequest generates a "aws/request.Request" representing the +// client's request for the Encrypt operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See Encrypt for more information on using the Encrypt +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the EncryptRequest method. +// req, resp := client.EncryptRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/Encrypt +func (c *KMS) EncryptRequest(input *EncryptInput) (req *request.Request, output *EncryptOutput) { + op := &request.Operation{ + Name: opEncrypt, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &EncryptInput{} + } + + output = &EncryptOutput{} + req = c.newRequest(op, input, output) + return +} + +// Encrypt API operation for AWS Key Management Service. +// +// Encrypts plaintext into ciphertext by using a customer master key (CMK). +// The Encrypt operation has two primary use cases: +// +// * You can encrypt small amounts of arbitrary data, such as a personal +// identifier or database password, or other sensitive information. +// +// * You can use the Encrypt operation to move encrypted data from one AWS +// Region to another. For example, in Region A, generate a data key and use +// the plaintext key to encrypt your data. Then, in Region A, use the Encrypt +// operation to encrypt the plaintext data key under a CMK in Region B. Now, +// you can move the encrypted data and the encrypted data key to Region B. +// When necessary, you can decrypt the encrypted data key and the encrypted +// data entirely within in Region B. +// +// You don't need to use the Encrypt operation to encrypt a data key. The GenerateDataKey +// and GenerateDataKeyPair operations return a plaintext data key and an encrypted +// copy of that data key. +// +// When you encrypt data, you must specify a symmetric or asymmetric CMK to +// use in the encryption operation. The CMK must have a KeyUsage value of ENCRYPT_DECRYPT. +// To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// If you use a symmetric CMK, you can use an encryption context to add additional +// security to your encryption operation. If you specify an EncryptionContext +// when encrypting data, you must specify the same encryption context (a case-sensitive +// exact match) when decrypting the data. Otherwise, the request to decrypt +// fails with an InvalidCiphertextException. For more information, see Encryption +// Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) +// in the AWS Key Management Service Developer Guide. +// +// If you specify an asymmetric CMK, you must also specify the encryption algorithm. +// The algorithm must be compatible with the CMK type. +// +// When you use an asymmetric CMK to encrypt or reencrypt data, be sure to record +// the CMK and encryption algorithm that you choose. You will be required to +// provide the same CMK and encryption algorithm when you decrypt the data. +// If the CMK and algorithm do not match the values used to encrypt the data, +// the decrypt operation fails. +// +// You are not required to supply the CMK ID and encryption algorithm when you +// decrypt with symmetric CMKs because AWS KMS stores this information in the +// ciphertext blob. AWS KMS cannot store metadata in ciphertext generated with +// asymmetric keys. The standard format for asymmetric key ciphertext does not +// include configurable fields. +// +// The maximum size of the data that you can encrypt varies with the type of +// CMK and the encryption algorithm that you choose. +// +// * Symmetric CMKs SYMMETRIC_DEFAULT: 4096 bytes +// +// * RSA_2048 RSAES_OAEP_SHA_1: 214 bytes RSAES_OAEP_SHA_256: 190 bytes +// +// * RSA_3072 RSAES_OAEP_SHA_1: 342 bytes RSAES_OAEP_SHA_256: 318 bytes +// +// * RSA_4096 RSAES_OAEP_SHA_1: 470 bytes RSAES_OAEP_SHA_256: 446 bytes +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// To perform this operation on a CMK in a different AWS account, specify the +// key ARN or alias ARN in the value of the KeyId parameter. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation Encrypt for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * KeyUnavailableException +// The request was rejected because the specified CMK was not available. You +// can retry the request. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidKeyUsageException +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/Encrypt +func (c *KMS) Encrypt(input *EncryptInput) (*EncryptOutput, error) { + req, out := c.EncryptRequest(input) + return out, req.Send() +} + +// EncryptWithContext is the same as Encrypt with the addition of +// the ability to pass a context and additional request options. +// +// See Encrypt for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) EncryptWithContext(ctx aws.Context, input *EncryptInput, opts ...request.Option) (*EncryptOutput, error) { + req, out := c.EncryptRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opGenerateDataKey = "GenerateDataKey" + +// GenerateDataKeyRequest generates a "aws/request.Request" representing the +// client's request for the GenerateDataKey operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See GenerateDataKey for more information on using the GenerateDataKey +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the GenerateDataKeyRequest method. +// req, resp := client.GenerateDataKeyRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GenerateDataKey +func (c *KMS) GenerateDataKeyRequest(input *GenerateDataKeyInput) (req *request.Request, output *GenerateDataKeyOutput) { + op := &request.Operation{ + Name: opGenerateDataKey, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &GenerateDataKeyInput{} + } + + output = &GenerateDataKeyOutput{} + req = c.newRequest(op, input, output) + return +} + +// GenerateDataKey API operation for AWS Key Management Service. +// +// Generates a unique symmetric data key for client-side encryption. This operation +// returns a plaintext copy of the data key and a copy that is encrypted under +// a customer master key (CMK) that you specify. You can use the plaintext key +// to encrypt your data outside of AWS KMS and store the encrypted data key +// with the encrypted data. +// +// GenerateDataKey returns a unique data key for each request. The bytes in +// the plaintext key are not related to the caller or the CMK. +// +// To generate a data key, specify the symmetric CMK that will be used to encrypt +// the data key. You cannot use an asymmetric CMK to generate data keys. To +// get the type of your CMK, use the DescribeKey operation. You must also specify +// the length of the data key. Use either the KeySpec or NumberOfBytes parameters +// (but not both). For 128-bit and 256-bit data keys, use the KeySpec parameter. +// +// To get only an encrypted copy of the data key, use GenerateDataKeyWithoutPlaintext. +// To generate an asymmetric data key pair, use the GenerateDataKeyPair or GenerateDataKeyPairWithoutPlaintext +// operation. To get a cryptographically secure random byte string, use GenerateRandom. +// +// You can use the optional encryption context to add additional security to +// the encryption operation. If you specify an EncryptionContext, you must specify +// the same encryption context (a case-sensitive exact match) when decrypting +// the encrypted data key. Otherwise, the request to decrypt fails with an InvalidCiphertextException. +// For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) +// in the AWS Key Management Service Developer Guide. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// How to use your data key +// +// We recommend that you use the following pattern to encrypt data locally in +// your application. You can write your own code or use a client-side encryption +// library, such as the AWS Encryption SDK (https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/), +// the Amazon DynamoDB Encryption Client (https://docs.aws.amazon.com/dynamodb-encryption-client/latest/devguide/), +// or Amazon S3 client-side encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html) +// to do these tasks for you. +// +// To encrypt data outside of AWS KMS: +// +// Use the GenerateDataKey operation to get a data key. +// +// Use the plaintext data key (in the Plaintext field of the response) to encrypt +// your data outside of AWS KMS. Then erase the plaintext data key from memory. +// +// Store the encrypted data key (in the CiphertextBlob field of the response) +// with the encrypted data. +// +// To decrypt data outside of AWS KMS: +// +// Use the Decrypt operation to decrypt the encrypted data key. The operation +// returns a plaintext copy of the data key. +// +// Use the plaintext data key to decrypt data outside of AWS KMS, then erase +// the plaintext data key from memory. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation GenerateDataKey for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * KeyUnavailableException +// The request was rejected because the specified CMK was not available. You +// can retry the request. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidKeyUsageException +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GenerateDataKey +func (c *KMS) GenerateDataKey(input *GenerateDataKeyInput) (*GenerateDataKeyOutput, error) { + req, out := c.GenerateDataKeyRequest(input) + return out, req.Send() +} + +// GenerateDataKeyWithContext is the same as GenerateDataKey with the addition of +// the ability to pass a context and additional request options. +// +// See GenerateDataKey for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) GenerateDataKeyWithContext(ctx aws.Context, input *GenerateDataKeyInput, opts ...request.Option) (*GenerateDataKeyOutput, error) { + req, out := c.GenerateDataKeyRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opGenerateDataKeyPair = "GenerateDataKeyPair" + +// GenerateDataKeyPairRequest generates a "aws/request.Request" representing the +// client's request for the GenerateDataKeyPair operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See GenerateDataKeyPair for more information on using the GenerateDataKeyPair +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the GenerateDataKeyPairRequest method. +// req, resp := client.GenerateDataKeyPairRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GenerateDataKeyPair +func (c *KMS) GenerateDataKeyPairRequest(input *GenerateDataKeyPairInput) (req *request.Request, output *GenerateDataKeyPairOutput) { + op := &request.Operation{ + Name: opGenerateDataKeyPair, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &GenerateDataKeyPairInput{} + } + + output = &GenerateDataKeyPairOutput{} + req = c.newRequest(op, input, output) + return +} + +// GenerateDataKeyPair API operation for AWS Key Management Service. +// +// Generates a unique asymmetric data key pair. The GenerateDataKeyPair operation +// returns a plaintext public key, a plaintext private key, and a copy of the +// private key that is encrypted under the symmetric CMK you specify. You can +// use the data key pair to perform asymmetric cryptography outside of AWS KMS. +// +// GenerateDataKeyPair returns a unique data key pair for each request. The +// bytes in the keys are not related to the caller or the CMK that is used to +// encrypt the private key. +// +// You can use the public key that GenerateDataKeyPair returns to encrypt data +// or verify a signature outside of AWS KMS. Then, store the encrypted private +// key with the data. When you are ready to decrypt data or sign a message, +// you can use the Decrypt operation to decrypt the encrypted private key. +// +// To generate a data key pair, you must specify a symmetric customer master +// key (CMK) to encrypt the private key in a data key pair. You cannot use an +// asymmetric CMK or a CMK in a custom key store. To get the type and origin +// of your CMK, use the DescribeKey operation. +// +// If you are using the data key pair to encrypt data, or for any operation +// where you don't immediately need a private key, consider using the GenerateDataKeyPairWithoutPlaintext +// operation. GenerateDataKeyPairWithoutPlaintext returns a plaintext public +// key and an encrypted private key, but omits the plaintext private key that +// you need only to decrypt ciphertext or sign a message. Later, when you need +// to decrypt the data or sign a message, use the Decrypt operation to decrypt +// the encrypted private key in the data key pair. +// +// You can use the optional encryption context to add additional security to +// the encryption operation. If you specify an EncryptionContext, you must specify +// the same encryption context (a case-sensitive exact match) when decrypting +// the encrypted data key. Otherwise, the request to decrypt fails with an InvalidCiphertextException. +// For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) +// in the AWS Key Management Service Developer Guide. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation GenerateDataKeyPair for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * KeyUnavailableException +// The request was rejected because the specified CMK was not available. You +// can retry the request. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidKeyUsageException +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GenerateDataKeyPair +func (c *KMS) GenerateDataKeyPair(input *GenerateDataKeyPairInput) (*GenerateDataKeyPairOutput, error) { + req, out := c.GenerateDataKeyPairRequest(input) + return out, req.Send() +} + +// GenerateDataKeyPairWithContext is the same as GenerateDataKeyPair with the addition of +// the ability to pass a context and additional request options. +// +// See GenerateDataKeyPair for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) GenerateDataKeyPairWithContext(ctx aws.Context, input *GenerateDataKeyPairInput, opts ...request.Option) (*GenerateDataKeyPairOutput, error) { + req, out := c.GenerateDataKeyPairRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opGenerateDataKeyPairWithoutPlaintext = "GenerateDataKeyPairWithoutPlaintext" + +// GenerateDataKeyPairWithoutPlaintextRequest generates a "aws/request.Request" representing the +// client's request for the GenerateDataKeyPairWithoutPlaintext operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See GenerateDataKeyPairWithoutPlaintext for more information on using the GenerateDataKeyPairWithoutPlaintext +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the GenerateDataKeyPairWithoutPlaintextRequest method. +// req, resp := client.GenerateDataKeyPairWithoutPlaintextRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GenerateDataKeyPairWithoutPlaintext +func (c *KMS) GenerateDataKeyPairWithoutPlaintextRequest(input *GenerateDataKeyPairWithoutPlaintextInput) (req *request.Request, output *GenerateDataKeyPairWithoutPlaintextOutput) { + op := &request.Operation{ + Name: opGenerateDataKeyPairWithoutPlaintext, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &GenerateDataKeyPairWithoutPlaintextInput{} + } + + output = &GenerateDataKeyPairWithoutPlaintextOutput{} + req = c.newRequest(op, input, output) + return +} + +// GenerateDataKeyPairWithoutPlaintext API operation for AWS Key Management Service. +// +// Generates a unique asymmetric data key pair. The GenerateDataKeyPairWithoutPlaintext +// operation returns a plaintext public key and a copy of the private key that +// is encrypted under the symmetric CMK you specify. Unlike GenerateDataKeyPair, +// this operation does not return a plaintext private key. +// +// To generate a data key pair, you must specify a symmetric customer master +// key (CMK) to encrypt the private key in the data key pair. You cannot use +// an asymmetric CMK or a CMK in a custom key store. To get the type and origin +// of your CMK, use the KeySpec field in the DescribeKey response. +// +// You can use the public key that GenerateDataKeyPairWithoutPlaintext returns +// to encrypt data or verify a signature outside of AWS KMS. Then, store the +// encrypted private key with the data. When you are ready to decrypt data or +// sign a message, you can use the Decrypt operation to decrypt the encrypted +// private key. +// +// GenerateDataKeyPairWithoutPlaintext returns a unique data key pair for each +// request. The bytes in the key are not related to the caller or CMK that is +// used to encrypt the private key. +// +// You can use the optional encryption context to add additional security to +// the encryption operation. If you specify an EncryptionContext, you must specify +// the same encryption context (a case-sensitive exact match) when decrypting +// the encrypted data key. Otherwise, the request to decrypt fails with an InvalidCiphertextException. +// For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) +// in the AWS Key Management Service Developer Guide. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation GenerateDataKeyPairWithoutPlaintext for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * KeyUnavailableException +// The request was rejected because the specified CMK was not available. You +// can retry the request. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidKeyUsageException +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GenerateDataKeyPairWithoutPlaintext +func (c *KMS) GenerateDataKeyPairWithoutPlaintext(input *GenerateDataKeyPairWithoutPlaintextInput) (*GenerateDataKeyPairWithoutPlaintextOutput, error) { + req, out := c.GenerateDataKeyPairWithoutPlaintextRequest(input) + return out, req.Send() +} + +// GenerateDataKeyPairWithoutPlaintextWithContext is the same as GenerateDataKeyPairWithoutPlaintext with the addition of +// the ability to pass a context and additional request options. +// +// See GenerateDataKeyPairWithoutPlaintext for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) GenerateDataKeyPairWithoutPlaintextWithContext(ctx aws.Context, input *GenerateDataKeyPairWithoutPlaintextInput, opts ...request.Option) (*GenerateDataKeyPairWithoutPlaintextOutput, error) { + req, out := c.GenerateDataKeyPairWithoutPlaintextRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opGenerateDataKeyWithoutPlaintext = "GenerateDataKeyWithoutPlaintext" + +// GenerateDataKeyWithoutPlaintextRequest generates a "aws/request.Request" representing the +// client's request for the GenerateDataKeyWithoutPlaintext operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See GenerateDataKeyWithoutPlaintext for more information on using the GenerateDataKeyWithoutPlaintext +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the GenerateDataKeyWithoutPlaintextRequest method. +// req, resp := client.GenerateDataKeyWithoutPlaintextRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GenerateDataKeyWithoutPlaintext +func (c *KMS) GenerateDataKeyWithoutPlaintextRequest(input *GenerateDataKeyWithoutPlaintextInput) (req *request.Request, output *GenerateDataKeyWithoutPlaintextOutput) { + op := &request.Operation{ + Name: opGenerateDataKeyWithoutPlaintext, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &GenerateDataKeyWithoutPlaintextInput{} + } + + output = &GenerateDataKeyWithoutPlaintextOutput{} + req = c.newRequest(op, input, output) + return +} + +// GenerateDataKeyWithoutPlaintext API operation for AWS Key Management Service. +// +// Generates a unique symmetric data key. This operation returns a data key +// that is encrypted under a customer master key (CMK) that you specify. To +// request an asymmetric data key pair, use the GenerateDataKeyPair or GenerateDataKeyPairWithoutPlaintext +// operations. +// +// GenerateDataKeyWithoutPlaintext is identical to the GenerateDataKey operation +// except that returns only the encrypted copy of the data key. This operation +// is useful for systems that need to encrypt data at some point, but not immediately. +// When you need to encrypt the data, you call the Decrypt operation on the +// encrypted copy of the key. +// +// It's also useful in distributed systems with different levels of trust. For +// example, you might store encrypted data in containers. One component of your +// system creates new containers and stores an encrypted data key with each +// container. Then, a different component puts the data into the containers. +// That component first decrypts the data key, uses the plaintext data key to +// encrypt data, puts the encrypted data into the container, and then destroys +// the plaintext data key. In this system, the component that creates the containers +// never sees the plaintext data key. +// +// GenerateDataKeyWithoutPlaintext returns a unique data key for each request. +// The bytes in the keys are not related to the caller or CMK that is used to +// encrypt the private key. +// +// To generate a data key, you must specify the symmetric customer master key +// (CMK) that is used to encrypt the data key. You cannot use an asymmetric +// CMK to generate a data key. To get the type of your CMK, use the DescribeKey +// operation. +// +// If the operation succeeds, you will find the encrypted copy of the data key +// in the CiphertextBlob field. +// +// You can use the optional encryption context to add additional security to +// the encryption operation. If you specify an EncryptionContext, you must specify +// the same encryption context (a case-sensitive exact match) when decrypting +// the encrypted data key. Otherwise, the request to decrypt fails with an InvalidCiphertextException. +// For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) +// in the AWS Key Management Service Developer Guide. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation GenerateDataKeyWithoutPlaintext for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * KeyUnavailableException +// The request was rejected because the specified CMK was not available. You +// can retry the request. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidKeyUsageException +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GenerateDataKeyWithoutPlaintext +func (c *KMS) GenerateDataKeyWithoutPlaintext(input *GenerateDataKeyWithoutPlaintextInput) (*GenerateDataKeyWithoutPlaintextOutput, error) { + req, out := c.GenerateDataKeyWithoutPlaintextRequest(input) + return out, req.Send() +} + +// GenerateDataKeyWithoutPlaintextWithContext is the same as GenerateDataKeyWithoutPlaintext with the addition of +// the ability to pass a context and additional request options. +// +// See GenerateDataKeyWithoutPlaintext for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) GenerateDataKeyWithoutPlaintextWithContext(ctx aws.Context, input *GenerateDataKeyWithoutPlaintextInput, opts ...request.Option) (*GenerateDataKeyWithoutPlaintextOutput, error) { + req, out := c.GenerateDataKeyWithoutPlaintextRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opGenerateRandom = "GenerateRandom" + +// GenerateRandomRequest generates a "aws/request.Request" representing the +// client's request for the GenerateRandom operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See GenerateRandom for more information on using the GenerateRandom +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the GenerateRandomRequest method. +// req, resp := client.GenerateRandomRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GenerateRandom +func (c *KMS) GenerateRandomRequest(input *GenerateRandomInput) (req *request.Request, output *GenerateRandomOutput) { + op := &request.Operation{ + Name: opGenerateRandom, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &GenerateRandomInput{} + } + + output = &GenerateRandomOutput{} + req = c.newRequest(op, input, output) + return +} + +// GenerateRandom API operation for AWS Key Management Service. +// +// Returns a random byte string that is cryptographically secure. +// +// By default, the random byte string is generated in AWS KMS. To generate the +// byte string in the AWS CloudHSM cluster that is associated with a custom +// key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html), +// specify the custom key store ID. +// +// For more information about entropy and random number generation, see the +// AWS Key Management Service Cryptographic Details (https://d0.awsstatic.com/whitepapers/KMS-Cryptographic-Details.pdf) +// whitepaper. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation GenerateRandom for usage and error information. +// +// Returned Error Types: +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * CustomKeyStoreNotFoundException +// The request was rejected because AWS KMS cannot find a custom key store with +// the specified key store name or ID. +// +// * CustomKeyStoreInvalidStateException +// The request was rejected because of the ConnectionState of the custom key +// store. To get the ConnectionState of a custom key store, use the DescribeCustomKeyStores +// operation. +// +// This exception is thrown under the following conditions: +// +// * You requested the CreateKey or GenerateRandom operation in a custom +// key store that is not connected. These operations are valid only when +// the custom key store ConnectionState is CONNECTED. +// +// * You requested the UpdateCustomKeyStore or DeleteCustomKeyStore operation +// on a custom key store that is not disconnected. This operation is valid +// only when the custom key store ConnectionState is DISCONNECTED. +// +// * You requested the ConnectCustomKeyStore operation on a custom key store +// with a ConnectionState of DISCONNECTING or FAILED. This operation is valid +// for all other ConnectionState values. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GenerateRandom +func (c *KMS) GenerateRandom(input *GenerateRandomInput) (*GenerateRandomOutput, error) { + req, out := c.GenerateRandomRequest(input) + return out, req.Send() +} + +// GenerateRandomWithContext is the same as GenerateRandom with the addition of +// the ability to pass a context and additional request options. +// +// See GenerateRandom for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) GenerateRandomWithContext(ctx aws.Context, input *GenerateRandomInput, opts ...request.Option) (*GenerateRandomOutput, error) { + req, out := c.GenerateRandomRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opGetKeyPolicy = "GetKeyPolicy" + +// GetKeyPolicyRequest generates a "aws/request.Request" representing the +// client's request for the GetKeyPolicy operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See GetKeyPolicy for more information on using the GetKeyPolicy +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the GetKeyPolicyRequest method. +// req, resp := client.GetKeyPolicyRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GetKeyPolicy +func (c *KMS) GetKeyPolicyRequest(input *GetKeyPolicyInput) (req *request.Request, output *GetKeyPolicyOutput) { + op := &request.Operation{ + Name: opGetKeyPolicy, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &GetKeyPolicyInput{} + } + + output = &GetKeyPolicyOutput{} + req = c.newRequest(op, input, output) + return +} + +// GetKeyPolicy API operation for AWS Key Management Service. +// +// Gets a key policy attached to the specified customer master key (CMK). You +// cannot perform this operation on a CMK in a different AWS account. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation GetKeyPolicy for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GetKeyPolicy +func (c *KMS) GetKeyPolicy(input *GetKeyPolicyInput) (*GetKeyPolicyOutput, error) { + req, out := c.GetKeyPolicyRequest(input) + return out, req.Send() +} + +// GetKeyPolicyWithContext is the same as GetKeyPolicy with the addition of +// the ability to pass a context and additional request options. +// +// See GetKeyPolicy for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) GetKeyPolicyWithContext(ctx aws.Context, input *GetKeyPolicyInput, opts ...request.Option) (*GetKeyPolicyOutput, error) { + req, out := c.GetKeyPolicyRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opGetKeyRotationStatus = "GetKeyRotationStatus" + +// GetKeyRotationStatusRequest generates a "aws/request.Request" representing the +// client's request for the GetKeyRotationStatus operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See GetKeyRotationStatus for more information on using the GetKeyRotationStatus +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the GetKeyRotationStatusRequest method. +// req, resp := client.GetKeyRotationStatusRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GetKeyRotationStatus +func (c *KMS) GetKeyRotationStatusRequest(input *GetKeyRotationStatusInput) (req *request.Request, output *GetKeyRotationStatusOutput) { + op := &request.Operation{ + Name: opGetKeyRotationStatus, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &GetKeyRotationStatusInput{} + } + + output = &GetKeyRotationStatusOutput{} + req = c.newRequest(op, input, output) + return +} + +// GetKeyRotationStatus API operation for AWS Key Management Service. +// +// Gets a Boolean value that indicates whether automatic rotation of the key +// material (https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html) +// is enabled for the specified customer master key (CMK). +// +// You cannot enable automatic rotation of asymmetric CMKs, CMKs with imported +// key material, or CMKs in a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html). +// The key rotation status for these CMKs is always false. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// * Disabled: The key rotation status does not change when you disable a +// CMK. However, while the CMK is disabled, AWS KMS does not rotate the backing +// key. +// +// * Pending deletion: While a CMK is pending deletion, its key rotation +// status is false and AWS KMS does not rotate the backing key. If you cancel +// the deletion, the original key rotation status is restored. +// +// To perform this operation on a CMK in a different AWS account, specify the +// key ARN in the value of the KeyId parameter. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation GetKeyRotationStatus for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GetKeyRotationStatus +func (c *KMS) GetKeyRotationStatus(input *GetKeyRotationStatusInput) (*GetKeyRotationStatusOutput, error) { + req, out := c.GetKeyRotationStatusRequest(input) + return out, req.Send() +} + +// GetKeyRotationStatusWithContext is the same as GetKeyRotationStatus with the addition of +// the ability to pass a context and additional request options. +// +// See GetKeyRotationStatus for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) GetKeyRotationStatusWithContext(ctx aws.Context, input *GetKeyRotationStatusInput, opts ...request.Option) (*GetKeyRotationStatusOutput, error) { + req, out := c.GetKeyRotationStatusRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opGetParametersForImport = "GetParametersForImport" + +// GetParametersForImportRequest generates a "aws/request.Request" representing the +// client's request for the GetParametersForImport operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See GetParametersForImport for more information on using the GetParametersForImport +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the GetParametersForImportRequest method. +// req, resp := client.GetParametersForImportRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GetParametersForImport +func (c *KMS) GetParametersForImportRequest(input *GetParametersForImportInput) (req *request.Request, output *GetParametersForImportOutput) { + op := &request.Operation{ + Name: opGetParametersForImport, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &GetParametersForImportInput{} + } + + output = &GetParametersForImportOutput{} + req = c.newRequest(op, input, output) + return +} + +// GetParametersForImport API operation for AWS Key Management Service. +// +// Returns the items you need to import key material into a symmetric, customer +// managed customer master key (CMK). For more information about importing key +// material into AWS KMS, see Importing Key Material (https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys.html) +// in the AWS Key Management Service Developer Guide. +// +// This operation returns a public key and an import token. Use the public key +// to encrypt the symmetric key material. Store the import token to send with +// a subsequent ImportKeyMaterial request. +// +// You must specify the key ID of the symmetric CMK into which you will import +// key material. This CMK's Origin must be EXTERNAL. You must also specify the +// wrapping algorithm and type of wrapping key (public key) that you will use +// to encrypt the key material. You cannot perform this operation on an asymmetric +// CMK or on any CMK in a different AWS account. +// +// To import key material, you must use the public key and import token from +// the same response. These items are valid for 24 hours. The expiration date +// and time appear in the GetParametersForImport response. You cannot use an +// expired token in an ImportKeyMaterial request. If your key and token expire, +// send another GetParametersForImport request. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation GetParametersForImport for usage and error information. +// +// Returned Error Types: +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GetParametersForImport +func (c *KMS) GetParametersForImport(input *GetParametersForImportInput) (*GetParametersForImportOutput, error) { + req, out := c.GetParametersForImportRequest(input) + return out, req.Send() +} + +// GetParametersForImportWithContext is the same as GetParametersForImport with the addition of +// the ability to pass a context and additional request options. +// +// See GetParametersForImport for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) GetParametersForImportWithContext(ctx aws.Context, input *GetParametersForImportInput, opts ...request.Option) (*GetParametersForImportOutput, error) { + req, out := c.GetParametersForImportRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opGetPublicKey = "GetPublicKey" + +// GetPublicKeyRequest generates a "aws/request.Request" representing the +// client's request for the GetPublicKey operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See GetPublicKey for more information on using the GetPublicKey +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the GetPublicKeyRequest method. +// req, resp := client.GetPublicKeyRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GetPublicKey +func (c *KMS) GetPublicKeyRequest(input *GetPublicKeyInput) (req *request.Request, output *GetPublicKeyOutput) { + op := &request.Operation{ + Name: opGetPublicKey, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &GetPublicKeyInput{} + } + + output = &GetPublicKeyOutput{} + req = c.newRequest(op, input, output) + return +} + +// GetPublicKey API operation for AWS Key Management Service. +// +// Returns the public key of an asymmetric CMK. Unlike the private key of a +// asymmetric CMK, which never leaves AWS KMS unencrypted, callers with kms:GetPublicKey +// permission can download the public key of an asymmetric CMK. You can share +// the public key to allow others to encrypt messages and verify signatures +// outside of AWS KMS. For information about symmetric and asymmetric CMKs, +// see Using Symmetric and Asymmetric CMKs (https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html) +// in the AWS Key Management Service Developer Guide. +// +// You do not need to download the public key. Instead, you can use the public +// key within AWS KMS by calling the Encrypt, ReEncrypt, or Verify operations +// with the identifier of an asymmetric CMK. When you use the public key within +// AWS KMS, you benefit from the authentication, authorization, and logging +// that are part of every AWS KMS operation. You also reduce of risk of encrypting +// data that cannot be decrypted. These features are not effective outside of +// AWS KMS. For details, see Special Considerations for Downloading Public Keys +// (https://docs.aws.amazon.com/kms/latest/developerguide/download-public-key.html#download-public-key-considerations). +// +// To help you use the public key safely outside of AWS KMS, GetPublicKey returns +// important information about the public key in the response, including: +// +// * CustomerMasterKeySpec (https://docs.aws.amazon.com/kms/latest/APIReference/API_GetPublicKey.html#KMS-GetPublicKey-response-CustomerMasterKeySpec): +// The type of key material in the public key, such as RSA_4096 or ECC_NIST_P521. +// +// * KeyUsage (https://docs.aws.amazon.com/kms/latest/APIReference/API_GetPublicKey.html#KMS-GetPublicKey-response-KeyUsage): +// Whether the key is used for encryption or signing. +// +// * EncryptionAlgorithms (https://docs.aws.amazon.com/kms/latest/APIReference/API_GetPublicKey.html#KMS-GetPublicKey-response-EncryptionAlgorithms) +// or SigningAlgorithms (https://docs.aws.amazon.com/kms/latest/APIReference/API_GetPublicKey.html#KMS-GetPublicKey-response-SigningAlgorithms): +// A list of the encryption algorithms or the signing algorithms for the +// key. +// +// Although AWS KMS cannot enforce these restrictions on external operations, +// it is crucial that you use this information to prevent the public key from +// being used improperly. For example, you can prevent a public signing key +// from being used encrypt data, or prevent a public key from being used with +// an encryption algorithm that is not supported by AWS KMS. You can also avoid +// errors, such as using the wrong signing algorithm in a verification operation. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation GetPublicKey for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * KeyUnavailableException +// The request was rejected because the specified CMK was not available. You +// can retry the request. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InvalidKeyUsageException +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/GetPublicKey +func (c *KMS) GetPublicKey(input *GetPublicKeyInput) (*GetPublicKeyOutput, error) { + req, out := c.GetPublicKeyRequest(input) + return out, req.Send() +} + +// GetPublicKeyWithContext is the same as GetPublicKey with the addition of +// the ability to pass a context and additional request options. +// +// See GetPublicKey for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) GetPublicKeyWithContext(ctx aws.Context, input *GetPublicKeyInput, opts ...request.Option) (*GetPublicKeyOutput, error) { + req, out := c.GetPublicKeyRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opImportKeyMaterial = "ImportKeyMaterial" + +// ImportKeyMaterialRequest generates a "aws/request.Request" representing the +// client's request for the ImportKeyMaterial operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See ImportKeyMaterial for more information on using the ImportKeyMaterial +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the ImportKeyMaterialRequest method. +// req, resp := client.ImportKeyMaterialRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ImportKeyMaterial +func (c *KMS) ImportKeyMaterialRequest(input *ImportKeyMaterialInput) (req *request.Request, output *ImportKeyMaterialOutput) { + op := &request.Operation{ + Name: opImportKeyMaterial, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &ImportKeyMaterialInput{} + } + + output = &ImportKeyMaterialOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// ImportKeyMaterial API operation for AWS Key Management Service. +// +// Imports key material into an existing symmetric AWS KMS customer master key +// (CMK) that was created without key material. After you successfully import +// key material into a CMK, you can reimport the same key material (https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys.html#reimport-key-material) +// into that CMK, but you cannot import different key material. +// +// You cannot perform this operation on an asymmetric CMK or on any CMK in a +// different AWS account. For more information about creating CMKs with no key +// material and then importing key material, see Importing Key Material (https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys.html) +// in the AWS Key Management Service Developer Guide. +// +// Before using this operation, call GetParametersForImport. Its response includes +// a public key and an import token. Use the public key to encrypt the key material. +// Then, submit the import token from the same GetParametersForImport response. +// +// When calling this operation, you must specify the following values: +// +// * The key ID or key ARN of a CMK with no key material. Its Origin must +// be EXTERNAL. To create a CMK with no key material, call CreateKey and +// set the value of its Origin parameter to EXTERNAL. To get the Origin of +// a CMK, call DescribeKey.) +// +// * The encrypted key material. To get the public key to encrypt the key +// material, call GetParametersForImport. +// +// * The import token that GetParametersForImport returned. You must use +// a public key and token from the same GetParametersForImport response. +// +// * Whether the key material expires and if so, when. If you set an expiration +// date, AWS KMS deletes the key material from the CMK on the specified date, +// and the CMK becomes unusable. To use the CMK again, you must reimport +// the same key material. The only way to change an expiration date is by +// reimporting the same key material and specifying a new expiration date. +// +// When this operation is successful, the key state of the CMK changes from +// PendingImport to Enabled, and you can use the CMK. +// +// If this operation fails, use the exception to help determine the problem. +// If the error is related to the key material, the import token, or wrapping +// key, use GetParametersForImport to get a new public key and import token +// for the CMK and repeat the import procedure. For help, see How To Import +// Key Material (https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys.html#importing-keys-overview) +// in the AWS Key Management Service Developer Guide. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation ImportKeyMaterial for usage and error information. +// +// Returned Error Types: +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// * InvalidCiphertextException +// From the Decrypt or ReEncrypt operation, the request was rejected because +// the specified ciphertext, or additional authenticated data incorporated into +// the ciphertext, such as the encryption context, is corrupted, missing, or +// otherwise invalid. +// +// From the ImportKeyMaterial operation, the request was rejected because AWS +// KMS could not decrypt the encrypted (wrapped) key material. +// +// * IncorrectKeyMaterialException +// The request was rejected because the key material in the request is, expired, +// invalid, or is not the same key material that was previously imported into +// this customer master key (CMK). +// +// * ExpiredImportTokenException +// The request was rejected because the specified import token is expired. Use +// GetParametersForImport to get a new import token and public key, use the +// new public key to encrypt the key material, and then try the request again. +// +// * InvalidImportTokenException +// The request was rejected because the provided import token is invalid or +// is associated with a different customer master key (CMK). +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ImportKeyMaterial +func (c *KMS) ImportKeyMaterial(input *ImportKeyMaterialInput) (*ImportKeyMaterialOutput, error) { + req, out := c.ImportKeyMaterialRequest(input) + return out, req.Send() +} + +// ImportKeyMaterialWithContext is the same as ImportKeyMaterial with the addition of +// the ability to pass a context and additional request options. +// +// See ImportKeyMaterial for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ImportKeyMaterialWithContext(ctx aws.Context, input *ImportKeyMaterialInput, opts ...request.Option) (*ImportKeyMaterialOutput, error) { + req, out := c.ImportKeyMaterialRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opListAliases = "ListAliases" + +// ListAliasesRequest generates a "aws/request.Request" representing the +// client's request for the ListAliases operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See ListAliases for more information on using the ListAliases +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the ListAliasesRequest method. +// req, resp := client.ListAliasesRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListAliases +func (c *KMS) ListAliasesRequest(input *ListAliasesInput) (req *request.Request, output *ListAliasesOutput) { + op := &request.Operation{ + Name: opListAliases, + HTTPMethod: "POST", + HTTPPath: "/", + Paginator: &request.Paginator{ + InputTokens: []string{"Marker"}, + OutputTokens: []string{"NextMarker"}, + LimitToken: "Limit", + TruncationToken: "Truncated", + }, + } + + if input == nil { + input = &ListAliasesInput{} + } + + output = &ListAliasesOutput{} + req = c.newRequest(op, input, output) + return +} + +// ListAliases API operation for AWS Key Management Service. +// +// Gets a list of aliases in the caller's AWS account and region. You cannot +// list aliases in other accounts. For more information about aliases, see CreateAlias. +// +// By default, the ListAliases command returns all aliases in the account and +// region. To get only the aliases that point to a particular customer master +// key (CMK), use the KeyId parameter. +// +// The ListAliases response can include aliases that you created and associated +// with your customer managed CMKs, and aliases that AWS created and associated +// with AWS managed CMKs in your account. You can recognize AWS aliases because +// their names have the format aws/, such as aws/dynamodb. +// +// The response might also include aliases that have no TargetKeyId field. These +// are predefined aliases that AWS has created but has not yet associated with +// a CMK. Aliases that AWS creates in your account, including predefined aliases, +// do not count against your AWS KMS aliases quota (https://docs.aws.amazon.com/kms/latest/developerguide/limits.html#aliases-limit). +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation ListAliases for usage and error information. +// +// Returned Error Types: +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidMarkerException +// The request was rejected because the marker that specifies where pagination +// should next begin is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListAliases +func (c *KMS) ListAliases(input *ListAliasesInput) (*ListAliasesOutput, error) { + req, out := c.ListAliasesRequest(input) + return out, req.Send() +} + +// ListAliasesWithContext is the same as ListAliases with the addition of +// the ability to pass a context and additional request options. +// +// See ListAliases for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ListAliasesWithContext(ctx aws.Context, input *ListAliasesInput, opts ...request.Option) (*ListAliasesOutput, error) { + req, out := c.ListAliasesRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +// ListAliasesPages iterates over the pages of a ListAliases operation, +// calling the "fn" function with the response data for each page. To stop +// iterating, return false from the fn function. +// +// See ListAliases method for more information on how to use this operation. +// +// Note: This operation can generate multiple requests to a service. +// +// // Example iterating over at most 3 pages of a ListAliases operation. +// pageNum := 0 +// err := client.ListAliasesPages(params, +// func(page *kms.ListAliasesOutput, lastPage bool) bool { +// pageNum++ +// fmt.Println(page) +// return pageNum <= 3 +// }) +// +func (c *KMS) ListAliasesPages(input *ListAliasesInput, fn func(*ListAliasesOutput, bool) bool) error { + return c.ListAliasesPagesWithContext(aws.BackgroundContext(), input, fn) +} + +// ListAliasesPagesWithContext same as ListAliasesPages except +// it takes a Context and allows setting request options on the pages. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ListAliasesPagesWithContext(ctx aws.Context, input *ListAliasesInput, fn func(*ListAliasesOutput, bool) bool, opts ...request.Option) error { + p := request.Pagination{ + NewRequest: func() (*request.Request, error) { + var inCpy *ListAliasesInput + if input != nil { + tmp := *input + inCpy = &tmp + } + req, _ := c.ListAliasesRequest(inCpy) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return req, nil + }, + } + + for p.Next() { + if !fn(p.Page().(*ListAliasesOutput), !p.HasNextPage()) { + break + } + } + + return p.Err() +} + +const opListGrants = "ListGrants" + +// ListGrantsRequest generates a "aws/request.Request" representing the +// client's request for the ListGrants operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See ListGrants for more information on using the ListGrants +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the ListGrantsRequest method. +// req, resp := client.ListGrantsRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListGrants +func (c *KMS) ListGrantsRequest(input *ListGrantsInput) (req *request.Request, output *ListGrantsResponse) { + op := &request.Operation{ + Name: opListGrants, + HTTPMethod: "POST", + HTTPPath: "/", + Paginator: &request.Paginator{ + InputTokens: []string{"Marker"}, + OutputTokens: []string{"NextMarker"}, + LimitToken: "Limit", + TruncationToken: "Truncated", + }, + } + + if input == nil { + input = &ListGrantsInput{} + } + + output = &ListGrantsResponse{} + req = c.newRequest(op, input, output) + return +} + +// ListGrants API operation for AWS Key Management Service. +// +// Gets a list of all grants for the specified customer master key (CMK). +// +// To perform this operation on a CMK in a different AWS account, specify the +// key ARN in the value of the KeyId parameter. +// +// The GranteePrincipal field in the ListGrants response usually contains the +// user or role designated as the grantee principal in the grant. However, when +// the grantee principal in the grant is an AWS service, the GranteePrincipal +// field contains the service principal (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html#principal-services), +// which might represent several different grantee principals. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation ListGrants for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidMarkerException +// The request was rejected because the marker that specifies where pagination +// should next begin is not valid. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListGrants +func (c *KMS) ListGrants(input *ListGrantsInput) (*ListGrantsResponse, error) { + req, out := c.ListGrantsRequest(input) + return out, req.Send() +} + +// ListGrantsWithContext is the same as ListGrants with the addition of +// the ability to pass a context and additional request options. +// +// See ListGrants for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ListGrantsWithContext(ctx aws.Context, input *ListGrantsInput, opts ...request.Option) (*ListGrantsResponse, error) { + req, out := c.ListGrantsRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +// ListGrantsPages iterates over the pages of a ListGrants operation, +// calling the "fn" function with the response data for each page. To stop +// iterating, return false from the fn function. +// +// See ListGrants method for more information on how to use this operation. +// +// Note: This operation can generate multiple requests to a service. +// +// // Example iterating over at most 3 pages of a ListGrants operation. +// pageNum := 0 +// err := client.ListGrantsPages(params, +// func(page *kms.ListGrantsResponse, lastPage bool) bool { +// pageNum++ +// fmt.Println(page) +// return pageNum <= 3 +// }) +// +func (c *KMS) ListGrantsPages(input *ListGrantsInput, fn func(*ListGrantsResponse, bool) bool) error { + return c.ListGrantsPagesWithContext(aws.BackgroundContext(), input, fn) +} + +// ListGrantsPagesWithContext same as ListGrantsPages except +// it takes a Context and allows setting request options on the pages. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ListGrantsPagesWithContext(ctx aws.Context, input *ListGrantsInput, fn func(*ListGrantsResponse, bool) bool, opts ...request.Option) error { + p := request.Pagination{ + NewRequest: func() (*request.Request, error) { + var inCpy *ListGrantsInput + if input != nil { + tmp := *input + inCpy = &tmp + } + req, _ := c.ListGrantsRequest(inCpy) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return req, nil + }, + } + + for p.Next() { + if !fn(p.Page().(*ListGrantsResponse), !p.HasNextPage()) { + break + } + } + + return p.Err() +} + +const opListKeyPolicies = "ListKeyPolicies" + +// ListKeyPoliciesRequest generates a "aws/request.Request" representing the +// client's request for the ListKeyPolicies operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See ListKeyPolicies for more information on using the ListKeyPolicies +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the ListKeyPoliciesRequest method. +// req, resp := client.ListKeyPoliciesRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListKeyPolicies +func (c *KMS) ListKeyPoliciesRequest(input *ListKeyPoliciesInput) (req *request.Request, output *ListKeyPoliciesOutput) { + op := &request.Operation{ + Name: opListKeyPolicies, + HTTPMethod: "POST", + HTTPPath: "/", + Paginator: &request.Paginator{ + InputTokens: []string{"Marker"}, + OutputTokens: []string{"NextMarker"}, + LimitToken: "Limit", + TruncationToken: "Truncated", + }, + } + + if input == nil { + input = &ListKeyPoliciesInput{} + } + + output = &ListKeyPoliciesOutput{} + req = c.newRequest(op, input, output) + return +} + +// ListKeyPolicies API operation for AWS Key Management Service. +// +// Gets the names of the key policies that are attached to a customer master +// key (CMK). This operation is designed to get policy names that you can use +// in a GetKeyPolicy operation. However, the only valid policy name is default. +// You cannot perform this operation on a CMK in a different AWS account. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation ListKeyPolicies for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListKeyPolicies +func (c *KMS) ListKeyPolicies(input *ListKeyPoliciesInput) (*ListKeyPoliciesOutput, error) { + req, out := c.ListKeyPoliciesRequest(input) + return out, req.Send() +} + +// ListKeyPoliciesWithContext is the same as ListKeyPolicies with the addition of +// the ability to pass a context and additional request options. +// +// See ListKeyPolicies for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ListKeyPoliciesWithContext(ctx aws.Context, input *ListKeyPoliciesInput, opts ...request.Option) (*ListKeyPoliciesOutput, error) { + req, out := c.ListKeyPoliciesRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +// ListKeyPoliciesPages iterates over the pages of a ListKeyPolicies operation, +// calling the "fn" function with the response data for each page. To stop +// iterating, return false from the fn function. +// +// See ListKeyPolicies method for more information on how to use this operation. +// +// Note: This operation can generate multiple requests to a service. +// +// // Example iterating over at most 3 pages of a ListKeyPolicies operation. +// pageNum := 0 +// err := client.ListKeyPoliciesPages(params, +// func(page *kms.ListKeyPoliciesOutput, lastPage bool) bool { +// pageNum++ +// fmt.Println(page) +// return pageNum <= 3 +// }) +// +func (c *KMS) ListKeyPoliciesPages(input *ListKeyPoliciesInput, fn func(*ListKeyPoliciesOutput, bool) bool) error { + return c.ListKeyPoliciesPagesWithContext(aws.BackgroundContext(), input, fn) +} + +// ListKeyPoliciesPagesWithContext same as ListKeyPoliciesPages except +// it takes a Context and allows setting request options on the pages. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ListKeyPoliciesPagesWithContext(ctx aws.Context, input *ListKeyPoliciesInput, fn func(*ListKeyPoliciesOutput, bool) bool, opts ...request.Option) error { + p := request.Pagination{ + NewRequest: func() (*request.Request, error) { + var inCpy *ListKeyPoliciesInput + if input != nil { + tmp := *input + inCpy = &tmp + } + req, _ := c.ListKeyPoliciesRequest(inCpy) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return req, nil + }, + } + + for p.Next() { + if !fn(p.Page().(*ListKeyPoliciesOutput), !p.HasNextPage()) { + break + } + } + + return p.Err() +} + +const opListKeys = "ListKeys" + +// ListKeysRequest generates a "aws/request.Request" representing the +// client's request for the ListKeys operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See ListKeys for more information on using the ListKeys +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the ListKeysRequest method. +// req, resp := client.ListKeysRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListKeys +func (c *KMS) ListKeysRequest(input *ListKeysInput) (req *request.Request, output *ListKeysOutput) { + op := &request.Operation{ + Name: opListKeys, + HTTPMethod: "POST", + HTTPPath: "/", + Paginator: &request.Paginator{ + InputTokens: []string{"Marker"}, + OutputTokens: []string{"NextMarker"}, + LimitToken: "Limit", + TruncationToken: "Truncated", + }, + } + + if input == nil { + input = &ListKeysInput{} + } + + output = &ListKeysOutput{} + req = c.newRequest(op, input, output) + return +} + +// ListKeys API operation for AWS Key Management Service. +// +// Gets a list of all customer master keys (CMKs) in the caller's AWS account +// and Region. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation ListKeys for usage and error information. +// +// Returned Error Types: +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidMarkerException +// The request was rejected because the marker that specifies where pagination +// should next begin is not valid. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListKeys +func (c *KMS) ListKeys(input *ListKeysInput) (*ListKeysOutput, error) { + req, out := c.ListKeysRequest(input) + return out, req.Send() +} + +// ListKeysWithContext is the same as ListKeys with the addition of +// the ability to pass a context and additional request options. +// +// See ListKeys for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ListKeysWithContext(ctx aws.Context, input *ListKeysInput, opts ...request.Option) (*ListKeysOutput, error) { + req, out := c.ListKeysRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +// ListKeysPages iterates over the pages of a ListKeys operation, +// calling the "fn" function with the response data for each page. To stop +// iterating, return false from the fn function. +// +// See ListKeys method for more information on how to use this operation. +// +// Note: This operation can generate multiple requests to a service. +// +// // Example iterating over at most 3 pages of a ListKeys operation. +// pageNum := 0 +// err := client.ListKeysPages(params, +// func(page *kms.ListKeysOutput, lastPage bool) bool { +// pageNum++ +// fmt.Println(page) +// return pageNum <= 3 +// }) +// +func (c *KMS) ListKeysPages(input *ListKeysInput, fn func(*ListKeysOutput, bool) bool) error { + return c.ListKeysPagesWithContext(aws.BackgroundContext(), input, fn) +} + +// ListKeysPagesWithContext same as ListKeysPages except +// it takes a Context and allows setting request options on the pages. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ListKeysPagesWithContext(ctx aws.Context, input *ListKeysInput, fn func(*ListKeysOutput, bool) bool, opts ...request.Option) error { + p := request.Pagination{ + NewRequest: func() (*request.Request, error) { + var inCpy *ListKeysInput + if input != nil { + tmp := *input + inCpy = &tmp + } + req, _ := c.ListKeysRequest(inCpy) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return req, nil + }, + } + + for p.Next() { + if !fn(p.Page().(*ListKeysOutput), !p.HasNextPage()) { + break + } + } + + return p.Err() +} + +const opListResourceTags = "ListResourceTags" + +// ListResourceTagsRequest generates a "aws/request.Request" representing the +// client's request for the ListResourceTags operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See ListResourceTags for more information on using the ListResourceTags +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the ListResourceTagsRequest method. +// req, resp := client.ListResourceTagsRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListResourceTags +func (c *KMS) ListResourceTagsRequest(input *ListResourceTagsInput) (req *request.Request, output *ListResourceTagsOutput) { + op := &request.Operation{ + Name: opListResourceTags, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &ListResourceTagsInput{} + } + + output = &ListResourceTagsOutput{} + req = c.newRequest(op, input, output) + return +} + +// ListResourceTags API operation for AWS Key Management Service. +// +// Returns a list of all tags for the specified customer master key (CMK). +// +// You cannot perform this operation on a CMK in a different AWS account. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation ListResourceTags for usage and error information. +// +// Returned Error Types: +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * InvalidMarkerException +// The request was rejected because the marker that specifies where pagination +// should next begin is not valid. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListResourceTags +func (c *KMS) ListResourceTags(input *ListResourceTagsInput) (*ListResourceTagsOutput, error) { + req, out := c.ListResourceTagsRequest(input) + return out, req.Send() +} + +// ListResourceTagsWithContext is the same as ListResourceTags with the addition of +// the ability to pass a context and additional request options. +// +// See ListResourceTags for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ListResourceTagsWithContext(ctx aws.Context, input *ListResourceTagsInput, opts ...request.Option) (*ListResourceTagsOutput, error) { + req, out := c.ListResourceTagsRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opListRetirableGrants = "ListRetirableGrants" + +// ListRetirableGrantsRequest generates a "aws/request.Request" representing the +// client's request for the ListRetirableGrants operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See ListRetirableGrants for more information on using the ListRetirableGrants +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the ListRetirableGrantsRequest method. +// req, resp := client.ListRetirableGrantsRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListRetirableGrants +func (c *KMS) ListRetirableGrantsRequest(input *ListRetirableGrantsInput) (req *request.Request, output *ListGrantsResponse) { + op := &request.Operation{ + Name: opListRetirableGrants, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &ListRetirableGrantsInput{} + } + + output = &ListGrantsResponse{} + req = c.newRequest(op, input, output) + return +} + +// ListRetirableGrants API operation for AWS Key Management Service. +// +// Returns a list of all grants for which the grant's RetiringPrincipal matches +// the one specified. +// +// A typical use is to list all grants that you are able to retire. To retire +// a grant, use RetireGrant. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation ListRetirableGrants for usage and error information. +// +// Returned Error Types: +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidMarkerException +// The request was rejected because the marker that specifies where pagination +// should next begin is not valid. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ListRetirableGrants +func (c *KMS) ListRetirableGrants(input *ListRetirableGrantsInput) (*ListGrantsResponse, error) { + req, out := c.ListRetirableGrantsRequest(input) + return out, req.Send() +} + +// ListRetirableGrantsWithContext is the same as ListRetirableGrants with the addition of +// the ability to pass a context and additional request options. +// +// See ListRetirableGrants for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ListRetirableGrantsWithContext(ctx aws.Context, input *ListRetirableGrantsInput, opts ...request.Option) (*ListGrantsResponse, error) { + req, out := c.ListRetirableGrantsRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opPutKeyPolicy = "PutKeyPolicy" + +// PutKeyPolicyRequest generates a "aws/request.Request" representing the +// client's request for the PutKeyPolicy operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See PutKeyPolicy for more information on using the PutKeyPolicy +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the PutKeyPolicyRequest method. +// req, resp := client.PutKeyPolicyRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/PutKeyPolicy +func (c *KMS) PutKeyPolicyRequest(input *PutKeyPolicyInput) (req *request.Request, output *PutKeyPolicyOutput) { + op := &request.Operation{ + Name: opPutKeyPolicy, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &PutKeyPolicyInput{} + } + + output = &PutKeyPolicyOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// PutKeyPolicy API operation for AWS Key Management Service. +// +// Attaches a key policy to the specified customer master key (CMK). You cannot +// perform this operation on a CMK in a different AWS account. +// +// For more information about key policies, see Key Policies (https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation PutKeyPolicy for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * MalformedPolicyDocumentException +// The request was rejected because the specified policy is not syntactically +// or semantically correct. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * UnsupportedOperationException +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * LimitExceededException +// The request was rejected because a quota was exceeded. For more information, +// see Quotas (https://docs.aws.amazon.com/kms/latest/developerguide/limits.html) +// in the AWS Key Management Service Developer Guide. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/PutKeyPolicy +func (c *KMS) PutKeyPolicy(input *PutKeyPolicyInput) (*PutKeyPolicyOutput, error) { + req, out := c.PutKeyPolicyRequest(input) + return out, req.Send() +} + +// PutKeyPolicyWithContext is the same as PutKeyPolicy with the addition of +// the ability to pass a context and additional request options. +// +// See PutKeyPolicy for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) PutKeyPolicyWithContext(ctx aws.Context, input *PutKeyPolicyInput, opts ...request.Option) (*PutKeyPolicyOutput, error) { + req, out := c.PutKeyPolicyRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opReEncrypt = "ReEncrypt" + +// ReEncryptRequest generates a "aws/request.Request" representing the +// client's request for the ReEncrypt operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See ReEncrypt for more information on using the ReEncrypt +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the ReEncryptRequest method. +// req, resp := client.ReEncryptRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ReEncrypt +func (c *KMS) ReEncryptRequest(input *ReEncryptInput) (req *request.Request, output *ReEncryptOutput) { + op := &request.Operation{ + Name: opReEncrypt, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &ReEncryptInput{} + } + + output = &ReEncryptOutput{} + req = c.newRequest(op, input, output) + return +} + +// ReEncrypt API operation for AWS Key Management Service. +// +// Decrypts ciphertext and then reencrypts it entirely within AWS KMS. You can +// use this operation to change the customer master key (CMK) under which data +// is encrypted, such as when you manually rotate (https://docs.aws.amazon.com/kms/latest/developerguide/rotate-keys.html#rotate-keys-manually) +// a CMK or change the CMK that protects a ciphertext. You can also use it to +// reencrypt ciphertext under the same CMK, such as to change the encryption +// context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) +// of a ciphertext. +// +// The ReEncrypt operation can decrypt ciphertext that was encrypted by using +// an AWS KMS CMK in an AWS KMS operation, such as Encrypt or GenerateDataKey. +// It can also decrypt ciphertext that was encrypted by using the public key +// of an asymmetric CMK (https://docs.aws.amazon.com/kms/latest/developerguide/symm-asymm-concepts.html#asymmetric-cmks) +// outside of AWS KMS. However, it cannot decrypt ciphertext produced by other +// libraries, such as the AWS Encryption SDK (https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/) +// or Amazon S3 client-side encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html). +// These libraries return a ciphertext format that is incompatible with AWS +// KMS. +// +// When you use the ReEncrypt operation, you need to provide information for +// the decrypt operation and the subsequent encrypt operation. +// +// * If your ciphertext was encrypted under an asymmetric CMK, you must identify +// the source CMK, that is, the CMK that encrypted the ciphertext. You must +// also supply the encryption algorithm that was used. This information is +// required to decrypt the data. +// +// * It is optional, but you can specify a source CMK even when the ciphertext +// was encrypted under a symmetric CMK. This ensures that the ciphertext +// is decrypted only by using a particular CMK. If the CMK that you specify +// cannot decrypt the ciphertext, the ReEncrypt operation fails. +// +// * To reencrypt the data, you must specify the destination CMK, that is, +// the CMK that re-encrypts the data after it is decrypted. You can select +// a symmetric or asymmetric CMK. If the destination CMK is an asymmetric +// CMK, you must also provide the encryption algorithm. The algorithm that +// you choose must be compatible with the CMK. When you use an asymmetric +// CMK to encrypt or reencrypt data, be sure to record the CMK and encryption +// algorithm that you choose. You will be required to provide the same CMK +// and encryption algorithm when you decrypt the data. If the CMK and algorithm +// do not match the values used to encrypt the data, the decrypt operation +// fails. You are not required to supply the CMK ID and encryption algorithm +// when you decrypt with symmetric CMKs because AWS KMS stores this information +// in the ciphertext blob. AWS KMS cannot store metadata in ciphertext generated +// with asymmetric keys. The standard format for asymmetric key ciphertext +// does not include configurable fields. +// +// Unlike other AWS KMS API operations, ReEncrypt callers must have two permissions: +// +// * kms:ReEncryptFrom permission on the source CMK +// +// * kms:ReEncryptTo permission on the destination CMK +// +// To permit reencryption from or to a CMK, include the "kms:ReEncrypt*" permission +// in your key policy (https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html). +// This permission is automatically included in the key policy when you use +// the console to create a CMK. But you must include it manually when you create +// a CMK programmatically or when you use the PutKeyPolicy operation to set +// a key policy. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation ReEncrypt for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * InvalidCiphertextException +// From the Decrypt or ReEncrypt operation, the request was rejected because +// the specified ciphertext, or additional authenticated data incorporated into +// the ciphertext, such as the encryption context, is corrupted, missing, or +// otherwise invalid. +// +// From the ImportKeyMaterial operation, the request was rejected because AWS +// KMS could not decrypt the encrypted (wrapped) key material. +// +// * KeyUnavailableException +// The request was rejected because the specified CMK was not available. You +// can retry the request. +// +// * IncorrectKeyException +// The request was rejected because the specified CMK cannot decrypt the data. +// The KeyId in a Decrypt request and the SourceKeyId in a ReEncrypt request +// must identify the same CMK that was used to encrypt the ciphertext. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidKeyUsageException +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ReEncrypt +func (c *KMS) ReEncrypt(input *ReEncryptInput) (*ReEncryptOutput, error) { + req, out := c.ReEncryptRequest(input) + return out, req.Send() +} + +// ReEncryptWithContext is the same as ReEncrypt with the addition of +// the ability to pass a context and additional request options. +// +// See ReEncrypt for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ReEncryptWithContext(ctx aws.Context, input *ReEncryptInput, opts ...request.Option) (*ReEncryptOutput, error) { + req, out := c.ReEncryptRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opRetireGrant = "RetireGrant" + +// RetireGrantRequest generates a "aws/request.Request" representing the +// client's request for the RetireGrant operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See RetireGrant for more information on using the RetireGrant +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the RetireGrantRequest method. +// req, resp := client.RetireGrantRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/RetireGrant +func (c *KMS) RetireGrantRequest(input *RetireGrantInput) (req *request.Request, output *RetireGrantOutput) { + op := &request.Operation{ + Name: opRetireGrant, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &RetireGrantInput{} + } + + output = &RetireGrantOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// RetireGrant API operation for AWS Key Management Service. +// +// Retires a grant. To clean up, you can retire a grant when you're done using +// it. You should revoke a grant when you intend to actively deny operations +// that depend on it. The following are permitted to call this API: +// +// * The AWS account (root user) under which the grant was created +// +// * The RetiringPrincipal, if present in the grant +// +// * The GranteePrincipal, if RetireGrant is an operation specified in the +// grant +// +// You must identify the grant to retire by its grant token or by a combination +// of the grant ID and the Amazon Resource Name (ARN) of the customer master +// key (CMK). A grant token is a unique variable-length base64-encoded string. +// A grant ID is a 64 character unique identifier of a grant. The CreateGrant +// operation returns both. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation RetireGrant for usage and error information. +// +// Returned Error Types: +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InvalidGrantIdException +// The request was rejected because the specified GrantId is not valid. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/RetireGrant +func (c *KMS) RetireGrant(input *RetireGrantInput) (*RetireGrantOutput, error) { + req, out := c.RetireGrantRequest(input) + return out, req.Send() +} + +// RetireGrantWithContext is the same as RetireGrant with the addition of +// the ability to pass a context and additional request options. +// +// See RetireGrant for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) RetireGrantWithContext(ctx aws.Context, input *RetireGrantInput, opts ...request.Option) (*RetireGrantOutput, error) { + req, out := c.RetireGrantRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opRevokeGrant = "RevokeGrant" + +// RevokeGrantRequest generates a "aws/request.Request" representing the +// client's request for the RevokeGrant operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See RevokeGrant for more information on using the RevokeGrant +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the RevokeGrantRequest method. +// req, resp := client.RevokeGrantRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/RevokeGrant +func (c *KMS) RevokeGrantRequest(input *RevokeGrantInput) (req *request.Request, output *RevokeGrantOutput) { + op := &request.Operation{ + Name: opRevokeGrant, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &RevokeGrantInput{} + } + + output = &RevokeGrantOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// RevokeGrant API operation for AWS Key Management Service. +// +// Revokes the specified grant for the specified customer master key (CMK). +// You can revoke a grant to actively deny operations that depend on it. +// +// To perform this operation on a CMK in a different AWS account, specify the +// key ARN in the value of the KeyId parameter. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation RevokeGrant for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * InvalidGrantIdException +// The request was rejected because the specified GrantId is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/RevokeGrant +func (c *KMS) RevokeGrant(input *RevokeGrantInput) (*RevokeGrantOutput, error) { + req, out := c.RevokeGrantRequest(input) + return out, req.Send() +} + +// RevokeGrantWithContext is the same as RevokeGrant with the addition of +// the ability to pass a context and additional request options. +// +// See RevokeGrant for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) RevokeGrantWithContext(ctx aws.Context, input *RevokeGrantInput, opts ...request.Option) (*RevokeGrantOutput, error) { + req, out := c.RevokeGrantRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opScheduleKeyDeletion = "ScheduleKeyDeletion" + +// ScheduleKeyDeletionRequest generates a "aws/request.Request" representing the +// client's request for the ScheduleKeyDeletion operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See ScheduleKeyDeletion for more information on using the ScheduleKeyDeletion +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the ScheduleKeyDeletionRequest method. +// req, resp := client.ScheduleKeyDeletionRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ScheduleKeyDeletion +func (c *KMS) ScheduleKeyDeletionRequest(input *ScheduleKeyDeletionInput) (req *request.Request, output *ScheduleKeyDeletionOutput) { + op := &request.Operation{ + Name: opScheduleKeyDeletion, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &ScheduleKeyDeletionInput{} + } + + output = &ScheduleKeyDeletionOutput{} + req = c.newRequest(op, input, output) + return +} + +// ScheduleKeyDeletion API operation for AWS Key Management Service. +// +// Schedules the deletion of a customer master key (CMK). You may provide a +// waiting period, specified in days, before deletion occurs. If you do not +// provide a waiting period, the default period of 30 days is used. When this +// operation is successful, the key state of the CMK changes to PendingDeletion. +// Before the waiting period ends, you can use CancelKeyDeletion to cancel the +// deletion of the CMK. After the waiting period ends, AWS KMS deletes the CMK +// and all AWS KMS data associated with it, including all aliases that refer +// to it. +// +// Deleting a CMK is a destructive and potentially dangerous operation. When +// a CMK is deleted, all data that was encrypted under the CMK is unrecoverable. +// To prevent the use of a CMK without deleting it, use DisableKey. +// +// If you schedule deletion of a CMK from a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html), +// when the waiting period expires, ScheduleKeyDeletion deletes the CMK from +// AWS KMS. Then AWS KMS makes a best effort to delete the key material from +// the associated AWS CloudHSM cluster. However, you might need to manually +// delete the orphaned key material (https://docs.aws.amazon.com/kms/latest/developerguide/fix-keystore.html#fix-keystore-orphaned-key) +// from the cluster and its backups. +// +// You cannot perform this operation on a CMK in a different AWS account. +// +// For more information about scheduling a CMK for deletion, see Deleting Customer +// Master Keys (https://docs.aws.amazon.com/kms/latest/developerguide/deleting-keys.html) +// in the AWS Key Management Service Developer Guide. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation ScheduleKeyDeletion for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/ScheduleKeyDeletion +func (c *KMS) ScheduleKeyDeletion(input *ScheduleKeyDeletionInput) (*ScheduleKeyDeletionOutput, error) { + req, out := c.ScheduleKeyDeletionRequest(input) + return out, req.Send() +} + +// ScheduleKeyDeletionWithContext is the same as ScheduleKeyDeletion with the addition of +// the ability to pass a context and additional request options. +// +// See ScheduleKeyDeletion for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) ScheduleKeyDeletionWithContext(ctx aws.Context, input *ScheduleKeyDeletionInput, opts ...request.Option) (*ScheduleKeyDeletionOutput, error) { + req, out := c.ScheduleKeyDeletionRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opSign = "Sign" + +// SignRequest generates a "aws/request.Request" representing the +// client's request for the Sign operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See Sign for more information on using the Sign +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the SignRequest method. +// req, resp := client.SignRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/Sign +func (c *KMS) SignRequest(input *SignInput) (req *request.Request, output *SignOutput) { + op := &request.Operation{ + Name: opSign, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &SignInput{} + } + + output = &SignOutput{} + req = c.newRequest(op, input, output) + return +} + +// Sign API operation for AWS Key Management Service. +// +// Creates a digital signature (https://en.wikipedia.org/wiki/Digital_signature) +// for a message or message digest by using the private key in an asymmetric +// CMK. To verify the signature, use the Verify operation, or use the public +// key in the same asymmetric CMK outside of AWS KMS. For information about +// symmetric and asymmetric CMKs, see Using Symmetric and Asymmetric CMKs (https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html) +// in the AWS Key Management Service Developer Guide. +// +// Digital signatures are generated and verified by using asymmetric key pair, +// such as an RSA or ECC pair that is represented by an asymmetric customer +// master key (CMK). The key owner (or an authorized user) uses their private +// key to sign a message. Anyone with the public key can verify that the message +// was signed with that particular private key and that the message hasn't changed +// since it was signed. +// +// To use the Sign operation, provide the following information: +// +// * Use the KeyId parameter to identify an asymmetric CMK with a KeyUsage +// value of SIGN_VERIFY. To get the KeyUsage value of a CMK, use the DescribeKey +// operation. The caller must have kms:Sign permission on the CMK. +// +// * Use the Message parameter to specify the message or message digest to +// sign. You can submit messages of up to 4096 bytes. To sign a larger message, +// generate a hash digest of the message, and then provide the hash digest +// in the Message parameter. To indicate whether the message is a full message +// or a digest, use the MessageType parameter. +// +// * Choose a signing algorithm that is compatible with the CMK. +// +// When signing a message, be sure to record the CMK and the signing algorithm. +// This information is required to verify the signature. +// +// To verify the signature that this operation generates, use the Verify operation. +// Or use the GetPublicKey operation to download the public key and then use +// the public key to verify the signature outside of AWS KMS. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation Sign for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * KeyUnavailableException +// The request was rejected because the specified CMK was not available. You +// can retry the request. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidKeyUsageException +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/Sign +func (c *KMS) Sign(input *SignInput) (*SignOutput, error) { + req, out := c.SignRequest(input) + return out, req.Send() +} + +// SignWithContext is the same as Sign with the addition of +// the ability to pass a context and additional request options. +// +// See Sign for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) SignWithContext(ctx aws.Context, input *SignInput, opts ...request.Option) (*SignOutput, error) { + req, out := c.SignRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opTagResource = "TagResource" + +// TagResourceRequest generates a "aws/request.Request" representing the +// client's request for the TagResource operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See TagResource for more information on using the TagResource +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the TagResourceRequest method. +// req, resp := client.TagResourceRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/TagResource +func (c *KMS) TagResourceRequest(input *TagResourceInput) (req *request.Request, output *TagResourceOutput) { + op := &request.Operation{ + Name: opTagResource, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &TagResourceInput{} + } + + output = &TagResourceOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// TagResource API operation for AWS Key Management Service. +// +// Adds or edits tags for a customer master key (CMK). You cannot perform this +// operation on a CMK in a different AWS account. +// +// Each tag consists of a tag key and a tag value. Tag keys and tag values are +// both required, but tag values can be empty (null) strings. +// +// You can only use a tag key once for each CMK. If you use the tag key again, +// AWS KMS replaces the current tag value with the specified value. +// +// For information about the rules that apply to tag keys and tag values, see +// User-Defined Tag Restrictions (https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html) +// in the AWS Billing and Cost Management User Guide. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation TagResource for usage and error information. +// +// Returned Error Types: +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// * LimitExceededException +// The request was rejected because a quota was exceeded. For more information, +// see Quotas (https://docs.aws.amazon.com/kms/latest/developerguide/limits.html) +// in the AWS Key Management Service Developer Guide. +// +// * TagException +// The request was rejected because one or more tags are not valid. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/TagResource +func (c *KMS) TagResource(input *TagResourceInput) (*TagResourceOutput, error) { + req, out := c.TagResourceRequest(input) + return out, req.Send() +} + +// TagResourceWithContext is the same as TagResource with the addition of +// the ability to pass a context and additional request options. +// +// See TagResource for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) TagResourceWithContext(ctx aws.Context, input *TagResourceInput, opts ...request.Option) (*TagResourceOutput, error) { + req, out := c.TagResourceRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opUntagResource = "UntagResource" + +// UntagResourceRequest generates a "aws/request.Request" representing the +// client's request for the UntagResource operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See UntagResource for more information on using the UntagResource +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the UntagResourceRequest method. +// req, resp := client.UntagResourceRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/UntagResource +func (c *KMS) UntagResourceRequest(input *UntagResourceInput) (req *request.Request, output *UntagResourceOutput) { + op := &request.Operation{ + Name: opUntagResource, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &UntagResourceInput{} + } + + output = &UntagResourceOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// UntagResource API operation for AWS Key Management Service. +// +// Removes the specified tags from the specified customer master key (CMK). +// You cannot perform this operation on a CMK in a different AWS account. +// +// To remove a tag, specify the tag key. To change the tag value of an existing +// tag key, use TagResource. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation UntagResource for usage and error information. +// +// Returned Error Types: +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// * TagException +// The request was rejected because one or more tags are not valid. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/UntagResource +func (c *KMS) UntagResource(input *UntagResourceInput) (*UntagResourceOutput, error) { + req, out := c.UntagResourceRequest(input) + return out, req.Send() +} + +// UntagResourceWithContext is the same as UntagResource with the addition of +// the ability to pass a context and additional request options. +// +// See UntagResource for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) UntagResourceWithContext(ctx aws.Context, input *UntagResourceInput, opts ...request.Option) (*UntagResourceOutput, error) { + req, out := c.UntagResourceRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opUpdateAlias = "UpdateAlias" + +// UpdateAliasRequest generates a "aws/request.Request" representing the +// client's request for the UpdateAlias operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See UpdateAlias for more information on using the UpdateAlias +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the UpdateAliasRequest method. +// req, resp := client.UpdateAliasRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/UpdateAlias +func (c *KMS) UpdateAliasRequest(input *UpdateAliasInput) (req *request.Request, output *UpdateAliasOutput) { + op := &request.Operation{ + Name: opUpdateAlias, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &UpdateAliasInput{} + } + + output = &UpdateAliasOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// UpdateAlias API operation for AWS Key Management Service. +// +// Associates an existing AWS KMS alias with a different customer master key +// (CMK). Each alias is associated with only one CMK at a time, although a CMK +// can have multiple aliases. The alias and the CMK must be in the same AWS +// account and region. You cannot perform this operation on an alias in a different +// AWS account. +// +// The current and new CMK must be the same type (both symmetric or both asymmetric), +// and they must have the same key usage (ENCRYPT_DECRYPT or SIGN_VERIFY). This +// restriction prevents errors in code that uses aliases. If you must assign +// an alias to a different type of CMK, use DeleteAlias to delete the old alias +// and CreateAlias to create a new alias. +// +// You cannot use UpdateAlias to change an alias name. To change an alias name, +// use DeleteAlias to delete the old alias and CreateAlias to create a new alias. +// +// Because an alias is not a property of a CMK, you can create, update, and +// delete the aliases of a CMK without affecting the CMK. Also, aliases do not +// appear in the response from the DescribeKey operation. To get the aliases +// of all CMKs in the account, use the ListAliases operation. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation UpdateAlias for usage and error information. +// +// Returned Error Types: +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * LimitExceededException +// The request was rejected because a quota was exceeded. For more information, +// see Quotas (https://docs.aws.amazon.com/kms/latest/developerguide/limits.html) +// in the AWS Key Management Service Developer Guide. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/UpdateAlias +func (c *KMS) UpdateAlias(input *UpdateAliasInput) (*UpdateAliasOutput, error) { + req, out := c.UpdateAliasRequest(input) + return out, req.Send() +} + +// UpdateAliasWithContext is the same as UpdateAlias with the addition of +// the ability to pass a context and additional request options. +// +// See UpdateAlias for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) UpdateAliasWithContext(ctx aws.Context, input *UpdateAliasInput, opts ...request.Option) (*UpdateAliasOutput, error) { + req, out := c.UpdateAliasRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opUpdateCustomKeyStore = "UpdateCustomKeyStore" + +// UpdateCustomKeyStoreRequest generates a "aws/request.Request" representing the +// client's request for the UpdateCustomKeyStore operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See UpdateCustomKeyStore for more information on using the UpdateCustomKeyStore +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the UpdateCustomKeyStoreRequest method. +// req, resp := client.UpdateCustomKeyStoreRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/UpdateCustomKeyStore +func (c *KMS) UpdateCustomKeyStoreRequest(input *UpdateCustomKeyStoreInput) (req *request.Request, output *UpdateCustomKeyStoreOutput) { + op := &request.Operation{ + Name: opUpdateCustomKeyStore, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &UpdateCustomKeyStoreInput{} + } + + output = &UpdateCustomKeyStoreOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// UpdateCustomKeyStore API operation for AWS Key Management Service. +// +// Changes the properties of a custom key store. Use the CustomKeyStoreId parameter +// to identify the custom key store you want to edit. Use the remaining parameters +// to change the properties of the custom key store. +// +// You can only update a custom key store that is disconnected. To disconnect +// the custom key store, use DisconnectCustomKeyStore. To reconnect the custom +// key store after the update completes, use ConnectCustomKeyStore. To find +// the connection state of a custom key store, use the DescribeCustomKeyStores +// operation. +// +// Use the parameters of UpdateCustomKeyStore to edit your keystore settings. +// +// * Use the NewCustomKeyStoreName parameter to change the friendly name +// of the custom key store to the value that you specify. +// +// * Use the KeyStorePassword parameter tell AWS KMS the current password +// of the kmsuser crypto user (CU) (https://docs.aws.amazon.com/kms/latest/developerguide/key-store-concepts.html#concept-kmsuser) +// in the associated AWS CloudHSM cluster. You can use this parameter to +// fix connection failures (https://docs.aws.amazon.com/kms/latest/developerguide/fix-keystore.html#fix-keystore-password) +// that occur when AWS KMS cannot log into the associated cluster because +// the kmsuser password has changed. This value does not change the password +// in the AWS CloudHSM cluster. +// +// * Use the CloudHsmClusterId parameter to associate the custom key store +// with a different, but related, AWS CloudHSM cluster. You can use this +// parameter to repair a custom key store if its AWS CloudHSM cluster becomes +// corrupted or is deleted, or when you need to create or restore a cluster +// from a backup. +// +// If the operation succeeds, it returns a JSON object with no properties. +// +// This operation is part of the Custom Key Store feature (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) +// feature in AWS KMS, which combines the convenience and extensive integration +// of AWS KMS with the isolation and control of a single-tenant key store. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation UpdateCustomKeyStore for usage and error information. +// +// Returned Error Types: +// * CustomKeyStoreNotFoundException +// The request was rejected because AWS KMS cannot find a custom key store with +// the specified key store name or ID. +// +// * CustomKeyStoreNameInUseException +// The request was rejected because the specified custom key store name is already +// assigned to another custom key store in the account. Try again with a custom +// key store name that is unique in the account. +// +// * CloudHsmClusterNotFoundException +// The request was rejected because AWS KMS cannot find the AWS CloudHSM cluster +// with the specified cluster ID. Retry the request with a different cluster +// ID. +// +// * CloudHsmClusterNotRelatedException +// The request was rejected because the specified AWS CloudHSM cluster has a +// different cluster certificate than the original cluster. You cannot use the +// operation to specify an unrelated cluster. +// +// Specify a cluster that shares a backup history with the original cluster. +// This includes clusters that were created from a backup of the current cluster, +// and clusters that were created from the same backup that produced the current +// cluster. +// +// Clusters that share a backup history have the same cluster certificate. To +// view the cluster certificate of a cluster, use the DescribeClusters (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_DescribeClusters.html) +// operation. +// +// * CustomKeyStoreInvalidStateException +// The request was rejected because of the ConnectionState of the custom key +// store. To get the ConnectionState of a custom key store, use the DescribeCustomKeyStores +// operation. +// +// This exception is thrown under the following conditions: +// +// * You requested the CreateKey or GenerateRandom operation in a custom +// key store that is not connected. These operations are valid only when +// the custom key store ConnectionState is CONNECTED. +// +// * You requested the UpdateCustomKeyStore or DeleteCustomKeyStore operation +// on a custom key store that is not disconnected. This operation is valid +// only when the custom key store ConnectionState is DISCONNECTED. +// +// * You requested the ConnectCustomKeyStore operation on a custom key store +// with a ConnectionState of DISCONNECTING or FAILED. This operation is valid +// for all other ConnectionState values. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * CloudHsmClusterNotActiveException +// The request was rejected because the AWS CloudHSM cluster that is associated +// with the custom key store is not active. Initialize and activate the cluster +// and try the command again. For detailed instructions, see Getting Started +// (https://docs.aws.amazon.com/cloudhsm/latest/userguide/getting-started.html) +// in the AWS CloudHSM User Guide. +// +// * CloudHsmClusterInvalidConfigurationException +// The request was rejected because the associated AWS CloudHSM cluster did +// not meet the configuration requirements for a custom key store. +// +// * The cluster must be configured with private subnets in at least two +// different Availability Zones in the Region. +// +// * The security group for the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) +// (cloudhsm-cluster--sg) must include inbound rules and outbound +// rules that allow TCP traffic on ports 2223-2225. The Source in the inbound +// rules and the Destination in the outbound rules must match the security +// group ID. These rules are set by default when you create the cluster. +// Do not delete or change them. To get information about a particular security +// group, use the DescribeSecurityGroups (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSecurityGroups.html) +// operation. +// +// * The cluster must contain at least as many HSMs as the operation requires. +// To add HSMs, use the AWS CloudHSM CreateHsm (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_CreateHsm.html) +// operation. For the CreateCustomKeyStore, UpdateCustomKeyStore, and CreateKey +// operations, the AWS CloudHSM cluster must have at least two active HSMs, +// each in a different Availability Zone. For the ConnectCustomKeyStore operation, +// the AWS CloudHSM must contain at least one active HSM. +// +// For information about the requirements for an AWS CloudHSM cluster that is +// associated with a custom key store, see Assemble the Prerequisites (https://docs.aws.amazon.com/kms/latest/developerguide/create-keystore.html#before-keystore) +// in the AWS Key Management Service Developer Guide. For information about +// creating a private subnet for an AWS CloudHSM cluster, see Create a Private +// Subnet (https://docs.aws.amazon.com/cloudhsm/latest/userguide/create-subnets.html) +// in the AWS CloudHSM User Guide. For information about cluster security groups, +// see Configure a Default Security Group (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) +// in the AWS CloudHSM User Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/UpdateCustomKeyStore +func (c *KMS) UpdateCustomKeyStore(input *UpdateCustomKeyStoreInput) (*UpdateCustomKeyStoreOutput, error) { + req, out := c.UpdateCustomKeyStoreRequest(input) + return out, req.Send() +} + +// UpdateCustomKeyStoreWithContext is the same as UpdateCustomKeyStore with the addition of +// the ability to pass a context and additional request options. +// +// See UpdateCustomKeyStore for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) UpdateCustomKeyStoreWithContext(ctx aws.Context, input *UpdateCustomKeyStoreInput, opts ...request.Option) (*UpdateCustomKeyStoreOutput, error) { + req, out := c.UpdateCustomKeyStoreRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opUpdateKeyDescription = "UpdateKeyDescription" + +// UpdateKeyDescriptionRequest generates a "aws/request.Request" representing the +// client's request for the UpdateKeyDescription operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See UpdateKeyDescription for more information on using the UpdateKeyDescription +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the UpdateKeyDescriptionRequest method. +// req, resp := client.UpdateKeyDescriptionRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/UpdateKeyDescription +func (c *KMS) UpdateKeyDescriptionRequest(input *UpdateKeyDescriptionInput) (req *request.Request, output *UpdateKeyDescriptionOutput) { + op := &request.Operation{ + Name: opUpdateKeyDescription, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &UpdateKeyDescriptionInput{} + } + + output = &UpdateKeyDescriptionOutput{} + req = c.newRequest(op, input, output) + req.Handlers.Unmarshal.Swap(jsonrpc.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + return +} + +// UpdateKeyDescription API operation for AWS Key Management Service. +// +// Updates the description of a customer master key (CMK). To see the description +// of a CMK, use DescribeKey. +// +// You cannot perform this operation on a CMK in a different AWS account. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation UpdateKeyDescription for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * InvalidArnException +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/UpdateKeyDescription +func (c *KMS) UpdateKeyDescription(input *UpdateKeyDescriptionInput) (*UpdateKeyDescriptionOutput, error) { + req, out := c.UpdateKeyDescriptionRequest(input) + return out, req.Send() +} + +// UpdateKeyDescriptionWithContext is the same as UpdateKeyDescription with the addition of +// the ability to pass a context and additional request options. +// +// See UpdateKeyDescription for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) UpdateKeyDescriptionWithContext(ctx aws.Context, input *UpdateKeyDescriptionInput, opts ...request.Option) (*UpdateKeyDescriptionOutput, error) { + req, out := c.UpdateKeyDescriptionRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +const opVerify = "Verify" + +// VerifyRequest generates a "aws/request.Request" representing the +// client's request for the Verify operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See Verify for more information on using the Verify +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the VerifyRequest method. +// req, resp := client.VerifyRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/Verify +func (c *KMS) VerifyRequest(input *VerifyInput) (req *request.Request, output *VerifyOutput) { + op := &request.Operation{ + Name: opVerify, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &VerifyInput{} + } + + output = &VerifyOutput{} + req = c.newRequest(op, input, output) + return +} + +// Verify API operation for AWS Key Management Service. +// +// Verifies a digital signature that was generated by the Sign operation. +// +// Verification confirms that an authorized user signed the message with the +// specified CMK and signing algorithm, and the message hasn't changed since +// it was signed. If the signature is verified, the value of the SignatureValid +// field in the response is True. If the signature verification fails, the Verify +// operation fails with an KMSInvalidSignatureException exception. +// +// A digital signature is generated by using the private key in an asymmetric +// CMK. The signature is verified by using the public key in the same asymmetric +// CMK. For information about symmetric and asymmetric CMKs, see Using Symmetric +// and Asymmetric CMKs (https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html) +// in the AWS Key Management Service Developer Guide. +// +// To verify a digital signature, you can use the Verify operation. Specify +// the same asymmetric CMK, message, and signing algorithm that were used to +// produce the signature. +// +// You can also verify the digital signature by using the public key of the +// CMK outside of AWS KMS. Use the GetPublicKey operation to download the public +// key in the asymmetric CMK and then use the public key to verify the signature +// outside of AWS KMS. The advantage of using the Verify operation is that it +// is performed within AWS KMS. As a result, it's easy to call, the operation +// is performed within the FIPS boundary, it is logged in AWS CloudTrail, and +// you can use key policy and IAM policy to determine who is authorized to use +// the CMK to verify signatures. +// +// The CMK that you use for this operation must be in a compatible key state. +// For details, see How Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Key Management Service's +// API operation Verify for usage and error information. +// +// Returned Error Types: +// * NotFoundException +// The request was rejected because the specified entity or resource could not +// be found. +// +// * DisabledException +// The request was rejected because the specified CMK is not enabled. +// +// * KeyUnavailableException +// The request was rejected because the specified CMK was not available. You +// can retry the request. +// +// * DependencyTimeoutException +// The system timed out while trying to fulfill the request. The request can +// be retried. +// +// * InvalidKeyUsageException +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +// +// * InvalidGrantTokenException +// The request was rejected because the specified grant token is not valid. +// +// * InternalException +// The request was rejected because an internal exception occurred. The request +// can be retried. +// +// * InvalidStateException +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +// +// * KMSInvalidSignatureException +// The request was rejected because the signature verification failed. Signature +// verification fails when it cannot confirm that signature was produced by +// signing the specified message with the specified CMK and signing algorithm. +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01/Verify +func (c *KMS) Verify(input *VerifyInput) (*VerifyOutput, error) { + req, out := c.VerifyRequest(input) + return out, req.Send() +} + +// VerifyWithContext is the same as Verify with the addition of +// the ability to pass a context and additional request options. +// +// See Verify for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *KMS) VerifyWithContext(ctx aws.Context, input *VerifyInput, opts ...request.Option) (*VerifyOutput, error) { + req, out := c.VerifyRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + +// Contains information about an alias. +type AliasListEntry struct { + _ struct{} `type:"structure"` + + // String that contains the key ARN. + AliasArn *string `min:"20" type:"string"` + + // String that contains the alias. This value begins with alias/. + AliasName *string `min:"1" type:"string"` + + // String that contains the key identifier referred to by the alias. + TargetKeyId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s AliasListEntry) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s AliasListEntry) GoString() string { + return s.String() +} + +// SetAliasArn sets the AliasArn field's value. +func (s *AliasListEntry) SetAliasArn(v string) *AliasListEntry { + s.AliasArn = &v + return s +} + +// SetAliasName sets the AliasName field's value. +func (s *AliasListEntry) SetAliasName(v string) *AliasListEntry { + s.AliasName = &v + return s +} + +// SetTargetKeyId sets the TargetKeyId field's value. +func (s *AliasListEntry) SetTargetKeyId(v string) *AliasListEntry { + s.TargetKeyId = &v + return s +} + +// The request was rejected because it attempted to create a resource that already +// exists. +type AlreadyExistsException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s AlreadyExistsException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s AlreadyExistsException) GoString() string { + return s.String() +} + +func newErrorAlreadyExistsException(v protocol.ResponseMetadata) error { + return &AlreadyExistsException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *AlreadyExistsException) Code() string { + return "AlreadyExistsException" +} + +// Message returns the exception's message. +func (s *AlreadyExistsException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *AlreadyExistsException) OrigErr() error { + return nil +} + +func (s *AlreadyExistsException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *AlreadyExistsException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *AlreadyExistsException) RequestID() string { + return s.RespMetadata.RequestID +} + +type CancelKeyDeletionInput struct { + _ struct{} `type:"structure"` + + // The unique identifier for the customer master key (CMK) for which to cancel + // deletion. + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s CancelKeyDeletionInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CancelKeyDeletionInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *CancelKeyDeletionInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "CancelKeyDeletionInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *CancelKeyDeletionInput) SetKeyId(v string) *CancelKeyDeletionInput { + s.KeyId = &v + return s +} + +type CancelKeyDeletionOutput struct { + _ struct{} `type:"structure"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the CMK whose deletion is canceled. + KeyId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s CancelKeyDeletionOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CancelKeyDeletionOutput) GoString() string { + return s.String() +} + +// SetKeyId sets the KeyId field's value. +func (s *CancelKeyDeletionOutput) SetKeyId(v string) *CancelKeyDeletionOutput { + s.KeyId = &v + return s +} + +// The request was rejected because the specified AWS CloudHSM cluster is already +// associated with a custom key store or it shares a backup history with a cluster +// that is associated with a custom key store. Each custom key store must be +// associated with a different AWS CloudHSM cluster. +// +// Clusters that share a backup history have the same cluster certificate. To +// view the cluster certificate of a cluster, use the DescribeClusters (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_DescribeClusters.html) +// operation. +type CloudHsmClusterInUseException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s CloudHsmClusterInUseException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CloudHsmClusterInUseException) GoString() string { + return s.String() +} + +func newErrorCloudHsmClusterInUseException(v protocol.ResponseMetadata) error { + return &CloudHsmClusterInUseException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *CloudHsmClusterInUseException) Code() string { + return "CloudHsmClusterInUseException" +} + +// Message returns the exception's message. +func (s *CloudHsmClusterInUseException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *CloudHsmClusterInUseException) OrigErr() error { + return nil +} + +func (s *CloudHsmClusterInUseException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *CloudHsmClusterInUseException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *CloudHsmClusterInUseException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the associated AWS CloudHSM cluster did +// not meet the configuration requirements for a custom key store. +// +// * The cluster must be configured with private subnets in at least two +// different Availability Zones in the Region. +// +// * The security group for the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) +// (cloudhsm-cluster--sg) must include inbound rules and outbound +// rules that allow TCP traffic on ports 2223-2225. The Source in the inbound +// rules and the Destination in the outbound rules must match the security +// group ID. These rules are set by default when you create the cluster. +// Do not delete or change them. To get information about a particular security +// group, use the DescribeSecurityGroups (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSecurityGroups.html) +// operation. +// +// * The cluster must contain at least as many HSMs as the operation requires. +// To add HSMs, use the AWS CloudHSM CreateHsm (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_CreateHsm.html) +// operation. For the CreateCustomKeyStore, UpdateCustomKeyStore, and CreateKey +// operations, the AWS CloudHSM cluster must have at least two active HSMs, +// each in a different Availability Zone. For the ConnectCustomKeyStore operation, +// the AWS CloudHSM must contain at least one active HSM. +// +// For information about the requirements for an AWS CloudHSM cluster that is +// associated with a custom key store, see Assemble the Prerequisites (https://docs.aws.amazon.com/kms/latest/developerguide/create-keystore.html#before-keystore) +// in the AWS Key Management Service Developer Guide. For information about +// creating a private subnet for an AWS CloudHSM cluster, see Create a Private +// Subnet (https://docs.aws.amazon.com/cloudhsm/latest/userguide/create-subnets.html) +// in the AWS CloudHSM User Guide. For information about cluster security groups, +// see Configure a Default Security Group (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) +// in the AWS CloudHSM User Guide . +type CloudHsmClusterInvalidConfigurationException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s CloudHsmClusterInvalidConfigurationException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CloudHsmClusterInvalidConfigurationException) GoString() string { + return s.String() +} + +func newErrorCloudHsmClusterInvalidConfigurationException(v protocol.ResponseMetadata) error { + return &CloudHsmClusterInvalidConfigurationException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *CloudHsmClusterInvalidConfigurationException) Code() string { + return "CloudHsmClusterInvalidConfigurationException" +} + +// Message returns the exception's message. +func (s *CloudHsmClusterInvalidConfigurationException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *CloudHsmClusterInvalidConfigurationException) OrigErr() error { + return nil +} + +func (s *CloudHsmClusterInvalidConfigurationException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *CloudHsmClusterInvalidConfigurationException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *CloudHsmClusterInvalidConfigurationException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the AWS CloudHSM cluster that is associated +// with the custom key store is not active. Initialize and activate the cluster +// and try the command again. For detailed instructions, see Getting Started +// (https://docs.aws.amazon.com/cloudhsm/latest/userguide/getting-started.html) +// in the AWS CloudHSM User Guide. +type CloudHsmClusterNotActiveException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s CloudHsmClusterNotActiveException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CloudHsmClusterNotActiveException) GoString() string { + return s.String() +} + +func newErrorCloudHsmClusterNotActiveException(v protocol.ResponseMetadata) error { + return &CloudHsmClusterNotActiveException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *CloudHsmClusterNotActiveException) Code() string { + return "CloudHsmClusterNotActiveException" +} + +// Message returns the exception's message. +func (s *CloudHsmClusterNotActiveException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *CloudHsmClusterNotActiveException) OrigErr() error { + return nil +} + +func (s *CloudHsmClusterNotActiveException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *CloudHsmClusterNotActiveException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *CloudHsmClusterNotActiveException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because AWS KMS cannot find the AWS CloudHSM cluster +// with the specified cluster ID. Retry the request with a different cluster +// ID. +type CloudHsmClusterNotFoundException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s CloudHsmClusterNotFoundException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CloudHsmClusterNotFoundException) GoString() string { + return s.String() +} + +func newErrorCloudHsmClusterNotFoundException(v protocol.ResponseMetadata) error { + return &CloudHsmClusterNotFoundException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *CloudHsmClusterNotFoundException) Code() string { + return "CloudHsmClusterNotFoundException" +} + +// Message returns the exception's message. +func (s *CloudHsmClusterNotFoundException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *CloudHsmClusterNotFoundException) OrigErr() error { + return nil +} + +func (s *CloudHsmClusterNotFoundException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *CloudHsmClusterNotFoundException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *CloudHsmClusterNotFoundException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the specified AWS CloudHSM cluster has a +// different cluster certificate than the original cluster. You cannot use the +// operation to specify an unrelated cluster. +// +// Specify a cluster that shares a backup history with the original cluster. +// This includes clusters that were created from a backup of the current cluster, +// and clusters that were created from the same backup that produced the current +// cluster. +// +// Clusters that share a backup history have the same cluster certificate. To +// view the cluster certificate of a cluster, use the DescribeClusters (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_DescribeClusters.html) +// operation. +type CloudHsmClusterNotRelatedException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s CloudHsmClusterNotRelatedException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CloudHsmClusterNotRelatedException) GoString() string { + return s.String() +} + +func newErrorCloudHsmClusterNotRelatedException(v protocol.ResponseMetadata) error { + return &CloudHsmClusterNotRelatedException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *CloudHsmClusterNotRelatedException) Code() string { + return "CloudHsmClusterNotRelatedException" +} + +// Message returns the exception's message. +func (s *CloudHsmClusterNotRelatedException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *CloudHsmClusterNotRelatedException) OrigErr() error { + return nil +} + +func (s *CloudHsmClusterNotRelatedException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *CloudHsmClusterNotRelatedException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *CloudHsmClusterNotRelatedException) RequestID() string { + return s.RespMetadata.RequestID +} + +type ConnectCustomKeyStoreInput struct { + _ struct{} `type:"structure"` + + // Enter the key store ID of the custom key store that you want to connect. + // To find the ID of a custom key store, use the DescribeCustomKeyStores operation. + // + // CustomKeyStoreId is a required field + CustomKeyStoreId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s ConnectCustomKeyStoreInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ConnectCustomKeyStoreInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ConnectCustomKeyStoreInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ConnectCustomKeyStoreInput"} + if s.CustomKeyStoreId == nil { + invalidParams.Add(request.NewErrParamRequired("CustomKeyStoreId")) + } + if s.CustomKeyStoreId != nil && len(*s.CustomKeyStoreId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CustomKeyStoreId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetCustomKeyStoreId sets the CustomKeyStoreId field's value. +func (s *ConnectCustomKeyStoreInput) SetCustomKeyStoreId(v string) *ConnectCustomKeyStoreInput { + s.CustomKeyStoreId = &v + return s +} + +type ConnectCustomKeyStoreOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s ConnectCustomKeyStoreOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ConnectCustomKeyStoreOutput) GoString() string { + return s.String() +} + +type CreateAliasInput struct { + _ struct{} `type:"structure"` + + // Specifies the alias name. This value must begin with alias/ followed by a + // name, such as alias/ExampleAlias. The alias name cannot begin with alias/aws/. + // The alias/aws/ prefix is reserved for AWS managed CMKs. + // + // AliasName is a required field + AliasName *string `min:"1" type:"string" required:"true"` + + // Identifies the CMK to which the alias refers. Specify the key ID or the Amazon + // Resource Name (ARN) of the CMK. You cannot specify another alias. For help + // finding the key ID and ARN, see Finding the Key ID and ARN (https://docs.aws.amazon.com/kms/latest/developerguide/viewing-keys.html#find-cmk-id-arn) + // in the AWS Key Management Service Developer Guide. + // + // TargetKeyId is a required field + TargetKeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s CreateAliasInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CreateAliasInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *CreateAliasInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "CreateAliasInput"} + if s.AliasName == nil { + invalidParams.Add(request.NewErrParamRequired("AliasName")) + } + if s.AliasName != nil && len(*s.AliasName) < 1 { + invalidParams.Add(request.NewErrParamMinLen("AliasName", 1)) + } + if s.TargetKeyId == nil { + invalidParams.Add(request.NewErrParamRequired("TargetKeyId")) + } + if s.TargetKeyId != nil && len(*s.TargetKeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("TargetKeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetAliasName sets the AliasName field's value. +func (s *CreateAliasInput) SetAliasName(v string) *CreateAliasInput { + s.AliasName = &v + return s +} + +// SetTargetKeyId sets the TargetKeyId field's value. +func (s *CreateAliasInput) SetTargetKeyId(v string) *CreateAliasInput { + s.TargetKeyId = &v + return s +} + +type CreateAliasOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s CreateAliasOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CreateAliasOutput) GoString() string { + return s.String() +} + +type CreateCustomKeyStoreInput struct { + _ struct{} `type:"structure"` + + // Identifies the AWS CloudHSM cluster for the custom key store. Enter the cluster + // ID of any active AWS CloudHSM cluster that is not already associated with + // a custom key store. To find the cluster ID, use the DescribeClusters (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_DescribeClusters.html) + // operation. + // + // CloudHsmClusterId is a required field + CloudHsmClusterId *string `min:"19" type:"string" required:"true"` + + // Specifies a friendly name for the custom key store. The name must be unique + // in your AWS account. + // + // CustomKeyStoreName is a required field + CustomKeyStoreName *string `min:"1" type:"string" required:"true"` + + // Enter the password of the kmsuser crypto user (CU) account (https://docs.aws.amazon.com/kms/latest/developerguide/key-store-concepts.html#concept-kmsuser) + // in the specified AWS CloudHSM cluster. AWS KMS logs into the cluster as this + // user to manage key material on your behalf. + // + // The password must be a string of 7 to 32 characters. Its value is case sensitive. + // + // This parameter tells AWS KMS the kmsuser account password; it does not change + // the password in the AWS CloudHSM cluster. + // + // KeyStorePassword is a required field + KeyStorePassword *string `min:"7" type:"string" required:"true" sensitive:"true"` + + // Enter the content of the trust anchor certificate for the cluster. This is + // the content of the customerCA.crt file that you created when you initialized + // the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/initialize-cluster.html). + // + // TrustAnchorCertificate is a required field + TrustAnchorCertificate *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s CreateCustomKeyStoreInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CreateCustomKeyStoreInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *CreateCustomKeyStoreInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "CreateCustomKeyStoreInput"} + if s.CloudHsmClusterId == nil { + invalidParams.Add(request.NewErrParamRequired("CloudHsmClusterId")) + } + if s.CloudHsmClusterId != nil && len(*s.CloudHsmClusterId) < 19 { + invalidParams.Add(request.NewErrParamMinLen("CloudHsmClusterId", 19)) + } + if s.CustomKeyStoreName == nil { + invalidParams.Add(request.NewErrParamRequired("CustomKeyStoreName")) + } + if s.CustomKeyStoreName != nil && len(*s.CustomKeyStoreName) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CustomKeyStoreName", 1)) + } + if s.KeyStorePassword == nil { + invalidParams.Add(request.NewErrParamRequired("KeyStorePassword")) + } + if s.KeyStorePassword != nil && len(*s.KeyStorePassword) < 7 { + invalidParams.Add(request.NewErrParamMinLen("KeyStorePassword", 7)) + } + if s.TrustAnchorCertificate == nil { + invalidParams.Add(request.NewErrParamRequired("TrustAnchorCertificate")) + } + if s.TrustAnchorCertificate != nil && len(*s.TrustAnchorCertificate) < 1 { + invalidParams.Add(request.NewErrParamMinLen("TrustAnchorCertificate", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetCloudHsmClusterId sets the CloudHsmClusterId field's value. +func (s *CreateCustomKeyStoreInput) SetCloudHsmClusterId(v string) *CreateCustomKeyStoreInput { + s.CloudHsmClusterId = &v + return s +} + +// SetCustomKeyStoreName sets the CustomKeyStoreName field's value. +func (s *CreateCustomKeyStoreInput) SetCustomKeyStoreName(v string) *CreateCustomKeyStoreInput { + s.CustomKeyStoreName = &v + return s +} + +// SetKeyStorePassword sets the KeyStorePassword field's value. +func (s *CreateCustomKeyStoreInput) SetKeyStorePassword(v string) *CreateCustomKeyStoreInput { + s.KeyStorePassword = &v + return s +} + +// SetTrustAnchorCertificate sets the TrustAnchorCertificate field's value. +func (s *CreateCustomKeyStoreInput) SetTrustAnchorCertificate(v string) *CreateCustomKeyStoreInput { + s.TrustAnchorCertificate = &v + return s +} + +type CreateCustomKeyStoreOutput struct { + _ struct{} `type:"structure"` + + // A unique identifier for the new custom key store. + CustomKeyStoreId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s CreateCustomKeyStoreOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CreateCustomKeyStoreOutput) GoString() string { + return s.String() +} + +// SetCustomKeyStoreId sets the CustomKeyStoreId field's value. +func (s *CreateCustomKeyStoreOutput) SetCustomKeyStoreId(v string) *CreateCustomKeyStoreOutput { + s.CustomKeyStoreId = &v + return s +} + +type CreateGrantInput struct { + _ struct{} `type:"structure"` + + // Allows a cryptographic operation (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) + // only when the encryption context matches or includes the encryption context + // specified in this structure. For more information about encryption context, + // see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) + // in the AWS Key Management Service Developer Guide . + Constraints *GrantConstraints `type:"structure"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // The principal that is given permission to perform the operations that the + // grant permits. + // + // To specify the principal, use the Amazon Resource Name (ARN) (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) + // of an AWS principal. Valid AWS principals include AWS accounts (root), IAM + // users, IAM roles, federated users, and assumed role users. For examples of + // the ARN syntax to use for specifying a principal, see AWS Identity and Access + // Management (IAM) (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-iam) + // in the Example ARNs section of the AWS General Reference. + // + // GranteePrincipal is a required field + GranteePrincipal *string `min:"1" type:"string" required:"true"` + + // The unique identifier for the customer master key (CMK) that the grant applies + // to. + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. To specify + // a CMK in a different AWS account, you must use the key ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // A friendly name for identifying the grant. Use this value to prevent the + // unintended creation of duplicate grants when retrying this request. + // + // When this value is absent, all CreateGrant requests result in a new grant + // with a unique GrantId even if all the supplied parameters are identical. + // This can result in unintended duplicates when you retry the CreateGrant request. + // + // When this value is present, you can retry a CreateGrant request with identical + // parameters; if the grant already exists, the original GrantId is returned + // without creating a new grant. Note that the returned grant token is unique + // with every CreateGrant request, even when a duplicate GrantId is returned. + // All grant tokens obtained in this way can be used interchangeably. + Name *string `min:"1" type:"string"` + + // A list of operations that the grant permits. + // + // Operations is a required field + Operations []*string `type:"list" required:"true"` + + // The principal that is given permission to retire the grant by using RetireGrant + // operation. + // + // To specify the principal, use the Amazon Resource Name (ARN) (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) + // of an AWS principal. Valid AWS principals include AWS accounts (root), IAM + // users, federated users, and assumed role users. For examples of the ARN syntax + // to use for specifying a principal, see AWS Identity and Access Management + // (IAM) (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-iam) + // in the Example ARNs section of the AWS General Reference. + RetiringPrincipal *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s CreateGrantInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CreateGrantInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *CreateGrantInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "CreateGrantInput"} + if s.GranteePrincipal == nil { + invalidParams.Add(request.NewErrParamRequired("GranteePrincipal")) + } + if s.GranteePrincipal != nil && len(*s.GranteePrincipal) < 1 { + invalidParams.Add(request.NewErrParamMinLen("GranteePrincipal", 1)) + } + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.Name != nil && len(*s.Name) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Name", 1)) + } + if s.Operations == nil { + invalidParams.Add(request.NewErrParamRequired("Operations")) + } + if s.RetiringPrincipal != nil && len(*s.RetiringPrincipal) < 1 { + invalidParams.Add(request.NewErrParamMinLen("RetiringPrincipal", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetConstraints sets the Constraints field's value. +func (s *CreateGrantInput) SetConstraints(v *GrantConstraints) *CreateGrantInput { + s.Constraints = v + return s +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *CreateGrantInput) SetGrantTokens(v []*string) *CreateGrantInput { + s.GrantTokens = v + return s +} + +// SetGranteePrincipal sets the GranteePrincipal field's value. +func (s *CreateGrantInput) SetGranteePrincipal(v string) *CreateGrantInput { + s.GranteePrincipal = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *CreateGrantInput) SetKeyId(v string) *CreateGrantInput { + s.KeyId = &v + return s +} + +// SetName sets the Name field's value. +func (s *CreateGrantInput) SetName(v string) *CreateGrantInput { + s.Name = &v + return s +} + +// SetOperations sets the Operations field's value. +func (s *CreateGrantInput) SetOperations(v []*string) *CreateGrantInput { + s.Operations = v + return s +} + +// SetRetiringPrincipal sets the RetiringPrincipal field's value. +func (s *CreateGrantInput) SetRetiringPrincipal(v string) *CreateGrantInput { + s.RetiringPrincipal = &v + return s +} + +type CreateGrantOutput struct { + _ struct{} `type:"structure"` + + // The unique identifier for the grant. + // + // You can use the GrantId in a subsequent RetireGrant or RevokeGrant operation. + GrantId *string `min:"1" type:"string"` + + // The grant token. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantToken *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s CreateGrantOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CreateGrantOutput) GoString() string { + return s.String() +} + +// SetGrantId sets the GrantId field's value. +func (s *CreateGrantOutput) SetGrantId(v string) *CreateGrantOutput { + s.GrantId = &v + return s +} + +// SetGrantToken sets the GrantToken field's value. +func (s *CreateGrantOutput) SetGrantToken(v string) *CreateGrantOutput { + s.GrantToken = &v + return s +} + +type CreateKeyInput struct { + _ struct{} `type:"structure"` + + // A flag to indicate whether to bypass the key policy lockout safety check. + // + // Setting this value to true increases the risk that the CMK becomes unmanageable. + // Do not set this value to true indiscriminately. + // + // For more information, refer to the scenario in the Default Key Policy (https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html#key-policy-default-allow-root-enable-iam) + // section in the AWS Key Management Service Developer Guide . + // + // Use this parameter only when you include a policy in the request and you + // intend to prevent the principal that is making the request from making a + // subsequent PutKeyPolicy request on the CMK. + // + // The default value is false. + BypassPolicyLockoutSafetyCheck *bool `type:"boolean"` + + // Creates the CMK in the specified custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) + // and the key material in its associated AWS CloudHSM cluster. To create a + // CMK in a custom key store, you must also specify the Origin parameter with + // a value of AWS_CLOUDHSM. The AWS CloudHSM cluster that is associated with + // the custom key store must have at least two active HSMs, each in a different + // Availability Zone in the Region. + // + // This parameter is valid only for symmetric CMKs. You cannot create an asymmetric + // CMK in a custom key store. + // + // To find the ID of a custom key store, use the DescribeCustomKeyStores operation. + // + // The response includes the custom key store ID and the ID of the AWS CloudHSM + // cluster. + // + // This operation is part of the Custom Key Store feature (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) + // feature in AWS KMS, which combines the convenience and extensive integration + // of AWS KMS with the isolation and control of a single-tenant key store. + CustomKeyStoreId *string `min:"1" type:"string"` + + // Specifies the type of CMK to create. The default value, SYMMETRIC_DEFAULT, + // creates a CMK with a 256-bit symmetric key for encryption and decryption. + // For help choosing a key spec for your CMK, see How to Choose Your CMK Configuration + // (https://docs.aws.amazon.com/kms/latest/developerguide/symm-asymm-choose.html) + // in the AWS Key Management Service Developer Guide. + // + // The CustomerMasterKeySpec determines whether the CMK contains a symmetric + // key or an asymmetric key pair. It also determines the encryption algorithms + // or signing algorithms that the CMK supports. You can't change the CustomerMasterKeySpec + // after the CMK is created. To further restrict the algorithms that can be + // used with the CMK, use a condition key in its key policy or IAM policy. For + // more information, see kms:EncryptionAlgorithm (https://docs.aws.amazon.com/kms/latest/developerguide/policy-conditions.html#conditions-kms-encryption-algorithm) + // or kms:Signing Algorithm (https://docs.aws.amazon.com/kms/latest/developerguide/policy-conditions.html#conditions-kms-signing-algorithm) + // in the AWS Key Management Service Developer Guide. + // + // AWS services that are integrated with AWS KMS (http://aws.amazon.com/kms/features/#AWS_Service_Integration) + // use symmetric CMKs to protect your data. These services do not support asymmetric + // CMKs. For help determining whether a CMK is symmetric or asymmetric, see + // Identifying Symmetric and Asymmetric CMKs (https://docs.aws.amazon.com/kms/latest/developerguide/find-symm-asymm.html) + // in the AWS Key Management Service Developer Guide. + // + // AWS KMS supports the following key specs for CMKs: + // + // * Symmetric key (default) SYMMETRIC_DEFAULT (AES-256-GCM) + // + // * Asymmetric RSA key pairs RSA_2048 RSA_3072 RSA_4096 + // + // * Asymmetric NIST-recommended elliptic curve key pairs ECC_NIST_P256 (secp256r1) + // ECC_NIST_P384 (secp384r1) ECC_NIST_P521 (secp521r1) + // + // * Other asymmetric elliptic curve key pairs ECC_SECG_P256K1 (secp256k1), + // commonly used for cryptocurrencies. + CustomerMasterKeySpec *string `type:"string" enum:"CustomerMasterKeySpec"` + + // A description of the CMK. + // + // Use a description that helps you decide whether the CMK is appropriate for + // a task. + Description *string `type:"string"` + + // Determines the cryptographic operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) + // for which you can use the CMK. The default value is ENCRYPT_DECRYPT. This + // parameter is required only for asymmetric CMKs. You can't change the KeyUsage + // value after the CMK is created. + // + // Select only one valid value. + // + // * For symmetric CMKs, omit the parameter or specify ENCRYPT_DECRYPT. + // + // * For asymmetric CMKs with RSA key material, specify ENCRYPT_DECRYPT or + // SIGN_VERIFY. + // + // * For asymmetric CMKs with ECC key material, specify SIGN_VERIFY. + KeyUsage *string `type:"string" enum:"KeyUsageType"` + + // The source of the key material for the CMK. You cannot change the origin + // after you create the CMK. The default is AWS_KMS, which means AWS KMS creates + // the key material. + // + // When the parameter value is EXTERNAL, AWS KMS creates a CMK without key material + // so that you can import key material from your existing key management infrastructure. + // For more information about importing key material into AWS KMS, see Importing + // Key Material (https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys.html) + // in the AWS Key Management Service Developer Guide. This value is valid only + // for symmetric CMKs. + // + // When the parameter value is AWS_CLOUDHSM, AWS KMS creates the CMK in an AWS + // KMS custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) + // and creates its key material in the associated AWS CloudHSM cluster. You + // must also use the CustomKeyStoreId parameter to identify the custom key store. + // This value is valid only for symmetric CMKs. + Origin *string `type:"string" enum:"OriginType"` + + // The key policy to attach to the CMK. + // + // If you provide a key policy, it must meet the following criteria: + // + // * If you don't set BypassPolicyLockoutSafetyCheck to true, the key policy + // must allow the principal that is making the CreateKey request to make + // a subsequent PutKeyPolicy request on the CMK. This reduces the risk that + // the CMK becomes unmanageable. For more information, refer to the scenario + // in the Default Key Policy (https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html#key-policy-default-allow-root-enable-iam) + // section of the AWS Key Management Service Developer Guide . + // + // * Each statement in the key policy must contain one or more principals. + // The principals in the key policy must exist and be visible to AWS KMS. + // When you create a new AWS principal (for example, an IAM user or role), + // you might need to enforce a delay before including the new principal in + // a key policy because the new principal might not be immediately visible + // to AWS KMS. For more information, see Changes that I make are not always + // immediately visible (https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_general.html#troubleshoot_general_eventual-consistency) + // in the AWS Identity and Access Management User Guide. + // + // If you do not provide a key policy, AWS KMS attaches a default key policy + // to the CMK. For more information, see Default Key Policy (https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html#key-policy-default) + // in the AWS Key Management Service Developer Guide. + // + // The key policy size quota is 32 kilobytes (32768 bytes). + Policy *string `min:"1" type:"string"` + + // One or more tags. Each tag consists of a tag key and a tag value. Both the + // tag key and the tag value are required, but the tag value can be an empty + // (null) string. + // + // When you add tags to an AWS resource, AWS generates a cost allocation report + // with usage and costs aggregated by tags. For information about adding, changing, + // deleting and listing tags for CMKs, see Tagging Keys (https://docs.aws.amazon.com/kms/latest/developerguide/tagging-keys.html). + // + // Use this parameter to tag the CMK when it is created. To add tags to an existing + // CMK, use the TagResource operation. + Tags []*Tag `type:"list"` +} + +// String returns the string representation +func (s CreateKeyInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CreateKeyInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *CreateKeyInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "CreateKeyInput"} + if s.CustomKeyStoreId != nil && len(*s.CustomKeyStoreId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CustomKeyStoreId", 1)) + } + if s.Policy != nil && len(*s.Policy) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Policy", 1)) + } + if s.Tags != nil { + for i, v := range s.Tags { + if v == nil { + continue + } + if err := v.Validate(); err != nil { + invalidParams.AddNested(fmt.Sprintf("%s[%v]", "Tags", i), err.(request.ErrInvalidParams)) + } + } + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetBypassPolicyLockoutSafetyCheck sets the BypassPolicyLockoutSafetyCheck field's value. +func (s *CreateKeyInput) SetBypassPolicyLockoutSafetyCheck(v bool) *CreateKeyInput { + s.BypassPolicyLockoutSafetyCheck = &v + return s +} + +// SetCustomKeyStoreId sets the CustomKeyStoreId field's value. +func (s *CreateKeyInput) SetCustomKeyStoreId(v string) *CreateKeyInput { + s.CustomKeyStoreId = &v + return s +} + +// SetCustomerMasterKeySpec sets the CustomerMasterKeySpec field's value. +func (s *CreateKeyInput) SetCustomerMasterKeySpec(v string) *CreateKeyInput { + s.CustomerMasterKeySpec = &v + return s +} + +// SetDescription sets the Description field's value. +func (s *CreateKeyInput) SetDescription(v string) *CreateKeyInput { + s.Description = &v + return s +} + +// SetKeyUsage sets the KeyUsage field's value. +func (s *CreateKeyInput) SetKeyUsage(v string) *CreateKeyInput { + s.KeyUsage = &v + return s +} + +// SetOrigin sets the Origin field's value. +func (s *CreateKeyInput) SetOrigin(v string) *CreateKeyInput { + s.Origin = &v + return s +} + +// SetPolicy sets the Policy field's value. +func (s *CreateKeyInput) SetPolicy(v string) *CreateKeyInput { + s.Policy = &v + return s +} + +// SetTags sets the Tags field's value. +func (s *CreateKeyInput) SetTags(v []*Tag) *CreateKeyInput { + s.Tags = v + return s +} + +type CreateKeyOutput struct { + _ struct{} `type:"structure"` + + // Metadata associated with the CMK. + KeyMetadata *KeyMetadata `type:"structure"` +} + +// String returns the string representation +func (s CreateKeyOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CreateKeyOutput) GoString() string { + return s.String() +} + +// SetKeyMetadata sets the KeyMetadata field's value. +func (s *CreateKeyOutput) SetKeyMetadata(v *KeyMetadata) *CreateKeyOutput { + s.KeyMetadata = v + return s +} + +// The request was rejected because the custom key store contains AWS KMS customer +// master keys (CMKs). After verifying that you do not need to use the CMKs, +// use the ScheduleKeyDeletion operation to delete the CMKs. After they are +// deleted, you can delete the custom key store. +type CustomKeyStoreHasCMKsException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s CustomKeyStoreHasCMKsException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CustomKeyStoreHasCMKsException) GoString() string { + return s.String() +} + +func newErrorCustomKeyStoreHasCMKsException(v protocol.ResponseMetadata) error { + return &CustomKeyStoreHasCMKsException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *CustomKeyStoreHasCMKsException) Code() string { + return "CustomKeyStoreHasCMKsException" +} + +// Message returns the exception's message. +func (s *CustomKeyStoreHasCMKsException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *CustomKeyStoreHasCMKsException) OrigErr() error { + return nil +} + +func (s *CustomKeyStoreHasCMKsException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *CustomKeyStoreHasCMKsException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *CustomKeyStoreHasCMKsException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because of the ConnectionState of the custom key +// store. To get the ConnectionState of a custom key store, use the DescribeCustomKeyStores +// operation. +// +// This exception is thrown under the following conditions: +// +// * You requested the CreateKey or GenerateRandom operation in a custom +// key store that is not connected. These operations are valid only when +// the custom key store ConnectionState is CONNECTED. +// +// * You requested the UpdateCustomKeyStore or DeleteCustomKeyStore operation +// on a custom key store that is not disconnected. This operation is valid +// only when the custom key store ConnectionState is DISCONNECTED. +// +// * You requested the ConnectCustomKeyStore operation on a custom key store +// with a ConnectionState of DISCONNECTING or FAILED. This operation is valid +// for all other ConnectionState values. +type CustomKeyStoreInvalidStateException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s CustomKeyStoreInvalidStateException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CustomKeyStoreInvalidStateException) GoString() string { + return s.String() +} + +func newErrorCustomKeyStoreInvalidStateException(v protocol.ResponseMetadata) error { + return &CustomKeyStoreInvalidStateException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *CustomKeyStoreInvalidStateException) Code() string { + return "CustomKeyStoreInvalidStateException" +} + +// Message returns the exception's message. +func (s *CustomKeyStoreInvalidStateException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *CustomKeyStoreInvalidStateException) OrigErr() error { + return nil +} + +func (s *CustomKeyStoreInvalidStateException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *CustomKeyStoreInvalidStateException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *CustomKeyStoreInvalidStateException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the specified custom key store name is already +// assigned to another custom key store in the account. Try again with a custom +// key store name that is unique in the account. +type CustomKeyStoreNameInUseException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s CustomKeyStoreNameInUseException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CustomKeyStoreNameInUseException) GoString() string { + return s.String() +} + +func newErrorCustomKeyStoreNameInUseException(v protocol.ResponseMetadata) error { + return &CustomKeyStoreNameInUseException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *CustomKeyStoreNameInUseException) Code() string { + return "CustomKeyStoreNameInUseException" +} + +// Message returns the exception's message. +func (s *CustomKeyStoreNameInUseException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *CustomKeyStoreNameInUseException) OrigErr() error { + return nil +} + +func (s *CustomKeyStoreNameInUseException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *CustomKeyStoreNameInUseException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *CustomKeyStoreNameInUseException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because AWS KMS cannot find a custom key store with +// the specified key store name or ID. +type CustomKeyStoreNotFoundException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s CustomKeyStoreNotFoundException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CustomKeyStoreNotFoundException) GoString() string { + return s.String() +} + +func newErrorCustomKeyStoreNotFoundException(v protocol.ResponseMetadata) error { + return &CustomKeyStoreNotFoundException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *CustomKeyStoreNotFoundException) Code() string { + return "CustomKeyStoreNotFoundException" +} + +// Message returns the exception's message. +func (s *CustomKeyStoreNotFoundException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *CustomKeyStoreNotFoundException) OrigErr() error { + return nil +} + +func (s *CustomKeyStoreNotFoundException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *CustomKeyStoreNotFoundException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *CustomKeyStoreNotFoundException) RequestID() string { + return s.RespMetadata.RequestID +} + +// Contains information about each custom key store in the custom key store +// list. +type CustomKeyStoresListEntry struct { + _ struct{} `type:"structure"` + + // A unique identifier for the AWS CloudHSM cluster that is associated with + // the custom key store. + CloudHsmClusterId *string `min:"19" type:"string"` + + // Describes the connection error. This field appears in the response only when + // the ConnectionState is FAILED. For help resolving these errors, see How to + // Fix a Connection Failure (https://docs.aws.amazon.com/kms/latest/developerguide/fix-keystore.html#fix-keystore-failed) + // in AWS Key Management Service Developer Guide. + // + // Valid values are: + // + // * CLUSTER_NOT_FOUND - AWS KMS cannot find the AWS CloudHSM cluster with + // the specified cluster ID. + // + // * INSUFFICIENT_CLOUDHSM_HSMS - The associated AWS CloudHSM cluster does + // not contain any active HSMs. To connect a custom key store to its AWS + // CloudHSM cluster, the cluster must contain at least one active HSM. + // + // * INTERNAL_ERROR - AWS KMS could not complete the request due to an internal + // error. Retry the request. For ConnectCustomKeyStore requests, disconnect + // the custom key store before trying to connect again. + // + // * INVALID_CREDENTIALS - AWS KMS does not have the correct password for + // the kmsuser crypto user in the AWS CloudHSM cluster. Before you can connect + // your custom key store to its AWS CloudHSM cluster, you must change the + // kmsuser account password and update the key store password value for the + // custom key store. + // + // * NETWORK_ERRORS - Network errors are preventing AWS KMS from connecting + // to the custom key store. + // + // * SUBNET_NOT_FOUND - A subnet in the AWS CloudHSM cluster configuration + // was deleted. If AWS KMS cannot find all of the subnets in the cluster + // configuration, attempts to connect the custom key store to the AWS CloudHSM + // cluster fail. To fix this error, create a cluster from a recent backup + // and associate it with your custom key store. (This process creates a new + // cluster configuration with a VPC and private subnets.) For details, see + // How to Fix a Connection Failure (https://docs.aws.amazon.com/kms/latest/developerguide/fix-keystore.html#fix-keystore-failed) + // in the AWS Key Management Service Developer Guide. + // + // * USER_LOCKED_OUT - The kmsuser CU account is locked out of the associated + // AWS CloudHSM cluster due to too many failed password attempts. Before + // you can connect your custom key store to its AWS CloudHSM cluster, you + // must change the kmsuser account password and update the key store password + // value for the custom key store. + // + // * USER_LOGGED_IN - The kmsuser CU account is logged into the the associated + // AWS CloudHSM cluster. This prevents AWS KMS from rotating the kmsuser + // account password and logging into the cluster. Before you can connect + // your custom key store to its AWS CloudHSM cluster, you must log the kmsuser + // CU out of the cluster. If you changed the kmsuser password to log into + // the cluster, you must also and update the key store password value for + // the custom key store. For help, see How to Log Out and Reconnect (https://docs.aws.amazon.com/kms/latest/developerguide/fix-keystore.html#login-kmsuser-2) + // in the AWS Key Management Service Developer Guide. + // + // * USER_NOT_FOUND - AWS KMS cannot find a kmsuser CU account in the associated + // AWS CloudHSM cluster. Before you can connect your custom key store to + // its AWS CloudHSM cluster, you must create a kmsuser CU account in the + // cluster, and then update the key store password value for the custom key + // store. + ConnectionErrorCode *string `type:"string" enum:"ConnectionErrorCodeType"` + + // Indicates whether the custom key store is connected to its AWS CloudHSM cluster. + // + // You can create and use CMKs in your custom key stores only when its connection + // state is CONNECTED. + // + // The value is DISCONNECTED if the key store has never been connected or you + // use the DisconnectCustomKeyStore operation to disconnect it. If the value + // is CONNECTED but you are having trouble using the custom key store, make + // sure that its associated AWS CloudHSM cluster is active and contains at least + // one active HSM. + // + // A value of FAILED indicates that an attempt to connect was unsuccessful. + // The ConnectionErrorCode field in the response indicates the cause of the + // failure. For help resolving a connection failure, see Troubleshooting a Custom + // Key Store (https://docs.aws.amazon.com/kms/latest/developerguide/fix-keystore.html) + // in the AWS Key Management Service Developer Guide. + ConnectionState *string `type:"string" enum:"ConnectionStateType"` + + // The date and time when the custom key store was created. + CreationDate *time.Time `type:"timestamp"` + + // A unique identifier for the custom key store. + CustomKeyStoreId *string `min:"1" type:"string"` + + // The user-specified friendly name for the custom key store. + CustomKeyStoreName *string `min:"1" type:"string"` + + // The trust anchor certificate of the associated AWS CloudHSM cluster. When + // you initialize the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/initialize-cluster.html#sign-csr), + // you create this certificate and save it in the customerCA.crt file. + TrustAnchorCertificate *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s CustomKeyStoresListEntry) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s CustomKeyStoresListEntry) GoString() string { + return s.String() +} + +// SetCloudHsmClusterId sets the CloudHsmClusterId field's value. +func (s *CustomKeyStoresListEntry) SetCloudHsmClusterId(v string) *CustomKeyStoresListEntry { + s.CloudHsmClusterId = &v + return s +} + +// SetConnectionErrorCode sets the ConnectionErrorCode field's value. +func (s *CustomKeyStoresListEntry) SetConnectionErrorCode(v string) *CustomKeyStoresListEntry { + s.ConnectionErrorCode = &v + return s +} + +// SetConnectionState sets the ConnectionState field's value. +func (s *CustomKeyStoresListEntry) SetConnectionState(v string) *CustomKeyStoresListEntry { + s.ConnectionState = &v + return s +} + +// SetCreationDate sets the CreationDate field's value. +func (s *CustomKeyStoresListEntry) SetCreationDate(v time.Time) *CustomKeyStoresListEntry { + s.CreationDate = &v + return s +} + +// SetCustomKeyStoreId sets the CustomKeyStoreId field's value. +func (s *CustomKeyStoresListEntry) SetCustomKeyStoreId(v string) *CustomKeyStoresListEntry { + s.CustomKeyStoreId = &v + return s +} + +// SetCustomKeyStoreName sets the CustomKeyStoreName field's value. +func (s *CustomKeyStoresListEntry) SetCustomKeyStoreName(v string) *CustomKeyStoresListEntry { + s.CustomKeyStoreName = &v + return s +} + +// SetTrustAnchorCertificate sets the TrustAnchorCertificate field's value. +func (s *CustomKeyStoresListEntry) SetTrustAnchorCertificate(v string) *CustomKeyStoresListEntry { + s.TrustAnchorCertificate = &v + return s +} + +type DecryptInput struct { + _ struct{} `type:"structure"` + + // Ciphertext to be decrypted. The blob includes metadata. + // + // CiphertextBlob is automatically base64 encoded/decoded by the SDK. + // + // CiphertextBlob is a required field + CiphertextBlob []byte `min:"1" type:"blob" required:"true"` + + // Specifies the encryption algorithm that will be used to decrypt the ciphertext. + // Specify the same algorithm that was used to encrypt the data. If you specify + // a different algorithm, the Decrypt operation fails. + // + // This parameter is required only when the ciphertext was encrypted under an + // asymmetric CMK. The default value, SYMMETRIC_DEFAULT, represents the only + // supported algorithm that is valid for symmetric CMKs. + EncryptionAlgorithm *string `type:"string" enum:"EncryptionAlgorithmSpec"` + + // Specifies the encryption context to use when decrypting the data. An encryption + // context is valid only for cryptographic operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) + // with a symmetric CMK. The standard asymmetric encryption algorithms that + // AWS KMS uses do not support an encryption context. + // + // An encryption context is a collection of non-secret key-value pairs that + // represents additional authenticated data. When you use an encryption context + // to encrypt data, you must specify the same (an exact case-sensitive match) + // encryption context to decrypt the data. An encryption context is optional + // when encrypting with a symmetric CMK, but it is highly recommended. + // + // For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) + // in the AWS Key Management Service Developer Guide. + EncryptionContext map[string]*string `type:"map"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // Specifies the customer master key (CMK) that AWS KMS will use to decrypt + // the ciphertext. Enter a key ID of the CMK that was used to encrypt the ciphertext. + // + // If you specify a KeyId value, the Decrypt operation succeeds only if the + // specified CMK was used to encrypt the ciphertext. + // + // This parameter is required only when the ciphertext was encrypted under an + // asymmetric CMK. Otherwise, AWS KMS uses the metadata that it adds to the + // ciphertext blob to determine which CMK was used to encrypt the ciphertext. + // However, you can use this parameter to ensure that a particular CMK (of any + // kind) is used to decrypt the ciphertext. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + KeyId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s DecryptInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DecryptInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *DecryptInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "DecryptInput"} + if s.CiphertextBlob == nil { + invalidParams.Add(request.NewErrParamRequired("CiphertextBlob")) + } + if s.CiphertextBlob != nil && len(s.CiphertextBlob) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CiphertextBlob", 1)) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetCiphertextBlob sets the CiphertextBlob field's value. +func (s *DecryptInput) SetCiphertextBlob(v []byte) *DecryptInput { + s.CiphertextBlob = v + return s +} + +// SetEncryptionAlgorithm sets the EncryptionAlgorithm field's value. +func (s *DecryptInput) SetEncryptionAlgorithm(v string) *DecryptInput { + s.EncryptionAlgorithm = &v + return s +} + +// SetEncryptionContext sets the EncryptionContext field's value. +func (s *DecryptInput) SetEncryptionContext(v map[string]*string) *DecryptInput { + s.EncryptionContext = v + return s +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *DecryptInput) SetGrantTokens(v []*string) *DecryptInput { + s.GrantTokens = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *DecryptInput) SetKeyId(v string) *DecryptInput { + s.KeyId = &v + return s +} + +type DecryptOutput struct { + _ struct{} `type:"structure"` + + // The encryption algorithm that was used to decrypt the ciphertext. + EncryptionAlgorithm *string `type:"string" enum:"EncryptionAlgorithmSpec"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the CMK that was used to decrypt the ciphertext. + KeyId *string `min:"1" type:"string"` + + // Decrypted plaintext data. When you use the HTTP API or the AWS CLI, the value + // is Base64-encoded. Otherwise, it is not Base64-encoded. + // + // Plaintext is automatically base64 encoded/decoded by the SDK. + Plaintext []byte `min:"1" type:"blob" sensitive:"true"` +} + +// String returns the string representation +func (s DecryptOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DecryptOutput) GoString() string { + return s.String() +} + +// SetEncryptionAlgorithm sets the EncryptionAlgorithm field's value. +func (s *DecryptOutput) SetEncryptionAlgorithm(v string) *DecryptOutput { + s.EncryptionAlgorithm = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *DecryptOutput) SetKeyId(v string) *DecryptOutput { + s.KeyId = &v + return s +} + +// SetPlaintext sets the Plaintext field's value. +func (s *DecryptOutput) SetPlaintext(v []byte) *DecryptOutput { + s.Plaintext = v + return s +} + +type DeleteAliasInput struct { + _ struct{} `type:"structure"` + + // The alias to be deleted. The alias name must begin with alias/ followed by + // the alias name, such as alias/ExampleAlias. + // + // AliasName is a required field + AliasName *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s DeleteAliasInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DeleteAliasInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *DeleteAliasInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "DeleteAliasInput"} + if s.AliasName == nil { + invalidParams.Add(request.NewErrParamRequired("AliasName")) + } + if s.AliasName != nil && len(*s.AliasName) < 1 { + invalidParams.Add(request.NewErrParamMinLen("AliasName", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetAliasName sets the AliasName field's value. +func (s *DeleteAliasInput) SetAliasName(v string) *DeleteAliasInput { + s.AliasName = &v + return s +} + +type DeleteAliasOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s DeleteAliasOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DeleteAliasOutput) GoString() string { + return s.String() +} + +type DeleteCustomKeyStoreInput struct { + _ struct{} `type:"structure"` + + // Enter the ID of the custom key store you want to delete. To find the ID of + // a custom key store, use the DescribeCustomKeyStores operation. + // + // CustomKeyStoreId is a required field + CustomKeyStoreId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s DeleteCustomKeyStoreInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DeleteCustomKeyStoreInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *DeleteCustomKeyStoreInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "DeleteCustomKeyStoreInput"} + if s.CustomKeyStoreId == nil { + invalidParams.Add(request.NewErrParamRequired("CustomKeyStoreId")) + } + if s.CustomKeyStoreId != nil && len(*s.CustomKeyStoreId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CustomKeyStoreId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetCustomKeyStoreId sets the CustomKeyStoreId field's value. +func (s *DeleteCustomKeyStoreInput) SetCustomKeyStoreId(v string) *DeleteCustomKeyStoreInput { + s.CustomKeyStoreId = &v + return s +} + +type DeleteCustomKeyStoreOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s DeleteCustomKeyStoreOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DeleteCustomKeyStoreOutput) GoString() string { + return s.String() +} + +type DeleteImportedKeyMaterialInput struct { + _ struct{} `type:"structure"` + + // Identifies the CMK from which you are deleting imported key material. The + // Origin of the CMK must be EXTERNAL. + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s DeleteImportedKeyMaterialInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DeleteImportedKeyMaterialInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *DeleteImportedKeyMaterialInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "DeleteImportedKeyMaterialInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *DeleteImportedKeyMaterialInput) SetKeyId(v string) *DeleteImportedKeyMaterialInput { + s.KeyId = &v + return s +} + +type DeleteImportedKeyMaterialOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s DeleteImportedKeyMaterialOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DeleteImportedKeyMaterialOutput) GoString() string { + return s.String() +} + +// The system timed out while trying to fulfill the request. The request can +// be retried. +type DependencyTimeoutException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s DependencyTimeoutException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DependencyTimeoutException) GoString() string { + return s.String() +} + +func newErrorDependencyTimeoutException(v protocol.ResponseMetadata) error { + return &DependencyTimeoutException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *DependencyTimeoutException) Code() string { + return "DependencyTimeoutException" +} + +// Message returns the exception's message. +func (s *DependencyTimeoutException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *DependencyTimeoutException) OrigErr() error { + return nil +} + +func (s *DependencyTimeoutException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *DependencyTimeoutException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *DependencyTimeoutException) RequestID() string { + return s.RespMetadata.RequestID +} + +type DescribeCustomKeyStoresInput struct { + _ struct{} `type:"structure"` + + // Gets only information about the specified custom key store. Enter the key + // store ID. + // + // By default, this operation gets information about all custom key stores in + // the account and region. To limit the output to a particular custom key store, + // you can use either the CustomKeyStoreId or CustomKeyStoreName parameter, + // but not both. + CustomKeyStoreId *string `min:"1" type:"string"` + + // Gets only information about the specified custom key store. Enter the friendly + // name of the custom key store. + // + // By default, this operation gets information about all custom key stores in + // the account and region. To limit the output to a particular custom key store, + // you can use either the CustomKeyStoreId or CustomKeyStoreName parameter, + // but not both. + CustomKeyStoreName *string `min:"1" type:"string"` + + // Use this parameter to specify the maximum number of items to return. When + // this value is present, AWS KMS does not return more than the specified number + // of items, but it might return fewer. + Limit *int64 `min:"1" type:"integer"` + + // Use this parameter in a subsequent request after you receive a response with + // truncated results. Set it to the value of NextMarker from the truncated response + // you just received. + Marker *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s DescribeCustomKeyStoresInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DescribeCustomKeyStoresInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *DescribeCustomKeyStoresInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "DescribeCustomKeyStoresInput"} + if s.CustomKeyStoreId != nil && len(*s.CustomKeyStoreId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CustomKeyStoreId", 1)) + } + if s.CustomKeyStoreName != nil && len(*s.CustomKeyStoreName) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CustomKeyStoreName", 1)) + } + if s.Limit != nil && *s.Limit < 1 { + invalidParams.Add(request.NewErrParamMinValue("Limit", 1)) + } + if s.Marker != nil && len(*s.Marker) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Marker", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetCustomKeyStoreId sets the CustomKeyStoreId field's value. +func (s *DescribeCustomKeyStoresInput) SetCustomKeyStoreId(v string) *DescribeCustomKeyStoresInput { + s.CustomKeyStoreId = &v + return s +} + +// SetCustomKeyStoreName sets the CustomKeyStoreName field's value. +func (s *DescribeCustomKeyStoresInput) SetCustomKeyStoreName(v string) *DescribeCustomKeyStoresInput { + s.CustomKeyStoreName = &v + return s +} + +// SetLimit sets the Limit field's value. +func (s *DescribeCustomKeyStoresInput) SetLimit(v int64) *DescribeCustomKeyStoresInput { + s.Limit = &v + return s +} + +// SetMarker sets the Marker field's value. +func (s *DescribeCustomKeyStoresInput) SetMarker(v string) *DescribeCustomKeyStoresInput { + s.Marker = &v + return s +} + +type DescribeCustomKeyStoresOutput struct { + _ struct{} `type:"structure"` + + // Contains metadata about each custom key store. + CustomKeyStores []*CustomKeyStoresListEntry `type:"list"` + + // When Truncated is true, this element is present and contains the value to + // use for the Marker parameter in a subsequent request. + NextMarker *string `min:"1" type:"string"` + + // A flag that indicates whether there are more items in the list. When this + // value is true, the list in this response is truncated. To get more items, + // pass the value of the NextMarker element in thisresponse to the Marker parameter + // in a subsequent request. + Truncated *bool `type:"boolean"` +} + +// String returns the string representation +func (s DescribeCustomKeyStoresOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DescribeCustomKeyStoresOutput) GoString() string { + return s.String() +} + +// SetCustomKeyStores sets the CustomKeyStores field's value. +func (s *DescribeCustomKeyStoresOutput) SetCustomKeyStores(v []*CustomKeyStoresListEntry) *DescribeCustomKeyStoresOutput { + s.CustomKeyStores = v + return s +} + +// SetNextMarker sets the NextMarker field's value. +func (s *DescribeCustomKeyStoresOutput) SetNextMarker(v string) *DescribeCustomKeyStoresOutput { + s.NextMarker = &v + return s +} + +// SetTruncated sets the Truncated field's value. +func (s *DescribeCustomKeyStoresOutput) SetTruncated(v bool) *DescribeCustomKeyStoresOutput { + s.Truncated = &v + return s +} + +type DescribeKeyInput struct { + _ struct{} `type:"structure"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // Describes the specified customer master key (CMK). + // + // If you specify a predefined AWS alias (an AWS alias with no key ID), KMS + // associates the alias with an AWS managed CMK (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#master_keys) + // and returns its KeyId and Arn in the response. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". To specify + // a CMK in a different AWS account, you must use the key ARN or alias ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s DescribeKeyInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DescribeKeyInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *DescribeKeyInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "DescribeKeyInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *DescribeKeyInput) SetGrantTokens(v []*string) *DescribeKeyInput { + s.GrantTokens = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *DescribeKeyInput) SetKeyId(v string) *DescribeKeyInput { + s.KeyId = &v + return s +} + +type DescribeKeyOutput struct { + _ struct{} `type:"structure"` + + // Metadata associated with the key. + KeyMetadata *KeyMetadata `type:"structure"` +} + +// String returns the string representation +func (s DescribeKeyOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DescribeKeyOutput) GoString() string { + return s.String() +} + +// SetKeyMetadata sets the KeyMetadata field's value. +func (s *DescribeKeyOutput) SetKeyMetadata(v *KeyMetadata) *DescribeKeyOutput { + s.KeyMetadata = v + return s +} + +type DisableKeyInput struct { + _ struct{} `type:"structure"` + + // A unique identifier for the customer master key (CMK). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s DisableKeyInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DisableKeyInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *DisableKeyInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "DisableKeyInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *DisableKeyInput) SetKeyId(v string) *DisableKeyInput { + s.KeyId = &v + return s +} + +type DisableKeyOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s DisableKeyOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DisableKeyOutput) GoString() string { + return s.String() +} + +type DisableKeyRotationInput struct { + _ struct{} `type:"structure"` + + // Identifies a symmetric customer master key (CMK). You cannot enable automatic + // rotation of asymmetric CMKs (https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html#asymmetric-cmks), + // CMKs with imported key material (https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys.html), + // or CMKs in a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s DisableKeyRotationInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DisableKeyRotationInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *DisableKeyRotationInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "DisableKeyRotationInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *DisableKeyRotationInput) SetKeyId(v string) *DisableKeyRotationInput { + s.KeyId = &v + return s +} + +type DisableKeyRotationOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s DisableKeyRotationOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DisableKeyRotationOutput) GoString() string { + return s.String() +} + +// The request was rejected because the specified CMK is not enabled. +type DisabledException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s DisabledException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DisabledException) GoString() string { + return s.String() +} + +func newErrorDisabledException(v protocol.ResponseMetadata) error { + return &DisabledException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *DisabledException) Code() string { + return "DisabledException" +} + +// Message returns the exception's message. +func (s *DisabledException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *DisabledException) OrigErr() error { + return nil +} + +func (s *DisabledException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *DisabledException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *DisabledException) RequestID() string { + return s.RespMetadata.RequestID +} + +type DisconnectCustomKeyStoreInput struct { + _ struct{} `type:"structure"` + + // Enter the ID of the custom key store you want to disconnect. To find the + // ID of a custom key store, use the DescribeCustomKeyStores operation. + // + // CustomKeyStoreId is a required field + CustomKeyStoreId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s DisconnectCustomKeyStoreInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DisconnectCustomKeyStoreInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *DisconnectCustomKeyStoreInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "DisconnectCustomKeyStoreInput"} + if s.CustomKeyStoreId == nil { + invalidParams.Add(request.NewErrParamRequired("CustomKeyStoreId")) + } + if s.CustomKeyStoreId != nil && len(*s.CustomKeyStoreId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CustomKeyStoreId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetCustomKeyStoreId sets the CustomKeyStoreId field's value. +func (s *DisconnectCustomKeyStoreInput) SetCustomKeyStoreId(v string) *DisconnectCustomKeyStoreInput { + s.CustomKeyStoreId = &v + return s +} + +type DisconnectCustomKeyStoreOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s DisconnectCustomKeyStoreOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s DisconnectCustomKeyStoreOutput) GoString() string { + return s.String() +} + +type EnableKeyInput struct { + _ struct{} `type:"structure"` + + // A unique identifier for the customer master key (CMK). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s EnableKeyInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s EnableKeyInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *EnableKeyInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "EnableKeyInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *EnableKeyInput) SetKeyId(v string) *EnableKeyInput { + s.KeyId = &v + return s +} + +type EnableKeyOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s EnableKeyOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s EnableKeyOutput) GoString() string { + return s.String() +} + +type EnableKeyRotationInput struct { + _ struct{} `type:"structure"` + + // Identifies a symmetric customer master key (CMK). You cannot enable automatic + // rotation of asymmetric CMKs, CMKs with imported key material, or CMKs in + // a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s EnableKeyRotationInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s EnableKeyRotationInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *EnableKeyRotationInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "EnableKeyRotationInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *EnableKeyRotationInput) SetKeyId(v string) *EnableKeyRotationInput { + s.KeyId = &v + return s +} + +type EnableKeyRotationOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s EnableKeyRotationOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s EnableKeyRotationOutput) GoString() string { + return s.String() +} + +type EncryptInput struct { + _ struct{} `type:"structure"` + + // Specifies the encryption algorithm that AWS KMS will use to encrypt the plaintext + // message. The algorithm must be compatible with the CMK that you specify. + // + // This parameter is required only for asymmetric CMKs. The default value, SYMMETRIC_DEFAULT, + // is the algorithm used for symmetric CMKs. If you are using an asymmetric + // CMK, we recommend RSAES_OAEP_SHA_256. + EncryptionAlgorithm *string `type:"string" enum:"EncryptionAlgorithmSpec"` + + // Specifies the encryption context that will be used to encrypt the data. An + // encryption context is valid only for cryptographic operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) + // with a symmetric CMK. The standard asymmetric encryption algorithms that + // AWS KMS uses do not support an encryption context. + // + // An encryption context is a collection of non-secret key-value pairs that + // represents additional authenticated data. When you use an encryption context + // to encrypt data, you must specify the same (an exact case-sensitive match) + // encryption context to decrypt the data. An encryption context is optional + // when encrypting with a symmetric CMK, but it is highly recommended. + // + // For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) + // in the AWS Key Management Service Developer Guide. + EncryptionContext map[string]*string `type:"map"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // A unique identifier for the customer master key (CMK). + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". To specify + // a CMK in a different AWS account, you must use the key ARN or alias ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // Data to be encrypted. + // + // Plaintext is automatically base64 encoded/decoded by the SDK. + // + // Plaintext is a required field + Plaintext []byte `min:"1" type:"blob" required:"true" sensitive:"true"` +} + +// String returns the string representation +func (s EncryptInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s EncryptInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *EncryptInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "EncryptInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.Plaintext == nil { + invalidParams.Add(request.NewErrParamRequired("Plaintext")) + } + if s.Plaintext != nil && len(s.Plaintext) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Plaintext", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetEncryptionAlgorithm sets the EncryptionAlgorithm field's value. +func (s *EncryptInput) SetEncryptionAlgorithm(v string) *EncryptInput { + s.EncryptionAlgorithm = &v + return s +} + +// SetEncryptionContext sets the EncryptionContext field's value. +func (s *EncryptInput) SetEncryptionContext(v map[string]*string) *EncryptInput { + s.EncryptionContext = v + return s +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *EncryptInput) SetGrantTokens(v []*string) *EncryptInput { + s.GrantTokens = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *EncryptInput) SetKeyId(v string) *EncryptInput { + s.KeyId = &v + return s +} + +// SetPlaintext sets the Plaintext field's value. +func (s *EncryptInput) SetPlaintext(v []byte) *EncryptInput { + s.Plaintext = v + return s +} + +type EncryptOutput struct { + _ struct{} `type:"structure"` + + // The encrypted plaintext. When you use the HTTP API or the AWS CLI, the value + // is Base64-encoded. Otherwise, it is not Base64-encoded. + // + // CiphertextBlob is automatically base64 encoded/decoded by the SDK. + CiphertextBlob []byte `min:"1" type:"blob"` + + // The encryption algorithm that was used to encrypt the plaintext. + EncryptionAlgorithm *string `type:"string" enum:"EncryptionAlgorithmSpec"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the CMK that was used to encrypt the plaintext. + KeyId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s EncryptOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s EncryptOutput) GoString() string { + return s.String() +} + +// SetCiphertextBlob sets the CiphertextBlob field's value. +func (s *EncryptOutput) SetCiphertextBlob(v []byte) *EncryptOutput { + s.CiphertextBlob = v + return s +} + +// SetEncryptionAlgorithm sets the EncryptionAlgorithm field's value. +func (s *EncryptOutput) SetEncryptionAlgorithm(v string) *EncryptOutput { + s.EncryptionAlgorithm = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *EncryptOutput) SetKeyId(v string) *EncryptOutput { + s.KeyId = &v + return s +} + +// The request was rejected because the specified import token is expired. Use +// GetParametersForImport to get a new import token and public key, use the +// new public key to encrypt the key material, and then try the request again. +type ExpiredImportTokenException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s ExpiredImportTokenException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ExpiredImportTokenException) GoString() string { + return s.String() +} + +func newErrorExpiredImportTokenException(v protocol.ResponseMetadata) error { + return &ExpiredImportTokenException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *ExpiredImportTokenException) Code() string { + return "ExpiredImportTokenException" +} + +// Message returns the exception's message. +func (s *ExpiredImportTokenException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *ExpiredImportTokenException) OrigErr() error { + return nil +} + +func (s *ExpiredImportTokenException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *ExpiredImportTokenException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *ExpiredImportTokenException) RequestID() string { + return s.RespMetadata.RequestID +} + +type GenerateDataKeyInput struct { + _ struct{} `type:"structure"` + + // Specifies the encryption context that will be used when encrypting the data + // key. + // + // An encryption context is a collection of non-secret key-value pairs that + // represents additional authenticated data. When you use an encryption context + // to encrypt data, you must specify the same (an exact case-sensitive match) + // encryption context to decrypt the data. An encryption context is optional + // when encrypting with a symmetric CMK, but it is highly recommended. + // + // For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) + // in the AWS Key Management Service Developer Guide. + EncryptionContext map[string]*string `type:"map"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // Identifies the symmetric CMK that encrypts the data key. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". To specify + // a CMK in a different AWS account, you must use the key ARN or alias ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // Specifies the length of the data key. Use AES_128 to generate a 128-bit symmetric + // key, or AES_256 to generate a 256-bit symmetric key. + // + // You must specify either the KeySpec or the NumberOfBytes parameter (but not + // both) in every GenerateDataKey request. + KeySpec *string `type:"string" enum:"DataKeySpec"` + + // Specifies the length of the data key in bytes. For example, use the value + // 64 to generate a 512-bit data key (64 bytes is 512 bits). For 128-bit (16-byte) + // and 256-bit (32-byte) data keys, use the KeySpec parameter. + // + // You must specify either the KeySpec or the NumberOfBytes parameter (but not + // both) in every GenerateDataKey request. + NumberOfBytes *int64 `min:"1" type:"integer"` +} + +// String returns the string representation +func (s GenerateDataKeyInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GenerateDataKeyInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *GenerateDataKeyInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "GenerateDataKeyInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.NumberOfBytes != nil && *s.NumberOfBytes < 1 { + invalidParams.Add(request.NewErrParamMinValue("NumberOfBytes", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetEncryptionContext sets the EncryptionContext field's value. +func (s *GenerateDataKeyInput) SetEncryptionContext(v map[string]*string) *GenerateDataKeyInput { + s.EncryptionContext = v + return s +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *GenerateDataKeyInput) SetGrantTokens(v []*string) *GenerateDataKeyInput { + s.GrantTokens = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *GenerateDataKeyInput) SetKeyId(v string) *GenerateDataKeyInput { + s.KeyId = &v + return s +} + +// SetKeySpec sets the KeySpec field's value. +func (s *GenerateDataKeyInput) SetKeySpec(v string) *GenerateDataKeyInput { + s.KeySpec = &v + return s +} + +// SetNumberOfBytes sets the NumberOfBytes field's value. +func (s *GenerateDataKeyInput) SetNumberOfBytes(v int64) *GenerateDataKeyInput { + s.NumberOfBytes = &v + return s +} + +type GenerateDataKeyOutput struct { + _ struct{} `type:"structure"` + + // The encrypted copy of the data key. When you use the HTTP API or the AWS + // CLI, the value is Base64-encoded. Otherwise, it is not Base64-encoded. + // + // CiphertextBlob is automatically base64 encoded/decoded by the SDK. + CiphertextBlob []byte `min:"1" type:"blob"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the CMK that encrypted the data key. + KeyId *string `min:"1" type:"string"` + + // The plaintext data key. When you use the HTTP API or the AWS CLI, the value + // is Base64-encoded. Otherwise, it is not Base64-encoded. Use this data key + // to encrypt your data outside of KMS. Then, remove it from memory as soon + // as possible. + // + // Plaintext is automatically base64 encoded/decoded by the SDK. + Plaintext []byte `min:"1" type:"blob" sensitive:"true"` +} + +// String returns the string representation +func (s GenerateDataKeyOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GenerateDataKeyOutput) GoString() string { + return s.String() +} + +// SetCiphertextBlob sets the CiphertextBlob field's value. +func (s *GenerateDataKeyOutput) SetCiphertextBlob(v []byte) *GenerateDataKeyOutput { + s.CiphertextBlob = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *GenerateDataKeyOutput) SetKeyId(v string) *GenerateDataKeyOutput { + s.KeyId = &v + return s +} + +// SetPlaintext sets the Plaintext field's value. +func (s *GenerateDataKeyOutput) SetPlaintext(v []byte) *GenerateDataKeyOutput { + s.Plaintext = v + return s +} + +type GenerateDataKeyPairInput struct { + _ struct{} `type:"structure"` + + // Specifies the encryption context that will be used when encrypting the private + // key in the data key pair. + // + // An encryption context is a collection of non-secret key-value pairs that + // represents additional authenticated data. When you use an encryption context + // to encrypt data, you must specify the same (an exact case-sensitive match) + // encryption context to decrypt the data. An encryption context is optional + // when encrypting with a symmetric CMK, but it is highly recommended. + // + // For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) + // in the AWS Key Management Service Developer Guide. + EncryptionContext map[string]*string `type:"map"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // Specifies the symmetric CMK that encrypts the private key in the data key + // pair. You cannot specify an asymmetric CMK or a CMK in a custom key store. + // To get the type and origin of your CMK, use the DescribeKey operation. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". To specify + // a CMK in a different AWS account, you must use the key ARN or alias ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // Determines the type of data key pair that is generated. + // + // The AWS KMS rule that restricts the use of asymmetric RSA CMKs to encrypt + // and decrypt or to sign and verify (but not both), and the rule that permits + // you to use ECC CMKs only to sign and verify, are not effective outside of + // AWS KMS. + // + // KeyPairSpec is a required field + KeyPairSpec *string `type:"string" required:"true" enum:"DataKeyPairSpec"` +} + +// String returns the string representation +func (s GenerateDataKeyPairInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GenerateDataKeyPairInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *GenerateDataKeyPairInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "GenerateDataKeyPairInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.KeyPairSpec == nil { + invalidParams.Add(request.NewErrParamRequired("KeyPairSpec")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetEncryptionContext sets the EncryptionContext field's value. +func (s *GenerateDataKeyPairInput) SetEncryptionContext(v map[string]*string) *GenerateDataKeyPairInput { + s.EncryptionContext = v + return s +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *GenerateDataKeyPairInput) SetGrantTokens(v []*string) *GenerateDataKeyPairInput { + s.GrantTokens = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *GenerateDataKeyPairInput) SetKeyId(v string) *GenerateDataKeyPairInput { + s.KeyId = &v + return s +} + +// SetKeyPairSpec sets the KeyPairSpec field's value. +func (s *GenerateDataKeyPairInput) SetKeyPairSpec(v string) *GenerateDataKeyPairInput { + s.KeyPairSpec = &v + return s +} + +type GenerateDataKeyPairOutput struct { + _ struct{} `type:"structure"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the CMK that encrypted the private key. + KeyId *string `min:"1" type:"string"` + + // The type of data key pair that was generated. + KeyPairSpec *string `type:"string" enum:"DataKeyPairSpec"` + + // The encrypted copy of the private key. When you use the HTTP API or the AWS + // CLI, the value is Base64-encoded. Otherwise, it is not Base64-encoded. + // + // PrivateKeyCiphertextBlob is automatically base64 encoded/decoded by the SDK. + PrivateKeyCiphertextBlob []byte `min:"1" type:"blob"` + + // The plaintext copy of the private key. When you use the HTTP API or the AWS + // CLI, the value is Base64-encoded. Otherwise, it is not Base64-encoded. + // + // PrivateKeyPlaintext is automatically base64 encoded/decoded by the SDK. + PrivateKeyPlaintext []byte `min:"1" type:"blob" sensitive:"true"` + + // The public key (in plaintext). + // + // PublicKey is automatically base64 encoded/decoded by the SDK. + PublicKey []byte `min:"1" type:"blob"` +} + +// String returns the string representation +func (s GenerateDataKeyPairOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GenerateDataKeyPairOutput) GoString() string { + return s.String() +} + +// SetKeyId sets the KeyId field's value. +func (s *GenerateDataKeyPairOutput) SetKeyId(v string) *GenerateDataKeyPairOutput { + s.KeyId = &v + return s +} + +// SetKeyPairSpec sets the KeyPairSpec field's value. +func (s *GenerateDataKeyPairOutput) SetKeyPairSpec(v string) *GenerateDataKeyPairOutput { + s.KeyPairSpec = &v + return s +} + +// SetPrivateKeyCiphertextBlob sets the PrivateKeyCiphertextBlob field's value. +func (s *GenerateDataKeyPairOutput) SetPrivateKeyCiphertextBlob(v []byte) *GenerateDataKeyPairOutput { + s.PrivateKeyCiphertextBlob = v + return s +} + +// SetPrivateKeyPlaintext sets the PrivateKeyPlaintext field's value. +func (s *GenerateDataKeyPairOutput) SetPrivateKeyPlaintext(v []byte) *GenerateDataKeyPairOutput { + s.PrivateKeyPlaintext = v + return s +} + +// SetPublicKey sets the PublicKey field's value. +func (s *GenerateDataKeyPairOutput) SetPublicKey(v []byte) *GenerateDataKeyPairOutput { + s.PublicKey = v + return s +} + +type GenerateDataKeyPairWithoutPlaintextInput struct { + _ struct{} `type:"structure"` + + // Specifies the encryption context that will be used when encrypting the private + // key in the data key pair. + // + // An encryption context is a collection of non-secret key-value pairs that + // represents additional authenticated data. When you use an encryption context + // to encrypt data, you must specify the same (an exact case-sensitive match) + // encryption context to decrypt the data. An encryption context is optional + // when encrypting with a symmetric CMK, but it is highly recommended. + // + // For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) + // in the AWS Key Management Service Developer Guide. + EncryptionContext map[string]*string `type:"map"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // Specifies the CMK that encrypts the private key in the data key pair. You + // must specify a symmetric CMK. You cannot use an asymmetric CMK or a CMK in + // a custom key store. To get the type and origin of your CMK, use the DescribeKey + // operation. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // Determines the type of data key pair that is generated. + // + // The AWS KMS rule that restricts the use of asymmetric RSA CMKs to encrypt + // and decrypt or to sign and verify (but not both), and the rule that permits + // you to use ECC CMKs only to sign and verify, are not effective outside of + // AWS KMS. + // + // KeyPairSpec is a required field + KeyPairSpec *string `type:"string" required:"true" enum:"DataKeyPairSpec"` +} + +// String returns the string representation +func (s GenerateDataKeyPairWithoutPlaintextInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GenerateDataKeyPairWithoutPlaintextInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *GenerateDataKeyPairWithoutPlaintextInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "GenerateDataKeyPairWithoutPlaintextInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.KeyPairSpec == nil { + invalidParams.Add(request.NewErrParamRequired("KeyPairSpec")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetEncryptionContext sets the EncryptionContext field's value. +func (s *GenerateDataKeyPairWithoutPlaintextInput) SetEncryptionContext(v map[string]*string) *GenerateDataKeyPairWithoutPlaintextInput { + s.EncryptionContext = v + return s +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *GenerateDataKeyPairWithoutPlaintextInput) SetGrantTokens(v []*string) *GenerateDataKeyPairWithoutPlaintextInput { + s.GrantTokens = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *GenerateDataKeyPairWithoutPlaintextInput) SetKeyId(v string) *GenerateDataKeyPairWithoutPlaintextInput { + s.KeyId = &v + return s +} + +// SetKeyPairSpec sets the KeyPairSpec field's value. +func (s *GenerateDataKeyPairWithoutPlaintextInput) SetKeyPairSpec(v string) *GenerateDataKeyPairWithoutPlaintextInput { + s.KeyPairSpec = &v + return s +} + +type GenerateDataKeyPairWithoutPlaintextOutput struct { + _ struct{} `type:"structure"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the CMK that encrypted the private key. + KeyId *string `min:"1" type:"string"` + + // The type of data key pair that was generated. + KeyPairSpec *string `type:"string" enum:"DataKeyPairSpec"` + + // The encrypted copy of the private key. When you use the HTTP API or the AWS + // CLI, the value is Base64-encoded. Otherwise, it is not Base64-encoded. + // + // PrivateKeyCiphertextBlob is automatically base64 encoded/decoded by the SDK. + PrivateKeyCiphertextBlob []byte `min:"1" type:"blob"` + + // The public key (in plaintext). + // + // PublicKey is automatically base64 encoded/decoded by the SDK. + PublicKey []byte `min:"1" type:"blob"` +} + +// String returns the string representation +func (s GenerateDataKeyPairWithoutPlaintextOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GenerateDataKeyPairWithoutPlaintextOutput) GoString() string { + return s.String() +} + +// SetKeyId sets the KeyId field's value. +func (s *GenerateDataKeyPairWithoutPlaintextOutput) SetKeyId(v string) *GenerateDataKeyPairWithoutPlaintextOutput { + s.KeyId = &v + return s +} + +// SetKeyPairSpec sets the KeyPairSpec field's value. +func (s *GenerateDataKeyPairWithoutPlaintextOutput) SetKeyPairSpec(v string) *GenerateDataKeyPairWithoutPlaintextOutput { + s.KeyPairSpec = &v + return s +} + +// SetPrivateKeyCiphertextBlob sets the PrivateKeyCiphertextBlob field's value. +func (s *GenerateDataKeyPairWithoutPlaintextOutput) SetPrivateKeyCiphertextBlob(v []byte) *GenerateDataKeyPairWithoutPlaintextOutput { + s.PrivateKeyCiphertextBlob = v + return s +} + +// SetPublicKey sets the PublicKey field's value. +func (s *GenerateDataKeyPairWithoutPlaintextOutput) SetPublicKey(v []byte) *GenerateDataKeyPairWithoutPlaintextOutput { + s.PublicKey = v + return s +} + +type GenerateDataKeyWithoutPlaintextInput struct { + _ struct{} `type:"structure"` + + // Specifies the encryption context that will be used when encrypting the data + // key. + // + // An encryption context is a collection of non-secret key-value pairs that + // represents additional authenticated data. When you use an encryption context + // to encrypt data, you must specify the same (an exact case-sensitive match) + // encryption context to decrypt the data. An encryption context is optional + // when encrypting with a symmetric CMK, but it is highly recommended. + // + // For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) + // in the AWS Key Management Service Developer Guide. + EncryptionContext map[string]*string `type:"map"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // The identifier of the symmetric customer master key (CMK) that encrypts the + // data key. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". To specify + // a CMK in a different AWS account, you must use the key ARN or alias ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // The length of the data key. Use AES_128 to generate a 128-bit symmetric key, + // or AES_256 to generate a 256-bit symmetric key. + KeySpec *string `type:"string" enum:"DataKeySpec"` + + // The length of the data key in bytes. For example, use the value 64 to generate + // a 512-bit data key (64 bytes is 512 bits). For common key lengths (128-bit + // and 256-bit symmetric keys), we recommend that you use the KeySpec field + // instead of this one. + NumberOfBytes *int64 `min:"1" type:"integer"` +} + +// String returns the string representation +func (s GenerateDataKeyWithoutPlaintextInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GenerateDataKeyWithoutPlaintextInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *GenerateDataKeyWithoutPlaintextInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "GenerateDataKeyWithoutPlaintextInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.NumberOfBytes != nil && *s.NumberOfBytes < 1 { + invalidParams.Add(request.NewErrParamMinValue("NumberOfBytes", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetEncryptionContext sets the EncryptionContext field's value. +func (s *GenerateDataKeyWithoutPlaintextInput) SetEncryptionContext(v map[string]*string) *GenerateDataKeyWithoutPlaintextInput { + s.EncryptionContext = v + return s +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *GenerateDataKeyWithoutPlaintextInput) SetGrantTokens(v []*string) *GenerateDataKeyWithoutPlaintextInput { + s.GrantTokens = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *GenerateDataKeyWithoutPlaintextInput) SetKeyId(v string) *GenerateDataKeyWithoutPlaintextInput { + s.KeyId = &v + return s +} + +// SetKeySpec sets the KeySpec field's value. +func (s *GenerateDataKeyWithoutPlaintextInput) SetKeySpec(v string) *GenerateDataKeyWithoutPlaintextInput { + s.KeySpec = &v + return s +} + +// SetNumberOfBytes sets the NumberOfBytes field's value. +func (s *GenerateDataKeyWithoutPlaintextInput) SetNumberOfBytes(v int64) *GenerateDataKeyWithoutPlaintextInput { + s.NumberOfBytes = &v + return s +} + +type GenerateDataKeyWithoutPlaintextOutput struct { + _ struct{} `type:"structure"` + + // The encrypted data key. When you use the HTTP API or the AWS CLI, the value + // is Base64-encoded. Otherwise, it is not Base64-encoded. + // + // CiphertextBlob is automatically base64 encoded/decoded by the SDK. + CiphertextBlob []byte `min:"1" type:"blob"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the CMK that encrypted the data key. + KeyId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s GenerateDataKeyWithoutPlaintextOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GenerateDataKeyWithoutPlaintextOutput) GoString() string { + return s.String() +} + +// SetCiphertextBlob sets the CiphertextBlob field's value. +func (s *GenerateDataKeyWithoutPlaintextOutput) SetCiphertextBlob(v []byte) *GenerateDataKeyWithoutPlaintextOutput { + s.CiphertextBlob = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *GenerateDataKeyWithoutPlaintextOutput) SetKeyId(v string) *GenerateDataKeyWithoutPlaintextOutput { + s.KeyId = &v + return s +} + +type GenerateRandomInput struct { + _ struct{} `type:"structure"` + + // Generates the random byte string in the AWS CloudHSM cluster that is associated + // with the specified custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html). + // To find the ID of a custom key store, use the DescribeCustomKeyStores operation. + CustomKeyStoreId *string `min:"1" type:"string"` + + // The length of the byte string. + NumberOfBytes *int64 `min:"1" type:"integer"` +} + +// String returns the string representation +func (s GenerateRandomInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GenerateRandomInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *GenerateRandomInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "GenerateRandomInput"} + if s.CustomKeyStoreId != nil && len(*s.CustomKeyStoreId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CustomKeyStoreId", 1)) + } + if s.NumberOfBytes != nil && *s.NumberOfBytes < 1 { + invalidParams.Add(request.NewErrParamMinValue("NumberOfBytes", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetCustomKeyStoreId sets the CustomKeyStoreId field's value. +func (s *GenerateRandomInput) SetCustomKeyStoreId(v string) *GenerateRandomInput { + s.CustomKeyStoreId = &v + return s +} + +// SetNumberOfBytes sets the NumberOfBytes field's value. +func (s *GenerateRandomInput) SetNumberOfBytes(v int64) *GenerateRandomInput { + s.NumberOfBytes = &v + return s +} + +type GenerateRandomOutput struct { + _ struct{} `type:"structure"` + + // The random byte string. When you use the HTTP API or the AWS CLI, the value + // is Base64-encoded. Otherwise, it is not Base64-encoded. + // + // Plaintext is automatically base64 encoded/decoded by the SDK. + Plaintext []byte `min:"1" type:"blob" sensitive:"true"` +} + +// String returns the string representation +func (s GenerateRandomOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GenerateRandomOutput) GoString() string { + return s.String() +} + +// SetPlaintext sets the Plaintext field's value. +func (s *GenerateRandomOutput) SetPlaintext(v []byte) *GenerateRandomOutput { + s.Plaintext = v + return s +} + +type GetKeyPolicyInput struct { + _ struct{} `type:"structure"` + + // A unique identifier for the customer master key (CMK). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // Specifies the name of the key policy. The only valid name is default. To + // get the names of key policies, use ListKeyPolicies. + // + // PolicyName is a required field + PolicyName *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s GetKeyPolicyInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GetKeyPolicyInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *GetKeyPolicyInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "GetKeyPolicyInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.PolicyName == nil { + invalidParams.Add(request.NewErrParamRequired("PolicyName")) + } + if s.PolicyName != nil && len(*s.PolicyName) < 1 { + invalidParams.Add(request.NewErrParamMinLen("PolicyName", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *GetKeyPolicyInput) SetKeyId(v string) *GetKeyPolicyInput { + s.KeyId = &v + return s +} + +// SetPolicyName sets the PolicyName field's value. +func (s *GetKeyPolicyInput) SetPolicyName(v string) *GetKeyPolicyInput { + s.PolicyName = &v + return s +} + +type GetKeyPolicyOutput struct { + _ struct{} `type:"structure"` + + // A key policy document in JSON format. + Policy *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s GetKeyPolicyOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GetKeyPolicyOutput) GoString() string { + return s.String() +} + +// SetPolicy sets the Policy field's value. +func (s *GetKeyPolicyOutput) SetPolicy(v string) *GetKeyPolicyOutput { + s.Policy = &v + return s +} + +type GetKeyRotationStatusInput struct { + _ struct{} `type:"structure"` + + // A unique identifier for the customer master key (CMK). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. To specify + // a CMK in a different AWS account, you must use the key ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s GetKeyRotationStatusInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GetKeyRotationStatusInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *GetKeyRotationStatusInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "GetKeyRotationStatusInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *GetKeyRotationStatusInput) SetKeyId(v string) *GetKeyRotationStatusInput { + s.KeyId = &v + return s +} + +type GetKeyRotationStatusOutput struct { + _ struct{} `type:"structure"` + + // A Boolean value that specifies whether key rotation is enabled. + KeyRotationEnabled *bool `type:"boolean"` +} + +// String returns the string representation +func (s GetKeyRotationStatusOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GetKeyRotationStatusOutput) GoString() string { + return s.String() +} + +// SetKeyRotationEnabled sets the KeyRotationEnabled field's value. +func (s *GetKeyRotationStatusOutput) SetKeyRotationEnabled(v bool) *GetKeyRotationStatusOutput { + s.KeyRotationEnabled = &v + return s +} + +type GetParametersForImportInput struct { + _ struct{} `type:"structure"` + + // The identifier of the symmetric CMK into which you will import key material. + // The Origin of the CMK must be EXTERNAL. + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // The algorithm you will use to encrypt the key material before importing it + // with ImportKeyMaterial. For more information, see Encrypt the Key Material + // (https://docs.aws.amazon.com/kms/latest/developerguide/importing-keys-encrypt-key-material.html) + // in the AWS Key Management Service Developer Guide. + // + // WrappingAlgorithm is a required field + WrappingAlgorithm *string `type:"string" required:"true" enum:"AlgorithmSpec"` + + // The type of wrapping key (public key) to return in the response. Only 2048-bit + // RSA public keys are supported. + // + // WrappingKeySpec is a required field + WrappingKeySpec *string `type:"string" required:"true" enum:"WrappingKeySpec"` +} + +// String returns the string representation +func (s GetParametersForImportInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GetParametersForImportInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *GetParametersForImportInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "GetParametersForImportInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.WrappingAlgorithm == nil { + invalidParams.Add(request.NewErrParamRequired("WrappingAlgorithm")) + } + if s.WrappingKeySpec == nil { + invalidParams.Add(request.NewErrParamRequired("WrappingKeySpec")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *GetParametersForImportInput) SetKeyId(v string) *GetParametersForImportInput { + s.KeyId = &v + return s +} + +// SetWrappingAlgorithm sets the WrappingAlgorithm field's value. +func (s *GetParametersForImportInput) SetWrappingAlgorithm(v string) *GetParametersForImportInput { + s.WrappingAlgorithm = &v + return s +} + +// SetWrappingKeySpec sets the WrappingKeySpec field's value. +func (s *GetParametersForImportInput) SetWrappingKeySpec(v string) *GetParametersForImportInput { + s.WrappingKeySpec = &v + return s +} + +type GetParametersForImportOutput struct { + _ struct{} `type:"structure"` + + // The import token to send in a subsequent ImportKeyMaterial request. + // + // ImportToken is automatically base64 encoded/decoded by the SDK. + ImportToken []byte `min:"1" type:"blob"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the CMK to use in a subsequent ImportKeyMaterial request. This is the + // same CMK specified in the GetParametersForImport request. + KeyId *string `min:"1" type:"string"` + + // The time at which the import token and public key are no longer valid. After + // this time, you cannot use them to make an ImportKeyMaterial request and you + // must send another GetParametersForImport request to get new ones. + ParametersValidTo *time.Time `type:"timestamp"` + + // The public key to use to encrypt the key material before importing it with + // ImportKeyMaterial. + // + // PublicKey is automatically base64 encoded/decoded by the SDK. + PublicKey []byte `min:"1" type:"blob" sensitive:"true"` +} + +// String returns the string representation +func (s GetParametersForImportOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GetParametersForImportOutput) GoString() string { + return s.String() +} + +// SetImportToken sets the ImportToken field's value. +func (s *GetParametersForImportOutput) SetImportToken(v []byte) *GetParametersForImportOutput { + s.ImportToken = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *GetParametersForImportOutput) SetKeyId(v string) *GetParametersForImportOutput { + s.KeyId = &v + return s +} + +// SetParametersValidTo sets the ParametersValidTo field's value. +func (s *GetParametersForImportOutput) SetParametersValidTo(v time.Time) *GetParametersForImportOutput { + s.ParametersValidTo = &v + return s +} + +// SetPublicKey sets the PublicKey field's value. +func (s *GetParametersForImportOutput) SetPublicKey(v []byte) *GetParametersForImportOutput { + s.PublicKey = v + return s +} + +type GetPublicKeyInput struct { + _ struct{} `type:"structure"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // Identifies the asymmetric CMK that includes the public key. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". To specify + // a CMK in a different AWS account, you must use the key ARN or alias ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s GetPublicKeyInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GetPublicKeyInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *GetPublicKeyInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "GetPublicKeyInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *GetPublicKeyInput) SetGrantTokens(v []*string) *GetPublicKeyInput { + s.GrantTokens = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *GetPublicKeyInput) SetKeyId(v string) *GetPublicKeyInput { + s.KeyId = &v + return s +} + +type GetPublicKeyOutput struct { + _ struct{} `type:"structure"` + + // The type of the of the public key that was downloaded. + CustomerMasterKeySpec *string `type:"string" enum:"CustomerMasterKeySpec"` + + // The encryption algorithms that AWS KMS supports for this key. + // + // This information is critical. If a public key encrypts data outside of AWS + // KMS by using an unsupported encryption algorithm, the ciphertext cannot be + // decrypted. + // + // This field appears in the response only when the KeyUsage of the public key + // is ENCRYPT_DECRYPT. + EncryptionAlgorithms []*string `type:"list"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the asymmetric CMK from which the public key was downloaded. + KeyId *string `min:"1" type:"string"` + + // The permitted use of the public key. Valid values are ENCRYPT_DECRYPT or + // SIGN_VERIFY. + // + // This information is critical. If a public key with SIGN_VERIFY key usage + // encrypts data outside of AWS KMS, the ciphertext cannot be decrypted. + KeyUsage *string `type:"string" enum:"KeyUsageType"` + + // The exported public key. + // + // The value is a DER-encoded X.509 public key, also known as SubjectPublicKeyInfo + // (SPKI), as defined in RFC 5280 (https://tools.ietf.org/html/rfc5280). When + // you use the HTTP API or the AWS CLI, the value is Base64-encoded. Otherwise, + // it is not Base64-encoded. + // + // PublicKey is automatically base64 encoded/decoded by the SDK. + PublicKey []byte `min:"1" type:"blob"` + + // The signing algorithms that AWS KMS supports for this key. + // + // This field appears in the response only when the KeyUsage of the public key + // is SIGN_VERIFY. + SigningAlgorithms []*string `type:"list"` +} + +// String returns the string representation +func (s GetPublicKeyOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GetPublicKeyOutput) GoString() string { + return s.String() +} + +// SetCustomerMasterKeySpec sets the CustomerMasterKeySpec field's value. +func (s *GetPublicKeyOutput) SetCustomerMasterKeySpec(v string) *GetPublicKeyOutput { + s.CustomerMasterKeySpec = &v + return s +} + +// SetEncryptionAlgorithms sets the EncryptionAlgorithms field's value. +func (s *GetPublicKeyOutput) SetEncryptionAlgorithms(v []*string) *GetPublicKeyOutput { + s.EncryptionAlgorithms = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *GetPublicKeyOutput) SetKeyId(v string) *GetPublicKeyOutput { + s.KeyId = &v + return s +} + +// SetKeyUsage sets the KeyUsage field's value. +func (s *GetPublicKeyOutput) SetKeyUsage(v string) *GetPublicKeyOutput { + s.KeyUsage = &v + return s +} + +// SetPublicKey sets the PublicKey field's value. +func (s *GetPublicKeyOutput) SetPublicKey(v []byte) *GetPublicKeyOutput { + s.PublicKey = v + return s +} + +// SetSigningAlgorithms sets the SigningAlgorithms field's value. +func (s *GetPublicKeyOutput) SetSigningAlgorithms(v []*string) *GetPublicKeyOutput { + s.SigningAlgorithms = v + return s +} + +// Use this structure to allow cryptographic operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) +// in the grant only when the operation request includes the specified encryption +// context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context). +// +// AWS KMS applies the grant constraints only to cryptographic operations that +// support an encryption context, that is, all cryptographic operations with +// a symmetric CMK (https://docs.aws.amazon.com/kms/latest/developerguide/symm-asymm-concepts.html#symmetric-cmks). +// Grant constraints are not applied to operations that do not support an encryption +// context, such as cryptographic operations with asymmetric CMKs and management +// operations, such as DescribeKey or ScheduleKeyDeletion. +// +// In a cryptographic operation, the encryption context in the decryption operation +// must be an exact, case-sensitive match for the keys and values in the encryption +// context of the encryption operation. Only the order of the pairs can vary. +// +// However, in a grant constraint, the key in each key-value pair is not case +// sensitive, but the value is case sensitive. +// +// To avoid confusion, do not use multiple encryption context pairs that differ +// only by case. To require a fully case-sensitive encryption context, use the +// kms:EncryptionContext: and kms:EncryptionContextKeys conditions in an IAM +// or key policy. For details, see kms:EncryptionContext: (https://docs.aws.amazon.com/kms/latest/developerguide/policy-conditions.html#conditions-kms-encryption-context) +// in the AWS Key Management Service Developer Guide . +type GrantConstraints struct { + _ struct{} `type:"structure"` + + // A list of key-value pairs that must match the encryption context in the cryptographic + // operation (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) + // request. The grant allows the operation only when the encryption context + // in the request is the same as the encryption context specified in this constraint. + EncryptionContextEquals map[string]*string `type:"map"` + + // A list of key-value pairs that must be included in the encryption context + // of the cryptographic operation (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) + // request. The grant allows the cryptographic operation only when the encryption + // context in the request includes the key-value pairs specified in this constraint, + // although it can include additional key-value pairs. + EncryptionContextSubset map[string]*string `type:"map"` +} + +// String returns the string representation +func (s GrantConstraints) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GrantConstraints) GoString() string { + return s.String() +} + +// SetEncryptionContextEquals sets the EncryptionContextEquals field's value. +func (s *GrantConstraints) SetEncryptionContextEquals(v map[string]*string) *GrantConstraints { + s.EncryptionContextEquals = v + return s +} + +// SetEncryptionContextSubset sets the EncryptionContextSubset field's value. +func (s *GrantConstraints) SetEncryptionContextSubset(v map[string]*string) *GrantConstraints { + s.EncryptionContextSubset = v + return s +} + +// Contains information about a grant. +type GrantListEntry struct { + _ struct{} `type:"structure"` + + // A list of key-value pairs that must be present in the encryption context + // of certain subsequent operations that the grant allows. + Constraints *GrantConstraints `type:"structure"` + + // The date and time when the grant was created. + CreationDate *time.Time `type:"timestamp"` + + // The unique identifier for the grant. + GrantId *string `min:"1" type:"string"` + + // The identity that gets the permissions in the grant. + // + // The GranteePrincipal field in the ListGrants response usually contains the + // user or role designated as the grantee principal in the grant. However, when + // the grantee principal in the grant is an AWS service, the GranteePrincipal + // field contains the service principal (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html#principal-services), + // which might represent several different grantee principals. + GranteePrincipal *string `min:"1" type:"string"` + + // The AWS account under which the grant was issued. + IssuingAccount *string `min:"1" type:"string"` + + // The unique identifier for the customer master key (CMK) to which the grant + // applies. + KeyId *string `min:"1" type:"string"` + + // The friendly name that identifies the grant. If a name was provided in the + // CreateGrant request, that name is returned. Otherwise this value is null. + Name *string `min:"1" type:"string"` + + // The list of operations permitted by the grant. + Operations []*string `type:"list"` + + // The principal that can retire the grant. + RetiringPrincipal *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s GrantListEntry) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GrantListEntry) GoString() string { + return s.String() +} + +// SetConstraints sets the Constraints field's value. +func (s *GrantListEntry) SetConstraints(v *GrantConstraints) *GrantListEntry { + s.Constraints = v + return s +} + +// SetCreationDate sets the CreationDate field's value. +func (s *GrantListEntry) SetCreationDate(v time.Time) *GrantListEntry { + s.CreationDate = &v + return s +} + +// SetGrantId sets the GrantId field's value. +func (s *GrantListEntry) SetGrantId(v string) *GrantListEntry { + s.GrantId = &v + return s +} + +// SetGranteePrincipal sets the GranteePrincipal field's value. +func (s *GrantListEntry) SetGranteePrincipal(v string) *GrantListEntry { + s.GranteePrincipal = &v + return s +} + +// SetIssuingAccount sets the IssuingAccount field's value. +func (s *GrantListEntry) SetIssuingAccount(v string) *GrantListEntry { + s.IssuingAccount = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *GrantListEntry) SetKeyId(v string) *GrantListEntry { + s.KeyId = &v + return s +} + +// SetName sets the Name field's value. +func (s *GrantListEntry) SetName(v string) *GrantListEntry { + s.Name = &v + return s +} + +// SetOperations sets the Operations field's value. +func (s *GrantListEntry) SetOperations(v []*string) *GrantListEntry { + s.Operations = v + return s +} + +// SetRetiringPrincipal sets the RetiringPrincipal field's value. +func (s *GrantListEntry) SetRetiringPrincipal(v string) *GrantListEntry { + s.RetiringPrincipal = &v + return s +} + +type ImportKeyMaterialInput struct { + _ struct{} `type:"structure"` + + // The encrypted key material to import. The key material must be encrypted + // with the public wrapping key that GetParametersForImport returned, using + // the wrapping algorithm that you specified in the same GetParametersForImport + // request. + // + // EncryptedKeyMaterial is automatically base64 encoded/decoded by the SDK. + // + // EncryptedKeyMaterial is a required field + EncryptedKeyMaterial []byte `min:"1" type:"blob" required:"true"` + + // Specifies whether the key material expires. The default is KEY_MATERIAL_EXPIRES, + // in which case you must include the ValidTo parameter. When this parameter + // is set to KEY_MATERIAL_DOES_NOT_EXPIRE, you must omit the ValidTo parameter. + ExpirationModel *string `type:"string" enum:"ExpirationModelType"` + + // The import token that you received in the response to a previous GetParametersForImport + // request. It must be from the same response that contained the public key + // that you used to encrypt the key material. + // + // ImportToken is automatically base64 encoded/decoded by the SDK. + // + // ImportToken is a required field + ImportToken []byte `min:"1" type:"blob" required:"true"` + + // The identifier of the symmetric CMK that receives the imported key material. + // The CMK's Origin must be EXTERNAL. This must be the same CMK specified in + // the KeyID parameter of the corresponding GetParametersForImport request. + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // The time at which the imported key material expires. When the key material + // expires, AWS KMS deletes the key material and the CMK becomes unusable. You + // must omit this parameter when the ExpirationModel parameter is set to KEY_MATERIAL_DOES_NOT_EXPIRE. + // Otherwise it is required. + ValidTo *time.Time `type:"timestamp"` +} + +// String returns the string representation +func (s ImportKeyMaterialInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ImportKeyMaterialInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ImportKeyMaterialInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ImportKeyMaterialInput"} + if s.EncryptedKeyMaterial == nil { + invalidParams.Add(request.NewErrParamRequired("EncryptedKeyMaterial")) + } + if s.EncryptedKeyMaterial != nil && len(s.EncryptedKeyMaterial) < 1 { + invalidParams.Add(request.NewErrParamMinLen("EncryptedKeyMaterial", 1)) + } + if s.ImportToken == nil { + invalidParams.Add(request.NewErrParamRequired("ImportToken")) + } + if s.ImportToken != nil && len(s.ImportToken) < 1 { + invalidParams.Add(request.NewErrParamMinLen("ImportToken", 1)) + } + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetEncryptedKeyMaterial sets the EncryptedKeyMaterial field's value. +func (s *ImportKeyMaterialInput) SetEncryptedKeyMaterial(v []byte) *ImportKeyMaterialInput { + s.EncryptedKeyMaterial = v + return s +} + +// SetExpirationModel sets the ExpirationModel field's value. +func (s *ImportKeyMaterialInput) SetExpirationModel(v string) *ImportKeyMaterialInput { + s.ExpirationModel = &v + return s +} + +// SetImportToken sets the ImportToken field's value. +func (s *ImportKeyMaterialInput) SetImportToken(v []byte) *ImportKeyMaterialInput { + s.ImportToken = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *ImportKeyMaterialInput) SetKeyId(v string) *ImportKeyMaterialInput { + s.KeyId = &v + return s +} + +// SetValidTo sets the ValidTo field's value. +func (s *ImportKeyMaterialInput) SetValidTo(v time.Time) *ImportKeyMaterialInput { + s.ValidTo = &v + return s +} + +type ImportKeyMaterialOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s ImportKeyMaterialOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ImportKeyMaterialOutput) GoString() string { + return s.String() +} + +// The request was rejected because the specified CMK cannot decrypt the data. +// The KeyId in a Decrypt request and the SourceKeyId in a ReEncrypt request +// must identify the same CMK that was used to encrypt the ciphertext. +type IncorrectKeyException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s IncorrectKeyException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s IncorrectKeyException) GoString() string { + return s.String() +} + +func newErrorIncorrectKeyException(v protocol.ResponseMetadata) error { + return &IncorrectKeyException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *IncorrectKeyException) Code() string { + return "IncorrectKeyException" +} + +// Message returns the exception's message. +func (s *IncorrectKeyException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *IncorrectKeyException) OrigErr() error { + return nil +} + +func (s *IncorrectKeyException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *IncorrectKeyException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *IncorrectKeyException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the key material in the request is, expired, +// invalid, or is not the same key material that was previously imported into +// this customer master key (CMK). +type IncorrectKeyMaterialException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s IncorrectKeyMaterialException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s IncorrectKeyMaterialException) GoString() string { + return s.String() +} + +func newErrorIncorrectKeyMaterialException(v protocol.ResponseMetadata) error { + return &IncorrectKeyMaterialException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *IncorrectKeyMaterialException) Code() string { + return "IncorrectKeyMaterialException" +} + +// Message returns the exception's message. +func (s *IncorrectKeyMaterialException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *IncorrectKeyMaterialException) OrigErr() error { + return nil +} + +func (s *IncorrectKeyMaterialException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *IncorrectKeyMaterialException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *IncorrectKeyMaterialException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the trust anchor certificate in the request +// is not the trust anchor certificate for the specified AWS CloudHSM cluster. +// +// When you initialize the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/initialize-cluster.html#sign-csr), +// you create the trust anchor certificate and save it in the customerCA.crt +// file. +type IncorrectTrustAnchorException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s IncorrectTrustAnchorException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s IncorrectTrustAnchorException) GoString() string { + return s.String() +} + +func newErrorIncorrectTrustAnchorException(v protocol.ResponseMetadata) error { + return &IncorrectTrustAnchorException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *IncorrectTrustAnchorException) Code() string { + return "IncorrectTrustAnchorException" +} + +// Message returns the exception's message. +func (s *IncorrectTrustAnchorException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *IncorrectTrustAnchorException) OrigErr() error { + return nil +} + +func (s *IncorrectTrustAnchorException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *IncorrectTrustAnchorException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *IncorrectTrustAnchorException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because an internal exception occurred. The request +// can be retried. +type InternalException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s InternalException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s InternalException) GoString() string { + return s.String() +} + +func newErrorInternalException(v protocol.ResponseMetadata) error { + return &InternalException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *InternalException) Code() string { + return "KMSInternalException" +} + +// Message returns the exception's message. +func (s *InternalException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *InternalException) OrigErr() error { + return nil +} + +func (s *InternalException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *InternalException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *InternalException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the specified alias name is not valid. +type InvalidAliasNameException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s InvalidAliasNameException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s InvalidAliasNameException) GoString() string { + return s.String() +} + +func newErrorInvalidAliasNameException(v protocol.ResponseMetadata) error { + return &InvalidAliasNameException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *InvalidAliasNameException) Code() string { + return "InvalidAliasNameException" +} + +// Message returns the exception's message. +func (s *InvalidAliasNameException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *InvalidAliasNameException) OrigErr() error { + return nil +} + +func (s *InvalidAliasNameException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *InvalidAliasNameException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *InvalidAliasNameException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because a specified ARN, or an ARN in a key policy, +// is not valid. +type InvalidArnException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s InvalidArnException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s InvalidArnException) GoString() string { + return s.String() +} + +func newErrorInvalidArnException(v protocol.ResponseMetadata) error { + return &InvalidArnException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *InvalidArnException) Code() string { + return "InvalidArnException" +} + +// Message returns the exception's message. +func (s *InvalidArnException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *InvalidArnException) OrigErr() error { + return nil +} + +func (s *InvalidArnException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *InvalidArnException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *InvalidArnException) RequestID() string { + return s.RespMetadata.RequestID +} + +// From the Decrypt or ReEncrypt operation, the request was rejected because +// the specified ciphertext, or additional authenticated data incorporated into +// the ciphertext, such as the encryption context, is corrupted, missing, or +// otherwise invalid. +// +// From the ImportKeyMaterial operation, the request was rejected because AWS +// KMS could not decrypt the encrypted (wrapped) key material. +type InvalidCiphertextException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s InvalidCiphertextException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s InvalidCiphertextException) GoString() string { + return s.String() +} + +func newErrorInvalidCiphertextException(v protocol.ResponseMetadata) error { + return &InvalidCiphertextException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *InvalidCiphertextException) Code() string { + return "InvalidCiphertextException" +} + +// Message returns the exception's message. +func (s *InvalidCiphertextException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *InvalidCiphertextException) OrigErr() error { + return nil +} + +func (s *InvalidCiphertextException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *InvalidCiphertextException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *InvalidCiphertextException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the specified GrantId is not valid. +type InvalidGrantIdException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s InvalidGrantIdException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s InvalidGrantIdException) GoString() string { + return s.String() +} + +func newErrorInvalidGrantIdException(v protocol.ResponseMetadata) error { + return &InvalidGrantIdException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *InvalidGrantIdException) Code() string { + return "InvalidGrantIdException" +} + +// Message returns the exception's message. +func (s *InvalidGrantIdException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *InvalidGrantIdException) OrigErr() error { + return nil +} + +func (s *InvalidGrantIdException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *InvalidGrantIdException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *InvalidGrantIdException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the specified grant token is not valid. +type InvalidGrantTokenException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s InvalidGrantTokenException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s InvalidGrantTokenException) GoString() string { + return s.String() +} + +func newErrorInvalidGrantTokenException(v protocol.ResponseMetadata) error { + return &InvalidGrantTokenException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *InvalidGrantTokenException) Code() string { + return "InvalidGrantTokenException" +} + +// Message returns the exception's message. +func (s *InvalidGrantTokenException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *InvalidGrantTokenException) OrigErr() error { + return nil +} + +func (s *InvalidGrantTokenException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *InvalidGrantTokenException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *InvalidGrantTokenException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the provided import token is invalid or +// is associated with a different customer master key (CMK). +type InvalidImportTokenException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s InvalidImportTokenException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s InvalidImportTokenException) GoString() string { + return s.String() +} + +func newErrorInvalidImportTokenException(v protocol.ResponseMetadata) error { + return &InvalidImportTokenException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *InvalidImportTokenException) Code() string { + return "InvalidImportTokenException" +} + +// Message returns the exception's message. +func (s *InvalidImportTokenException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *InvalidImportTokenException) OrigErr() error { + return nil +} + +func (s *InvalidImportTokenException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *InvalidImportTokenException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *InvalidImportTokenException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected for one of the following reasons: +// +// * The KeyUsage value of the CMK is incompatible with the API operation. +// +// * The encryption algorithm or signing algorithm specified for the operation +// is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). +// +// For encrypting, decrypting, re-encrypting, and generating data keys, the +// KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage +// must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. +// +// To find the encryption or signing algorithms supported for a particular CMK, +// use the DescribeKey operation. +type InvalidKeyUsageException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s InvalidKeyUsageException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s InvalidKeyUsageException) GoString() string { + return s.String() +} + +func newErrorInvalidKeyUsageException(v protocol.ResponseMetadata) error { + return &InvalidKeyUsageException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *InvalidKeyUsageException) Code() string { + return "InvalidKeyUsageException" +} + +// Message returns the exception's message. +func (s *InvalidKeyUsageException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *InvalidKeyUsageException) OrigErr() error { + return nil +} + +func (s *InvalidKeyUsageException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *InvalidKeyUsageException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *InvalidKeyUsageException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the marker that specifies where pagination +// should next begin is not valid. +type InvalidMarkerException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s InvalidMarkerException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s InvalidMarkerException) GoString() string { + return s.String() +} + +func newErrorInvalidMarkerException(v protocol.ResponseMetadata) error { + return &InvalidMarkerException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *InvalidMarkerException) Code() string { + return "InvalidMarkerException" +} + +// Message returns the exception's message. +func (s *InvalidMarkerException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *InvalidMarkerException) OrigErr() error { + return nil +} + +func (s *InvalidMarkerException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *InvalidMarkerException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *InvalidMarkerException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the state of the specified resource is not +// valid for this request. +// +// For more information about how key state affects the use of a CMK, see How +// Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) +// in the AWS Key Management Service Developer Guide . +type InvalidStateException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s InvalidStateException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s InvalidStateException) GoString() string { + return s.String() +} + +func newErrorInvalidStateException(v protocol.ResponseMetadata) error { + return &InvalidStateException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *InvalidStateException) Code() string { + return "KMSInvalidStateException" +} + +// Message returns the exception's message. +func (s *InvalidStateException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *InvalidStateException) OrigErr() error { + return nil +} + +func (s *InvalidStateException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *InvalidStateException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *InvalidStateException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the signature verification failed. Signature +// verification fails when it cannot confirm that signature was produced by +// signing the specified message with the specified CMK and signing algorithm. +type KMSInvalidSignatureException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s KMSInvalidSignatureException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s KMSInvalidSignatureException) GoString() string { + return s.String() +} + +func newErrorKMSInvalidSignatureException(v protocol.ResponseMetadata) error { + return &KMSInvalidSignatureException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *KMSInvalidSignatureException) Code() string { + return "KMSInvalidSignatureException" +} + +// Message returns the exception's message. +func (s *KMSInvalidSignatureException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *KMSInvalidSignatureException) OrigErr() error { + return nil +} + +func (s *KMSInvalidSignatureException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *KMSInvalidSignatureException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *KMSInvalidSignatureException) RequestID() string { + return s.RespMetadata.RequestID +} + +// Contains information about each entry in the key list. +type KeyListEntry struct { + _ struct{} `type:"structure"` + + // ARN of the key. + KeyArn *string `min:"20" type:"string"` + + // Unique identifier of the key. + KeyId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s KeyListEntry) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s KeyListEntry) GoString() string { + return s.String() +} + +// SetKeyArn sets the KeyArn field's value. +func (s *KeyListEntry) SetKeyArn(v string) *KeyListEntry { + s.KeyArn = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *KeyListEntry) SetKeyId(v string) *KeyListEntry { + s.KeyId = &v + return s +} + +// Contains metadata about a customer master key (CMK). +// +// This data type is used as a response element for the CreateKey and DescribeKey +// operations. +type KeyMetadata struct { + _ struct{} `type:"structure"` + + // The twelve-digit account ID of the AWS account that owns the CMK. + AWSAccountId *string `type:"string"` + + // The Amazon Resource Name (ARN) of the CMK. For examples, see AWS Key Management + // Service (AWS KMS) (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-kms) + // in the Example ARNs section of the AWS General Reference. + Arn *string `min:"20" type:"string"` + + // The cluster ID of the AWS CloudHSM cluster that contains the key material + // for the CMK. When you create a CMK in a custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html), + // AWS KMS creates the key material for the CMK in the associated AWS CloudHSM + // cluster. This value is present only when the CMK is created in a custom key + // store. + CloudHsmClusterId *string `min:"19" type:"string"` + + // The date and time when the CMK was created. + CreationDate *time.Time `type:"timestamp"` + + // A unique identifier for the custom key store (https://docs.aws.amazon.com/kms/latest/developerguide/custom-key-store-overview.html) + // that contains the CMK. This value is present only when the CMK is created + // in a custom key store. + CustomKeyStoreId *string `min:"1" type:"string"` + + // Describes the type of key material in the CMK. + CustomerMasterKeySpec *string `type:"string" enum:"CustomerMasterKeySpec"` + + // The date and time after which AWS KMS deletes the CMK. This value is present + // only when KeyState is PendingDeletion. + DeletionDate *time.Time `type:"timestamp"` + + // The description of the CMK. + Description *string `type:"string"` + + // Specifies whether the CMK is enabled. When KeyState is Enabled this value + // is true, otherwise it is false. + Enabled *bool `type:"boolean"` + + // The encryption algorithms that the CMK supports. You cannot use the CMK with + // other encryption algorithms within AWS KMS. + // + // This field appears only when the KeyUsage of the CMK is ENCRYPT_DECRYPT. + EncryptionAlgorithms []*string `type:"list"` + + // Specifies whether the CMK's key material expires. This value is present only + // when Origin is EXTERNAL, otherwise this value is omitted. + ExpirationModel *string `type:"string" enum:"ExpirationModelType"` + + // The globally unique identifier for the CMK. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // The manager of the CMK. CMKs in your AWS account are either customer managed + // or AWS managed. For more information about the difference, see Customer Master + // Keys (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#master_keys) + // in the AWS Key Management Service Developer Guide. + KeyManager *string `type:"string" enum:"KeyManagerType"` + + // The current status of the CMK. + // + // For more information about how key state affects the use of a CMK, see Key + // state: Effect on your CMK (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) + // in the AWS Key Management Service Developer Guide. + KeyState *string `type:"string" enum:"KeyState"` + + // The cryptographic operations (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#cryptographic-operations) + // for which you can use the CMK. + KeyUsage *string `type:"string" enum:"KeyUsageType"` + + // The source of the CMK's key material. When this value is AWS_KMS, AWS KMS + // created the key material. When this value is EXTERNAL, the key material was + // imported from your existing key management infrastructure or the CMK lacks + // key material. When this value is AWS_CLOUDHSM, the key material was created + // in the AWS CloudHSM cluster associated with a custom key store. + Origin *string `type:"string" enum:"OriginType"` + + // The signing algorithms that the CMK supports. You cannot use the CMK with + // other signing algorithms within AWS KMS. + // + // This field appears only when the KeyUsage of the CMK is SIGN_VERIFY. + SigningAlgorithms []*string `type:"list"` + + // The time at which the imported key material expires. When the key material + // expires, AWS KMS deletes the key material and the CMK becomes unusable. This + // value is present only for CMKs whose Origin is EXTERNAL and whose ExpirationModel + // is KEY_MATERIAL_EXPIRES, otherwise this value is omitted. + ValidTo *time.Time `type:"timestamp"` +} + +// String returns the string representation +func (s KeyMetadata) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s KeyMetadata) GoString() string { + return s.String() +} + +// SetAWSAccountId sets the AWSAccountId field's value. +func (s *KeyMetadata) SetAWSAccountId(v string) *KeyMetadata { + s.AWSAccountId = &v + return s +} + +// SetArn sets the Arn field's value. +func (s *KeyMetadata) SetArn(v string) *KeyMetadata { + s.Arn = &v + return s +} + +// SetCloudHsmClusterId sets the CloudHsmClusterId field's value. +func (s *KeyMetadata) SetCloudHsmClusterId(v string) *KeyMetadata { + s.CloudHsmClusterId = &v + return s +} + +// SetCreationDate sets the CreationDate field's value. +func (s *KeyMetadata) SetCreationDate(v time.Time) *KeyMetadata { + s.CreationDate = &v + return s +} + +// SetCustomKeyStoreId sets the CustomKeyStoreId field's value. +func (s *KeyMetadata) SetCustomKeyStoreId(v string) *KeyMetadata { + s.CustomKeyStoreId = &v + return s +} + +// SetCustomerMasterKeySpec sets the CustomerMasterKeySpec field's value. +func (s *KeyMetadata) SetCustomerMasterKeySpec(v string) *KeyMetadata { + s.CustomerMasterKeySpec = &v + return s +} + +// SetDeletionDate sets the DeletionDate field's value. +func (s *KeyMetadata) SetDeletionDate(v time.Time) *KeyMetadata { + s.DeletionDate = &v + return s +} + +// SetDescription sets the Description field's value. +func (s *KeyMetadata) SetDescription(v string) *KeyMetadata { + s.Description = &v + return s +} + +// SetEnabled sets the Enabled field's value. +func (s *KeyMetadata) SetEnabled(v bool) *KeyMetadata { + s.Enabled = &v + return s +} + +// SetEncryptionAlgorithms sets the EncryptionAlgorithms field's value. +func (s *KeyMetadata) SetEncryptionAlgorithms(v []*string) *KeyMetadata { + s.EncryptionAlgorithms = v + return s +} + +// SetExpirationModel sets the ExpirationModel field's value. +func (s *KeyMetadata) SetExpirationModel(v string) *KeyMetadata { + s.ExpirationModel = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *KeyMetadata) SetKeyId(v string) *KeyMetadata { + s.KeyId = &v + return s +} + +// SetKeyManager sets the KeyManager field's value. +func (s *KeyMetadata) SetKeyManager(v string) *KeyMetadata { + s.KeyManager = &v + return s +} + +// SetKeyState sets the KeyState field's value. +func (s *KeyMetadata) SetKeyState(v string) *KeyMetadata { + s.KeyState = &v + return s +} + +// SetKeyUsage sets the KeyUsage field's value. +func (s *KeyMetadata) SetKeyUsage(v string) *KeyMetadata { + s.KeyUsage = &v + return s +} + +// SetOrigin sets the Origin field's value. +func (s *KeyMetadata) SetOrigin(v string) *KeyMetadata { + s.Origin = &v + return s +} + +// SetSigningAlgorithms sets the SigningAlgorithms field's value. +func (s *KeyMetadata) SetSigningAlgorithms(v []*string) *KeyMetadata { + s.SigningAlgorithms = v + return s +} + +// SetValidTo sets the ValidTo field's value. +func (s *KeyMetadata) SetValidTo(v time.Time) *KeyMetadata { + s.ValidTo = &v + return s +} + +// The request was rejected because the specified CMK was not available. You +// can retry the request. +type KeyUnavailableException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s KeyUnavailableException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s KeyUnavailableException) GoString() string { + return s.String() +} + +func newErrorKeyUnavailableException(v protocol.ResponseMetadata) error { + return &KeyUnavailableException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *KeyUnavailableException) Code() string { + return "KeyUnavailableException" +} + +// Message returns the exception's message. +func (s *KeyUnavailableException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *KeyUnavailableException) OrigErr() error { + return nil +} + +func (s *KeyUnavailableException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *KeyUnavailableException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *KeyUnavailableException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because a quota was exceeded. For more information, +// see Quotas (https://docs.aws.amazon.com/kms/latest/developerguide/limits.html) +// in the AWS Key Management Service Developer Guide. +type LimitExceededException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s LimitExceededException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s LimitExceededException) GoString() string { + return s.String() +} + +func newErrorLimitExceededException(v protocol.ResponseMetadata) error { + return &LimitExceededException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *LimitExceededException) Code() string { + return "LimitExceededException" +} + +// Message returns the exception's message. +func (s *LimitExceededException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *LimitExceededException) OrigErr() error { + return nil +} + +func (s *LimitExceededException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *LimitExceededException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *LimitExceededException) RequestID() string { + return s.RespMetadata.RequestID +} + +type ListAliasesInput struct { + _ struct{} `type:"structure"` + + // Lists only aliases that refer to the specified CMK. The value of this parameter + // can be the ID or Amazon Resource Name (ARN) of a CMK in the caller's account + // and region. You cannot use an alias name or alias ARN in this value. + // + // This parameter is optional. If you omit it, ListAliases returns all aliases + // in the account and region. + KeyId *string `min:"1" type:"string"` + + // Use this parameter to specify the maximum number of items to return. When + // this value is present, AWS KMS does not return more than the specified number + // of items, but it might return fewer. + // + // This value is optional. If you include a value, it must be between 1 and + // 100, inclusive. If you do not include a value, it defaults to 50. + Limit *int64 `min:"1" type:"integer"` + + // Use this parameter in a subsequent request after you receive a response with + // truncated results. Set it to the value of NextMarker from the truncated response + // you just received. + Marker *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s ListAliasesInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListAliasesInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ListAliasesInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ListAliasesInput"} + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.Limit != nil && *s.Limit < 1 { + invalidParams.Add(request.NewErrParamMinValue("Limit", 1)) + } + if s.Marker != nil && len(*s.Marker) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Marker", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *ListAliasesInput) SetKeyId(v string) *ListAliasesInput { + s.KeyId = &v + return s +} + +// SetLimit sets the Limit field's value. +func (s *ListAliasesInput) SetLimit(v int64) *ListAliasesInput { + s.Limit = &v + return s +} + +// SetMarker sets the Marker field's value. +func (s *ListAliasesInput) SetMarker(v string) *ListAliasesInput { + s.Marker = &v + return s +} + +type ListAliasesOutput struct { + _ struct{} `type:"structure"` + + // A list of aliases. + Aliases []*AliasListEntry `type:"list"` + + // When Truncated is true, this element is present and contains the value to + // use for the Marker parameter in a subsequent request. + NextMarker *string `min:"1" type:"string"` + + // A flag that indicates whether there are more items in the list. When this + // value is true, the list in this response is truncated. To get more items, + // pass the value of the NextMarker element in thisresponse to the Marker parameter + // in a subsequent request. + Truncated *bool `type:"boolean"` +} + +// String returns the string representation +func (s ListAliasesOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListAliasesOutput) GoString() string { + return s.String() +} + +// SetAliases sets the Aliases field's value. +func (s *ListAliasesOutput) SetAliases(v []*AliasListEntry) *ListAliasesOutput { + s.Aliases = v + return s +} + +// SetNextMarker sets the NextMarker field's value. +func (s *ListAliasesOutput) SetNextMarker(v string) *ListAliasesOutput { + s.NextMarker = &v + return s +} + +// SetTruncated sets the Truncated field's value. +func (s *ListAliasesOutput) SetTruncated(v bool) *ListAliasesOutput { + s.Truncated = &v + return s +} + +type ListGrantsInput struct { + _ struct{} `type:"structure"` + + // A unique identifier for the customer master key (CMK). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. To specify + // a CMK in a different AWS account, you must use the key ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // Use this parameter to specify the maximum number of items to return. When + // this value is present, AWS KMS does not return more than the specified number + // of items, but it might return fewer. + // + // This value is optional. If you include a value, it must be between 1 and + // 100, inclusive. If you do not include a value, it defaults to 50. + Limit *int64 `min:"1" type:"integer"` + + // Use this parameter in a subsequent request after you receive a response with + // truncated results. Set it to the value of NextMarker from the truncated response + // you just received. + Marker *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s ListGrantsInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListGrantsInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ListGrantsInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ListGrantsInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.Limit != nil && *s.Limit < 1 { + invalidParams.Add(request.NewErrParamMinValue("Limit", 1)) + } + if s.Marker != nil && len(*s.Marker) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Marker", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *ListGrantsInput) SetKeyId(v string) *ListGrantsInput { + s.KeyId = &v + return s +} + +// SetLimit sets the Limit field's value. +func (s *ListGrantsInput) SetLimit(v int64) *ListGrantsInput { + s.Limit = &v + return s +} + +// SetMarker sets the Marker field's value. +func (s *ListGrantsInput) SetMarker(v string) *ListGrantsInput { + s.Marker = &v + return s +} + +type ListGrantsResponse struct { + _ struct{} `type:"structure"` + + // A list of grants. + Grants []*GrantListEntry `type:"list"` + + // When Truncated is true, this element is present and contains the value to + // use for the Marker parameter in a subsequent request. + NextMarker *string `min:"1" type:"string"` + + // A flag that indicates whether there are more items in the list. When this + // value is true, the list in this response is truncated. To get more items, + // pass the value of the NextMarker element in thisresponse to the Marker parameter + // in a subsequent request. + Truncated *bool `type:"boolean"` +} + +// String returns the string representation +func (s ListGrantsResponse) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListGrantsResponse) GoString() string { + return s.String() +} + +// SetGrants sets the Grants field's value. +func (s *ListGrantsResponse) SetGrants(v []*GrantListEntry) *ListGrantsResponse { + s.Grants = v + return s +} + +// SetNextMarker sets the NextMarker field's value. +func (s *ListGrantsResponse) SetNextMarker(v string) *ListGrantsResponse { + s.NextMarker = &v + return s +} + +// SetTruncated sets the Truncated field's value. +func (s *ListGrantsResponse) SetTruncated(v bool) *ListGrantsResponse { + s.Truncated = &v + return s +} + +type ListKeyPoliciesInput struct { + _ struct{} `type:"structure"` + + // A unique identifier for the customer master key (CMK). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // Use this parameter to specify the maximum number of items to return. When + // this value is present, AWS KMS does not return more than the specified number + // of items, but it might return fewer. + // + // This value is optional. If you include a value, it must be between 1 and + // 1000, inclusive. If you do not include a value, it defaults to 100. + // + // Only one policy can be attached to a key. + Limit *int64 `min:"1" type:"integer"` + + // Use this parameter in a subsequent request after you receive a response with + // truncated results. Set it to the value of NextMarker from the truncated response + // you just received. + Marker *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s ListKeyPoliciesInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListKeyPoliciesInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ListKeyPoliciesInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ListKeyPoliciesInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.Limit != nil && *s.Limit < 1 { + invalidParams.Add(request.NewErrParamMinValue("Limit", 1)) + } + if s.Marker != nil && len(*s.Marker) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Marker", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *ListKeyPoliciesInput) SetKeyId(v string) *ListKeyPoliciesInput { + s.KeyId = &v + return s +} + +// SetLimit sets the Limit field's value. +func (s *ListKeyPoliciesInput) SetLimit(v int64) *ListKeyPoliciesInput { + s.Limit = &v + return s +} + +// SetMarker sets the Marker field's value. +func (s *ListKeyPoliciesInput) SetMarker(v string) *ListKeyPoliciesInput { + s.Marker = &v + return s +} + +type ListKeyPoliciesOutput struct { + _ struct{} `type:"structure"` + + // When Truncated is true, this element is present and contains the value to + // use for the Marker parameter in a subsequent request. + NextMarker *string `min:"1" type:"string"` + + // A list of key policy names. The only valid value is default. + PolicyNames []*string `type:"list"` + + // A flag that indicates whether there are more items in the list. When this + // value is true, the list in this response is truncated. To get more items, + // pass the value of the NextMarker element in thisresponse to the Marker parameter + // in a subsequent request. + Truncated *bool `type:"boolean"` +} + +// String returns the string representation +func (s ListKeyPoliciesOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListKeyPoliciesOutput) GoString() string { + return s.String() +} + +// SetNextMarker sets the NextMarker field's value. +func (s *ListKeyPoliciesOutput) SetNextMarker(v string) *ListKeyPoliciesOutput { + s.NextMarker = &v + return s +} + +// SetPolicyNames sets the PolicyNames field's value. +func (s *ListKeyPoliciesOutput) SetPolicyNames(v []*string) *ListKeyPoliciesOutput { + s.PolicyNames = v + return s +} + +// SetTruncated sets the Truncated field's value. +func (s *ListKeyPoliciesOutput) SetTruncated(v bool) *ListKeyPoliciesOutput { + s.Truncated = &v + return s +} + +type ListKeysInput struct { + _ struct{} `type:"structure"` + + // Use this parameter to specify the maximum number of items to return. When + // this value is present, AWS KMS does not return more than the specified number + // of items, but it might return fewer. + // + // This value is optional. If you include a value, it must be between 1 and + // 1000, inclusive. If you do not include a value, it defaults to 100. + Limit *int64 `min:"1" type:"integer"` + + // Use this parameter in a subsequent request after you receive a response with + // truncated results. Set it to the value of NextMarker from the truncated response + // you just received. + Marker *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s ListKeysInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListKeysInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ListKeysInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ListKeysInput"} + if s.Limit != nil && *s.Limit < 1 { + invalidParams.Add(request.NewErrParamMinValue("Limit", 1)) + } + if s.Marker != nil && len(*s.Marker) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Marker", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetLimit sets the Limit field's value. +func (s *ListKeysInput) SetLimit(v int64) *ListKeysInput { + s.Limit = &v + return s +} + +// SetMarker sets the Marker field's value. +func (s *ListKeysInput) SetMarker(v string) *ListKeysInput { + s.Marker = &v + return s +} + +type ListKeysOutput struct { + _ struct{} `type:"structure"` + + // A list of customer master keys (CMKs). + Keys []*KeyListEntry `type:"list"` + + // When Truncated is true, this element is present and contains the value to + // use for the Marker parameter in a subsequent request. + NextMarker *string `min:"1" type:"string"` + + // A flag that indicates whether there are more items in the list. When this + // value is true, the list in this response is truncated. To get more items, + // pass the value of the NextMarker element in thisresponse to the Marker parameter + // in a subsequent request. + Truncated *bool `type:"boolean"` +} + +// String returns the string representation +func (s ListKeysOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListKeysOutput) GoString() string { + return s.String() +} + +// SetKeys sets the Keys field's value. +func (s *ListKeysOutput) SetKeys(v []*KeyListEntry) *ListKeysOutput { + s.Keys = v + return s +} + +// SetNextMarker sets the NextMarker field's value. +func (s *ListKeysOutput) SetNextMarker(v string) *ListKeysOutput { + s.NextMarker = &v + return s +} + +// SetTruncated sets the Truncated field's value. +func (s *ListKeysOutput) SetTruncated(v bool) *ListKeysOutput { + s.Truncated = &v + return s +} + +type ListResourceTagsInput struct { + _ struct{} `type:"structure"` + + // A unique identifier for the customer master key (CMK). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // Use this parameter to specify the maximum number of items to return. When + // this value is present, AWS KMS does not return more than the specified number + // of items, but it might return fewer. + // + // This value is optional. If you include a value, it must be between 1 and + // 50, inclusive. If you do not include a value, it defaults to 50. + Limit *int64 `min:"1" type:"integer"` + + // Use this parameter in a subsequent request after you receive a response with + // truncated results. Set it to the value of NextMarker from the truncated response + // you just received. + // + // Do not attempt to construct this value. Use only the value of NextMarker + // from the truncated response you just received. + Marker *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s ListResourceTagsInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListResourceTagsInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ListResourceTagsInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ListResourceTagsInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.Limit != nil && *s.Limit < 1 { + invalidParams.Add(request.NewErrParamMinValue("Limit", 1)) + } + if s.Marker != nil && len(*s.Marker) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Marker", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *ListResourceTagsInput) SetKeyId(v string) *ListResourceTagsInput { + s.KeyId = &v + return s +} + +// SetLimit sets the Limit field's value. +func (s *ListResourceTagsInput) SetLimit(v int64) *ListResourceTagsInput { + s.Limit = &v + return s +} + +// SetMarker sets the Marker field's value. +func (s *ListResourceTagsInput) SetMarker(v string) *ListResourceTagsInput { + s.Marker = &v + return s +} + +type ListResourceTagsOutput struct { + _ struct{} `type:"structure"` + + // When Truncated is true, this element is present and contains the value to + // use for the Marker parameter in a subsequent request. + // + // Do not assume or infer any information from this value. + NextMarker *string `min:"1" type:"string"` + + // A list of tags. Each tag consists of a tag key and a tag value. + Tags []*Tag `type:"list"` + + // A flag that indicates whether there are more items in the list. When this + // value is true, the list in this response is truncated. To get more items, + // pass the value of the NextMarker element in thisresponse to the Marker parameter + // in a subsequent request. + Truncated *bool `type:"boolean"` +} + +// String returns the string representation +func (s ListResourceTagsOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListResourceTagsOutput) GoString() string { + return s.String() +} + +// SetNextMarker sets the NextMarker field's value. +func (s *ListResourceTagsOutput) SetNextMarker(v string) *ListResourceTagsOutput { + s.NextMarker = &v + return s +} + +// SetTags sets the Tags field's value. +func (s *ListResourceTagsOutput) SetTags(v []*Tag) *ListResourceTagsOutput { + s.Tags = v + return s +} + +// SetTruncated sets the Truncated field's value. +func (s *ListResourceTagsOutput) SetTruncated(v bool) *ListResourceTagsOutput { + s.Truncated = &v + return s +} + +type ListRetirableGrantsInput struct { + _ struct{} `type:"structure"` + + // Use this parameter to specify the maximum number of items to return. When + // this value is present, AWS KMS does not return more than the specified number + // of items, but it might return fewer. + // + // This value is optional. If you include a value, it must be between 1 and + // 100, inclusive. If you do not include a value, it defaults to 50. + Limit *int64 `min:"1" type:"integer"` + + // Use this parameter in a subsequent request after you receive a response with + // truncated results. Set it to the value of NextMarker from the truncated response + // you just received. + Marker *string `min:"1" type:"string"` + + // The retiring principal for which to list grants. + // + // To specify the retiring principal, use the Amazon Resource Name (ARN) (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) + // of an AWS principal. Valid AWS principals include AWS accounts (root), IAM + // users, federated users, and assumed role users. For examples of the ARN syntax + // for specifying a principal, see AWS Identity and Access Management (IAM) + // (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html#arn-syntax-iam) + // in the Example ARNs section of the Amazon Web Services General Reference. + // + // RetiringPrincipal is a required field + RetiringPrincipal *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s ListRetirableGrantsInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ListRetirableGrantsInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ListRetirableGrantsInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ListRetirableGrantsInput"} + if s.Limit != nil && *s.Limit < 1 { + invalidParams.Add(request.NewErrParamMinValue("Limit", 1)) + } + if s.Marker != nil && len(*s.Marker) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Marker", 1)) + } + if s.RetiringPrincipal == nil { + invalidParams.Add(request.NewErrParamRequired("RetiringPrincipal")) + } + if s.RetiringPrincipal != nil && len(*s.RetiringPrincipal) < 1 { + invalidParams.Add(request.NewErrParamMinLen("RetiringPrincipal", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetLimit sets the Limit field's value. +func (s *ListRetirableGrantsInput) SetLimit(v int64) *ListRetirableGrantsInput { + s.Limit = &v + return s +} + +// SetMarker sets the Marker field's value. +func (s *ListRetirableGrantsInput) SetMarker(v string) *ListRetirableGrantsInput { + s.Marker = &v + return s +} + +// SetRetiringPrincipal sets the RetiringPrincipal field's value. +func (s *ListRetirableGrantsInput) SetRetiringPrincipal(v string) *ListRetirableGrantsInput { + s.RetiringPrincipal = &v + return s +} + +// The request was rejected because the specified policy is not syntactically +// or semantically correct. +type MalformedPolicyDocumentException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s MalformedPolicyDocumentException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s MalformedPolicyDocumentException) GoString() string { + return s.String() +} + +func newErrorMalformedPolicyDocumentException(v protocol.ResponseMetadata) error { + return &MalformedPolicyDocumentException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *MalformedPolicyDocumentException) Code() string { + return "MalformedPolicyDocumentException" +} + +// Message returns the exception's message. +func (s *MalformedPolicyDocumentException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *MalformedPolicyDocumentException) OrigErr() error { + return nil +} + +func (s *MalformedPolicyDocumentException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *MalformedPolicyDocumentException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *MalformedPolicyDocumentException) RequestID() string { + return s.RespMetadata.RequestID +} + +// The request was rejected because the specified entity or resource could not +// be found. +type NotFoundException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s NotFoundException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s NotFoundException) GoString() string { + return s.String() +} + +func newErrorNotFoundException(v protocol.ResponseMetadata) error { + return &NotFoundException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *NotFoundException) Code() string { + return "NotFoundException" +} + +// Message returns the exception's message. +func (s *NotFoundException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *NotFoundException) OrigErr() error { + return nil +} + +func (s *NotFoundException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *NotFoundException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *NotFoundException) RequestID() string { + return s.RespMetadata.RequestID +} + +type PutKeyPolicyInput struct { + _ struct{} `type:"structure"` + + // A flag to indicate whether to bypass the key policy lockout safety check. + // + // Setting this value to true increases the risk that the CMK becomes unmanageable. + // Do not set this value to true indiscriminately. + // + // For more information, refer to the scenario in the Default Key Policy (https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html#key-policy-default-allow-root-enable-iam) + // section in the AWS Key Management Service Developer Guide. + // + // Use this parameter only when you intend to prevent the principal that is + // making the request from making a subsequent PutKeyPolicy request on the CMK. + // + // The default value is false. + BypassPolicyLockoutSafetyCheck *bool `type:"boolean"` + + // A unique identifier for the customer master key (CMK). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // The key policy to attach to the CMK. + // + // The key policy must meet the following criteria: + // + // * If you don't set BypassPolicyLockoutSafetyCheck to true, the key policy + // must allow the principal that is making the PutKeyPolicy request to make + // a subsequent PutKeyPolicy request on the CMK. This reduces the risk that + // the CMK becomes unmanageable. For more information, refer to the scenario + // in the Default Key Policy (https://docs.aws.amazon.com/kms/latest/developerguide/key-policies.html#key-policy-default-allow-root-enable-iam) + // section of the AWS Key Management Service Developer Guide. + // + // * Each statement in the key policy must contain one or more principals. + // The principals in the key policy must exist and be visible to AWS KMS. + // When you create a new AWS principal (for example, an IAM user or role), + // you might need to enforce a delay before including the new principal in + // a key policy because the new principal might not be immediately visible + // to AWS KMS. For more information, see Changes that I make are not always + // immediately visible (https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_general.html#troubleshoot_general_eventual-consistency) + // in the AWS Identity and Access Management User Guide. + // + // The key policy cannot exceed 32 kilobytes (32768 bytes). For more information, + // see Resource Quotas (https://docs.aws.amazon.com/kms/latest/developerguide/resource-limits.html) + // in the AWS Key Management Service Developer Guide. + // + // Policy is a required field + Policy *string `min:"1" type:"string" required:"true"` + + // The name of the key policy. The only valid value is default. + // + // PolicyName is a required field + PolicyName *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s PutKeyPolicyInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s PutKeyPolicyInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *PutKeyPolicyInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "PutKeyPolicyInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.Policy == nil { + invalidParams.Add(request.NewErrParamRequired("Policy")) + } + if s.Policy != nil && len(*s.Policy) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Policy", 1)) + } + if s.PolicyName == nil { + invalidParams.Add(request.NewErrParamRequired("PolicyName")) + } + if s.PolicyName != nil && len(*s.PolicyName) < 1 { + invalidParams.Add(request.NewErrParamMinLen("PolicyName", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetBypassPolicyLockoutSafetyCheck sets the BypassPolicyLockoutSafetyCheck field's value. +func (s *PutKeyPolicyInput) SetBypassPolicyLockoutSafetyCheck(v bool) *PutKeyPolicyInput { + s.BypassPolicyLockoutSafetyCheck = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *PutKeyPolicyInput) SetKeyId(v string) *PutKeyPolicyInput { + s.KeyId = &v + return s +} + +// SetPolicy sets the Policy field's value. +func (s *PutKeyPolicyInput) SetPolicy(v string) *PutKeyPolicyInput { + s.Policy = &v + return s +} + +// SetPolicyName sets the PolicyName field's value. +func (s *PutKeyPolicyInput) SetPolicyName(v string) *PutKeyPolicyInput { + s.PolicyName = &v + return s +} + +type PutKeyPolicyOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s PutKeyPolicyOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s PutKeyPolicyOutput) GoString() string { + return s.String() +} + +type ReEncryptInput struct { + _ struct{} `type:"structure"` + + // Ciphertext of the data to reencrypt. + // + // CiphertextBlob is automatically base64 encoded/decoded by the SDK. + // + // CiphertextBlob is a required field + CiphertextBlob []byte `min:"1" type:"blob" required:"true"` + + // Specifies the encryption algorithm that AWS KMS will use to reecrypt the + // data after it has decrypted it. The default value, SYMMETRIC_DEFAULT, represents + // the encryption algorithm used for symmetric CMKs. + // + // This parameter is required only when the destination CMK is an asymmetric + // CMK. + DestinationEncryptionAlgorithm *string `type:"string" enum:"EncryptionAlgorithmSpec"` + + // Specifies that encryption context to use when the reencrypting the data. + // + // A destination encryption context is valid only when the destination CMK is + // a symmetric CMK. The standard ciphertext format for asymmetric CMKs does + // not include fields for metadata. + // + // An encryption context is a collection of non-secret key-value pairs that + // represents additional authenticated data. When you use an encryption context + // to encrypt data, you must specify the same (an exact case-sensitive match) + // encryption context to decrypt the data. An encryption context is optional + // when encrypting with a symmetric CMK, but it is highly recommended. + // + // For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) + // in the AWS Key Management Service Developer Guide. + DestinationEncryptionContext map[string]*string `type:"map"` + + // A unique identifier for the CMK that is used to reencrypt the data. Specify + // a symmetric or asymmetric CMK with a KeyUsage value of ENCRYPT_DECRYPT. To + // find the KeyUsage value of a CMK, use the DescribeKey operation. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". To specify + // a CMK in a different AWS account, you must use the key ARN or alias ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + // + // DestinationKeyId is a required field + DestinationKeyId *string `min:"1" type:"string" required:"true"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // Specifies the encryption algorithm that AWS KMS will use to decrypt the ciphertext + // before it is reencrypted. The default value, SYMMETRIC_DEFAULT, represents + // the algorithm used for symmetric CMKs. + // + // Specify the same algorithm that was used to encrypt the ciphertext. If you + // specify a different algorithm, the decrypt attempt fails. + // + // This parameter is required only when the ciphertext was encrypted under an + // asymmetric CMK. + SourceEncryptionAlgorithm *string `type:"string" enum:"EncryptionAlgorithmSpec"` + + // Specifies the encryption context to use to decrypt the ciphertext. Enter + // the same encryption context that was used to encrypt the ciphertext. + // + // An encryption context is a collection of non-secret key-value pairs that + // represents additional authenticated data. When you use an encryption context + // to encrypt data, you must specify the same (an exact case-sensitive match) + // encryption context to decrypt the data. An encryption context is optional + // when encrypting with a symmetric CMK, but it is highly recommended. + // + // For more information, see Encryption Context (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#encrypt_context) + // in the AWS Key Management Service Developer Guide. + SourceEncryptionContext map[string]*string `type:"map"` + + // A unique identifier for the CMK that is used to decrypt the ciphertext before + // it reencrypts it using the destination CMK. + // + // This parameter is required only when the ciphertext was encrypted under an + // asymmetric CMK. Otherwise, AWS KMS uses the metadata that it adds to the + // ciphertext blob to determine which CMK was used to encrypt the ciphertext. + // However, you can use this parameter to ensure that a particular CMK (of any + // kind) is used to decrypt the ciphertext before it is reencrypted. + // + // If you specify a KeyId value, the decrypt part of the ReEncrypt operation + // succeeds only if the specified CMK was used to encrypt the ciphertext. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + SourceKeyId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s ReEncryptInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ReEncryptInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ReEncryptInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ReEncryptInput"} + if s.CiphertextBlob == nil { + invalidParams.Add(request.NewErrParamRequired("CiphertextBlob")) + } + if s.CiphertextBlob != nil && len(s.CiphertextBlob) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CiphertextBlob", 1)) + } + if s.DestinationKeyId == nil { + invalidParams.Add(request.NewErrParamRequired("DestinationKeyId")) + } + if s.DestinationKeyId != nil && len(*s.DestinationKeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("DestinationKeyId", 1)) + } + if s.SourceKeyId != nil && len(*s.SourceKeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("SourceKeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetCiphertextBlob sets the CiphertextBlob field's value. +func (s *ReEncryptInput) SetCiphertextBlob(v []byte) *ReEncryptInput { + s.CiphertextBlob = v + return s +} + +// SetDestinationEncryptionAlgorithm sets the DestinationEncryptionAlgorithm field's value. +func (s *ReEncryptInput) SetDestinationEncryptionAlgorithm(v string) *ReEncryptInput { + s.DestinationEncryptionAlgorithm = &v + return s +} + +// SetDestinationEncryptionContext sets the DestinationEncryptionContext field's value. +func (s *ReEncryptInput) SetDestinationEncryptionContext(v map[string]*string) *ReEncryptInput { + s.DestinationEncryptionContext = v + return s +} + +// SetDestinationKeyId sets the DestinationKeyId field's value. +func (s *ReEncryptInput) SetDestinationKeyId(v string) *ReEncryptInput { + s.DestinationKeyId = &v + return s +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *ReEncryptInput) SetGrantTokens(v []*string) *ReEncryptInput { + s.GrantTokens = v + return s +} + +// SetSourceEncryptionAlgorithm sets the SourceEncryptionAlgorithm field's value. +func (s *ReEncryptInput) SetSourceEncryptionAlgorithm(v string) *ReEncryptInput { + s.SourceEncryptionAlgorithm = &v + return s +} + +// SetSourceEncryptionContext sets the SourceEncryptionContext field's value. +func (s *ReEncryptInput) SetSourceEncryptionContext(v map[string]*string) *ReEncryptInput { + s.SourceEncryptionContext = v + return s +} + +// SetSourceKeyId sets the SourceKeyId field's value. +func (s *ReEncryptInput) SetSourceKeyId(v string) *ReEncryptInput { + s.SourceKeyId = &v + return s +} + +type ReEncryptOutput struct { + _ struct{} `type:"structure"` + + // The reencrypted data. When you use the HTTP API or the AWS CLI, the value + // is Base64-encoded. Otherwise, it is not Base64-encoded. + // + // CiphertextBlob is automatically base64 encoded/decoded by the SDK. + CiphertextBlob []byte `min:"1" type:"blob"` + + // The encryption algorithm that was used to reencrypt the data. + DestinationEncryptionAlgorithm *string `type:"string" enum:"EncryptionAlgorithmSpec"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the CMK that was used to reencrypt the data. + KeyId *string `min:"1" type:"string"` + + // The encryption algorithm that was used to decrypt the ciphertext before it + // was reencrypted. + SourceEncryptionAlgorithm *string `type:"string" enum:"EncryptionAlgorithmSpec"` + + // Unique identifier of the CMK used to originally encrypt the data. + SourceKeyId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s ReEncryptOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ReEncryptOutput) GoString() string { + return s.String() +} + +// SetCiphertextBlob sets the CiphertextBlob field's value. +func (s *ReEncryptOutput) SetCiphertextBlob(v []byte) *ReEncryptOutput { + s.CiphertextBlob = v + return s +} + +// SetDestinationEncryptionAlgorithm sets the DestinationEncryptionAlgorithm field's value. +func (s *ReEncryptOutput) SetDestinationEncryptionAlgorithm(v string) *ReEncryptOutput { + s.DestinationEncryptionAlgorithm = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *ReEncryptOutput) SetKeyId(v string) *ReEncryptOutput { + s.KeyId = &v + return s +} + +// SetSourceEncryptionAlgorithm sets the SourceEncryptionAlgorithm field's value. +func (s *ReEncryptOutput) SetSourceEncryptionAlgorithm(v string) *ReEncryptOutput { + s.SourceEncryptionAlgorithm = &v + return s +} + +// SetSourceKeyId sets the SourceKeyId field's value. +func (s *ReEncryptOutput) SetSourceKeyId(v string) *ReEncryptOutput { + s.SourceKeyId = &v + return s +} + +type RetireGrantInput struct { + _ struct{} `type:"structure"` + + // Unique identifier of the grant to retire. The grant ID is returned in the + // response to a CreateGrant operation. + // + // * Grant ID Example - 0123456789012345678901234567890123456789012345678901234567890123 + GrantId *string `min:"1" type:"string"` + + // Token that identifies the grant to be retired. + GrantToken *string `min:"1" type:"string"` + + // The Amazon Resource Name (ARN) of the CMK associated with the grant. + // + // For example: arn:aws:kms:us-east-2:444455556666:key/1234abcd-12ab-34cd-56ef-1234567890ab + KeyId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s RetireGrantInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s RetireGrantInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *RetireGrantInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "RetireGrantInput"} + if s.GrantId != nil && len(*s.GrantId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("GrantId", 1)) + } + if s.GrantToken != nil && len(*s.GrantToken) < 1 { + invalidParams.Add(request.NewErrParamMinLen("GrantToken", 1)) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetGrantId sets the GrantId field's value. +func (s *RetireGrantInput) SetGrantId(v string) *RetireGrantInput { + s.GrantId = &v + return s +} + +// SetGrantToken sets the GrantToken field's value. +func (s *RetireGrantInput) SetGrantToken(v string) *RetireGrantInput { + s.GrantToken = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *RetireGrantInput) SetKeyId(v string) *RetireGrantInput { + s.KeyId = &v + return s +} + +type RetireGrantOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s RetireGrantOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s RetireGrantOutput) GoString() string { + return s.String() +} + +type RevokeGrantInput struct { + _ struct{} `type:"structure"` + + // Identifier of the grant to be revoked. + // + // GrantId is a required field + GrantId *string `min:"1" type:"string" required:"true"` + + // A unique identifier for the customer master key associated with the grant. + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. To specify + // a CMK in a different AWS account, you must use the key ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s RevokeGrantInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s RevokeGrantInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *RevokeGrantInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "RevokeGrantInput"} + if s.GrantId == nil { + invalidParams.Add(request.NewErrParamRequired("GrantId")) + } + if s.GrantId != nil && len(*s.GrantId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("GrantId", 1)) + } + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetGrantId sets the GrantId field's value. +func (s *RevokeGrantInput) SetGrantId(v string) *RevokeGrantInput { + s.GrantId = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *RevokeGrantInput) SetKeyId(v string) *RevokeGrantInput { + s.KeyId = &v + return s +} + +type RevokeGrantOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s RevokeGrantOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s RevokeGrantOutput) GoString() string { + return s.String() +} + +type ScheduleKeyDeletionInput struct { + _ struct{} `type:"structure"` + + // The unique identifier of the customer master key (CMK) to delete. + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // The waiting period, specified in number of days. After the waiting period + // ends, AWS KMS deletes the customer master key (CMK). + // + // This value is optional. If you include a value, it must be between 7 and + // 30, inclusive. If you do not include a value, it defaults to 30. + PendingWindowInDays *int64 `min:"1" type:"integer"` +} + +// String returns the string representation +func (s ScheduleKeyDeletionInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ScheduleKeyDeletionInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ScheduleKeyDeletionInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ScheduleKeyDeletionInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.PendingWindowInDays != nil && *s.PendingWindowInDays < 1 { + invalidParams.Add(request.NewErrParamMinValue("PendingWindowInDays", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *ScheduleKeyDeletionInput) SetKeyId(v string) *ScheduleKeyDeletionInput { + s.KeyId = &v + return s +} + +// SetPendingWindowInDays sets the PendingWindowInDays field's value. +func (s *ScheduleKeyDeletionInput) SetPendingWindowInDays(v int64) *ScheduleKeyDeletionInput { + s.PendingWindowInDays = &v + return s +} + +type ScheduleKeyDeletionOutput struct { + _ struct{} `type:"structure"` + + // The date and time after which AWS KMS deletes the customer master key (CMK). + DeletionDate *time.Time `type:"timestamp"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the CMK whose deletion is scheduled. + KeyId *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s ScheduleKeyDeletionOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ScheduleKeyDeletionOutput) GoString() string { + return s.String() +} + +// SetDeletionDate sets the DeletionDate field's value. +func (s *ScheduleKeyDeletionOutput) SetDeletionDate(v time.Time) *ScheduleKeyDeletionOutput { + s.DeletionDate = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *ScheduleKeyDeletionOutput) SetKeyId(v string) *ScheduleKeyDeletionOutput { + s.KeyId = &v + return s +} + +type SignInput struct { + _ struct{} `type:"structure"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // Identifies an asymmetric CMK. AWS KMS uses the private key in the asymmetric + // CMK to sign the message. The KeyUsage type of the CMK must be SIGN_VERIFY. + // To find the KeyUsage of a CMK, use the DescribeKey operation. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". To specify + // a CMK in a different AWS account, you must use the key ARN or alias ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // Specifies the message or message digest to sign. Messages can be 0-4096 bytes. + // To sign a larger message, provide the message digest. + // + // If you provide a message, AWS KMS generates a hash digest of the message + // and then signs it. + // + // Message is automatically base64 encoded/decoded by the SDK. + // + // Message is a required field + Message []byte `min:"1" type:"blob" required:"true" sensitive:"true"` + + // Tells AWS KMS whether the value of the Message parameter is a message or + // message digest. The default value, RAW, indicates a message. To indicate + // a message digest, enter DIGEST. + MessageType *string `type:"string" enum:"MessageType"` + + // Specifies the signing algorithm to use when signing the message. + // + // Choose an algorithm that is compatible with the type and size of the specified + // asymmetric CMK. + // + // SigningAlgorithm is a required field + SigningAlgorithm *string `type:"string" required:"true" enum:"SigningAlgorithmSpec"` +} + +// String returns the string representation +func (s SignInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s SignInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *SignInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "SignInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.Message == nil { + invalidParams.Add(request.NewErrParamRequired("Message")) + } + if s.Message != nil && len(s.Message) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Message", 1)) + } + if s.SigningAlgorithm == nil { + invalidParams.Add(request.NewErrParamRequired("SigningAlgorithm")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *SignInput) SetGrantTokens(v []*string) *SignInput { + s.GrantTokens = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *SignInput) SetKeyId(v string) *SignInput { + s.KeyId = &v + return s +} + +// SetMessage sets the Message field's value. +func (s *SignInput) SetMessage(v []byte) *SignInput { + s.Message = v + return s +} + +// SetMessageType sets the MessageType field's value. +func (s *SignInput) SetMessageType(v string) *SignInput { + s.MessageType = &v + return s +} + +// SetSigningAlgorithm sets the SigningAlgorithm field's value. +func (s *SignInput) SetSigningAlgorithm(v string) *SignInput { + s.SigningAlgorithm = &v + return s +} + +type SignOutput struct { + _ struct{} `type:"structure"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the asymmetric CMK that was used to sign the message. + KeyId *string `min:"1" type:"string"` + + // The cryptographic signature that was generated for the message. + // + // * When used with the supported RSA signing algorithms, the encoding of + // this value is defined by PKCS #1 in RFC 8017 (https://tools.ietf.org/html/rfc8017). + // + // * When used with the ECDSA_SHA_256, ECDSA_SHA_384, or ECDSA_SHA_512 signing + // algorithms, this value is a DER-encoded object as defined by ANS X9.62–2005 + // and RFC 3279 Section 2.2.3 (https://tools.ietf.org/html/rfc3279#section-2.2.3). + // This is the most commonly used signature format and is appropriate for + // most uses. + // + // When you use the HTTP API or the AWS CLI, the value is Base64-encoded. Otherwise, + // it is not Base64-encoded. + // + // Signature is automatically base64 encoded/decoded by the SDK. + Signature []byte `min:"1" type:"blob"` + + // The signing algorithm that was used to sign the message. + SigningAlgorithm *string `type:"string" enum:"SigningAlgorithmSpec"` +} + +// String returns the string representation +func (s SignOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s SignOutput) GoString() string { + return s.String() +} + +// SetKeyId sets the KeyId field's value. +func (s *SignOutput) SetKeyId(v string) *SignOutput { + s.KeyId = &v + return s +} + +// SetSignature sets the Signature field's value. +func (s *SignOutput) SetSignature(v []byte) *SignOutput { + s.Signature = v + return s +} + +// SetSigningAlgorithm sets the SigningAlgorithm field's value. +func (s *SignOutput) SetSigningAlgorithm(v string) *SignOutput { + s.SigningAlgorithm = &v + return s +} + +// A key-value pair. A tag consists of a tag key and a tag value. Tag keys and +// tag values are both required, but tag values can be empty (null) strings. +// +// For information about the rules that apply to tag keys and tag values, see +// User-Defined Tag Restrictions (https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html) +// in the AWS Billing and Cost Management User Guide. +type Tag struct { + _ struct{} `type:"structure"` + + // The key of the tag. + // + // TagKey is a required field + TagKey *string `min:"1" type:"string" required:"true"` + + // The value of the tag. + // + // TagValue is a required field + TagValue *string `type:"string" required:"true"` +} + +// String returns the string representation +func (s Tag) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s Tag) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *Tag) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "Tag"} + if s.TagKey == nil { + invalidParams.Add(request.NewErrParamRequired("TagKey")) + } + if s.TagKey != nil && len(*s.TagKey) < 1 { + invalidParams.Add(request.NewErrParamMinLen("TagKey", 1)) + } + if s.TagValue == nil { + invalidParams.Add(request.NewErrParamRequired("TagValue")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetTagKey sets the TagKey field's value. +func (s *Tag) SetTagKey(v string) *Tag { + s.TagKey = &v + return s +} + +// SetTagValue sets the TagValue field's value. +func (s *Tag) SetTagValue(v string) *Tag { + s.TagValue = &v + return s +} + +// The request was rejected because one or more tags are not valid. +type TagException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s TagException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s TagException) GoString() string { + return s.String() +} + +func newErrorTagException(v protocol.ResponseMetadata) error { + return &TagException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *TagException) Code() string { + return "TagException" +} + +// Message returns the exception's message. +func (s *TagException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *TagException) OrigErr() error { + return nil +} + +func (s *TagException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *TagException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *TagException) RequestID() string { + return s.RespMetadata.RequestID +} + +type TagResourceInput struct { + _ struct{} `type:"structure"` + + // A unique identifier for the CMK you are tagging. + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // One or more tags. Each tag consists of a tag key and a tag value. + // + // Tags is a required field + Tags []*Tag `type:"list" required:"true"` +} + +// String returns the string representation +func (s TagResourceInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s TagResourceInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *TagResourceInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "TagResourceInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.Tags == nil { + invalidParams.Add(request.NewErrParamRequired("Tags")) + } + if s.Tags != nil { + for i, v := range s.Tags { + if v == nil { + continue + } + if err := v.Validate(); err != nil { + invalidParams.AddNested(fmt.Sprintf("%s[%v]", "Tags", i), err.(request.ErrInvalidParams)) + } + } + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *TagResourceInput) SetKeyId(v string) *TagResourceInput { + s.KeyId = &v + return s +} + +// SetTags sets the Tags field's value. +func (s *TagResourceInput) SetTags(v []*Tag) *TagResourceInput { + s.Tags = v + return s +} + +type TagResourceOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s TagResourceOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s TagResourceOutput) GoString() string { + return s.String() +} + +// The request was rejected because a specified parameter is not supported or +// a specified resource is not valid for this operation. +type UnsupportedOperationException struct { + _ struct{} `type:"structure"` + RespMetadata protocol.ResponseMetadata `json:"-" xml:"-"` + + Message_ *string `locationName:"message" type:"string"` +} + +// String returns the string representation +func (s UnsupportedOperationException) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s UnsupportedOperationException) GoString() string { + return s.String() +} + +func newErrorUnsupportedOperationException(v protocol.ResponseMetadata) error { + return &UnsupportedOperationException{ + RespMetadata: v, + } +} + +// Code returns the exception type name. +func (s *UnsupportedOperationException) Code() string { + return "UnsupportedOperationException" +} + +// Message returns the exception's message. +func (s *UnsupportedOperationException) Message() string { + if s.Message_ != nil { + return *s.Message_ + } + return "" +} + +// OrigErr always returns nil, satisfies awserr.Error interface. +func (s *UnsupportedOperationException) OrigErr() error { + return nil +} + +func (s *UnsupportedOperationException) Error() string { + return fmt.Sprintf("%s: %s", s.Code(), s.Message()) +} + +// Status code returns the HTTP status code for the request's response error. +func (s *UnsupportedOperationException) StatusCode() int { + return s.RespMetadata.StatusCode +} + +// RequestID returns the service's response RequestID for request. +func (s *UnsupportedOperationException) RequestID() string { + return s.RespMetadata.RequestID +} + +type UntagResourceInput struct { + _ struct{} `type:"structure"` + + // A unique identifier for the CMK from which you are removing tags. + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // One or more tag keys. Specify only the tag keys, not the tag values. + // + // TagKeys is a required field + TagKeys []*string `type:"list" required:"true"` +} + +// String returns the string representation +func (s UntagResourceInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s UntagResourceInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *UntagResourceInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "UntagResourceInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.TagKeys == nil { + invalidParams.Add(request.NewErrParamRequired("TagKeys")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKeyId sets the KeyId field's value. +func (s *UntagResourceInput) SetKeyId(v string) *UntagResourceInput { + s.KeyId = &v + return s +} + +// SetTagKeys sets the TagKeys field's value. +func (s *UntagResourceInput) SetTagKeys(v []*string) *UntagResourceInput { + s.TagKeys = v + return s +} + +type UntagResourceOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s UntagResourceOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s UntagResourceOutput) GoString() string { + return s.String() +} + +type UpdateAliasInput struct { + _ struct{} `type:"structure"` + + // Identifies the alias that is changing its CMK. This value must begin with + // alias/ followed by the alias name, such as alias/ExampleAlias. You cannot + // use UpdateAlias to change the alias name. + // + // AliasName is a required field + AliasName *string `min:"1" type:"string" required:"true"` + + // Identifies the CMK to associate with the alias. When the update operation + // completes, the alias will point to this CMK. + // + // The CMK must be in the same AWS account and Region as the alias. Also, the + // new target CMK must be the same type as the current target CMK (both symmetric + // or both asymmetric) and they must have the same key usage. + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // To verify that the alias is mapped to the correct CMK, use ListAliases. + // + // TargetKeyId is a required field + TargetKeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s UpdateAliasInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s UpdateAliasInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *UpdateAliasInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "UpdateAliasInput"} + if s.AliasName == nil { + invalidParams.Add(request.NewErrParamRequired("AliasName")) + } + if s.AliasName != nil && len(*s.AliasName) < 1 { + invalidParams.Add(request.NewErrParamMinLen("AliasName", 1)) + } + if s.TargetKeyId == nil { + invalidParams.Add(request.NewErrParamRequired("TargetKeyId")) + } + if s.TargetKeyId != nil && len(*s.TargetKeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("TargetKeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetAliasName sets the AliasName field's value. +func (s *UpdateAliasInput) SetAliasName(v string) *UpdateAliasInput { + s.AliasName = &v + return s +} + +// SetTargetKeyId sets the TargetKeyId field's value. +func (s *UpdateAliasInput) SetTargetKeyId(v string) *UpdateAliasInput { + s.TargetKeyId = &v + return s +} + +type UpdateAliasOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s UpdateAliasOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s UpdateAliasOutput) GoString() string { + return s.String() +} + +type UpdateCustomKeyStoreInput struct { + _ struct{} `type:"structure"` + + // Associates the custom key store with a related AWS CloudHSM cluster. + // + // Enter the cluster ID of the cluster that you used to create the custom key + // store or a cluster that shares a backup history and has the same cluster + // certificate as the original cluster. You cannot use this parameter to associate + // a custom key store with an unrelated cluster. In addition, the replacement + // cluster must fulfill the requirements (https://docs.aws.amazon.com/kms/latest/developerguide/create-keystore.html#before-keystore) + // for a cluster associated with a custom key store. To view the cluster certificate + // of a cluster, use the DescribeClusters (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_DescribeClusters.html) + // operation. + CloudHsmClusterId *string `min:"19" type:"string"` + + // Identifies the custom key store that you want to update. Enter the ID of + // the custom key store. To find the ID of a custom key store, use the DescribeCustomKeyStores + // operation. + // + // CustomKeyStoreId is a required field + CustomKeyStoreId *string `min:"1" type:"string" required:"true"` + + // Enter the current password of the kmsuser crypto user (CU) in the AWS CloudHSM + // cluster that is associated with the custom key store. + // + // This parameter tells AWS KMS the current password of the kmsuser crypto user + // (CU). It does not set or change the password of any users in the AWS CloudHSM + // cluster. + KeyStorePassword *string `min:"7" type:"string" sensitive:"true"` + + // Changes the friendly name of the custom key store to the value that you specify. + // The custom key store name must be unique in the AWS account. + NewCustomKeyStoreName *string `min:"1" type:"string"` +} + +// String returns the string representation +func (s UpdateCustomKeyStoreInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s UpdateCustomKeyStoreInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *UpdateCustomKeyStoreInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "UpdateCustomKeyStoreInput"} + if s.CloudHsmClusterId != nil && len(*s.CloudHsmClusterId) < 19 { + invalidParams.Add(request.NewErrParamMinLen("CloudHsmClusterId", 19)) + } + if s.CustomKeyStoreId == nil { + invalidParams.Add(request.NewErrParamRequired("CustomKeyStoreId")) + } + if s.CustomKeyStoreId != nil && len(*s.CustomKeyStoreId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("CustomKeyStoreId", 1)) + } + if s.KeyStorePassword != nil && len(*s.KeyStorePassword) < 7 { + invalidParams.Add(request.NewErrParamMinLen("KeyStorePassword", 7)) + } + if s.NewCustomKeyStoreName != nil && len(*s.NewCustomKeyStoreName) < 1 { + invalidParams.Add(request.NewErrParamMinLen("NewCustomKeyStoreName", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetCloudHsmClusterId sets the CloudHsmClusterId field's value. +func (s *UpdateCustomKeyStoreInput) SetCloudHsmClusterId(v string) *UpdateCustomKeyStoreInput { + s.CloudHsmClusterId = &v + return s +} + +// SetCustomKeyStoreId sets the CustomKeyStoreId field's value. +func (s *UpdateCustomKeyStoreInput) SetCustomKeyStoreId(v string) *UpdateCustomKeyStoreInput { + s.CustomKeyStoreId = &v + return s +} + +// SetKeyStorePassword sets the KeyStorePassword field's value. +func (s *UpdateCustomKeyStoreInput) SetKeyStorePassword(v string) *UpdateCustomKeyStoreInput { + s.KeyStorePassword = &v + return s +} + +// SetNewCustomKeyStoreName sets the NewCustomKeyStoreName field's value. +func (s *UpdateCustomKeyStoreInput) SetNewCustomKeyStoreName(v string) *UpdateCustomKeyStoreInput { + s.NewCustomKeyStoreName = &v + return s +} + +type UpdateCustomKeyStoreOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s UpdateCustomKeyStoreOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s UpdateCustomKeyStoreOutput) GoString() string { + return s.String() +} + +type UpdateKeyDescriptionInput struct { + _ struct{} `type:"structure"` + + // New description for the CMK. + // + // Description is a required field + Description *string `type:"string" required:"true"` + + // A unique identifier for the customer master key (CMK). + // + // Specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` +} + +// String returns the string representation +func (s UpdateKeyDescriptionInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s UpdateKeyDescriptionInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *UpdateKeyDescriptionInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "UpdateKeyDescriptionInput"} + if s.Description == nil { + invalidParams.Add(request.NewErrParamRequired("Description")) + } + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetDescription sets the Description field's value. +func (s *UpdateKeyDescriptionInput) SetDescription(v string) *UpdateKeyDescriptionInput { + s.Description = &v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *UpdateKeyDescriptionInput) SetKeyId(v string) *UpdateKeyDescriptionInput { + s.KeyId = &v + return s +} + +type UpdateKeyDescriptionOutput struct { + _ struct{} `type:"structure"` +} + +// String returns the string representation +func (s UpdateKeyDescriptionOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s UpdateKeyDescriptionOutput) GoString() string { + return s.String() +} + +type VerifyInput struct { + _ struct{} `type:"structure"` + + // A list of grant tokens. + // + // For more information, see Grant Tokens (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#grant_token) + // in the AWS Key Management Service Developer Guide. + GrantTokens []*string `type:"list"` + + // Identifies the asymmetric CMK that will be used to verify the signature. + // This must be the same CMK that was used to generate the signature. If you + // specify a different CMK, the signature verification fails. + // + // To specify a CMK, use its key ID, Amazon Resource Name (ARN), alias name, + // or alias ARN. When using an alias name, prefix it with "alias/". To specify + // a CMK in a different AWS account, you must use the key ARN or alias ARN. + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Alias name: alias/ExampleAlias + // + // * Alias ARN: arn:aws:kms:us-east-2:111122223333:alias/ExampleAlias + // + // To get the key ID and key ARN for a CMK, use ListKeys or DescribeKey. To + // get the alias name and alias ARN, use ListAliases. + // + // KeyId is a required field + KeyId *string `min:"1" type:"string" required:"true"` + + // Specifies the message that was signed. You can submit a raw message of up + // to 4096 bytes, or a hash digest of the message. If you submit a digest, use + // the MessageType parameter with a value of DIGEST. + // + // If the message specified here is different from the message that was signed, + // the signature verification fails. A message and its hash digest are considered + // to be the same message. + // + // Message is automatically base64 encoded/decoded by the SDK. + // + // Message is a required field + Message []byte `min:"1" type:"blob" required:"true" sensitive:"true"` + + // Tells AWS KMS whether the value of the Message parameter is a message or + // message digest. The default value, RAW, indicates a message. To indicate + // a message digest, enter DIGEST. + // + // Use the DIGEST value only when the value of the Message parameter is a message + // digest. If you use the DIGEST value with a raw message, the security of the + // verification operation can be compromised. + MessageType *string `type:"string" enum:"MessageType"` + + // The signature that the Sign operation generated. + // + // Signature is automatically base64 encoded/decoded by the SDK. + // + // Signature is a required field + Signature []byte `min:"1" type:"blob" required:"true"` + + // The signing algorithm that was used to sign the message. If you submit a + // different algorithm, the signature verification fails. + // + // SigningAlgorithm is a required field + SigningAlgorithm *string `type:"string" required:"true" enum:"SigningAlgorithmSpec"` +} + +// String returns the string representation +func (s VerifyInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s VerifyInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *VerifyInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "VerifyInput"} + if s.KeyId == nil { + invalidParams.Add(request.NewErrParamRequired("KeyId")) + } + if s.KeyId != nil && len(*s.KeyId) < 1 { + invalidParams.Add(request.NewErrParamMinLen("KeyId", 1)) + } + if s.Message == nil { + invalidParams.Add(request.NewErrParamRequired("Message")) + } + if s.Message != nil && len(s.Message) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Message", 1)) + } + if s.Signature == nil { + invalidParams.Add(request.NewErrParamRequired("Signature")) + } + if s.Signature != nil && len(s.Signature) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Signature", 1)) + } + if s.SigningAlgorithm == nil { + invalidParams.Add(request.NewErrParamRequired("SigningAlgorithm")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetGrantTokens sets the GrantTokens field's value. +func (s *VerifyInput) SetGrantTokens(v []*string) *VerifyInput { + s.GrantTokens = v + return s +} + +// SetKeyId sets the KeyId field's value. +func (s *VerifyInput) SetKeyId(v string) *VerifyInput { + s.KeyId = &v + return s +} + +// SetMessage sets the Message field's value. +func (s *VerifyInput) SetMessage(v []byte) *VerifyInput { + s.Message = v + return s +} + +// SetMessageType sets the MessageType field's value. +func (s *VerifyInput) SetMessageType(v string) *VerifyInput { + s.MessageType = &v + return s +} + +// SetSignature sets the Signature field's value. +func (s *VerifyInput) SetSignature(v []byte) *VerifyInput { + s.Signature = v + return s +} + +// SetSigningAlgorithm sets the SigningAlgorithm field's value. +func (s *VerifyInput) SetSigningAlgorithm(v string) *VerifyInput { + s.SigningAlgorithm = &v + return s +} + +type VerifyOutput struct { + _ struct{} `type:"structure"` + + // The Amazon Resource Name (key ARN (https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id-key-ARN)) + // of the asymmetric CMK that was used to verify the signature. + KeyId *string `min:"1" type:"string"` + + // A Boolean value that indicates whether the signature was verified. A value + // of True indicates that the Signature was produced by signing the Message + // with the specified KeyID and SigningAlgorithm. If the signature is not verified, + // the Verify operation fails with a KMSInvalidSignatureException exception. + SignatureValid *bool `type:"boolean"` + + // The signing algorithm that was used to verify the signature. + SigningAlgorithm *string `type:"string" enum:"SigningAlgorithmSpec"` +} + +// String returns the string representation +func (s VerifyOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s VerifyOutput) GoString() string { + return s.String() +} + +// SetKeyId sets the KeyId field's value. +func (s *VerifyOutput) SetKeyId(v string) *VerifyOutput { + s.KeyId = &v + return s +} + +// SetSignatureValid sets the SignatureValid field's value. +func (s *VerifyOutput) SetSignatureValid(v bool) *VerifyOutput { + s.SignatureValid = &v + return s +} + +// SetSigningAlgorithm sets the SigningAlgorithm field's value. +func (s *VerifyOutput) SetSigningAlgorithm(v string) *VerifyOutput { + s.SigningAlgorithm = &v + return s +} + +const ( + // AlgorithmSpecRsaesPkcs1V15 is a AlgorithmSpec enum value + AlgorithmSpecRsaesPkcs1V15 = "RSAES_PKCS1_V1_5" + + // AlgorithmSpecRsaesOaepSha1 is a AlgorithmSpec enum value + AlgorithmSpecRsaesOaepSha1 = "RSAES_OAEP_SHA_1" + + // AlgorithmSpecRsaesOaepSha256 is a AlgorithmSpec enum value + AlgorithmSpecRsaesOaepSha256 = "RSAES_OAEP_SHA_256" +) + +const ( + // ConnectionErrorCodeTypeInvalidCredentials is a ConnectionErrorCodeType enum value + ConnectionErrorCodeTypeInvalidCredentials = "INVALID_CREDENTIALS" + + // ConnectionErrorCodeTypeClusterNotFound is a ConnectionErrorCodeType enum value + ConnectionErrorCodeTypeClusterNotFound = "CLUSTER_NOT_FOUND" + + // ConnectionErrorCodeTypeNetworkErrors is a ConnectionErrorCodeType enum value + ConnectionErrorCodeTypeNetworkErrors = "NETWORK_ERRORS" + + // ConnectionErrorCodeTypeInternalError is a ConnectionErrorCodeType enum value + ConnectionErrorCodeTypeInternalError = "INTERNAL_ERROR" + + // ConnectionErrorCodeTypeInsufficientCloudhsmHsms is a ConnectionErrorCodeType enum value + ConnectionErrorCodeTypeInsufficientCloudhsmHsms = "INSUFFICIENT_CLOUDHSM_HSMS" + + // ConnectionErrorCodeTypeUserLockedOut is a ConnectionErrorCodeType enum value + ConnectionErrorCodeTypeUserLockedOut = "USER_LOCKED_OUT" + + // ConnectionErrorCodeTypeUserNotFound is a ConnectionErrorCodeType enum value + ConnectionErrorCodeTypeUserNotFound = "USER_NOT_FOUND" + + // ConnectionErrorCodeTypeUserLoggedIn is a ConnectionErrorCodeType enum value + ConnectionErrorCodeTypeUserLoggedIn = "USER_LOGGED_IN" + + // ConnectionErrorCodeTypeSubnetNotFound is a ConnectionErrorCodeType enum value + ConnectionErrorCodeTypeSubnetNotFound = "SUBNET_NOT_FOUND" +) + +const ( + // ConnectionStateTypeConnected is a ConnectionStateType enum value + ConnectionStateTypeConnected = "CONNECTED" + + // ConnectionStateTypeConnecting is a ConnectionStateType enum value + ConnectionStateTypeConnecting = "CONNECTING" + + // ConnectionStateTypeFailed is a ConnectionStateType enum value + ConnectionStateTypeFailed = "FAILED" + + // ConnectionStateTypeDisconnected is a ConnectionStateType enum value + ConnectionStateTypeDisconnected = "DISCONNECTED" + + // ConnectionStateTypeDisconnecting is a ConnectionStateType enum value + ConnectionStateTypeDisconnecting = "DISCONNECTING" +) + +const ( + // CustomerMasterKeySpecRsa2048 is a CustomerMasterKeySpec enum value + CustomerMasterKeySpecRsa2048 = "RSA_2048" + + // CustomerMasterKeySpecRsa3072 is a CustomerMasterKeySpec enum value + CustomerMasterKeySpecRsa3072 = "RSA_3072" + + // CustomerMasterKeySpecRsa4096 is a CustomerMasterKeySpec enum value + CustomerMasterKeySpecRsa4096 = "RSA_4096" + + // CustomerMasterKeySpecEccNistP256 is a CustomerMasterKeySpec enum value + CustomerMasterKeySpecEccNistP256 = "ECC_NIST_P256" + + // CustomerMasterKeySpecEccNistP384 is a CustomerMasterKeySpec enum value + CustomerMasterKeySpecEccNistP384 = "ECC_NIST_P384" + + // CustomerMasterKeySpecEccNistP521 is a CustomerMasterKeySpec enum value + CustomerMasterKeySpecEccNistP521 = "ECC_NIST_P521" + + // CustomerMasterKeySpecEccSecgP256k1 is a CustomerMasterKeySpec enum value + CustomerMasterKeySpecEccSecgP256k1 = "ECC_SECG_P256K1" + + // CustomerMasterKeySpecSymmetricDefault is a CustomerMasterKeySpec enum value + CustomerMasterKeySpecSymmetricDefault = "SYMMETRIC_DEFAULT" +) + +const ( + // DataKeyPairSpecRsa2048 is a DataKeyPairSpec enum value + DataKeyPairSpecRsa2048 = "RSA_2048" + + // DataKeyPairSpecRsa3072 is a DataKeyPairSpec enum value + DataKeyPairSpecRsa3072 = "RSA_3072" + + // DataKeyPairSpecRsa4096 is a DataKeyPairSpec enum value + DataKeyPairSpecRsa4096 = "RSA_4096" + + // DataKeyPairSpecEccNistP256 is a DataKeyPairSpec enum value + DataKeyPairSpecEccNistP256 = "ECC_NIST_P256" + + // DataKeyPairSpecEccNistP384 is a DataKeyPairSpec enum value + DataKeyPairSpecEccNistP384 = "ECC_NIST_P384" + + // DataKeyPairSpecEccNistP521 is a DataKeyPairSpec enum value + DataKeyPairSpecEccNistP521 = "ECC_NIST_P521" + + // DataKeyPairSpecEccSecgP256k1 is a DataKeyPairSpec enum value + DataKeyPairSpecEccSecgP256k1 = "ECC_SECG_P256K1" +) + +const ( + // DataKeySpecAes256 is a DataKeySpec enum value + DataKeySpecAes256 = "AES_256" + + // DataKeySpecAes128 is a DataKeySpec enum value + DataKeySpecAes128 = "AES_128" +) + +const ( + // EncryptionAlgorithmSpecSymmetricDefault is a EncryptionAlgorithmSpec enum value + EncryptionAlgorithmSpecSymmetricDefault = "SYMMETRIC_DEFAULT" + + // EncryptionAlgorithmSpecRsaesOaepSha1 is a EncryptionAlgorithmSpec enum value + EncryptionAlgorithmSpecRsaesOaepSha1 = "RSAES_OAEP_SHA_1" + + // EncryptionAlgorithmSpecRsaesOaepSha256 is a EncryptionAlgorithmSpec enum value + EncryptionAlgorithmSpecRsaesOaepSha256 = "RSAES_OAEP_SHA_256" +) + +const ( + // ExpirationModelTypeKeyMaterialExpires is a ExpirationModelType enum value + ExpirationModelTypeKeyMaterialExpires = "KEY_MATERIAL_EXPIRES" + + // ExpirationModelTypeKeyMaterialDoesNotExpire is a ExpirationModelType enum value + ExpirationModelTypeKeyMaterialDoesNotExpire = "KEY_MATERIAL_DOES_NOT_EXPIRE" +) + +const ( + // GrantOperationDecrypt is a GrantOperation enum value + GrantOperationDecrypt = "Decrypt" + + // GrantOperationEncrypt is a GrantOperation enum value + GrantOperationEncrypt = "Encrypt" + + // GrantOperationGenerateDataKey is a GrantOperation enum value + GrantOperationGenerateDataKey = "GenerateDataKey" + + // GrantOperationGenerateDataKeyWithoutPlaintext is a GrantOperation enum value + GrantOperationGenerateDataKeyWithoutPlaintext = "GenerateDataKeyWithoutPlaintext" + + // GrantOperationReEncryptFrom is a GrantOperation enum value + GrantOperationReEncryptFrom = "ReEncryptFrom" + + // GrantOperationReEncryptTo is a GrantOperation enum value + GrantOperationReEncryptTo = "ReEncryptTo" + + // GrantOperationSign is a GrantOperation enum value + GrantOperationSign = "Sign" + + // GrantOperationVerify is a GrantOperation enum value + GrantOperationVerify = "Verify" + + // GrantOperationGetPublicKey is a GrantOperation enum value + GrantOperationGetPublicKey = "GetPublicKey" + + // GrantOperationCreateGrant is a GrantOperation enum value + GrantOperationCreateGrant = "CreateGrant" + + // GrantOperationRetireGrant is a GrantOperation enum value + GrantOperationRetireGrant = "RetireGrant" + + // GrantOperationDescribeKey is a GrantOperation enum value + GrantOperationDescribeKey = "DescribeKey" + + // GrantOperationGenerateDataKeyPair is a GrantOperation enum value + GrantOperationGenerateDataKeyPair = "GenerateDataKeyPair" + + // GrantOperationGenerateDataKeyPairWithoutPlaintext is a GrantOperation enum value + GrantOperationGenerateDataKeyPairWithoutPlaintext = "GenerateDataKeyPairWithoutPlaintext" +) + +const ( + // KeyManagerTypeAws is a KeyManagerType enum value + KeyManagerTypeAws = "AWS" + + // KeyManagerTypeCustomer is a KeyManagerType enum value + KeyManagerTypeCustomer = "CUSTOMER" +) + +const ( + // KeyStateEnabled is a KeyState enum value + KeyStateEnabled = "Enabled" + + // KeyStateDisabled is a KeyState enum value + KeyStateDisabled = "Disabled" + + // KeyStatePendingDeletion is a KeyState enum value + KeyStatePendingDeletion = "PendingDeletion" + + // KeyStatePendingImport is a KeyState enum value + KeyStatePendingImport = "PendingImport" + + // KeyStateUnavailable is a KeyState enum value + KeyStateUnavailable = "Unavailable" +) + +const ( + // KeyUsageTypeSignVerify is a KeyUsageType enum value + KeyUsageTypeSignVerify = "SIGN_VERIFY" + + // KeyUsageTypeEncryptDecrypt is a KeyUsageType enum value + KeyUsageTypeEncryptDecrypt = "ENCRYPT_DECRYPT" +) + +const ( + // MessageTypeRaw is a MessageType enum value + MessageTypeRaw = "RAW" + + // MessageTypeDigest is a MessageType enum value + MessageTypeDigest = "DIGEST" +) + +const ( + // OriginTypeAwsKms is a OriginType enum value + OriginTypeAwsKms = "AWS_KMS" + + // OriginTypeExternal is a OriginType enum value + OriginTypeExternal = "EXTERNAL" + + // OriginTypeAwsCloudhsm is a OriginType enum value + OriginTypeAwsCloudhsm = "AWS_CLOUDHSM" +) + +const ( + // SigningAlgorithmSpecRsassaPssSha256 is a SigningAlgorithmSpec enum value + SigningAlgorithmSpecRsassaPssSha256 = "RSASSA_PSS_SHA_256" + + // SigningAlgorithmSpecRsassaPssSha384 is a SigningAlgorithmSpec enum value + SigningAlgorithmSpecRsassaPssSha384 = "RSASSA_PSS_SHA_384" + + // SigningAlgorithmSpecRsassaPssSha512 is a SigningAlgorithmSpec enum value + SigningAlgorithmSpecRsassaPssSha512 = "RSASSA_PSS_SHA_512" + + // SigningAlgorithmSpecRsassaPkcs1V15Sha256 is a SigningAlgorithmSpec enum value + SigningAlgorithmSpecRsassaPkcs1V15Sha256 = "RSASSA_PKCS1_V1_5_SHA_256" + + // SigningAlgorithmSpecRsassaPkcs1V15Sha384 is a SigningAlgorithmSpec enum value + SigningAlgorithmSpecRsassaPkcs1V15Sha384 = "RSASSA_PKCS1_V1_5_SHA_384" + + // SigningAlgorithmSpecRsassaPkcs1V15Sha512 is a SigningAlgorithmSpec enum value + SigningAlgorithmSpecRsassaPkcs1V15Sha512 = "RSASSA_PKCS1_V1_5_SHA_512" + + // SigningAlgorithmSpecEcdsaSha256 is a SigningAlgorithmSpec enum value + SigningAlgorithmSpecEcdsaSha256 = "ECDSA_SHA_256" + + // SigningAlgorithmSpecEcdsaSha384 is a SigningAlgorithmSpec enum value + SigningAlgorithmSpecEcdsaSha384 = "ECDSA_SHA_384" + + // SigningAlgorithmSpecEcdsaSha512 is a SigningAlgorithmSpec enum value + SigningAlgorithmSpecEcdsaSha512 = "ECDSA_SHA_512" +) + +const ( + // WrappingKeySpecRsa2048 is a WrappingKeySpec enum value + WrappingKeySpecRsa2048 = "RSA_2048" +) diff --git a/github.com/aws/aws-sdk-go/service/kms/doc.go b/github.com/aws/aws-sdk-go/service/kms/doc.go new file mode 100644 index 0000000000..c4c2125020 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/kms/doc.go @@ -0,0 +1,98 @@ +// Code generated by private/model/cli/gen-api/main.go. DO NOT EDIT. + +// Package kms provides the client and types for making API +// requests to AWS Key Management Service. +// +// AWS Key Management Service (AWS KMS) is an encryption and key management +// web service. This guide describes the AWS KMS operations that you can call +// programmatically. For general information about AWS KMS, see the AWS Key +// Management Service Developer Guide (https://docs.aws.amazon.com/kms/latest/developerguide/). +// +// AWS provides SDKs that consist of libraries and sample code for various programming +// languages and platforms (Java, Ruby, .Net, macOS, Android, etc.). The SDKs +// provide a convenient way to create programmatic access to AWS KMS and other +// AWS services. For example, the SDKs take care of tasks such as signing requests +// (see below), managing errors, and retrying requests automatically. For more +// information about the AWS SDKs, including how to download and install them, +// see Tools for Amazon Web Services (http://aws.amazon.com/tools/). +// +// We recommend that you use the AWS SDKs to make programmatic API calls to +// AWS KMS. +// +// Clients must support TLS (Transport Layer Security) 1.0. We recommend TLS +// 1.2. Clients must also support cipher suites with Perfect Forward Secrecy +// (PFS) such as Ephemeral Diffie-Hellman (DHE) or Elliptic Curve Ephemeral +// Diffie-Hellman (ECDHE). Most modern systems such as Java 7 and later support +// these modes. +// +// Signing Requests +// +// Requests must be signed by using an access key ID and a secret access key. +// We strongly recommend that you do not use your AWS account (root) access +// key ID and secret key for everyday work with AWS KMS. Instead, use the access +// key ID and secret access key for an IAM user. You can also use the AWS Security +// Token Service to generate temporary security credentials that you can use +// to sign requests. +// +// All AWS KMS operations require Signature Version 4 (https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html). +// +// Logging API Requests +// +// AWS KMS supports AWS CloudTrail, a service that logs AWS API calls and related +// events for your AWS account and delivers them to an Amazon S3 bucket that +// you specify. By using the information collected by CloudTrail, you can determine +// what requests were made to AWS KMS, who made the request, when it was made, +// and so on. To learn more about CloudTrail, including how to turn it on and +// find your log files, see the AWS CloudTrail User Guide (https://docs.aws.amazon.com/awscloudtrail/latest/userguide/). +// +// Additional Resources +// +// For more information about credentials and request signing, see the following: +// +// * AWS Security Credentials (https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html) +// - This topic provides general information about the types of credentials +// used for accessing AWS. +// +// * Temporary Security Credentials (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html) +// - This section of the IAM User Guide describes how to create and use temporary +// security credentials. +// +// * Signature Version 4 Signing Process (https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html) +// - This set of topics walks you through the process of signing a request +// using an access key ID and a secret access key. +// +// Commonly Used API Operations +// +// Of the API operations discussed in this guide, the following will prove the +// most useful for most applications. You will likely perform operations other +// than these, such as creating keys and assigning policies, by using the console. +// +// * Encrypt +// +// * Decrypt +// +// * GenerateDataKey +// +// * GenerateDataKeyWithoutPlaintext +// +// See https://docs.aws.amazon.com/goto/WebAPI/kms-2014-11-01 for more information on this service. +// +// See kms package documentation for more information. +// https://docs.aws.amazon.com/sdk-for-go/api/service/kms/ +// +// Using the Client +// +// To contact AWS Key Management Service with the SDK use the New function to create +// a new service client. With that client you can make API requests to the service. +// These clients are safe to use concurrently. +// +// See the SDK's documentation for more information on how to use the SDK. +// https://docs.aws.amazon.com/sdk-for-go/api/ +// +// See aws.Config documentation for more information on configuring SDK clients. +// https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config +// +// See the AWS Key Management Service client KMS for more +// information on creating client for this service. +// https://docs.aws.amazon.com/sdk-for-go/api/service/kms/#New +package kms diff --git a/github.com/aws/aws-sdk-go/service/kms/errors.go b/github.com/aws/aws-sdk-go/service/kms/errors.go new file mode 100644 index 0000000000..911bf576eb --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/kms/errors.go @@ -0,0 +1,367 @@ +// Code generated by private/model/cli/gen-api/main.go. DO NOT EDIT. + +package kms + +import ( + "github.com/aws/aws-sdk-go/private/protocol" +) + +const ( + + // ErrCodeAlreadyExistsException for service response error code + // "AlreadyExistsException". + // + // The request was rejected because it attempted to create a resource that already + // exists. + ErrCodeAlreadyExistsException = "AlreadyExistsException" + + // ErrCodeCloudHsmClusterInUseException for service response error code + // "CloudHsmClusterInUseException". + // + // The request was rejected because the specified AWS CloudHSM cluster is already + // associated with a custom key store or it shares a backup history with a cluster + // that is associated with a custom key store. Each custom key store must be + // associated with a different AWS CloudHSM cluster. + // + // Clusters that share a backup history have the same cluster certificate. To + // view the cluster certificate of a cluster, use the DescribeClusters (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_DescribeClusters.html) + // operation. + ErrCodeCloudHsmClusterInUseException = "CloudHsmClusterInUseException" + + // ErrCodeCloudHsmClusterInvalidConfigurationException for service response error code + // "CloudHsmClusterInvalidConfigurationException". + // + // The request was rejected because the associated AWS CloudHSM cluster did + // not meet the configuration requirements for a custom key store. + // + // * The cluster must be configured with private subnets in at least two + // different Availability Zones in the Region. + // + // * The security group for the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) + // (cloudhsm-cluster--sg) must include inbound rules and outbound + // rules that allow TCP traffic on ports 2223-2225. The Source in the inbound + // rules and the Destination in the outbound rules must match the security + // group ID. These rules are set by default when you create the cluster. + // Do not delete or change them. To get information about a particular security + // group, use the DescribeSecurityGroups (https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeSecurityGroups.html) + // operation. + // + // * The cluster must contain at least as many HSMs as the operation requires. + // To add HSMs, use the AWS CloudHSM CreateHsm (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_CreateHsm.html) + // operation. For the CreateCustomKeyStore, UpdateCustomKeyStore, and CreateKey + // operations, the AWS CloudHSM cluster must have at least two active HSMs, + // each in a different Availability Zone. For the ConnectCustomKeyStore operation, + // the AWS CloudHSM must contain at least one active HSM. + // + // For information about the requirements for an AWS CloudHSM cluster that is + // associated with a custom key store, see Assemble the Prerequisites (https://docs.aws.amazon.com/kms/latest/developerguide/create-keystore.html#before-keystore) + // in the AWS Key Management Service Developer Guide. For information about + // creating a private subnet for an AWS CloudHSM cluster, see Create a Private + // Subnet (https://docs.aws.amazon.com/cloudhsm/latest/userguide/create-subnets.html) + // in the AWS CloudHSM User Guide. For information about cluster security groups, + // see Configure a Default Security Group (https://docs.aws.amazon.com/cloudhsm/latest/userguide/configure-sg.html) + // in the AWS CloudHSM User Guide . + ErrCodeCloudHsmClusterInvalidConfigurationException = "CloudHsmClusterInvalidConfigurationException" + + // ErrCodeCloudHsmClusterNotActiveException for service response error code + // "CloudHsmClusterNotActiveException". + // + // The request was rejected because the AWS CloudHSM cluster that is associated + // with the custom key store is not active. Initialize and activate the cluster + // and try the command again. For detailed instructions, see Getting Started + // (https://docs.aws.amazon.com/cloudhsm/latest/userguide/getting-started.html) + // in the AWS CloudHSM User Guide. + ErrCodeCloudHsmClusterNotActiveException = "CloudHsmClusterNotActiveException" + + // ErrCodeCloudHsmClusterNotFoundException for service response error code + // "CloudHsmClusterNotFoundException". + // + // The request was rejected because AWS KMS cannot find the AWS CloudHSM cluster + // with the specified cluster ID. Retry the request with a different cluster + // ID. + ErrCodeCloudHsmClusterNotFoundException = "CloudHsmClusterNotFoundException" + + // ErrCodeCloudHsmClusterNotRelatedException for service response error code + // "CloudHsmClusterNotRelatedException". + // + // The request was rejected because the specified AWS CloudHSM cluster has a + // different cluster certificate than the original cluster. You cannot use the + // operation to specify an unrelated cluster. + // + // Specify a cluster that shares a backup history with the original cluster. + // This includes clusters that were created from a backup of the current cluster, + // and clusters that were created from the same backup that produced the current + // cluster. + // + // Clusters that share a backup history have the same cluster certificate. To + // view the cluster certificate of a cluster, use the DescribeClusters (https://docs.aws.amazon.com/cloudhsm/latest/APIReference/API_DescribeClusters.html) + // operation. + ErrCodeCloudHsmClusterNotRelatedException = "CloudHsmClusterNotRelatedException" + + // ErrCodeCustomKeyStoreHasCMKsException for service response error code + // "CustomKeyStoreHasCMKsException". + // + // The request was rejected because the custom key store contains AWS KMS customer + // master keys (CMKs). After verifying that you do not need to use the CMKs, + // use the ScheduleKeyDeletion operation to delete the CMKs. After they are + // deleted, you can delete the custom key store. + ErrCodeCustomKeyStoreHasCMKsException = "CustomKeyStoreHasCMKsException" + + // ErrCodeCustomKeyStoreInvalidStateException for service response error code + // "CustomKeyStoreInvalidStateException". + // + // The request was rejected because of the ConnectionState of the custom key + // store. To get the ConnectionState of a custom key store, use the DescribeCustomKeyStores + // operation. + // + // This exception is thrown under the following conditions: + // + // * You requested the CreateKey or GenerateRandom operation in a custom + // key store that is not connected. These operations are valid only when + // the custom key store ConnectionState is CONNECTED. + // + // * You requested the UpdateCustomKeyStore or DeleteCustomKeyStore operation + // on a custom key store that is not disconnected. This operation is valid + // only when the custom key store ConnectionState is DISCONNECTED. + // + // * You requested the ConnectCustomKeyStore operation on a custom key store + // with a ConnectionState of DISCONNECTING or FAILED. This operation is valid + // for all other ConnectionState values. + ErrCodeCustomKeyStoreInvalidStateException = "CustomKeyStoreInvalidStateException" + + // ErrCodeCustomKeyStoreNameInUseException for service response error code + // "CustomKeyStoreNameInUseException". + // + // The request was rejected because the specified custom key store name is already + // assigned to another custom key store in the account. Try again with a custom + // key store name that is unique in the account. + ErrCodeCustomKeyStoreNameInUseException = "CustomKeyStoreNameInUseException" + + // ErrCodeCustomKeyStoreNotFoundException for service response error code + // "CustomKeyStoreNotFoundException". + // + // The request was rejected because AWS KMS cannot find a custom key store with + // the specified key store name or ID. + ErrCodeCustomKeyStoreNotFoundException = "CustomKeyStoreNotFoundException" + + // ErrCodeDependencyTimeoutException for service response error code + // "DependencyTimeoutException". + // + // The system timed out while trying to fulfill the request. The request can + // be retried. + ErrCodeDependencyTimeoutException = "DependencyTimeoutException" + + // ErrCodeDisabledException for service response error code + // "DisabledException". + // + // The request was rejected because the specified CMK is not enabled. + ErrCodeDisabledException = "DisabledException" + + // ErrCodeExpiredImportTokenException for service response error code + // "ExpiredImportTokenException". + // + // The request was rejected because the specified import token is expired. Use + // GetParametersForImport to get a new import token and public key, use the + // new public key to encrypt the key material, and then try the request again. + ErrCodeExpiredImportTokenException = "ExpiredImportTokenException" + + // ErrCodeIncorrectKeyException for service response error code + // "IncorrectKeyException". + // + // The request was rejected because the specified CMK cannot decrypt the data. + // The KeyId in a Decrypt request and the SourceKeyId in a ReEncrypt request + // must identify the same CMK that was used to encrypt the ciphertext. + ErrCodeIncorrectKeyException = "IncorrectKeyException" + + // ErrCodeIncorrectKeyMaterialException for service response error code + // "IncorrectKeyMaterialException". + // + // The request was rejected because the key material in the request is, expired, + // invalid, or is not the same key material that was previously imported into + // this customer master key (CMK). + ErrCodeIncorrectKeyMaterialException = "IncorrectKeyMaterialException" + + // ErrCodeIncorrectTrustAnchorException for service response error code + // "IncorrectTrustAnchorException". + // + // The request was rejected because the trust anchor certificate in the request + // is not the trust anchor certificate for the specified AWS CloudHSM cluster. + // + // When you initialize the cluster (https://docs.aws.amazon.com/cloudhsm/latest/userguide/initialize-cluster.html#sign-csr), + // you create the trust anchor certificate and save it in the customerCA.crt + // file. + ErrCodeIncorrectTrustAnchorException = "IncorrectTrustAnchorException" + + // ErrCodeInternalException for service response error code + // "KMSInternalException". + // + // The request was rejected because an internal exception occurred. The request + // can be retried. + ErrCodeInternalException = "KMSInternalException" + + // ErrCodeInvalidAliasNameException for service response error code + // "InvalidAliasNameException". + // + // The request was rejected because the specified alias name is not valid. + ErrCodeInvalidAliasNameException = "InvalidAliasNameException" + + // ErrCodeInvalidArnException for service response error code + // "InvalidArnException". + // + // The request was rejected because a specified ARN, or an ARN in a key policy, + // is not valid. + ErrCodeInvalidArnException = "InvalidArnException" + + // ErrCodeInvalidCiphertextException for service response error code + // "InvalidCiphertextException". + // + // From the Decrypt or ReEncrypt operation, the request was rejected because + // the specified ciphertext, or additional authenticated data incorporated into + // the ciphertext, such as the encryption context, is corrupted, missing, or + // otherwise invalid. + // + // From the ImportKeyMaterial operation, the request was rejected because AWS + // KMS could not decrypt the encrypted (wrapped) key material. + ErrCodeInvalidCiphertextException = "InvalidCiphertextException" + + // ErrCodeInvalidGrantIdException for service response error code + // "InvalidGrantIdException". + // + // The request was rejected because the specified GrantId is not valid. + ErrCodeInvalidGrantIdException = "InvalidGrantIdException" + + // ErrCodeInvalidGrantTokenException for service response error code + // "InvalidGrantTokenException". + // + // The request was rejected because the specified grant token is not valid. + ErrCodeInvalidGrantTokenException = "InvalidGrantTokenException" + + // ErrCodeInvalidImportTokenException for service response error code + // "InvalidImportTokenException". + // + // The request was rejected because the provided import token is invalid or + // is associated with a different customer master key (CMK). + ErrCodeInvalidImportTokenException = "InvalidImportTokenException" + + // ErrCodeInvalidKeyUsageException for service response error code + // "InvalidKeyUsageException". + // + // The request was rejected for one of the following reasons: + // + // * The KeyUsage value of the CMK is incompatible with the API operation. + // + // * The encryption algorithm or signing algorithm specified for the operation + // is incompatible with the type of key material in the CMK (CustomerMasterKeySpec). + // + // For encrypting, decrypting, re-encrypting, and generating data keys, the + // KeyUsage must be ENCRYPT_DECRYPT. For signing and verifying, the KeyUsage + // must be SIGN_VERIFY. To find the KeyUsage of a CMK, use the DescribeKey operation. + // + // To find the encryption or signing algorithms supported for a particular CMK, + // use the DescribeKey operation. + ErrCodeInvalidKeyUsageException = "InvalidKeyUsageException" + + // ErrCodeInvalidMarkerException for service response error code + // "InvalidMarkerException". + // + // The request was rejected because the marker that specifies where pagination + // should next begin is not valid. + ErrCodeInvalidMarkerException = "InvalidMarkerException" + + // ErrCodeInvalidStateException for service response error code + // "KMSInvalidStateException". + // + // The request was rejected because the state of the specified resource is not + // valid for this request. + // + // For more information about how key state affects the use of a CMK, see How + // Key State Affects Use of a Customer Master Key (https://docs.aws.amazon.com/kms/latest/developerguide/key-state.html) + // in the AWS Key Management Service Developer Guide . + ErrCodeInvalidStateException = "KMSInvalidStateException" + + // ErrCodeKMSInvalidSignatureException for service response error code + // "KMSInvalidSignatureException". + // + // The request was rejected because the signature verification failed. Signature + // verification fails when it cannot confirm that signature was produced by + // signing the specified message with the specified CMK and signing algorithm. + ErrCodeKMSInvalidSignatureException = "KMSInvalidSignatureException" + + // ErrCodeKeyUnavailableException for service response error code + // "KeyUnavailableException". + // + // The request was rejected because the specified CMK was not available. You + // can retry the request. + ErrCodeKeyUnavailableException = "KeyUnavailableException" + + // ErrCodeLimitExceededException for service response error code + // "LimitExceededException". + // + // The request was rejected because a quota was exceeded. For more information, + // see Quotas (https://docs.aws.amazon.com/kms/latest/developerguide/limits.html) + // in the AWS Key Management Service Developer Guide. + ErrCodeLimitExceededException = "LimitExceededException" + + // ErrCodeMalformedPolicyDocumentException for service response error code + // "MalformedPolicyDocumentException". + // + // The request was rejected because the specified policy is not syntactically + // or semantically correct. + ErrCodeMalformedPolicyDocumentException = "MalformedPolicyDocumentException" + + // ErrCodeNotFoundException for service response error code + // "NotFoundException". + // + // The request was rejected because the specified entity or resource could not + // be found. + ErrCodeNotFoundException = "NotFoundException" + + // ErrCodeTagException for service response error code + // "TagException". + // + // The request was rejected because one or more tags are not valid. + ErrCodeTagException = "TagException" + + // ErrCodeUnsupportedOperationException for service response error code + // "UnsupportedOperationException". + // + // The request was rejected because a specified parameter is not supported or + // a specified resource is not valid for this operation. + ErrCodeUnsupportedOperationException = "UnsupportedOperationException" +) + +var exceptionFromCode = map[string]func(protocol.ResponseMetadata) error{ + "AlreadyExistsException": newErrorAlreadyExistsException, + "CloudHsmClusterInUseException": newErrorCloudHsmClusterInUseException, + "CloudHsmClusterInvalidConfigurationException": newErrorCloudHsmClusterInvalidConfigurationException, + "CloudHsmClusterNotActiveException": newErrorCloudHsmClusterNotActiveException, + "CloudHsmClusterNotFoundException": newErrorCloudHsmClusterNotFoundException, + "CloudHsmClusterNotRelatedException": newErrorCloudHsmClusterNotRelatedException, + "CustomKeyStoreHasCMKsException": newErrorCustomKeyStoreHasCMKsException, + "CustomKeyStoreInvalidStateException": newErrorCustomKeyStoreInvalidStateException, + "CustomKeyStoreNameInUseException": newErrorCustomKeyStoreNameInUseException, + "CustomKeyStoreNotFoundException": newErrorCustomKeyStoreNotFoundException, + "DependencyTimeoutException": newErrorDependencyTimeoutException, + "DisabledException": newErrorDisabledException, + "ExpiredImportTokenException": newErrorExpiredImportTokenException, + "IncorrectKeyException": newErrorIncorrectKeyException, + "IncorrectKeyMaterialException": newErrorIncorrectKeyMaterialException, + "IncorrectTrustAnchorException": newErrorIncorrectTrustAnchorException, + "KMSInternalException": newErrorInternalException, + "InvalidAliasNameException": newErrorInvalidAliasNameException, + "InvalidArnException": newErrorInvalidArnException, + "InvalidCiphertextException": newErrorInvalidCiphertextException, + "InvalidGrantIdException": newErrorInvalidGrantIdException, + "InvalidGrantTokenException": newErrorInvalidGrantTokenException, + "InvalidImportTokenException": newErrorInvalidImportTokenException, + "InvalidKeyUsageException": newErrorInvalidKeyUsageException, + "InvalidMarkerException": newErrorInvalidMarkerException, + "KMSInvalidStateException": newErrorInvalidStateException, + "KMSInvalidSignatureException": newErrorKMSInvalidSignatureException, + "KeyUnavailableException": newErrorKeyUnavailableException, + "LimitExceededException": newErrorLimitExceededException, + "MalformedPolicyDocumentException": newErrorMalformedPolicyDocumentException, + "NotFoundException": newErrorNotFoundException, + "TagException": newErrorTagException, + "UnsupportedOperationException": newErrorUnsupportedOperationException, +} diff --git a/github.com/aws/aws-sdk-go/service/kms/service.go b/github.com/aws/aws-sdk-go/service/kms/service.go new file mode 100644 index 0000000000..50ca0c092e --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/kms/service.go @@ -0,0 +1,103 @@ +// Code generated by private/model/cli/gen-api/main.go. DO NOT EDIT. + +package kms + +import ( + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/client" + "github.com/aws/aws-sdk-go/aws/client/metadata" + "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/aws/signer/v4" + "github.com/aws/aws-sdk-go/private/protocol" + "github.com/aws/aws-sdk-go/private/protocol/jsonrpc" +) + +// KMS provides the API operation methods for making requests to +// AWS Key Management Service. See this package's package overview docs +// for details on the service. +// +// KMS methods are safe to use concurrently. It is not safe to +// modify mutate any of the struct's properties though. +type KMS struct { + *client.Client +} + +// Used for custom client initialization logic +var initClient func(*client.Client) + +// Used for custom request initialization logic +var initRequest func(*request.Request) + +// Service information constants +const ( + ServiceName = "kms" // Name of service. + EndpointsID = ServiceName // ID to lookup a service endpoint with. + ServiceID = "KMS" // ServiceID is a unique identifier of a specific service. +) + +// New creates a new instance of the KMS client with a session. +// If additional configuration is needed for the client instance use the optional +// aws.Config parameter to add your extra config. +// +// Example: +// mySession := session.Must(session.NewSession()) +// +// // Create a KMS client from just a session. +// svc := kms.New(mySession) +// +// // Create a KMS client with additional configuration +// svc := kms.New(mySession, aws.NewConfig().WithRegion("us-west-2")) +func New(p client.ConfigProvider, cfgs ...*aws.Config) *KMS { + c := p.ClientConfig(EndpointsID, cfgs...) + return newClient(*c.Config, c.Handlers, c.PartitionID, c.Endpoint, c.SigningRegion, c.SigningName) +} + +// newClient creates, initializes and returns a new service client instance. +func newClient(cfg aws.Config, handlers request.Handlers, partitionID, endpoint, signingRegion, signingName string) *KMS { + svc := &KMS{ + Client: client.New( + cfg, + metadata.ClientInfo{ + ServiceName: ServiceName, + ServiceID: ServiceID, + SigningName: signingName, + SigningRegion: signingRegion, + PartitionID: partitionID, + Endpoint: endpoint, + APIVersion: "2014-11-01", + JSONVersion: "1.1", + TargetPrefix: "TrentService", + }, + handlers, + ), + } + + // Handlers + svc.Handlers.Sign.PushBackNamed(v4.SignRequestHandler) + svc.Handlers.Build.PushBackNamed(jsonrpc.BuildHandler) + svc.Handlers.Unmarshal.PushBackNamed(jsonrpc.UnmarshalHandler) + svc.Handlers.UnmarshalMeta.PushBackNamed(jsonrpc.UnmarshalMetaHandler) + svc.Handlers.UnmarshalError.PushBackNamed( + protocol.NewUnmarshalErrorHandler(jsonrpc.NewUnmarshalTypedError(exceptionFromCode)).NamedHandler(), + ) + + // Run custom client initialization if present + if initClient != nil { + initClient(svc.Client) + } + + return svc +} + +// newRequest creates a new request for a KMS operation and runs any +// custom request initialization. +func (c *KMS) newRequest(op *request.Operation, params, data interface{}) *request.Request { + req := c.NewRequest(op, params, data) + + // Run custom request initialization if present + if initRequest != nil { + initRequest(req) + } + + return req +} diff --git a/github.com/aws/aws-sdk-go/service/s3/api.go b/github.com/aws/aws-sdk-go/service/s3/api.go index 3b3c2e380c..82defe7f2c 100644 --- a/github.com/aws/aws-sdk-go/service/s3/api.go +++ b/github.com/aws/aws-sdk-go/service/s3/api.go @@ -7,7 +7,6 @@ import ( "fmt" "io" "sync" - "sync/atomic" "time" "github.com/aws/aws-sdk-go/aws" @@ -15,11 +14,13 @@ import ( "github.com/aws/aws-sdk-go/aws/awsutil" "github.com/aws/aws-sdk-go/aws/client" "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/private/checksum" "github.com/aws/aws-sdk-go/private/protocol" "github.com/aws/aws-sdk-go/private/protocol/eventstream" "github.com/aws/aws-sdk-go/private/protocol/eventstream/eventstreamapi" "github.com/aws/aws-sdk-go/private/protocol/rest" "github.com/aws/aws-sdk-go/private/protocol/restxml" + "github.com/aws/aws-sdk-go/service/s3/internal/arn" ) const opAbortMultipartUpload = "AbortMultipartUpload" @@ -66,11 +67,31 @@ func (c *S3) AbortMultipartUploadRequest(input *AbortMultipartUploadInput) (req // AbortMultipartUpload API operation for Amazon Simple Storage Service. // -// Aborts a multipart upload. +// This operation aborts a multipart upload. After a multipart upload is aborted, +// no additional parts can be uploaded using that upload ID. The storage consumed +// by any previously uploaded parts will be freed. However, if any part uploads +// are currently in progress, those part uploads might or might not succeed. +// As a result, it might be necessary to abort a given multipart upload multiple +// times in order to completely free all storage consumed by all parts. // // To verify that all parts have been removed, so you don't get charged for -// the part storage, you should call the List Parts operation and ensure the -// parts list is empty. +// the part storage, you should call the ListParts operation and ensure that +// the parts list is empty. +// +// For information about permissions required to use the multipart upload API, +// see Multipart Upload API and Permissions (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuAndPermissions.html). +// +// The following operations are related to AbortMultipartUpload: +// +// * CreateMultipartUpload +// +// * UploadPart +// +// * CompleteMultipartUpload +// +// * ListParts +// +// * ListMultipartUploads // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -151,6 +172,64 @@ func (c *S3) CompleteMultipartUploadRequest(input *CompleteMultipartUploadInput) // // Completes a multipart upload by assembling previously uploaded parts. // +// You first initiate the multipart upload and then upload all parts using the +// UploadPart operation. After successfully uploading all relevant parts of +// an upload, you call this operation to complete the upload. Upon receiving +// this request, Amazon S3 concatenates all the parts in ascending order by +// part number to create a new object. In the Complete Multipart Upload request, +// you must provide the parts list. You must ensure that the parts list is complete. +// This operation concatenates the parts that you provide in the list. For each +// part in the list, you must provide the part number and the ETag value, returned +// after that part was uploaded. +// +// Processing of a Complete Multipart Upload request could take several minutes +// to complete. After Amazon S3 begins processing the request, it sends an HTTP +// response header that specifies a 200 OK response. While processing is in +// progress, Amazon S3 periodically sends white space characters to keep the +// connection from timing out. Because a request could fail after the initial +// 200 OK response has been sent, it is important that you check the response +// body to determine whether the request succeeded. +// +// Note that if CompleteMultipartUpload fails, applications should be prepared +// to retry the failed requests. For more information, see Amazon S3 Error Best +// Practices (https://docs.aws.amazon.com/AmazonS3/latest/dev/ErrorBestPractices.html). +// +// For more information about multipart uploads, see Uploading Objects Using +// Multipart Upload (https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html). +// +// For information about permissions required to use the multipart upload API, +// see Multipart Upload API and Permissions (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuAndPermissions.html). +// +// GetBucketLifecycle has the following special errors: +// +// * Error code: EntityTooSmall Description: Your proposed upload is smaller +// than the minimum allowed object size. Each part must be at least 5 MB +// in size, except the last part. 400 Bad Request +// +// * Error code: InvalidPart Description: One or more of the specified parts +// could not be found. The part might not have been uploaded, or the specified +// entity tag might not have matched the part's entity tag. 400 Bad Request +// +// * Error code: InvalidPartOrder Description: The list of parts was not +// in ascending order. The parts list must be specified in order by part +// number. 400 Bad Request +// +// * Error code: NoSuchUpload Description: The specified multipart upload +// does not exist. The upload ID might be invalid, or the multipart upload +// might have been aborted or completed. 404 Not Found +// +// The following operations are related to CompleteMultipartUpload: +// +// * CreateMultipartUpload +// +// * UploadPart +// +// * AbortMultipartUpload +// +// * ListParts +// +// * ListMultipartUploads +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -225,6 +304,153 @@ func (c *S3) CopyObjectRequest(input *CopyObjectInput) (req *request.Request, ou // // Creates a copy of an object that is already stored in Amazon S3. // +// You can store individual objects of up to 5 TB in Amazon S3. You create a +// copy of your object up to 5 GB in size in a single atomic operation using +// this API. However, to copy an object greater than 5 GB, you must use the +// multipart upload Upload Part - Copy API. For more information, see Copy Object +// Using the REST Multipart Upload API (https://docs.aws.amazon.com/AmazonS3/latest/dev/CopyingObjctsUsingRESTMPUapi.html). +// +// All copy requests must be authenticated. Additionally, you must have read +// access to the source object and write access to the destination bucket. For +// more information, see REST Authentication (https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html). +// Both the Region that you want to copy the object from and the Region that +// you want to copy the object to must be enabled for your account. +// +// A copy request might return an error when Amazon S3 receives the copy request +// or while Amazon S3 is copying the files. If the error occurs before the copy +// operation starts, you receive a standard Amazon S3 error. If the error occurs +// during the copy operation, the error response is embedded in the 200 OK response. +// This means that a 200 OK response can contain either a success or an error. +// Design your application to parse the contents of the response and handle +// it appropriately. +// +// If the copy is successful, you receive a response with information about +// the copied object. +// +// If the request is an HTTP 1.1 request, the response is chunk encoded. If +// it were not, it would not contain the content-length, and you would need +// to read the entire body. +// +// The copy request charge is based on the storage class and Region that you +// specify for the destination object. For pricing information, see Amazon S3 +// pricing (https://aws.amazon.com/s3/pricing/). +// +// Amazon S3 transfer acceleration does not support cross-Region copies. If +// you request a cross-Region copy using a transfer acceleration endpoint, you +// get a 400 Bad Request error. For more information, see Transfer Acceleration +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html). +// +// Metadata +// +// When copying an object, you can preserve all metadata (default) or specify +// new metadata. However, the ACL is not preserved and is set to private for +// the user making the request. To override the default ACL setting, specify +// a new ACL when generating a copy request. For more information, see Using +// ACLs (https://docs.aws.amazon.com/AmazonS3/latest/dev/S3_ACLs_UsingACLs.html). +// +// To specify whether you want the object metadata copied from the source object +// or replaced with metadata provided in the request, you can optionally add +// the x-amz-metadata-directive header. When you grant permissions, you can +// use the s3:x-amz-metadata-directive condition key to enforce certain metadata +// behavior when objects are uploaded. For more information, see Specifying +// Conditions in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/amazon-s3-policy-keys.html) +// in the Amazon S3 Developer Guide. For a complete list of Amazon S3-specific +// condition keys, see Actions, Resources, and Condition Keys for Amazon S3 +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/list_amazons3.html). +// +// x-amz-copy-source-if Headers +// +// To only copy an object under certain conditions, such as whether the Etag +// matches or whether the object was modified before or after a specified date, +// use the following request parameters: +// +// * x-amz-copy-source-if-match +// +// * x-amz-copy-source-if-none-match +// +// * x-amz-copy-source-if-unmodified-since +// +// * x-amz-copy-source-if-modified-since +// +// If both the x-amz-copy-source-if-match and x-amz-copy-source-if-unmodified-since +// headers are present in the request and evaluate as follows, Amazon S3 returns +// 200 OK and copies the data: +// +// * x-amz-copy-source-if-match condition evaluates to true +// +// * x-amz-copy-source-if-unmodified-since condition evaluates to false +// +// If both the x-amz-copy-source-if-none-match and x-amz-copy-source-if-modified-since +// headers are present in the request and evaluate as follows, Amazon S3 returns +// the 412 Precondition Failed response code: +// +// * x-amz-copy-source-if-none-match condition evaluates to false +// +// * x-amz-copy-source-if-modified-since condition evaluates to true +// +// All headers with the x-amz- prefix, including x-amz-copy-source, must be +// signed. +// +// Encryption +// +// The source object that you are copying can be encrypted or unencrypted. The +// source object can be encrypted with server-side encryption using AWS managed +// encryption keys (SSE-S3 or SSE-KMS) or by using a customer-provided encryption +// key. With server-side encryption, Amazon S3 encrypts your data as it writes +// it to disks in its data centers and decrypts the data when you access it. +// +// You can optionally use the appropriate encryption-related headers to request +// server-side encryption for the target object. You have the option to provide +// your own encryption key or use SSE-S3 or SSE-KMS, regardless of the form +// of server-side encryption that was used to encrypt the source object. You +// can even request encryption if the source object was not encrypted. For more +// information about server-side encryption, see Using Server-Side Encryption +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html). +// +// Access Control List (ACL)-Specific Request Headers +// +// When copying an object, you can optionally use headers to grant ACL-based +// permissions. By default, all objects are private. Only the owner has full +// access control. When adding a new object, you can grant permissions to individual +// AWS accounts or to predefined groups defined by Amazon S3. These permissions +// are then added to the ACL on the object. For more information, see Access +// Control List (ACL) Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html) +// and Managing ACLs Using the REST API (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-using-rest-api.html). +// +// Storage Class Options +// +// You can use the CopyObject operation to change the storage class of an object +// that is already stored in Amazon S3 using the StorageClass parameter. For +// more information, see Storage Classes (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html) +// in the Amazon S3 Service Developer Guide. +// +// Versioning +// +// By default, x-amz-copy-source identifies the current version of an object +// to copy. If the current version is a delete marker, Amazon S3 behaves as +// if the object was deleted. To copy a different version, use the versionId +// subresource. +// +// If you enable versioning on the target bucket, Amazon S3 generates a unique +// version ID for the object being copied. This version ID is different from +// the version ID of the source object. Amazon S3 returns the version ID of +// the copied object in the x-amz-version-id response header in the response. +// +// If you do not enable versioning or suspend it on the target bucket, the version +// ID that Amazon S3 generates is always null. +// +// If the source object's storage class is GLACIER, you must restore a copy +// of this object before you can use it as a source object for the copy operation. +// For more information, see . +// +// The following operations are related to CopyObject: +// +// * PutObject +// +// * GetObject +// +// For more information, see Copying Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/CopyingObjectsExamples.html). +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -235,7 +461,7 @@ func (c *S3) CopyObjectRequest(input *CopyObjectInput) (req *request.Request, ou // Returned Error Codes: // * ErrCodeObjectNotInActiveTierError "ObjectNotInActiveTierError" // The source object of the COPY operation is not in the active tier and is -// only stored in Amazon Glacier. +// only stored in Amazon S3 Glacier. // // See also, https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/CopyObject func (c *S3) CopyObject(input *CopyObjectInput) (*CopyObjectOutput, error) { @@ -303,7 +529,67 @@ func (c *S3) CreateBucketRequest(input *CreateBucketInput) (req *request.Request // CreateBucket API operation for Amazon Simple Storage Service. // -// Creates a new bucket. +// Creates a new bucket. To create a bucket, you must register with Amazon S3 +// and have a valid AWS Access Key ID to authenticate requests. Anonymous requests +// are never allowed to create buckets. By creating the bucket, you become the +// bucket owner. +// +// Not every string is an acceptable bucket name. For information on bucket +// naming restrictions, see Working with Amazon S3 Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html). +// +// By default, the bucket is created in the US East (N. Virginia) Region. You +// can optionally specify a Region in the request body. You might choose a Region +// to optimize latency, minimize costs, or address regulatory requirements. +// For example, if you reside in Europe, you will probably find it advantageous +// to create buckets in the Europe (Ireland) Region. For more information, see +// How to Select a Region for Your Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro). +// +// If you send your create bucket request to the s3.amazonaws.com endpoint, +// the request goes to the us-east-1 Region. Accordingly, the signature calculations +// in Signature Version 4 must use us-east-1 as the Region, even if the location +// constraint in the request specifies another Region where the bucket is to +// be created. If you create a bucket in a Region other than US East (N. Virginia), +// your application must be able to handle 307 redirect. For more information, +// see Virtual Hosting of Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html). +// +// When creating a bucket using this operation, you can optionally specify the +// accounts or groups that should be granted specific permissions on the bucket. +// There are two ways to grant the appropriate permissions using the request +// headers. +// +// * Specify a canned ACL using the x-amz-acl request header. Amazon S3 supports +// a set of predefined ACLs, known as canned ACLs. Each canned ACL has a +// predefined set of grantees and permissions. For more information, see +// Canned ACL (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL). +// +// * Specify access permissions explicitly using the x-amz-grant-read, x-amz-grant-write, +// x-amz-grant-read-acp, x-amz-grant-write-acp, and x-amz-grant-full-control +// headers. These headers map to the set of permissions Amazon S3 supports +// in an ACL. For more information, see Access Control List (ACL) Overview +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html). You +// specify each grantee as a type=value pair, where the type is one of the +// following: id – if the value specified is the canonical user ID of an +// AWS account uri – if you are granting permissions to a predefined group +// emailAddress – if the value specified is the email address of an AWS +// account Using email addresses to specify a grantee is only supported in +// the following AWS Regions: US East (N. Virginia) US West (N. California) +// US West (Oregon) Asia Pacific (Singapore) Asia Pacific (Sydney) Asia Pacific +// (Tokyo) Europe (Ireland) South America (São Paulo) For a list of all +// the Amazon S3 supported Regions and endpoints, see Regions and Endpoints +// (https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) in +// the AWS General Reference. For example, the following x-amz-grant-read +// header grants the AWS accounts identified by account IDs permissions to +// read object data and its metadata: x-amz-grant-read: id="11112222333", +// id="444455556666" +// +// You can use either a canned ACL or specify access permissions explicitly. +// You cannot do both. +// +// The following operations are related to CreateBucket: +// +// * PutObject +// +// * DeleteBucket // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -318,6 +604,11 @@ func (c *S3) CreateBucketRequest(input *CreateBucketInput) (req *request.Request // by all users of the system. Please select a different name and try again. // // * ErrCodeBucketAlreadyOwnedByYou "BucketAlreadyOwnedByYou" +// The bucket you tried to create already exists, and you own it. Amazon S3 +// returns this error in all AWS Regions except in the North Virginia Region. +// For legacy compatibility, if you re-create an existing bucket that you already +// own in the North Virginia Region, Amazon S3 returns 200 OK and resets the +// bucket access control lists (ACLs). // // See also, https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/CreateBucket func (c *S3) CreateBucket(input *CreateBucketInput) (*CreateBucketOutput, error) { @@ -385,13 +676,154 @@ func (c *S3) CreateMultipartUploadRequest(input *CreateMultipartUploadInput) (re // CreateMultipartUpload API operation for Amazon Simple Storage Service. // -// Initiates a multipart upload and returns an upload ID. -// -// Note: After you initiate multipart upload and upload one or more parts, you -// must either complete or abort multipart upload in order to stop getting charged -// for storage of the uploaded parts. Only after you either complete or abort -// multipart upload, Amazon S3 frees up the parts storage and stops charging -// you for the parts storage. +// This operation initiates a multipart upload and returns an upload ID. This +// upload ID is used to associate all of the parts in the specific multipart +// upload. You specify this upload ID in each of your subsequent upload part +// requests (see UploadPart). You also include this upload ID in the final request +// to either complete or abort the multipart upload request. +// +// For more information about multipart uploads, see Multipart Upload Overview +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html). +// +// If you have configured a lifecycle rule to abort incomplete multipart uploads, +// the upload must complete within the number of days specified in the bucket +// lifecycle configuration. Otherwise, the incomplete multipart upload becomes +// eligible for an abort operation and Amazon S3 aborts the multipart upload. +// For more information, see Aborting Incomplete Multipart Uploads Using a Bucket +// Lifecycle Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-abort-incomplete-mpu-lifecycle-config). +// +// For information about the permissions required to use the multipart upload +// API, see Multipart Upload API and Permissions (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuAndPermissions.html). +// +// For request signing, multipart upload is just a series of regular requests. +// You initiate a multipart upload, send one or more requests to upload parts, +// and then complete the multipart upload process. You sign each request individually. +// There is nothing special about signing multipart upload requests. For more +// information about signing, see Authenticating Requests (AWS Signature Version +// 4) (https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). +// +// After you initiate a multipart upload and upload one or more parts, to stop +// being charged for storing the uploaded parts, you must either complete or +// abort the multipart upload. Amazon S3 frees up the space used to store the +// parts and stop charging you for storing them only after you either complete +// or abort a multipart upload. +// +// You can optionally request server-side encryption. For server-side encryption, +// Amazon S3 encrypts your data as it writes it to disks in its data centers +// and decrypts it when you access it. You can provide your own encryption key, +// or use AWS Key Management Service (AWS KMS) customer master keys (CMKs) or +// Amazon S3-managed encryption keys. If you choose to provide your own encryption +// key, the request headers you provide in UploadPart) and UploadPartCopy) requests +// must match the headers you used in the request to initiate the upload by +// using CreateMultipartUpload. +// +// To perform a multipart upload with encryption using an AWS KMS CMK, the requester +// must have permission to the kms:Encrypt, kms:Decrypt, kms:ReEncrypt*, kms:GenerateDataKey*, +// and kms:DescribeKey actions on the key. These permissions are required because +// Amazon S3 must decrypt and read data from the encrypted file parts before +// it completes the multipart upload. +// +// If your AWS Identity and Access Management (IAM) user or role is in the same +// AWS account as the AWS KMS CMK, then you must have these permissions on the +// key policy. If your IAM user or role belongs to a different account than +// the key, then you must have the permissions on both the key policy and your +// IAM user or role. +// +// For more information, see Protecting Data Using Server-Side Encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html). +// +// Access Permissions +// +// When copying an object, you can optionally specify the accounts or groups +// that should be granted specific permissions on the new object. There are +// two ways to grant the permissions using the request headers: +// +// * Specify a canned ACL with the x-amz-acl request header. For more information, +// see Canned ACL (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL). +// +// * Specify access permissions explicitly with the x-amz-grant-read, x-amz-grant-read-acp, +// x-amz-grant-write-acp, and x-amz-grant-full-control headers. These parameters +// map to the set of permissions that Amazon S3 supports in an ACL. For more +// information, see Access Control List (ACL) Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html). +// +// You can use either a canned ACL or specify access permissions explicitly. +// You cannot do both. +// +// Server-Side- Encryption-Specific Request Headers +// +// You can optionally tell Amazon S3 to encrypt data at rest using server-side +// encryption. Server-side encryption is for data encryption at rest. Amazon +// S3 encrypts your data as it writes it to disks in its data centers and decrypts +// it when you access it. The option you use depends on whether you want to +// use AWS managed encryption keys or provide your own encryption key. +// +// * Use encryption keys managed by Amazon S3 or customer master keys (CMKs) +// stored in AWS Key Management Service (AWS KMS) – If you want AWS to +// manage the keys used to encrypt data, specify the following headers in +// the request. x-amz-server-side​-encryption x-amz-server-side-encryption-aws-kms-key-id +// x-amz-server-side-encryption-context If you specify x-amz-server-side-encryption:aws:kms, +// but don't provide x-amz-server-side-encryption-aws-kms-key-id, Amazon +// S3 uses the AWS managed CMK in AWS KMS to protect the data. All GET and +// PUT requests for an object protected by AWS KMS fail if you don't make +// them with SSL or by using SigV4. For more information about server-side +// encryption with CMKs stored in AWS KMS (SSE-KMS), see Protecting Data +// Using Server-Side Encryption with CMKs stored in AWS KMS (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). +// +// * Use customer-provided encryption keys – If you want to manage your +// own encryption keys, provide all the following headers in the request. +// x-amz-server-side​-encryption​-customer-algorithm x-amz-server-side​-encryption​-customer-key +// x-amz-server-side​-encryption​-customer-key-MD5 For more information +// about server-side encryption with CMKs stored in AWS KMS (SSE-KMS), see +// Protecting Data Using Server-Side Encryption with CMKs stored in AWS KMS +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html). +// +// Access-Control-List (ACL)-Specific Request Headers +// +// You also can use the following access control–related headers with this +// operation. By default, all objects are private. Only the owner has full access +// control. When adding a new object, you can grant permissions to individual +// AWS accounts or to predefined groups defined by Amazon S3. These permissions +// are then added to the access control list (ACL) on the object. For more information, +// see Using ACLs (https://docs.aws.amazon.com/AmazonS3/latest/dev/S3_ACLs_UsingACLs.html). +// With this operation, you can grant access permissions using one of the following +// two methods: +// +// * Specify a canned ACL (x-amz-acl) — Amazon S3 supports a set of predefined +// ACLs, known as canned ACLs. Each canned ACL has a predefined set of grantees +// and permissions. For more information, see Canned ACL (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL). +// +// * Specify access permissions explicitly — To explicitly grant access +// permissions to specific AWS accounts or groups, use the following headers. +// Each header maps to specific permissions that Amazon S3 supports in an +// ACL. For more information, see Access Control List (ACL) Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html). +// In the header, you specify a list of grantees who get the specific permission. +// To grant permissions explicitly, use: x-amz-grant-read x-amz-grant-write +// x-amz-grant-read-acp x-amz-grant-write-acp x-amz-grant-full-control You +// specify each grantee as a type=value pair, where the type is one of the +// following: id – if the value specified is the canonical user ID of an +// AWS account uri – if you are granting permissions to a predefined group +// emailAddress – if the value specified is the email address of an AWS +// account Using email addresses to specify a grantee is only supported in +// the following AWS Regions: US East (N. Virginia) US West (N. California) +// US West (Oregon) Asia Pacific (Singapore) Asia Pacific (Sydney) Asia Pacific +// (Tokyo) Europe (Ireland) South America (São Paulo) For a list of all +// the Amazon S3 supported Regions and endpoints, see Regions and Endpoints +// (https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) in +// the AWS General Reference. For example, the following x-amz-grant-read +// header grants the AWS accounts identified by account IDs permissions to +// read object data and its metadata: x-amz-grant-read: id="11112222333", +// id="444455556666" +// +// The following operations are related to CreateMultipartUpload: +// +// * UploadPart +// +// * CompleteMultipartUpload +// +// * AbortMultipartUpload +// +// * ListParts +// +// * ListMultipartUploads // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -466,8 +898,14 @@ func (c *S3) DeleteBucketRequest(input *DeleteBucketInput) (req *request.Request // DeleteBucket API operation for Amazon Simple Storage Service. // -// Deletes the bucket. All objects (including all object versions and Delete -// Markers) in the bucket must be deleted before the bucket itself can be deleted. +// Deletes the bucket. All objects (including all object versions and delete +// markers) in the bucket must be deleted before the bucket itself can be deleted. +// +// Related Resources +// +// * +// +// * // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -545,6 +983,23 @@ func (c *S3) DeleteBucketAnalyticsConfigurationRequest(input *DeleteBucketAnalyt // Deletes an analytics configuration for the bucket (specified by the analytics // configuration ID). // +// To use this operation, you must have permissions to perform the s3:PutAnalyticsConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// For information about the Amazon S3 analytics feature, see Amazon S3 Analytics +// – Storage Class Analysis (https://docs.aws.amazon.com/AmazonS3/latest/dev/analytics-storage-class.html). +// +// The following operations are related to DeleteBucketAnalyticsConfiguration: +// +// * +// +// * +// +// * +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -618,7 +1073,20 @@ func (c *S3) DeleteBucketCorsRequest(input *DeleteBucketCorsInput) (req *request // DeleteBucketCors API operation for Amazon Simple Storage Service. // -// Deletes the CORS configuration information set for the bucket. +// Deletes the cors configuration information set for the bucket. +// +// To use this operation, you must have permission to perform the s3:PutBucketCORS +// action. The bucket owner has this permission by default and can grant this +// permission to others. +// +// For information about cors, see Enabling Cross-Origin Resource Sharing (https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Related Resources: +// +// * +// +// * RESTOPTIONSobject // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -693,7 +1161,23 @@ func (c *S3) DeleteBucketEncryptionRequest(input *DeleteBucketEncryptionInput) ( // DeleteBucketEncryption API operation for Amazon Simple Storage Service. // -// Deletes the server-side encryption configuration from the bucket. +// This implementation of the DELETE operation removes default encryption from +// the bucket. For information about the Amazon S3 default encryption feature, +// see Amazon S3 Default Bucket Encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// To use this operation, you must have permissions to perform the s3:PutEncryptionConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Related Resources +// +// * PutBucketEncryption +// +// * GetBucketEncryption // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -771,6 +1255,23 @@ func (c *S3) DeleteBucketInventoryConfigurationRequest(input *DeleteBucketInvent // Deletes an inventory configuration (identified by the inventory ID) from // the bucket. // +// To use this operation, you must have permissions to perform the s3:PutInventoryConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// For information about the Amazon S3 inventory feature, see Amazon S3 Inventory +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html). +// +// Operations related to DeleteBucketInventoryConfiguration include: +// +// * GetBucketInventoryConfiguration +// +// * PutBucketInventoryConfiguration +// +// * ListBucketInventoryConfigurations +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -844,7 +1345,27 @@ func (c *S3) DeleteBucketLifecycleRequest(input *DeleteBucketLifecycleInput) (re // DeleteBucketLifecycle API operation for Amazon Simple Storage Service. // -// Deletes the lifecycle configuration from the bucket. +// Deletes the lifecycle configuration from the specified bucket. Amazon S3 +// removes all the lifecycle configuration rules in the lifecycle subresource +// associated with the bucket. Your objects never expire, and Amazon S3 no longer +// automatically deletes any objects on the basis of rules contained in the +// deleted lifecycle configuration. +// +// To use this operation, you must have permission to perform the s3:PutLifecycleConfiguration +// action. By default, the bucket owner has this permission and the bucket owner +// can grant this permission to others. +// +// There is usually some time lag before lifecycle configuration deletion is +// fully propagated to all the Amazon S3 systems. +// +// For more information about the object expiration, see Elements to Describe +// Lifecycle Actions (https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html#intro-lifecycle-rules-actions). +// +// Related actions include: +// +// * PutBucketLifecycleConfiguration +// +// * GetBucketLifecycleConfiguration // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -919,8 +1440,28 @@ func (c *S3) DeleteBucketMetricsConfigurationRequest(input *DeleteBucketMetricsC // DeleteBucketMetricsConfiguration API operation for Amazon Simple Storage Service. // -// Deletes a metrics configuration (specified by the metrics configuration ID) -// from the bucket. +// Deletes a metrics configuration for the Amazon CloudWatch request metrics +// (specified by the metrics configuration ID) from the bucket. Note that this +// doesn't include the daily storage metrics. +// +// To use this operation, you must have permissions to perform the s3:PutMetricsConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// For information about CloudWatch request metrics for Amazon S3, see Monitoring +// Metrics with Amazon CloudWatch (https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html). +// +// The following operations are related to DeleteBucketMetricsConfiguration: +// +// * GetBucketMetricsConfiguration +// +// * PutBucketMetricsConfiguration +// +// * ListBucketMetricsConfigurations +// +// * Monitoring Metrics with Amazon CloudWatch (https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html) // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -995,7 +1536,29 @@ func (c *S3) DeleteBucketPolicyRequest(input *DeleteBucketPolicyInput) (req *req // DeleteBucketPolicy API operation for Amazon Simple Storage Service. // -// Deletes the policy from the bucket. +// This implementation of the DELETE operation uses the policy subresource to +// delete the policy of a specified bucket. If you are using an identity other +// than the root user of the AWS account that owns the bucket, the calling identity +// must have the DeleteBucketPolicy permissions on the specified bucket and +// belong to the bucket owner's account to use this operation. +// +// If you don't have DeleteBucketPolicy permissions, Amazon S3 returns a 403 +// Access Denied error. If you have the correct permissions, but you're not +// using an identity that belongs to the bucket owner's account, Amazon S3 returns +// a 405 Method Not Allowed error. +// +// As a security precaution, the root user of the AWS account that owns a bucket +// can always use this operation, even if the policy explicitly denies the root +// user the ability to perform this action. +// +// For more information about bucket policies, see Using Bucket Policies and +// UserPolicies (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html). +// +// The following operations are related to DeleteBucketPolicy +// +// * CreateBucket +// +// * DeleteObject // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -1070,10 +1633,26 @@ func (c *S3) DeleteBucketReplicationRequest(input *DeleteBucketReplicationInput) // DeleteBucketReplication API operation for Amazon Simple Storage Service. // -// Deletes the replication configuration from the bucket. For information about -// replication configuration, see Cross-Region Replication (CRR) ( https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) +// Deletes the replication configuration from the bucket. +// +// To use this operation, you must have permissions to perform the s3:PutReplicationConfiguration +// action. The bucket owner has these permissions by default and can grant it +// to others. For more information about permissions, see Permissions Related +// to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// It can take a while for the deletion of a replication configuration to fully +// propagate. +// +// For information about replication configuration, see Replication (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.html) // in the Amazon S3 Developer Guide. // +// The following operations are related to DeleteBucketReplication: +// +// * PutBucketReplication +// +// * GetBucketReplication +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -1149,6 +1728,16 @@ func (c *S3) DeleteBucketTaggingRequest(input *DeleteBucketTaggingInput) (req *r // // Deletes the tags from the bucket. // +// To use this operation, you must have permission to perform the s3:PutBucketTagging +// action. By default, the bucket owner has this permission and can grant this +// permission to others. +// +// The following operations are related to DeleteBucketTagging: +// +// * GetBucketTagging +// +// * PutBucketTagging +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -1222,7 +1811,26 @@ func (c *S3) DeleteBucketWebsiteRequest(input *DeleteBucketWebsiteInput) (req *r // DeleteBucketWebsite API operation for Amazon Simple Storage Service. // -// This operation removes the website configuration from the bucket. +// This operation removes the website configuration for a bucket. Amazon S3 +// returns a 200 OK response upon successfully deleting a website configuration +// on the specified bucket. You will get a 200 OK response if the website configuration +// you are trying to delete does not exist on the bucket. Amazon S3 returns +// a 404 response if the bucket specified in the request does not exist. +// +// This DELETE operation requires the S3:DeleteBucketWebsite permission. By +// default, only the bucket owner can delete the website configuration attached +// to a bucket. However, bucket owners can grant other users permission to delete +// the website configuration by writing a bucket policy granting them the S3:DeleteBucketWebsite +// permission. +// +// For more information about hosting websites, see Hosting Websites on Amazon +// S3 (https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html). +// +// The following operations are related to DeleteBucketWebsite: +// +// * GetBucketWebsite +// +// * PutBucketWebsite // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -1300,6 +1908,29 @@ func (c *S3) DeleteObjectRequest(input *DeleteObjectInput) (req *request.Request // marker, which becomes the latest version of the object. If there isn't a // null version, Amazon S3 does not remove any objects. // +// To remove a specific version, you must be the bucket owner and you must use +// the version Id subresource. Using this subresource permanently deletes the +// version. If the object deleted is a delete marker, Amazon S3 sets the response +// header, x-amz-delete-marker, to true. +// +// If the object you want to delete is in a bucket where the bucket versioning +// configuration is MFA Delete enabled, you must include the x-amz-mfa request +// header in the DELETE versionId request. Requests that include x-amz-mfa must +// use HTTPS. +// +// For more information about MFA Delete, see Using MFA Delete (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMFADelete.html). +// To see sample requests that use versioning, see Sample Request (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectDELETE.html#ExampleVersionObjectDelete). +// +// You can delete objects by explicitly calling the DELETE Object API or configure +// its lifecycle (PutBucketLifecycle) to enable Amazon S3 to remove them for +// you. If you want to block users or accounts from removing or deleting objects +// from your bucket, you must deny them the s3:DeleteObject, s3:DeleteObjectVersion, +// and s3:PutLifeCycleConfiguration actions. +// +// The following operation is related to DeleteObject: +// +// * PutObject +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -1372,7 +2003,21 @@ func (c *S3) DeleteObjectTaggingRequest(input *DeleteObjectTaggingInput) (req *r // DeleteObjectTagging API operation for Amazon Simple Storage Service. // -// Removes the tag-set from an existing object. +// Removes the entire tag set from the specified object. For more information +// about managing object tags, see Object Tagging (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html). +// +// To use this operation, you must have permission to perform the s3:DeleteObjectTagging +// action. +// +// To delete tags of a specific object version, add the versionId query parameter +// in the request. You will need permission for the s3:DeleteObjectVersionTagging +// action. +// +// The following operations are related to DeleteBucketMetricsConfiguration: +// +// * PutObjectTagging +// +// * GetObjectTagging // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -1441,13 +2086,57 @@ func (c *S3) DeleteObjectsRequest(input *DeleteObjectsInput) (req *request.Reque output = &DeleteObjectsOutput{} req = c.newRequest(op, input, output) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // DeleteObjects API operation for Amazon Simple Storage Service. // // This operation enables you to delete multiple objects from a bucket using -// a single HTTP request. You may specify up to 1000 keys. +// a single HTTP request. If you know the object keys that you want to delete, +// then this operation provides a suitable alternative to sending individual +// delete requests, reducing per-request overhead. +// +// The request contains a list of up to 1000 keys that you want to delete. In +// the XML, you provide the object key names, and optionally, version IDs if +// you want to delete a specific version of the object from a versioning-enabled +// bucket. For each key, Amazon S3 performs a delete operation and returns the +// result of that delete, success, or failure, in the response. Note that if +// the object specified in the request is not found, Amazon S3 returns the result +// as deleted. +// +// The operation supports two modes for the response: verbose and quiet. By +// default, the operation uses verbose mode in which the response includes the +// result of deletion of each key in your request. In quiet mode the response +// includes only keys where the delete operation encountered an error. For a +// successful deletion, the operation does not return any information about +// the delete in the response body. +// +// When performing this operation on an MFA Delete enabled bucket, that attempts +// to delete any versioned objects, you must include an MFA token. If you do +// not provide one, the entire request will fail, even if there are non-versioned +// objects you are trying to delete. If you provide an invalid token, whether +// there are versioned keys in the request or not, the entire Multi-Object Delete +// request will fail. For information about MFA Delete, see MFA Delete (https://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html#MultiFactorAuthenticationDelete). +// +// Finally, the Content-MD5 header is required for all Multi-Object Delete requests. +// Amazon S3 uses the header value to ensure that your request body has not +// been altered in transit. +// +// The following operations are related to DeleteObjects: +// +// * CreateMultipartUpload +// +// * UploadPart +// +// * CompleteMultipartUpload +// +// * ListParts +// +// * AbortMultipartUpload // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -1522,7 +2211,21 @@ func (c *S3) DeletePublicAccessBlockRequest(input *DeletePublicAccessBlockInput) // DeletePublicAccessBlock API operation for Amazon Simple Storage Service. // -// Removes the PublicAccessBlock configuration from an Amazon S3 bucket. +// Removes the PublicAccessBlock configuration for an Amazon S3 bucket. To use +// this operation, you must have the s3:PutBucketPublicAccessBlock permission. +// For more information about permissions, see Permissions Related to Bucket +// Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// The following operations are related to DeletePublicAccessBlock: +// +// * Using Amazon S3 Block Public Access (https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html) +// +// * GetPublicAccessBlock +// +// * PutPublicAccessBlock +// +// * GetBucketPolicyStatus // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -1596,7 +2299,32 @@ func (c *S3) GetBucketAccelerateConfigurationRequest(input *GetBucketAccelerateC // GetBucketAccelerateConfiguration API operation for Amazon Simple Storage Service. // -// Returns the accelerate configuration of a bucket. +// This implementation of the GET operation uses the accelerate subresource +// to return the Transfer Acceleration state of a bucket, which is either Enabled +// or Suspended. Amazon S3 Transfer Acceleration is a bucket-level feature that +// enables you to perform faster data transfers to and from Amazon S3. +// +// To use this operation, you must have permission to perform the s3:GetAccelerateConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// You set the Transfer Acceleration state of an existing bucket to Enabled +// or Suspended by using the PutBucketAccelerateConfiguration operation. +// +// A GET accelerate request does not return a state value for a bucket that +// has no transfer acceleration state. A bucket has no Transfer Acceleration +// state if a state has never been set on the bucket. +// +// For more information about transfer acceleration, see Transfer Acceleration +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Related Resources +// +// * PutBucketAccelerateConfiguration // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -1670,7 +2398,15 @@ func (c *S3) GetBucketAclRequest(input *GetBucketAclInput) (req *request.Request // GetBucketAcl API operation for Amazon Simple Storage Service. // -// Gets the access control policy for the bucket. +// This implementation of the GET operation uses the acl subresource to return +// the access control list (ACL) of a bucket. To use GET to return the ACL of +// the bucket, you must have READ_ACP access to the bucket. If READ_ACP permission +// is granted to the anonymous user, you can return the ACL of the bucket without +// using an authorization header. +// +// Related Resources +// +// * // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -1744,8 +2480,27 @@ func (c *S3) GetBucketAnalyticsConfigurationRequest(input *GetBucketAnalyticsCon // GetBucketAnalyticsConfiguration API operation for Amazon Simple Storage Service. // -// Gets an analytics configuration for the bucket (specified by the analytics -// configuration ID). +// This implementation of the GET operation returns an analytics configuration +// (identified by the analytics configuration ID) from the bucket. +// +// To use this operation, you must have permissions to perform the s3:GetAnalyticsConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// For information about Amazon S3 analytics feature, see Amazon S3 Analytics +// – Storage Class Analysis (https://docs.aws.amazon.com/AmazonS3/latest/dev/analytics-storage-class.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Related Resources +// +// * +// +// * +// +// * // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -1819,7 +2574,20 @@ func (c *S3) GetBucketCorsRequest(input *GetBucketCorsInput) (req *request.Reque // GetBucketCors API operation for Amazon Simple Storage Service. // -// Returns the CORS configuration for the bucket. +// Returns the cors configuration information set for the bucket. +// +// To use this operation, you must have permission to perform the s3:GetBucketCORS +// action. By default, the bucket owner has this permission and can grant it +// to others. +// +// For more information about cors, see Enabling Cross-Origin Resource Sharing +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html). +// +// The following operations are related to GetBucketCors: +// +// * PutBucketCors +// +// * DeleteBucketCors // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -1893,7 +2661,21 @@ func (c *S3) GetBucketEncryptionRequest(input *GetBucketEncryptionInput) (req *r // GetBucketEncryption API operation for Amazon Simple Storage Service. // -// Returns the server-side encryption configuration of a bucket. +// Returns the default encryption configuration for an Amazon S3 bucket. For +// information about the Amazon S3 default encryption feature, see Amazon S3 +// Default Bucket Encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html). +// +// To use this operation, you must have permission to perform the s3:GetEncryptionConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// The following operations are related to GetBucketEncryption: +// +// * PutBucketEncryption +// +// * DeleteBucketEncryption // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -1967,8 +2749,25 @@ func (c *S3) GetBucketInventoryConfigurationRequest(input *GetBucketInventoryCon // GetBucketInventoryConfiguration API operation for Amazon Simple Storage Service. // -// Returns an inventory configuration (identified by the inventory ID) from -// the bucket. +// Returns an inventory configuration (identified by the inventory configuration +// ID) from the bucket. +// +// To use this operation, you must have permissions to perform the s3:GetInventoryConfiguration +// action. The bucket owner has this permission by default and can grant this +// permission to others. For more information about permissions, see Permissions +// Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// For information about the Amazon S3 inventory feature, see Amazon S3 Inventory +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html). +// +// The following operations are related to GetBucketInventoryConfiguration: +// +// * DeleteBucketInventoryConfiguration +// +// * ListBucketInventoryConfigurations +// +// * PutBucketInventoryConfiguration // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -2047,7 +2846,34 @@ func (c *S3) GetBucketLifecycleRequest(input *GetBucketLifecycleInput) (req *req // GetBucketLifecycle API operation for Amazon Simple Storage Service. // -// Deprecated, see the GetBucketLifecycleConfiguration operation. +// +// For an updated version of this API, see GetBucketLifecycleConfiguration. +// If you configured a bucket lifecycle using the filter element, you should +// see the updated version of this topic. This topic is provided for backward +// compatibility. +// +// Returns the lifecycle configuration information set on the bucket. For information +// about lifecycle configuration, see Object Lifecycle Management (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html). +// +// To use this operation, you must have permission to perform the s3:GetLifecycleConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// GetBucketLifecycle has the following special error: +// +// * Error code: NoSuchLifecycleConfiguration Description: The lifecycle +// configuration does not exist. HTTP Status Code: 404 Not Found SOAP Fault +// Code Prefix: Client +// +// The following operations are related to GetBucketLifecycle: +// +// * GetBucketLifecycleConfiguration +// +// * PutBucketLifecycle +// +// * DeleteBucketLifecycle // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -2125,7 +2951,37 @@ func (c *S3) GetBucketLifecycleConfigurationRequest(input *GetBucketLifecycleCon // GetBucketLifecycleConfiguration API operation for Amazon Simple Storage Service. // -// Returns the lifecycle configuration information set on the bucket. +// +// Bucket lifecycle configuration now supports specifying a lifecycle rule using +// an object key name prefix, one or more object tags, or a combination of both. +// Accordingly, this section describes the latest API. The response describes +// the new filter element that you can use to specify a filter to select a subset +// of objects to which the rule applies. If you are still using previous version +// of the lifecycle configuration, it works. For the earlier API description, +// see GetBucketLifecycle. +// +// Returns the lifecycle configuration information set on the bucket. For information +// about lifecycle configuration, see Object Lifecycle Management (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html). +// +// To use this operation, you must have permission to perform the s3:GetLifecycleConfiguration +// action. The bucket owner has this permission, by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// GetBucketLifecycleConfiguration has the following special error: +// +// * Error code: NoSuchLifecycleConfiguration Description: The lifecycle +// configuration does not exist. HTTP Status Code: 404 Not Found SOAP Fault +// Code Prefix: Client +// +// The following operations are related to GetBucketLifecycleConfiguration: +// +// * GetBucketLifecycle +// +// * PutBucketLifecycle +// +// * DeleteBucketLifecycle // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -2199,7 +3055,17 @@ func (c *S3) GetBucketLocationRequest(input *GetBucketLocationInput) (req *reque // GetBucketLocation API operation for Amazon Simple Storage Service. // -// Returns the region the bucket resides in. +// Returns the Region the bucket resides in. You set the bucket's Region using +// the LocationConstraint request parameter in a CreateBucket request. For more +// information, see CreateBucket. +// +// To use this implementation of the operation, you must be the bucket owner. +// +// The following operations are related to GetBucketLocation: +// +// * GetObject +// +// * CreateBucket // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -2276,6 +3142,12 @@ func (c *S3) GetBucketLoggingRequest(input *GetBucketLoggingInput) (req *request // Returns the logging status of a bucket and the permissions users have to // view and modify that status. To use GET, you must be the bucket owner. // +// The following operations are related to GetBucketLogging: +// +// * CreateBucket +// +// * PutBucketLogging +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -2349,7 +3221,26 @@ func (c *S3) GetBucketMetricsConfigurationRequest(input *GetBucketMetricsConfigu // GetBucketMetricsConfiguration API operation for Amazon Simple Storage Service. // // Gets a metrics configuration (specified by the metrics configuration ID) -// from the bucket. +// from the bucket. Note that this doesn't include the daily storage metrics. +// +// To use this operation, you must have permissions to perform the s3:GetMetricsConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// For information about CloudWatch request metrics for Amazon S3, see Monitoring +// Metrics with Amazon CloudWatch (https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html). +// +// The following operations are related to GetBucketMetricsConfiguration: +// +// * PutBucketMetricsConfiguration +// +// * DeleteBucketMetricsConfiguration +// +// * ListBucketMetricsConfigurations +// +// * Monitoring Metrics with Amazon CloudWatch (https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html) // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -2428,7 +3319,7 @@ func (c *S3) GetBucketNotificationRequest(input *GetBucketNotificationConfigurat // GetBucketNotification API operation for Amazon Simple Storage Service. // -// Deprecated, see the GetBucketNotificationConfiguration operation. +// No longer used, see GetBucketNotificationConfiguration. // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -2508,6 +3399,22 @@ func (c *S3) GetBucketNotificationConfigurationRequest(input *GetBucketNotificat // // Returns the notification configuration of a bucket. // +// If notifications are not enabled on the bucket, the operation returns an +// empty NotificationConfiguration element. +// +// By default, you must be the bucket owner to read the notification configuration +// of a bucket. However, the bucket owner can use a bucket policy to grant permission +// to other users to read this configuration with the s3:GetBucketNotification +// permission. +// +// For more information about setting and reading the notification configuration +// on a bucket, see Setting Up Notification of Bucket Events (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html). +// For more information about bucket policies, see Using Bucket Policies (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html). +// +// The following operation is related to GetBucketNotification: +// +// * PutBucketNotification +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -2580,7 +3487,26 @@ func (c *S3) GetBucketPolicyRequest(input *GetBucketPolicyInput) (req *request.R // GetBucketPolicy API operation for Amazon Simple Storage Service. // -// Returns the policy of a specified bucket. +// Returns the policy of a specified bucket. If you are using an identity other +// than the root user of the AWS account that owns the bucket, the calling identity +// must have the GetBucketPolicy permissions on the specified bucket and belong +// to the bucket owner's account in order to use this operation. +// +// If you don't have GetBucketPolicy permissions, Amazon S3 returns a 403 Access +// Denied error. If you have the correct permissions, but you're not using an +// identity that belongs to the bucket owner's account, Amazon S3 returns a +// 405 Method Not Allowed error. +// +// As a security precaution, the root user of the AWS account that owns a bucket +// can always use this operation, even if the policy explicitly denies the root +// user the ability to perform this action. +// +// For more information about bucket policies, see Using Bucket Policies and +// User Policies (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html). +// +// The following operation is related to GetBucketPolicy: +// +// * GetObject // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -2655,7 +3581,22 @@ func (c *S3) GetBucketPolicyStatusRequest(input *GetBucketPolicyStatusInput) (re // GetBucketPolicyStatus API operation for Amazon Simple Storage Service. // // Retrieves the policy status for an Amazon S3 bucket, indicating whether the -// bucket is public. +// bucket is public. In order to use this operation, you must have the s3:GetBucketPolicyStatus +// permission. For more information about Amazon S3 permissions, see Specifying +// Permissions in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html). +// +// For more information about when Amazon S3 considers a bucket public, see +// The Meaning of "Public" (https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html#access-control-block-public-access-policy-status). +// +// The following operations are related to GetBucketPolicyStatus: +// +// * Using Amazon S3 Block Public Access (https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html) +// +// * GetPublicAccessBlock +// +// * PutPublicAccessBlock +// +// * DeletePublicAccessBlock // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -2735,6 +3676,25 @@ func (c *S3) GetBucketReplicationRequest(input *GetBucketReplicationInput) (req // to all Amazon S3 systems. Therefore, a get request soon after put or delete // can return a wrong result. // +// For information about replication configuration, see Replication (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// This operation requires permissions for the s3:GetReplicationConfiguration +// action. For more information about permissions, see Using Bucket Policies +// and User Policies (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html). +// +// If you include the Filter element in a replication configuration, you must +// also include the DeleteMarkerReplication and Priority elements. The response +// also returns those elements. +// +// For information about GetBucketReplication errors, see ReplicationErrorCodeList +// +// The following operations are related to GetBucketReplication: +// +// * PutBucketReplication +// +// * DeleteBucketReplication +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -2807,7 +3767,13 @@ func (c *S3) GetBucketRequestPaymentRequest(input *GetBucketRequestPaymentInput) // GetBucketRequestPayment API operation for Amazon Simple Storage Service. // -// Returns the request payment configuration of a bucket. +// Returns the request payment configuration of a bucket. To use this version +// of the operation, you must be the bucket owner. For more information, see +// Requester Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html). +// +// The following operations are related to GetBucketRequestPayment: +// +// * ListObjects // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -2883,6 +3849,21 @@ func (c *S3) GetBucketTaggingRequest(input *GetBucketTaggingInput) (req *request // // Returns the tag set associated with the bucket. // +// To use this operation, you must have permission to perform the s3:GetBucketTagging +// action. By default, the bucket owner has this permission and can grant this +// permission to others. +// +// GetBucketTagging has the following special error: +// +// * Error code: NoSuchTagSetError Description: There is no tag set associated +// with the bucket. +// +// The following operations are related to GetBucketTagging: +// +// * PutBucketTagging +// +// * DeleteBucketTagging +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -2957,8 +3938,22 @@ func (c *S3) GetBucketVersioningRequest(input *GetBucketVersioningInput) (req *r // // Returns the versioning state of a bucket. // -// Returns awserr.Error for service API and SDK errors. Use runtime type assertions -// with awserr.Error's Code and Message methods to get detailed information about +// To retrieve the versioning state of a bucket, you must be the bucket owner. +// +// This implementation also returns the MFA Delete status of the versioning +// state. If the MFA Delete status is enabled, the bucket owner must use an +// authentication device to change the versioning state of the bucket. +// +// The following operations are related to GetBucketVersioning: +// +// * GetObject +// +// * PutObject +// +// * DeleteObject +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about // the error. // // See the AWS API reference guide for Amazon Simple Storage Service's @@ -3029,7 +4024,21 @@ func (c *S3) GetBucketWebsiteRequest(input *GetBucketWebsiteInput) (req *request // GetBucketWebsite API operation for Amazon Simple Storage Service. // -// Returns the website configuration for a bucket. +// Returns the website configuration for a bucket. To host website on Amazon +// S3, you can configure a bucket as website by adding a website configuration. +// For more information about hosting websites, see Hosting Websites on Amazon +// S3 (https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html). +// +// This GET operation requires the S3:GetBucketWebsite permission. By default, +// only the bucket owner can read the bucket website configuration. However, +// bucket owners can allow other users to read the website configuration by +// writing a bucket policy granting them the S3:GetBucketWebsite permission. +// +// The following operations are related to DeleteBucketWebsite: +// +// * DeleteBucketWebsite +// +// * PutBucketWebsite // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3103,7 +4112,130 @@ func (c *S3) GetObjectRequest(input *GetObjectInput) (req *request.Request, outp // GetObject API operation for Amazon Simple Storage Service. // -// Retrieves objects from Amazon S3. +// Retrieves objects from Amazon S3. To use GET, you must have READ access to +// the object. If you grant READ access to the anonymous user, you can return +// the object without using an authorization header. +// +// An Amazon S3 bucket has no directory hierarchy such as you would find in +// a typical computer file system. You can, however, create a logical hierarchy +// by using object key names that imply a folder structure. For example, instead +// of naming an object sample.jpg, you can name it photos/2006/February/sample.jpg. +// +// To get an object from such a logical hierarchy, specify the full key name +// for the object in the GET operation. For a virtual hosted-style request example, +// if you have the object photos/2006/February/sample.jpg, specify the resource +// as /photos/2006/February/sample.jpg. For a path-style request example, if +// you have the object photos/2006/February/sample.jpg in the bucket named examplebucket, +// specify the resource as /examplebucket/photos/2006/February/sample.jpg. For +// more information about request types, see HTTP Host Header Bucket Specification +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#VirtualHostingSpecifyBucket). +// +// To distribute large files to many people, you can save bandwidth costs by +// using BitTorrent. For more information, see Amazon S3 Torrent (https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html). +// For more information about returning the ACL of an object, see GetObjectAcl. +// +// If the object you are retrieving is stored in the GLACIER or DEEP_ARCHIVE +// storage classes, before you can retrieve the object you must first restore +// a copy using . Otherwise, this operation returns an InvalidObjectStateError +// error. For information about restoring archived objects, see Restoring Archived +// Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/restoring-objects.html). +// +// Encryption request headers, like x-amz-server-side-encryption, should not +// be sent for GET requests if your object uses server-side encryption with +// CMKs stored in AWS KMS (SSE-KMS) or server-side encryption with Amazon S3–managed +// encryption keys (SSE-S3). If your object does use these types of keys, you’ll +// get an HTTP 400 BadRequest error. +// +// If you encrypt an object by using server-side encryption with customer-provided +// encryption keys (SSE-C) when you store the object in Amazon S3, then when +// you GET the object, you must use the following headers: +// +// * x-amz-server-side​-encryption​-customer-algorithm +// +// * x-amz-server-side​-encryption​-customer-key +// +// * x-amz-server-side​-encryption​-customer-key-MD5 +// +// For more information about SSE-C, see Server-Side Encryption (Using Customer-Provided +// Encryption Keys) (https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html). +// +// Assuming you have permission to read object tags (permission for the s3:GetObjectVersionTagging +// action), the response also returns the x-amz-tagging-count header that provides +// the count of number of tags associated with the object. You can use GetObjectTagging +// to retrieve the tag set associated with an object. +// +// Permissions +// +// You need the s3:GetObject permission for this operation. For more information, +// see Specifying Permissions in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html). +// If the object you request does not exist, the error Amazon S3 returns depends +// on whether you also have the s3:ListBucket permission. +// +// * If you have the s3:ListBucket permission on the bucket, Amazon S3 will +// return an HTTP status code 404 ("no such key") error. +// +// * If you don’t have the s3:ListBucket permission, Amazon S3 will return +// an HTTP status code 403 ("access denied") error. +// +// Versioning +// +// By default, the GET operation returns the current version of an object. To +// return a different version, use the versionId subresource. +// +// If the current version of the object is a delete marker, Amazon S3 behaves +// as if the object was deleted and includes x-amz-delete-marker: true in the +// response. +// +// For more information about versioning, see PutBucketVersioning. +// +// Overriding Response Header Values +// +// There are times when you want to override certain response header values +// in a GET response. For example, you might override the Content-Disposition +// response header value in your GET request. +// +// You can override values for a set of response headers using the following +// query parameters. These response header values are sent only on a successful +// request, that is, when status code 200 OK is returned. The set of headers +// you can override using these parameters is a subset of the headers that Amazon +// S3 accepts when you create an object. The response headers that you can override +// for the GET response are Content-Type, Content-Language, Expires, Cache-Control, +// Content-Disposition, and Content-Encoding. To override these header values +// in the GET response, you use the following request parameters. +// +// You must sign the request, either using an Authorization header or a presigned +// URL, when using these parameters. They cannot be used with an unsigned (anonymous) +// request. +// +// * response-content-type +// +// * response-content-language +// +// * response-expires +// +// * response-cache-control +// +// * response-content-disposition +// +// * response-content-encoding +// +// Additional Considerations about Request Headers +// +// If both of the If-Match and If-Unmodified-Since headers are present in the +// request as follows: If-Match condition evaluates to true, and; If-Unmodified-Since +// condition evaluates to false; then, S3 returns 200 OK and the data requested. +// +// If both of the If-None-Match and If-Modified-Since headers are present in +// the request as follows:If-None-Match condition evaluates to false, and; If-Modified-Since +// condition evaluates to true; then, S3 returns 304 Not Modified response code. +// +// For more information about conditional requests, see RFC 7232 (https://tools.ietf.org/html/rfc7232). +// +// The following operations are related to GetObject: +// +// * ListBuckets +// +// * GetObjectAcl // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3182,7 +4314,21 @@ func (c *S3) GetObjectAclRequest(input *GetObjectAclInput) (req *request.Request // GetObjectAcl API operation for Amazon Simple Storage Service. // -// Returns the access control list (ACL) of an object. +// Returns the access control list (ACL) of an object. To use this operation, +// you must have READ_ACP access to the object. +// +// Versioning +// +// By default, GET returns ACL information about the current version of an object. +// To return ACL information about a different version, use the versionId subresource. +// +// The following operations are related to GetObjectAcl: +// +// * GetObject +// +// * DeleteObject +// +// * PutObject // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3261,7 +4407,8 @@ func (c *S3) GetObjectLegalHoldRequest(input *GetObjectLegalHoldInput) (req *req // GetObjectLegalHold API operation for Amazon Simple Storage Service. // -// Gets an object's current Legal Hold status. +// Gets an object's current Legal Hold status. For more information, see Locking +// Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html). // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3337,7 +4484,8 @@ func (c *S3) GetObjectLockConfigurationRequest(input *GetObjectLockConfiguration // // Gets the Object Lock configuration for a bucket. The rule specified in the // Object Lock configuration will be applied by default to every new object -// placed in the specified bucket. +// placed in the specified bucket. For more information, see Locking Objects +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html). // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3411,7 +4559,8 @@ func (c *S3) GetObjectRetentionRequest(input *GetObjectRetentionInput) (req *req // GetObjectRetention API operation for Amazon Simple Storage Service. // -// Retrieves an object's retention settings. +// Retrieves an object's retention settings. For more information, see Locking +// Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html). // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3485,7 +4634,25 @@ func (c *S3) GetObjectTaggingRequest(input *GetObjectTaggingInput) (req *request // GetObjectTagging API operation for Amazon Simple Storage Service. // -// Returns the tag-set of an object. +// Returns the tag-set of an object. You send the GET request against the tagging +// subresource associated with the object. +// +// To use this operation, you must have permission to perform the s3:GetObjectTagging +// action. By default, the GET operation returns information about current version +// of an object. For a versioned bucket, you can have multiple versions of an +// object in your bucket. To retrieve tags of any other version, use the versionId +// query parameter. You also need permission for the s3:GetObjectVersionTagging +// action. +// +// By default, the bucket owner has this permission and can grant this permission +// to others. +// +// For information about the Amazon S3 object tagging feature, see Object Tagging +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html). +// +// The following operation is related to GetObjectTagging: +// +// * PutObjectTagging // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3559,7 +4726,19 @@ func (c *S3) GetObjectTorrentRequest(input *GetObjectTorrentInput) (req *request // GetObjectTorrent API operation for Amazon Simple Storage Service. // -// Return torrent files from a bucket. +// Return torrent files from a bucket. BitTorrent can save you bandwidth when +// you're distributing large files. For more information about BitTorrent, see +// Amazon S3 Torrent (https://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.html). +// +// You can get torrent only for objects that are less than 5 GB in size and +// that are not encrypted using server-side encryption with customer-provided +// encryption key. +// +// To use GET, you must have READ access to the object. +// +// The following operation is related to GetObjectTorrent: +// +// * GetObject // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3633,7 +4812,30 @@ func (c *S3) GetPublicAccessBlockRequest(input *GetPublicAccessBlockInput) (req // GetPublicAccessBlock API operation for Amazon Simple Storage Service. // -// Retrieves the PublicAccessBlock configuration for an Amazon S3 bucket. +// Retrieves the PublicAccessBlock configuration for an Amazon S3 bucket. To +// use this operation, you must have the s3:GetBucketPublicAccessBlock permission. +// For more information about Amazon S3 permissions, see Specifying Permissions +// in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html). +// +// When Amazon S3 evaluates the PublicAccessBlock configuration for a bucket +// or an object, it checks the PublicAccessBlock configuration for both the +// bucket (or the bucket that contains the object) and the bucket owner's account. +// If the PublicAccessBlock settings are different between the bucket and the +// account, Amazon S3 uses the most restrictive combination of the bucket-level +// and account-level settings. +// +// For more information about when Amazon S3 considers a bucket or an object +// public, see The Meaning of "Public" (https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html#access-control-block-public-access-policy-status). +// +// The following operations are related to GetPublicAccessBlock: +// +// * Using Amazon S3 Block Public Access (https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html) +// +// * PutPublicAccessBlock +// +// * GetPublicAccessBlock +// +// * DeletePublicAccessBlock // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3709,7 +4911,15 @@ func (c *S3) HeadBucketRequest(input *HeadBucketInput) (req *request.Request, ou // HeadBucket API operation for Amazon Simple Storage Service. // // This operation is useful to determine if a bucket exists and you have permission -// to access it. +// to access it. The operation returns a 200 OK if the bucket exists and you +// have permission to access it. Otherwise, the operation might return responses +// such as 404 Not Found and 403 Forbidden. +// +// To use this operation, you must have permissions to perform the s3:ListBucket +// action. The bucket owner has this permission by default and can grant this +// permission to others. For more information about permissions, see Permissions +// Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3792,6 +5002,63 @@ func (c *S3) HeadObjectRequest(input *HeadObjectInput) (req *request.Request, ou // object itself. This operation is useful if you're only interested in an object's // metadata. To use HEAD, you must have READ access to the object. // +// A HEAD request has the same options as a GET operation on an object. The +// response is identical to the GET response except that there is no response +// body. +// +// If you encrypt an object by using server-side encryption with customer-provided +// encryption keys (SSE-C) when you store the object in Amazon S3, then when +// you retrieve the metadata from the object, you must use the following headers: +// +// * x-amz-server-side​-encryption​-customer-algorithm +// +// * x-amz-server-side​-encryption​-customer-key +// +// * x-amz-server-side​-encryption​-customer-key-MD5 +// +// For more information about SSE-C, see Server-Side Encryption (Using Customer-Provided +// Encryption Keys) (https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html). +// +// Encryption request headers, like x-amz-server-side-encryption, should not +// be sent for GET requests if your object uses server-side encryption with +// CMKs stored in AWS KMS (SSE-KMS) or server-side encryption with Amazon S3–managed +// encryption keys (SSE-S3). If your object does use these types of keys, you’ll +// get an HTTP 400 BadRequest error. +// +// Request headers are limited to 8 KB in size. For more information, see Common +// Request Headers (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonRequestHeaders.html). +// +// Consider the following when using request headers: +// +// * Consideration 1 – If both of the If-Match and If-Unmodified-Since +// headers are present in the request as follows: If-Match condition evaluates +// to true, and; If-Unmodified-Since condition evaluates to false; Then Amazon +// S3 returns 200 OK and the data requested. +// +// * Consideration 2 – If both of the If-None-Match and If-Modified-Since +// headers are present in the request as follows: If-None-Match condition +// evaluates to false, and; If-Modified-Since condition evaluates to true; +// Then Amazon S3 returns the 304 Not Modified response code. +// +// For more information about conditional requests, see RFC 7232 (https://tools.ietf.org/html/rfc7232). +// +// Permissions +// +// You need the s3:GetObject permission for this operation. For more information, +// see Specifying Permissions in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html). +// If the object you request does not exist, the error Amazon S3 returns depends +// on whether you also have the s3:ListBucket permission. +// +// * If you have the s3:ListBucket permission on the bucket, Amazon S3 returns +// an HTTP status code 404 ("no such key") error. +// +// * If you don’t have the s3:ListBucket permission, Amazon S3 returns +// an HTTP status code 403 ("access denied") error. +// +// The following operation is related to HeadObject: +// +// * GetObject +// // See http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#RESTErrorResponses // for more information on returned errors. // @@ -3867,7 +5134,33 @@ func (c *S3) ListBucketAnalyticsConfigurationsRequest(input *ListBucketAnalytics // ListBucketAnalyticsConfigurations API operation for Amazon Simple Storage Service. // -// Lists the analytics configurations for the bucket. +// Lists the analytics configurations for the bucket. You can have up to 1,000 +// analytics configurations per bucket. +// +// This operation supports list pagination and does not return more than 100 +// configurations at a time. You should always check the IsTruncated element +// in the response. If there are no more configurations to list, IsTruncated +// is set to false. If there are more configurations to list, IsTruncated is +// set to true, and there will be a value in NextContinuationToken. You use +// the NextContinuationToken value to continue the pagination of the list by +// passing the value in continuation-token in the request to GET the next page. +// +// To use this operation, you must have permissions to perform the s3:GetAnalyticsConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// For information about Amazon S3 analytics feature, see Amazon S3 Analytics +// – Storage Class Analysis (https://docs.aws.amazon.com/AmazonS3/latest/dev/analytics-storage-class.html). +// +// The following operations are related to ListBucketAnalyticsConfigurations: +// +// * GetBucketAnalyticsConfiguration +// +// * DeleteBucketAnalyticsConfiguration +// +// * PutBucketAnalyticsConfiguration // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -3941,7 +5234,33 @@ func (c *S3) ListBucketInventoryConfigurationsRequest(input *ListBucketInventory // ListBucketInventoryConfigurations API operation for Amazon Simple Storage Service. // -// Returns a list of inventory configurations for the bucket. +// Returns a list of inventory configurations for the bucket. You can have up +// to 1,000 analytics configurations per bucket. +// +// This operation supports list pagination and does not return more than 100 +// configurations at a time. Always check the IsTruncated element in the response. +// If there are no more configurations to list, IsTruncated is set to false. +// If there are more configurations to list, IsTruncated is set to true, and +// there is a value in NextContinuationToken. You use the NextContinuationToken +// value to continue the pagination of the list by passing the value in continuation-token +// in the request to GET the next page. +// +// To use this operation, you must have permissions to perform the s3:GetInventoryConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// For information about the Amazon S3 inventory feature, see Amazon S3 Inventory +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html) +// +// The following operations are related to ListBucketInventoryConfigurations: +// +// * GetBucketInventoryConfiguration +// +// * DeleteBucketInventoryConfiguration +// +// * PutBucketInventoryConfiguration // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -4015,7 +5334,34 @@ func (c *S3) ListBucketMetricsConfigurationsRequest(input *ListBucketMetricsConf // ListBucketMetricsConfigurations API operation for Amazon Simple Storage Service. // -// Lists the metrics configurations for the bucket. +// Lists the metrics configurations for the bucket. The metrics configurations +// are only for the request metrics of the bucket and do not provide information +// on daily storage metrics. You can have up to 1,000 configurations per bucket. +// +// This operation supports list pagination and does not return more than 100 +// configurations at a time. Always check the IsTruncated element in the response. +// If there are no more configurations to list, IsTruncated is set to false. +// If there are more configurations to list, IsTruncated is set to true, and +// there is a value in NextContinuationToken. You use the NextContinuationToken +// value to continue the pagination of the list by passing the value in continuation-token +// in the request to GET the next page. +// +// To use this operation, you must have permissions to perform the s3:GetMetricsConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// For more information about metrics configurations and CloudWatch request +// metrics, see Monitoring Metrics with Amazon CloudWatch (https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html). +// +// The following operations are related to ListBucketMetricsConfigurations: +// +// * PutBucketMetricsConfiguration +// +// * GetBucketMetricsConfiguration +// +// * DeleteBucketMetricsConfiguration // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -4169,7 +5515,40 @@ func (c *S3) ListMultipartUploadsRequest(input *ListMultipartUploadsInput) (req // ListMultipartUploads API operation for Amazon Simple Storage Service. // -// This operation lists in-progress multipart uploads. +// This operation lists in-progress multipart uploads. An in-progress multipart +// upload is a multipart upload that has been initiated using the Initiate Multipart +// Upload request, but has not yet been completed or aborted. +// +// This operation returns at most 1,000 multipart uploads in the response. 1,000 +// multipart uploads is the maximum number of uploads a response can include, +// which is also the default value. You can further limit the number of uploads +// in a response by specifying the max-uploads parameter in the response. If +// additional multipart uploads satisfy the list criteria, the response will +// contain an IsTruncated element with the value true. To list the additional +// multipart uploads, use the key-marker and upload-id-marker request parameters. +// +// In the response, the uploads are sorted by key. If your application has initiated +// more than one multipart upload using the same object key, then uploads in +// the response are first sorted by key. Additionally, uploads are sorted in +// ascending order within each key by the upload initiation time. +// +// For more information on multipart uploads, see Uploading Objects Using Multipart +// Upload (https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html). +// +// For information on permissions required to use the multipart upload API, +// see Multipart Upload API and Permissions (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuAndPermissions.html). +// +// The following operations are related to ListMultipartUploads: +// +// * CreateMultipartUpload +// +// * UploadPart +// +// * CompleteMultipartUpload +// +// * ListParts +// +// * AbortMultipartUpload // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -4210,7 +5589,7 @@ func (c *S3) ListMultipartUploadsWithContext(ctx aws.Context, input *ListMultipa // // Example iterating over at most 3 pages of a ListMultipartUploads operation. // pageNum := 0 // err := client.ListMultipartUploadsPages(params, -// func(page *ListMultipartUploadsOutput, lastPage bool) bool { +// func(page *s3.ListMultipartUploadsOutput, lastPage bool) bool { // pageNum++ // fmt.Println(page) // return pageNum <= 3 @@ -4242,10 +5621,12 @@ func (c *S3) ListMultipartUploadsPagesWithContext(ctx aws.Context, input *ListMu }, } - cont := true - for p.Next() && cont { - cont = fn(p.Page().(*ListMultipartUploadsOutput), !p.HasNextPage()) + for p.Next() { + if !fn(p.Page().(*ListMultipartUploadsOutput), !p.HasNextPage()) { + break + } } + return p.Err() } @@ -4299,7 +5680,24 @@ func (c *S3) ListObjectVersionsRequest(input *ListObjectVersionsInput) (req *req // ListObjectVersions API operation for Amazon Simple Storage Service. // -// Returns metadata about all of the versions of objects in a bucket. +// Returns metadata about all of the versions of objects in a bucket. You can +// also use request parameters as selection criteria to return metadata about +// a subset of all the object versions. +// +// A 200 OK response can contain valid or invalid XML. Make sure to design your +// application to parse the contents of the response and handle it appropriately. +// +// To use this operation, you must have READ access to the bucket. +// +// The following operations are related to ListObjectVersions: +// +// * ListObjectsV2 +// +// * GetObject +// +// * PutObject +// +// * DeleteObject // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -4340,7 +5738,7 @@ func (c *S3) ListObjectVersionsWithContext(ctx aws.Context, input *ListObjectVer // // Example iterating over at most 3 pages of a ListObjectVersions operation. // pageNum := 0 // err := client.ListObjectVersionsPages(params, -// func(page *ListObjectVersionsOutput, lastPage bool) bool { +// func(page *s3.ListObjectVersionsOutput, lastPage bool) bool { // pageNum++ // fmt.Println(page) // return pageNum <= 3 @@ -4372,10 +5770,12 @@ func (c *S3) ListObjectVersionsPagesWithContext(ctx aws.Context, input *ListObje }, } - cont := true - for p.Next() && cont { - cont = fn(p.Page().(*ListObjectVersionsOutput), !p.HasNextPage()) + for p.Next() { + if !fn(p.Page().(*ListObjectVersionsOutput), !p.HasNextPage()) { + break + } } + return p.Err() } @@ -4429,9 +5829,27 @@ func (c *S3) ListObjectsRequest(input *ListObjectsInput) (req *request.Request, // ListObjects API operation for Amazon Simple Storage Service. // -// Returns some or all (up to 1000) of the objects in a bucket. You can use +// Returns some or all (up to 1,000) of the objects in a bucket. You can use // the request parameters as selection criteria to return a subset of the objects -// in a bucket. +// in a bucket. A 200 OK response can contain valid or invalid XML. Be sure +// to design your application to parse the contents of the response and handle +// it appropriately. +// +// This API has been revised. We recommend that you use the newer version, ListObjectsV2, +// when developing applications. For backward compatibility, Amazon S3 continues +// to support ListObjects. +// +// The following operations are related to ListObjects: +// +// * ListObjectsV2 +// +// * GetObject +// +// * PutObject +// +// * CreateBucket +// +// * ListBuckets // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -4477,7 +5895,7 @@ func (c *S3) ListObjectsWithContext(ctx aws.Context, input *ListObjectsInput, op // // Example iterating over at most 3 pages of a ListObjects operation. // pageNum := 0 // err := client.ListObjectsPages(params, -// func(page *ListObjectsOutput, lastPage bool) bool { +// func(page *s3.ListObjectsOutput, lastPage bool) bool { // pageNum++ // fmt.Println(page) // return pageNum <= 3 @@ -4509,10 +5927,12 @@ func (c *S3) ListObjectsPagesWithContext(ctx aws.Context, input *ListObjectsInpu }, } - cont := true - for p.Next() && cont { - cont = fn(p.Page().(*ListObjectsOutput), !p.HasNextPage()) + for p.Next() { + if !fn(p.Page().(*ListObjectsOutput), !p.HasNextPage()) { + break + } } + return p.Err() } @@ -4566,10 +5986,34 @@ func (c *S3) ListObjectsV2Request(input *ListObjectsV2Input) (req *request.Reque // ListObjectsV2 API operation for Amazon Simple Storage Service. // -// Returns some or all (up to 1000) of the objects in a bucket. You can use +// Returns some or all (up to 1,000) of the objects in a bucket. You can use // the request parameters as selection criteria to return a subset of the objects -// in a bucket. Note: ListObjectsV2 is the revised List Objects API and we recommend -// you use this revised API for new application development. +// in a bucket. A 200 OK response can contain valid or invalid XML. Make sure +// to design your application to parse the contents of the response and handle +// it appropriately. +// +// To use this operation, you must have READ access to the bucket. +// +// To use this operation in an AWS Identity and Access Management (IAM) policy, +// you must have permissions to perform the s3:ListBucket action. The bucket +// owner has this permission by default and can grant this permission to others. +// For more information about permissions, see Permissions Related to Bucket +// Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// This section describes the latest revision of the API. We recommend that +// you use this revised API for application development. For backward compatibility, +// Amazon S3 continues to support the prior version of this API, ListObjects. +// +// To get a list of your buckets, see ListBuckets. +// +// The following operations are related to ListObjectsV2: +// +// * GetObject +// +// * PutObject +// +// * CreateBucket // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -4615,7 +6059,7 @@ func (c *S3) ListObjectsV2WithContext(ctx aws.Context, input *ListObjectsV2Input // // Example iterating over at most 3 pages of a ListObjectsV2 operation. // pageNum := 0 // err := client.ListObjectsV2Pages(params, -// func(page *ListObjectsV2Output, lastPage bool) bool { +// func(page *s3.ListObjectsV2Output, lastPage bool) bool { // pageNum++ // fmt.Println(page) // return pageNum <= 3 @@ -4647,10 +6091,12 @@ func (c *S3) ListObjectsV2PagesWithContext(ctx aws.Context, input *ListObjectsV2 }, } - cont := true - for p.Next() && cont { - cont = fn(p.Page().(*ListObjectsV2Output), !p.HasNextPage()) + for p.Next() { + if !fn(p.Page().(*ListObjectsV2Output), !p.HasNextPage()) { + break + } } + return p.Err() } @@ -4705,6 +6151,33 @@ func (c *S3) ListPartsRequest(input *ListPartsInput) (req *request.Request, outp // ListParts API operation for Amazon Simple Storage Service. // // Lists the parts that have been uploaded for a specific multipart upload. +// This operation must include the upload ID, which you obtain by sending the +// initiate multipart upload request (see CreateMultipartUpload). This request +// returns a maximum of 1,000 uploaded parts. The default number of parts returned +// is 1,000 parts. You can restrict the number of parts returned by specifying +// the max-parts request parameter. If your multipart upload consists of more +// than 1,000 parts, the response returns an IsTruncated field with the value +// of true, and a NextPartNumberMarker element. In subsequent ListParts requests +// you can include the part-number-marker query string parameter and set its +// value to the NextPartNumberMarker field value from the previous response. +// +// For more information on multipart uploads, see Uploading Objects Using Multipart +// Upload (https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html). +// +// For information on permissions required to use the multipart upload API, +// see Multipart Upload API and Permissions (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuAndPermissions.html). +// +// The following operations are related to ListParts: +// +// * CreateMultipartUpload +// +// * UploadPart +// +// * CompleteMultipartUpload +// +// * AbortMultipartUpload +// +// * ListMultipartUploads // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -4745,7 +6218,7 @@ func (c *S3) ListPartsWithContext(ctx aws.Context, input *ListPartsInput, opts . // // Example iterating over at most 3 pages of a ListParts operation. // pageNum := 0 // err := client.ListPartsPages(params, -// func(page *ListPartsOutput, lastPage bool) bool { +// func(page *s3.ListPartsOutput, lastPage bool) bool { // pageNum++ // fmt.Println(page) // return pageNum <= 3 @@ -4777,10 +6250,12 @@ func (c *S3) ListPartsPagesWithContext(ctx aws.Context, input *ListPartsInput, f }, } - cont := true - for p.Next() && cont { - cont = fn(p.Page().(*ListPartsOutput), !p.HasNextPage()) + for p.Next() { + if !fn(p.Page().(*ListPartsOutput), !p.HasNextPage()) { + break + } } + return p.Err() } @@ -4829,7 +6304,41 @@ func (c *S3) PutBucketAccelerateConfigurationRequest(input *PutBucketAccelerateC // PutBucketAccelerateConfiguration API operation for Amazon Simple Storage Service. // -// Sets the accelerate configuration of an existing bucket. +// Sets the accelerate configuration of an existing bucket. Amazon S3 Transfer +// Acceleration is a bucket-level feature that enables you to perform faster +// data transfers to Amazon S3. +// +// To use this operation, you must have permission to perform the s3:PutAccelerateConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// The Transfer Acceleration state of a bucket can be set to one of the following +// two values: +// +// * Enabled – Enables accelerated data transfers to the bucket. +// +// * Suspended – Disables accelerated data transfers to the bucket. +// +// The GetBucketAccelerateConfiguration operation returns the transfer acceleration +// state of a bucket. +// +// After setting the Transfer Acceleration state of a bucket to Enabled, it +// might take up to thirty minutes before the data transfer rates to the bucket +// increase. +// +// The name of the bucket used for Transfer Acceleration must be DNS-compliant +// and must not contain periods ("."). +// +// For more information about transfer acceleration, see Transfer Acceleration +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html). +// +// The following operations are related to PutBucketAccelerateConfiguration: +// +// * GetBucketAccelerateConfiguration +// +// * CreateBucket // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -4899,12 +6408,101 @@ func (c *S3) PutBucketAclRequest(input *PutBucketAclInput) (req *request.Request output = &PutBucketAclOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutBucketAcl API operation for Amazon Simple Storage Service. // -// Sets the permissions on a bucket using access control lists (ACL). +// Sets the permissions on an existing bucket using access control lists (ACL). +// For more information, see Using ACLs (https://docs.aws.amazon.com/AmazonS3/latest/dev/S3_ACLs_UsingACLs.html). +// To set the ACL of a bucket, you must have WRITE_ACP permission. +// +// You can use one of the following two ways to set a bucket's permissions: +// +// * Specify the ACL in the request body +// +// * Specify permissions using request headers +// +// You cannot specify access permission using both the body and the request +// headers. +// +// Depending on your application needs, you may choose to set the ACL on a bucket +// using either the request body or the headers. For example, if you have an +// existing application that updates a bucket ACL using the request body, then +// you can continue to use that approach. +// +// Access Permissions +// +// You can set access permissions using one of the following methods: +// +// * Specify a canned ACL with the x-amz-acl request header. Amazon S3 supports +// a set of predefined ACLs, known as canned ACLs. Each canned ACL has a +// predefined set of grantees and permissions. Specify the canned ACL name +// as the value of x-amz-acl. If you use this header, you cannot use other +// access control-specific headers in your request. For more information, +// see Canned ACL (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL). +// +// * Specify access permissions explicitly with the x-amz-grant-read, x-amz-grant-read-acp, +// x-amz-grant-write-acp, and x-amz-grant-full-control headers. When using +// these headers, you specify explicit access permissions and grantees (AWS +// accounts or Amazon S3 groups) who will receive the permission. If you +// use these ACL-specific headers, you cannot use the x-amz-acl header to +// set a canned ACL. These parameters map to the set of permissions that +// Amazon S3 supports in an ACL. For more information, see Access Control +// List (ACL) Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html). +// You specify each grantee as a type=value pair, where the type is one of +// the following: id – if the value specified is the canonical user ID +// of an AWS account uri – if you are granting permissions to a predefined +// group emailAddress – if the value specified is the email address of +// an AWS account Using email addresses to specify a grantee is only supported +// in the following AWS Regions: US East (N. Virginia) US West (N. California) +// US West (Oregon) Asia Pacific (Singapore) Asia Pacific (Sydney) Asia Pacific +// (Tokyo) Europe (Ireland) South America (São Paulo) For a list of all +// the Amazon S3 supported Regions and endpoints, see Regions and Endpoints +// (https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) in +// the AWS General Reference. For example, the following x-amz-grant-write +// header grants create, overwrite, and delete objects permission to LogDelivery +// group predefined by Amazon S3 and two AWS accounts identified by their +// email addresses. x-amz-grant-write: uri="http://acs.amazonaws.com/groups/s3/LogDelivery", +// id="111122223333", id="555566667777" +// +// You can use either a canned ACL or specify access permissions explicitly. +// You cannot do both. +// +// Grantee Values +// +// You can specify the person (grantee) to whom you're assigning access rights +// (using request elements) in the following ways: +// +// * By the person's ID: <>ID<><>GranteesEmail<> +// DisplayName is optional and ignored in the request +// +// * By URI: <>http://acs.amazonaws.com/groups/global/AuthenticatedUsers<> +// +// * By Email address: <>Grantees@email.com<>lt;/Grantee> +// The grantee is resolved to the CanonicalUser and, in a response to a GET +// Object acl request, appears as the CanonicalUser. Using email addresses +// to specify a grantee is only supported in the following AWS Regions: US +// East (N. Virginia) US West (N. California) US West (Oregon) Asia Pacific +// (Singapore) Asia Pacific (Sydney) Asia Pacific (Tokyo) Europe (Ireland) +// South America (São Paulo) For a list of all the Amazon S3 supported Regions +// and endpoints, see Regions and Endpoints (https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) +// in the AWS General Reference. +// +// Related Resources +// +// * CreateBucket +// +// * DeleteBucket +// +// * GetObjectAcl // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -4980,7 +6578,50 @@ func (c *S3) PutBucketAnalyticsConfigurationRequest(input *PutBucketAnalyticsCon // PutBucketAnalyticsConfiguration API operation for Amazon Simple Storage Service. // // Sets an analytics configuration for the bucket (specified by the analytics -// configuration ID). +// configuration ID). You can have up to 1,000 analytics configurations per +// bucket. +// +// You can choose to have storage class analysis export analysis reports sent +// to a comma-separated values (CSV) flat file. See the DataExport request element. +// Reports are updated daily and are based on the object filters that you configure. +// When selecting data export, you specify a destination bucket and an optional +// destination prefix where the file is written. You can export the data to +// a destination bucket in a different account. However, the destination bucket +// must be in the same Region as the bucket that you are making the PUT analytics +// configuration to. For more information, see Amazon S3 Analytics – Storage +// Class Analysis (https://docs.aws.amazon.com/AmazonS3/latest/dev/analytics-storage-class.html). +// +// You must create a bucket policy on the destination bucket where the exported +// file is written to grant permissions to Amazon S3 to write objects to the +// bucket. For an example policy, see Granting Permissions for Amazon S3 Inventory +// and Storage Class Analysis (https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-9). +// +// To use this operation, you must have permissions to perform the s3:PutAnalyticsConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// Special Errors +// +// * HTTP Error: HTTP 400 Bad Request Code: InvalidArgument Cause: Invalid +// argument. +// +// * HTTP Error: HTTP 400 Bad Request Code: TooManyConfigurations Cause: +// You are attempting to create a new configuration but have already reached +// the 1,000-configuration limit. +// +// * HTTP Error: HTTP 403 Forbidden Code: AccessDenied Cause: You are not +// the owner of the specified bucket, or you do not have the s3:PutAnalyticsConfiguration +// bucket permission to set the configuration on the bucket. +// +// Related Resources +// +// * +// +// * +// +// * // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5050,12 +6691,58 @@ func (c *S3) PutBucketCorsRequest(input *PutBucketCorsInput) (req *request.Reque output = &PutBucketCorsOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutBucketCors API operation for Amazon Simple Storage Service. // -// Sets the CORS configuration for a bucket. +// Sets the cors configuration for your bucket. If the configuration exists, +// Amazon S3 replaces it. +// +// To use this operation, you must be allowed to perform the s3:PutBucketCORS +// action. By default, the bucket owner has this permission and can grant it +// to others. +// +// You set this configuration on a bucket so that the bucket can service cross-origin +// requests. For example, you might want to enable a request whose origin is +// http://www.example.com to access your Amazon S3 bucket at my.example.bucket.com +// by using the browser's XMLHttpRequest capability. +// +// To enable cross-origin resource sharing (CORS) on a bucket, you add the cors +// subresource to the bucket. The cors subresource is an XML document in which +// you configure rules that identify origins and the HTTP methods that can be +// executed on your bucket. The document is limited to 64 KB in size. +// +// When Amazon S3 receives a cross-origin request (or a pre-flight OPTIONS request) +// against a bucket, it evaluates the cors configuration on the bucket and uses +// the first CORSRule rule that matches the incoming browser request to enable +// a cross-origin request. For a rule to match, the following conditions must +// be met: +// +// * The request's Origin header must match AllowedOrigin elements. +// +// * The request method (for example, GET, PUT, HEAD, and so on) or the Access-Control-Request-Method +// header in case of a pre-flight OPTIONS request must be one of the AllowedMethod +// elements. +// +// * Every header specified in the Access-Control-Request-Headers request +// header of a pre-flight request must match an AllowedHeader element. +// +// For more information about CORS, go to Enabling Cross-Origin Resource Sharing +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html) in the Amazon +// Simple Storage Service Developer Guide. +// +// Related Resources +// +// * GetBucketCors +// +// * DeleteBucketCors +// +// * RESTOPTIONSobject // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5125,13 +6812,38 @@ func (c *S3) PutBucketEncryptionRequest(input *PutBucketEncryptionInput) (req *r output = &PutBucketEncryptionOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutBucketEncryption API operation for Amazon Simple Storage Service. // -// Creates a new server-side encryption configuration (or replaces an existing -// one, if present). +// This implementation of the PUT operation uses the encryption subresource +// to set the default encryption state of an existing bucket. +// +// This implementation of the PUT operation sets default encryption for a bucket +// using server-side encryption with Amazon S3-managed keys SSE-S3 or AWS KMS +// customer master keys (CMKs) (SSE-KMS). For information about the Amazon S3 +// default encryption feature, see Amazon S3 Default Bucket Encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html). +// +// This operation requires AWS Signature Version 4. For more information, see +// Authenticating Requests (AWS Signature Version 4) (sig-v4-authenticating-requests.html). +// +// To use this operation, you must have permissions to perform the s3:PutEncryptionConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Related Resources +// +// * GetBucketEncryption +// +// * DeleteBucketEncryption // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5206,8 +6918,54 @@ func (c *S3) PutBucketInventoryConfigurationRequest(input *PutBucketInventoryCon // PutBucketInventoryConfiguration API operation for Amazon Simple Storage Service. // -// Adds an inventory configuration (identified by the inventory ID) from the -// bucket. +// This implementation of the PUT operation adds an inventory configuration +// (identified by the inventory ID) to the bucket. You can have up to 1,000 +// inventory configurations per bucket. +// +// Amazon S3 inventory generates inventories of the objects in the bucket on +// a daily or weekly basis, and the results are published to a flat file. The +// bucket that is inventoried is called the source bucket, and the bucket where +// the inventory flat file is stored is called the destination bucket. The destination +// bucket must be in the same AWS Region as the source bucket. +// +// When you configure an inventory for a source bucket, you specify the destination +// bucket where you want the inventory to be stored, and whether to generate +// the inventory daily or weekly. You can also configure what object metadata +// to include and whether to inventory all object versions or only current versions. +// For more information, see Amazon S3 Inventory (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// You must create a bucket policy on the destination bucket to grant permissions +// to Amazon S3 to write objects to the bucket in the defined location. For +// an example policy, see Granting Permissions for Amazon S3 Inventory and Storage +// Class Analysis (https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-9). +// +// To use this operation, you must have permissions to perform the s3:PutInventoryConfiguration +// action. The bucket owner has this permission by default and can grant this +// permission to others. For more information about permissions, see Permissions +// Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Special Errors +// +// * HTTP 400 Bad Request Error Code: InvalidArgument Cause: Invalid Argument +// +// * HTTP 400 Bad Request Error Code: TooManyConfigurations Cause: You are +// attempting to create a new configuration but have already reached the +// 1,000-configuration limit. +// +// * HTTP 403 Forbidden Error Code: AccessDenied Cause: You are not the owner +// of the specified bucket, or you do not have the s3:PutInventoryConfiguration +// bucket permission to set the configuration on the bucket. +// +// Related Resources +// +// * GetBucketInventoryConfiguration +// +// * DeleteBucketInventoryConfiguration +// +// * ListBucketInventoryConfigurations // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5282,12 +7040,64 @@ func (c *S3) PutBucketLifecycleRequest(input *PutBucketLifecycleInput) (req *req output = &PutBucketLifecycleOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutBucketLifecycle API operation for Amazon Simple Storage Service. // -// Deprecated, see the PutBucketLifecycleConfiguration operation. +// +// For an updated version of this API, see PutBucketLifecycleConfiguration. +// This version has been deprecated. Existing lifecycle configurations will +// work. For new lifecycle configurations, use the updated API. +// +// Creates a new lifecycle configuration for the bucket or replaces an existing +// lifecycle configuration. For information about lifecycle configuration, see +// Object Lifecycle Management (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// By default, all Amazon S3 resources, including buckets, objects, and related +// subresources (for example, lifecycle configuration and website configuration) +// are private. Only the resource owner, the AWS account that created the resource, +// can access it. The resource owner can optionally grant access permissions +// to others by writing an access policy. For this operation, users must get +// the s3:PutLifecycleConfiguration permission. +// +// You can also explicitly deny permissions. Explicit denial also supersedes +// any other permissions. If you want to prevent users or accounts from removing +// or deleting objects from your bucket, you must deny them permissions for +// the following actions: +// +// * s3:DeleteObject +// +// * s3:DeleteObjectVersion +// +// * s3:PutLifecycleConfiguration +// +// For more information about permissions, see Managing Access Permissions to +// your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// For more examples of transitioning objects to storage classes such as STANDARD_IA +// or ONEZONE_IA, see Examples of Lifecycle Configuration (https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html#lifecycle-configuration-examples). +// +// Related Resources +// +// * GetBucketLifecycle(Deprecated) +// +// * GetBucketLifecycleConfiguration +// +// * +// +// * By default, a resource owner—in this case, a bucket owner, which is +// the AWS account that created the bucket—can perform any of the operations. +// A resource owner can also grant others permission to perform the operation. +// For more information, see the following topics in the Amazon Simple Storage +// Service Developer Guide: Specifying Permissions in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html) +// Managing Access Permissions to your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html) // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5361,13 +7171,78 @@ func (c *S3) PutBucketLifecycleConfigurationRequest(input *PutBucketLifecycleCon output = &PutBucketLifecycleConfigurationOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutBucketLifecycleConfiguration API operation for Amazon Simple Storage Service. // -// Sets lifecycle configuration for your bucket. If a lifecycle configuration -// exists, it replaces it. +// Creates a new lifecycle configuration for the bucket or replaces an existing +// lifecycle configuration. For information about lifecycle configuration, see +// Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// Bucket lifecycle configuration now supports specifying a lifecycle rule using +// an object key name prefix, one or more object tags, or a combination of both. +// Accordingly, this section describes the latest API. The previous version +// of the API supported filtering based only on an object key name prefix, which +// is supported for backward compatibility. For the related API description, +// see PutBucketLifecycle. +// +// Rules +// +// You specify the lifecycle configuration in your request body. The lifecycle +// configuration is specified as XML consisting of one or more rules. Each rule +// consists of the following: +// +// * Filter identifying a subset of objects to which the rule applies. The +// filter can be based on a key name prefix, object tags, or a combination +// of both. +// +// * Status whether the rule is in effect. +// +// * One or more lifecycle transition and expiration actions that you want +// Amazon S3 to perform on the objects identified by the filter. If the state +// of your bucket is versioning-enabled or versioning-suspended, you can +// have many versions of the same object (one current version and zero or +// more noncurrent versions). Amazon S3 provides predefined actions that +// you can specify for current and noncurrent object versions. +// +// For more information, see Object Lifecycle Management (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) +// and Lifecycle Configuration Elements (https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html). +// +// Permissions +// +// By default, all Amazon S3 resources are private, including buckets, objects, +// and related subresources (for example, lifecycle configuration and website +// configuration). Only the resource owner (that is, the AWS account that created +// it) can access the resource. The resource owner can optionally grant access +// permissions to others by writing an access policy. For this operation, a +// user must get the s3:PutLifecycleConfiguration permission. +// +// You can also explicitly deny permissions. Explicit deny also supersedes any +// other permissions. If you want to block users or accounts from removing or +// deleting objects from your bucket, you must deny them permissions for the +// following actions: +// +// * s3:DeleteObject +// +// * s3:DeleteObjectVersion +// +// * s3:PutLifecycleConfiguration +// +// For more information about permissions, see Managing Access Permissions to +// Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// The following are related to PutBucketLifecycleConfiguration: +// +// * Examples of Lifecycle Configuration (https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-configuration-examples.html) +// +// * GetBucketLifecycleConfiguration +// +// * DeleteBucketLifecycle // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5437,21 +7312,68 @@ func (c *S3) PutBucketLoggingRequest(input *PutBucketLoggingInput) (req *request output = &PutBucketLoggingOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutBucketLogging API operation for Amazon Simple Storage Service. // // Set the logging parameters for a bucket and to specify permissions for who -// can view and modify the logging parameters. To set the logging status of +// can view and modify the logging parameters. All logs are saved to buckets +// in the same AWS Region as the source bucket. To set the logging status of // a bucket, you must be the bucket owner. // -// Returns awserr.Error for service API and SDK errors. Use runtime type assertions -// with awserr.Error's Code and Message methods to get detailed information about -// the error. +// The bucket owner is automatically granted FULL_CONTROL to all logs. You use +// the Grantee request element to grant access to other people. The Permissions +// request element specifies the kind of access the grantee has to the logs. // -// See the AWS API reference guide for Amazon Simple Storage Service's -// API operation PutBucketLogging for usage and error information. +// Grantee Values +// +// You can specify the person (grantee) to whom you're assigning access rights +// (using request elements) in the following ways: +// +// * By the person's ID: <>ID<><>GranteesEmail<> +// DisplayName is optional and ignored in the request. +// +// * By Email address: <>Grantees@email.com<> +// The grantee is resolved to the CanonicalUser and, in a response to a GET +// Object acl request, appears as the CanonicalUser. +// +// * By URI: <>http://acs.amazonaws.com/groups/global/AuthenticatedUsers<> +// +// To enable logging, you use LoggingEnabled and its children request elements. +// To disable logging, you use an empty BucketLoggingStatus request element: +// +// +// +// For more information about server access logging, see Server Access Logging +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html). +// +// For more information about creating a bucket, see CreateBucket. For more +// information about returning the logging status of a bucket, see GetBucketLogging. +// +// The following operations are related to PutBucketLogging: +// +// * PutObject +// +// * DeleteBucket +// +// * CreateBucket +// +// * GetBucketLogging +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for Amazon Simple Storage Service's +// API operation PutBucketLogging for usage and error information. // See also, https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/PutBucketLogging func (c *S3) PutBucketLogging(input *PutBucketLoggingInput) (*PutBucketLoggingOutput, error) { req, out := c.PutBucketLoggingRequest(input) @@ -5520,7 +7442,33 @@ func (c *S3) PutBucketMetricsConfigurationRequest(input *PutBucketMetricsConfigu // PutBucketMetricsConfiguration API operation for Amazon Simple Storage Service. // // Sets a metrics configuration (specified by the metrics configuration ID) -// for the bucket. +// for the bucket. You can have up to 1,000 metrics configurations per bucket. +// If you're updating an existing metrics configuration, note that this is a +// full replacement of the existing metrics configuration. If you don't include +// the elements you want to keep, they are erased. +// +// To use this operation, you must have permissions to perform the s3:PutMetricsConfiguration +// action. The bucket owner has this permission by default. The bucket owner +// can grant this permission to others. For more information about permissions, +// see Permissions Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// For information about CloudWatch request metrics for Amazon S3, see Monitoring +// Metrics with Amazon CloudWatch (https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html). +// +// The following operations are related to PutBucketMetricsConfiguration: +// +// * DeleteBucketMetricsConfiguration +// +// * PutBucketMetricsConfiguration +// +// * ListBucketMetricsConfigurations +// +// GetBucketLifecycle has the following special error: +// +// * Error code: TooManyConfigurations Description: You are attempting to +// create a new configuration but have already reached the 1,000-configuration +// limit. HTTP Status Code: HTTP 400 Bad Request // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5595,12 +7543,16 @@ func (c *S3) PutBucketNotificationRequest(input *PutBucketNotificationInput) (re output = &PutBucketNotificationOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutBucketNotification API operation for Amazon Simple Storage Service. // -// Deprecated, see the PutBucketNotificationConfiguraiton operation. +// No longer used, see the PutBucketNotificationConfiguration operation. // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5679,7 +7631,55 @@ func (c *S3) PutBucketNotificationConfigurationRequest(input *PutBucketNotificat // PutBucketNotificationConfiguration API operation for Amazon Simple Storage Service. // -// Enables notifications of specified events for a bucket. +// Enables notifications of specified events for a bucket. For more information +// about event notifications, see Configuring Event Notifications (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html). +// +// Using this API, you can replace an existing notification configuration. The +// configuration is an XML file that defines the event types that you want Amazon +// S3 to publish and the destination where you want Amazon S3 to publish an +// event notification when it detects an event of the specified type. +// +// By default, your bucket has no event notifications configured. That is, the +// notification configuration will be an empty NotificationConfiguration. +// +// +// +// +// +// This operation replaces the existing notification configuration with the +// configuration you include in the request body. +// +// After Amazon S3 receives this request, it first verifies that any Amazon +// Simple Notification Service (Amazon SNS) or Amazon Simple Queue Service (Amazon +// SQS) destination exists, and that the bucket owner has permission to publish +// to it by sending a test notification. In the case of AWS Lambda destinations, +// Amazon S3 verifies that the Lambda function permissions grant Amazon S3 permission +// to invoke the function from the Amazon S3 bucket. For more information, see +// Configuring Notifications for Amazon S3 Events (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html). +// +// You can disable notifications by adding the empty NotificationConfiguration +// element. +// +// By default, only the bucket owner can configure notifications on a bucket. +// However, bucket owners can use a bucket policy to grant permission to other +// users to set this configuration with s3:PutBucketNotification permission. +// +// The PUT notification is an atomic operation. For example, suppose your notification +// configuration includes SNS topic, SQS queue, and Lambda function configurations. +// When you send a PUT request with this configuration, Amazon S3 sends test +// messages to your SNS topic. If the message fails, the entire PUT operation +// will fail, and Amazon S3 will not add the configuration to your bucket. +// +// Responses +// +// If the configuration in the request body includes only one TopicConfiguration +// specifying only the s3:ReducedRedundancyLostObject event type, the response +// will also include the x-amz-sns-test-message-id header containing the message +// ID of the test notification sent to the topic. +// +// The following operation is related to PutBucketNotificationConfiguration: +// +// * GetBucketNotificationConfiguration // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5749,13 +7749,37 @@ func (c *S3) PutBucketPolicyRequest(input *PutBucketPolicyInput) (req *request.R output = &PutBucketPolicyOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutBucketPolicy API operation for Amazon Simple Storage Service. // -// Replaces a policy on a bucket. If the bucket already has a policy, the one -// in this request completely replaces it. +// Applies an Amazon S3 bucket policy to an Amazon S3 bucket. If you are using +// an identity other than the root user of the AWS account that owns the bucket, +// the calling identity must have the PutBucketPolicy permissions on the specified +// bucket and belong to the bucket owner's account in order to use this operation. +// +// If you don't have PutBucketPolicy permissions, Amazon S3 returns a 403 Access +// Denied error. If you have the correct permissions, but you're not using an +// identity that belongs to the bucket owner's account, Amazon S3 returns a +// 405 Method Not Allowed error. +// +// As a security precaution, the root user of the AWS account that owns a bucket +// can always use this operation, even if the policy explicitly denies the root +// user the ability to perform this action. +// +// For more information about bucket policies, see Using Bucket Policies and +// User Policies (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-iam-policies.html). +// +// The following operations are related to PutBucketPolicy: +// +// * CreateBucket +// +// * DeleteBucket // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5825,15 +7849,66 @@ func (c *S3) PutBucketReplicationRequest(input *PutBucketReplicationInput) (req output = &PutBucketReplicationOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutBucketReplication API operation for Amazon Simple Storage Service. // // Creates a replication configuration or replaces an existing one. For more -// information, see Cross-Region Replication (CRR) ( https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) +// information, see Replication (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.html) // in the Amazon S3 Developer Guide. // +// To perform this operation, the user or role performing the operation must +// have the iam:PassRole (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_passrole.html) +// permission. +// +// Specify the replication configuration in the request body. In the replication +// configuration, you provide the name of the destination bucket where you want +// Amazon S3 to replicate objects, the IAM role that Amazon S3 can assume to +// replicate objects on your behalf, and other relevant information. +// +// A replication configuration must include at least one rule, and can contain +// a maximum of 1,000. Each rule identifies a subset of objects to replicate +// by filtering the objects in the source bucket. To choose additional subsets +// of objects to replicate, add a rule for each subset. All rules must specify +// the same destination bucket. +// +// To specify a subset of the objects in the source bucket to apply a replication +// rule to, add the Filter element as a child of the Rule element. You can filter +// objects based on an object key prefix, one or more object tags, or both. +// When you add the Filter element in the configuration, you must also add the +// following elements: DeleteMarkerReplication, Status, and Priority. +// +// For information about enabling versioning on a bucket, see Using Versioning +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/Versioning.html). +// +// By default, a resource owner, in this case the AWS account that created the +// bucket, can perform this operation. The resource owner can also grant others +// permissions to perform the operation. For more information about permissions, +// see Specifying Permissions in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// Handling Replication of Encrypted Objects +// +// By default, Amazon S3 doesn't replicate objects that are stored at rest using +// server-side encryption with CMKs stored in AWS KMS. To replicate AWS KMS-encrypted +// objects, add the following: SourceSelectionCriteria, SseKmsEncryptedObjects, +// Status, EncryptionConfiguration, and ReplicaKmsKeyID. For information about +// replication configuration, see Replicating Objects Created with SSE Using +// CMKs stored in AWS KMS (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-config-for-kms-objects.html). +// +// For information on PutBucketReplication errors, see ReplicationErrorCodeList +// +// The following operations are related to PutBucketReplication: +// +// * GetBucketReplication +// +// * DeleteBucketReplication +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -5902,6 +7977,10 @@ func (c *S3) PutBucketRequestPaymentRequest(input *PutBucketRequestPaymentInput) output = &PutBucketRequestPaymentOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } @@ -5910,8 +7989,14 @@ func (c *S3) PutBucketRequestPaymentRequest(input *PutBucketRequestPaymentInput) // Sets the request payment configuration for a bucket. By default, the bucket // owner pays for downloads from the bucket. This configuration parameter enables // the bucket owner (only) to specify that the person requesting the download -// will be charged for the download. Documentation on requester pays buckets -// can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html +// will be charged for the download. For more information, see Requester Pays +// Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html). +// +// The following operations are related to PutBucketRequestPayment: +// +// * CreateBucket +// +// * GetBucketRequestPayment // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -5981,6 +8066,10 @@ func (c *S3) PutBucketTaggingRequest(input *PutBucketTaggingInput) (req *request output = &PutBucketTaggingOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } @@ -5988,6 +8077,47 @@ func (c *S3) PutBucketTaggingRequest(input *PutBucketTaggingInput) (req *request // // Sets the tags for a bucket. // +// Use tags to organize your AWS bill to reflect your own cost structure. To +// do this, sign up to get your AWS account bill with tag key values included. +// Then, to see the cost of combined resources, organize your billing information +// according to resources with the same tag key values. For example, you can +// tag several resources with a specific application name, and then organize +// your billing information to see the total cost of that application across +// several services. For more information, see Cost Allocation and Tagging (https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/cost-alloc-tags.html). +// +// Within a bucket, if you add a tag that has the same key as an existing tag, +// the new value overwrites the old value. For more information, see Using Cost +// Allocation in Amazon S3 Bucket Tags (https://docs.aws.amazon.com/AmazonS3/latest/dev/CostAllocTagging.html). +// +// To use this operation, you must have permissions to perform the s3:PutBucketTagging +// action. The bucket owner has this permission by default and can grant this +// permission to others. For more information about permissions, see Permissions +// Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html). +// +// PutBucketTagging has the following special errors: +// +// * Error code: InvalidTagError Description: The tag provided was not a +// valid tag. This error can occur if the tag did not pass input validation. +// For information about tag restrictions, see User-Defined Tag Restrictions +// (https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html) +// and AWS-Generated Cost Allocation Tag Restrictions (https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/aws-tag-restrictions.html). +// +// * Error code: MalformedXMLError Description: The XML provided does not +// match the schema. +// +// * Error code: OperationAbortedError Description: A conflicting conditional +// operation is currently in progress against this resource. Please try again. +// +// * Error code: InternalError Description: The service was unable to apply +// the provided tag to the bucket. +// +// The following operations are related to PutBucketTagging: +// +// * GetBucketTagging +// +// * DeleteBucketTagging +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -6056,6 +8186,10 @@ func (c *S3) PutBucketVersioningRequest(input *PutBucketVersioningInput) (req *r output = &PutBucketVersioningOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } @@ -6064,6 +8198,38 @@ func (c *S3) PutBucketVersioningRequest(input *PutBucketVersioningInput) (req *r // Sets the versioning state of an existing bucket. To set the versioning state, // you must be the bucket owner. // +// You can set the versioning state with one of the following values: +// +// Enabled—Enables versioning for the objects in the bucket. All objects added +// to the bucket receive a unique version ID. +// +// Suspended—Disables versioning for the objects in the bucket. All objects +// added to the bucket receive the version ID null. +// +// If the versioning state has never been set on a bucket, it has no versioning +// state; a GetBucketVersioning request does not return a versioning state value. +// +// If the bucket owner enables MFA Delete in the bucket versioning configuration, +// the bucket owner must include the x-amz-mfa request header and the Status +// and the MfaDelete request elements in a request to set the versioning state +// of the bucket. +// +// If you have an object expiration lifecycle policy in your non-versioned bucket +// and you want to maintain the same permanent delete behavior when you enable +// versioning, you must add a noncurrent expiration policy. The noncurrent expiration +// lifecycle policy will manage the deletes of the noncurrent object versions +// in the version-enabled bucket. (A version-enabled bucket maintains one current +// and zero or more noncurrent object versions.) For more information, see Lifecycle +// and Versioning (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html#lifecycle-and-other-bucket-config). +// +// Related Resources +// +// * CreateBucket +// +// * DeleteBucket +// +// * GetBucketVersioning +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -6132,12 +8298,81 @@ func (c *S3) PutBucketWebsiteRequest(input *PutBucketWebsiteInput) (req *request output = &PutBucketWebsiteOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutBucketWebsite API operation for Amazon Simple Storage Service. // -// Set the website configuration for a bucket. +// Sets the configuration of the website that is specified in the website subresource. +// To configure a bucket as a website, you can add this subresource on the bucket +// with website configuration information such as the file name of the index +// document and any redirect rules. For more information, see Hosting Websites +// on Amazon S3 (https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html). +// +// This PUT operation requires the S3:PutBucketWebsite permission. By default, +// only the bucket owner can configure the website attached to a bucket; however, +// bucket owners can allow other users to set the website configuration by writing +// a bucket policy that grants them the S3:PutBucketWebsite permission. +// +// To redirect all website requests sent to the bucket's website endpoint, you +// add a website configuration with the following elements. Because all requests +// are sent to another website, you don't need to provide index document name +// for the bucket. +// +// * WebsiteConfiguration +// +// * RedirectAllRequestsTo +// +// * HostName +// +// * Protocol +// +// If you want granular control over redirects, you can use the following elements +// to add routing rules that describe conditions for redirecting requests and +// information about the redirect destination. In this case, the website configuration +// must provide an index document for the bucket, because some requests might +// not be redirected. +// +// * WebsiteConfiguration +// +// * IndexDocument +// +// * Suffix +// +// * ErrorDocument +// +// * Key +// +// * RoutingRules +// +// * RoutingRule +// +// * Condition +// +// * HttpErrorCodeReturnedEquals +// +// * KeyPrefixEquals +// +// * Redirect +// +// * Protocol +// +// * HostName +// +// * ReplaceKeyPrefixWith +// +// * ReplaceKeyWith +// +// * HttpRedirectCode +// +// Amazon S3 has a limitation of 50 routing rules per website configuration. +// If you require more than 50 routing rules, you can use object redirect. For +// more information, see Configuring an Object Redirect (https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html) +// in the Amazon Simple Storage Service Developer Guide. // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -6211,7 +8446,70 @@ func (c *S3) PutObjectRequest(input *PutObjectInput) (req *request.Request, outp // PutObject API operation for Amazon Simple Storage Service. // -// Adds an object to a bucket. +// Adds an object to a bucket. You must have WRITE permissions on a bucket to +// add an object to it. +// +// Amazon S3 never adds partial objects; if you receive a success response, +// Amazon S3 added the entire object to the bucket. +// +// Amazon S3 is a distributed system. If it receives multiple write requests +// for the same object simultaneously, it overwrites all but the last object +// written. Amazon S3 does not provide object locking; if you need this, make +// sure to build it into your application layer or use versioning instead. +// +// To ensure that data is not corrupted traversing the network, use the Content-MD5 +// header. When you use this header, Amazon S3 checks the object against the +// provided MD5 value and, if they do not match, returns an error. Additionally, +// you can calculate the MD5 while putting an object to Amazon S3 and compare +// the returned ETag to the calculated MD5 value. +// +// The Content-MD5 header is required for any request to upload an object with +// a retention period configured using Amazon S3 Object Lock. For more information +// about Amazon S3 Object Lock, see Amazon S3 Object Lock Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock-overview.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Server-side Encryption +// +// You can optionally request server-side encryption. With server-side encryption, +// Amazon S3 encrypts your data as it writes it to disks in its data centers +// and decrypts the data when you access it. You have the option to provide +// your own encryption key or use AWS managed encryption keys. For more information, +// see Using Server-Side Encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html). +// +// Access Control List (ACL)-Specific Request Headers +// +// You can use headers to grant ACL- based permissions. By default, all objects +// are private. Only the owner has full access control. When adding a new object, +// you can grant permissions to individual AWS accounts or to predefined groups +// defined by Amazon S3. These permissions are then added to the ACL on the +// object. For more information, see Access Control List (ACL) Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html) +// and Managing ACLs Using the REST API (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-using-rest-api.html). +// +// Storage Class Options +// +// By default, Amazon S3 uses the STANDARD storage class to store newly created +// objects. The STANDARD storage class provides high durability and high availability. +// Depending on performance needs, you can specify a different storage class. +// For more information, see Storage Classes (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html) +// in the Amazon S3 Service Developer Guide. +// +// Versioning +// +// If you enable versioning for a bucket, Amazon S3 automatically generates +// a unique version ID for the object being stored. Amazon S3 returns this ID +// in the response. When you enable versioning for a bucket, if Amazon S3 receives +// multiple write requests for the same object simultaneously, it stores all +// of the objects. +// +// For more information about versioning, see Adding Objects to Versioning Enabled +// Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/AddingObjectstoVersioningEnabledBuckets.html). +// For information about returning the versioning state of a bucket, see GetBucketVersioning. +// +// Related Resources +// +// * CopyObject +// +// * DeleteObject // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -6280,13 +8578,97 @@ func (c *S3) PutObjectAclRequest(input *PutObjectAclInput) (req *request.Request output = &PutObjectAclOutput{} req = c.newRequest(op, input, output) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutObjectAcl API operation for Amazon Simple Storage Service. // -// uses the acl subresource to set the access control list (ACL) permissions -// for an object that already exists in a bucket +// Uses the acl subresource to set the access control list (ACL) permissions +// for an object that already exists in a bucket. You must have WRITE_ACP permission +// to set the ACL of an object. +// +// Depending on your application needs, you can choose to set the ACL on an +// object using either the request body or the headers. For example, if you +// have an existing application that updates a bucket ACL using the request +// body, you can continue to use that approach. For more information, see Access +// Control List (ACL) Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html) +// in the Amazon S3 Developer Guide. +// +// Access Permissions +// +// You can set access permissions using one of the following methods: +// +// * Specify a canned ACL with the x-amz-acl request header. Amazon S3 supports +// a set of predefined ACLs, known as canned ACLs. Each canned ACL has a +// predefined set of grantees and permissions. Specify the canned ACL name +// as the value of x-amz-acl. If you use this header, you cannot use other +// access control-specific headers in your request. For more information, +// see Canned ACL (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL). +// +// * Specify access permissions explicitly with the x-amz-grant-read, x-amz-grant-read-acp, +// x-amz-grant-write-acp, and x-amz-grant-full-control headers. When using +// these headers, you specify explicit access permissions and grantees (AWS +// accounts or Amazon S3 groups) who will receive the permission. If you +// use these ACL-specific headers, you cannot use x-amz-acl header to set +// a canned ACL. These parameters map to the set of permissions that Amazon +// S3 supports in an ACL. For more information, see Access Control List (ACL) +// Overview (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html). +// You specify each grantee as a type=value pair, where the type is one of +// the following: id – if the value specified is the canonical user ID +// of an AWS account uri – if you are granting permissions to a predefined +// group emailAddress – if the value specified is the email address of +// an AWS account Using email addresses to specify a grantee is only supported +// in the following AWS Regions: US East (N. Virginia) US West (N. California) +// US West (Oregon) Asia Pacific (Singapore) Asia Pacific (Sydney) Asia Pacific +// (Tokyo) Europe (Ireland) South America (São Paulo) For a list of all +// the Amazon S3 supported Regions and endpoints, see Regions and Endpoints +// (https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) in +// the AWS General Reference. For example, the following x-amz-grant-read +// header grants list objects permission to the two AWS accounts identified +// by their email addresses. x-amz-grant-read: emailAddress="xyz@amazon.com", +// emailAddress="abc@amazon.com" +// +// You can use either a canned ACL or specify access permissions explicitly. +// You cannot do both. +// +// Grantee Values +// +// You can specify the person (grantee) to whom you're assigning access rights +// (using request elements) in the following ways: +// +// * By the person's ID: <>ID<><>GranteesEmail<> +// DisplayName is optional and ignored in the request. +// +// * By URI: <>http://acs.amazonaws.com/groups/global/AuthenticatedUsers<> +// +// * By Email address: <>Grantees@email.com<>lt;/Grantee> +// The grantee is resolved to the CanonicalUser and, in a response to a GET +// Object acl request, appears as the CanonicalUser. Using email addresses +// to specify a grantee is only supported in the following AWS Regions: US +// East (N. Virginia) US West (N. California) US West (Oregon) Asia Pacific +// (Singapore) Asia Pacific (Sydney) Asia Pacific (Tokyo) Europe (Ireland) +// South America (São Paulo) For a list of all the Amazon S3 supported Regions +// and endpoints, see Regions and Endpoints (https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) +// in the AWS General Reference. +// +// Versioning +// +// The ACL of an object is set at the object version level. By default, PUT +// sets the ACL of the current version of an object. To set the ACL of a different +// version, use the versionId subresource. +// +// Related Resources +// +// * CopyObject +// +// * GetObject // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -6360,6 +8742,10 @@ func (c *S3) PutObjectLegalHoldRequest(input *PutObjectLegalHoldInput) (req *req output = &PutObjectLegalHoldOutput{} req = c.newRequest(op, input, output) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } @@ -6367,6 +8753,10 @@ func (c *S3) PutObjectLegalHoldRequest(input *PutObjectLegalHoldInput) (req *req // // Applies a Legal Hold configuration to the specified object. // +// Related Resources +// +// * Locking Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html) +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -6434,6 +8824,10 @@ func (c *S3) PutObjectLockConfigurationRequest(input *PutObjectLockConfiguration output = &PutObjectLockConfigurationOutput{} req = c.newRequest(op, input, output) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } @@ -6443,6 +8837,13 @@ func (c *S3) PutObjectLockConfigurationRequest(input *PutObjectLockConfiguration // in the Object Lock configuration will be applied by default to every new // object placed in the specified bucket. // +// DefaultRetention requires either Days or Years. You can't specify both at +// the same time. +// +// Related Resources +// +// * Locking Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html) +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -6510,6 +8911,10 @@ func (c *S3) PutObjectRetentionRequest(input *PutObjectRetentionInput) (req *req output = &PutObjectRetentionOutput{} req = c.newRequest(op, input, output) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } @@ -6517,6 +8922,10 @@ func (c *S3) PutObjectRetentionRequest(input *PutObjectRetentionInput) (req *req // // Places an Object Retention configuration on an object. // +// Related Resources +// +// * Locking Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html) +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -6584,12 +8993,53 @@ func (c *S3) PutObjectTaggingRequest(input *PutObjectTaggingInput) (req *request output = &PutObjectTaggingOutput{} req = c.newRequest(op, input, output) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutObjectTagging API operation for Amazon Simple Storage Service. // -// Sets the supplied tag-set to an object that already exists in a bucket +// Sets the supplied tag-set to an object that already exists in a bucket. +// +// A tag is a key-value pair. You can associate tags with an object by sending +// a PUT request against the tagging subresource that is associated with the +// object. You can retrieve tags by sending a GET request. For more information, +// see GetObjectTagging. +// +// For tagging-related restrictions related to characters and encodings, see +// Tag Restrictions (https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/allocation-tag-restrictions.html). +// Note that Amazon S3 limits the maximum number of tags to 10 tags per object. +// +// To use this operation, you must have permission to perform the s3:PutObjectTagging +// action. By default, the bucket owner has this permission and can grant this +// permission to others. +// +// To put tags of any other version, use the versionId query parameter. You +// also need permission for the s3:PutObjectVersionTagging action. +// +// For information about the Amazon S3 object tagging feature, see Object Tagging +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html). +// +// Special Errors +// +// * Code: InvalidTagError Cause: The tag provided was not a valid tag. This +// error can occur if the tag did not pass input validation. For more information, +// see Object Tagging (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-tagging.html). +// +// * Code: MalformedXMLError Cause: The XML provided does not match the schema. +// +// * Code: OperationAbortedError Cause: A conflicting conditional operation +// is currently in progress against this resource. Please try again. +// +// * Code: InternalError Cause: The service was unable to apply the provided +// tag to the object. +// +// Related Resources +// +// * GetObjectTagging // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -6659,13 +9109,39 @@ func (c *S3) PutPublicAccessBlockRequest(input *PutPublicAccessBlockInput) (req output = &PutPublicAccessBlockOutput{} req = c.newRequest(op, input, output) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, protocol.UnmarshalDiscardBodyHandler) + req.Handlers.Build.PushBackNamed(request.NamedHandler{ + Name: "contentMd5Handler", + Fn: checksum.AddBodyContentMD5Handler, + }) return } // PutPublicAccessBlock API operation for Amazon Simple Storage Service. // // Creates or modifies the PublicAccessBlock configuration for an Amazon S3 -// bucket. +// bucket. To use this operation, you must have the s3:PutBucketPublicAccessBlock +// permission. For more information about Amazon S3 permissions, see Specifying +// Permissions in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html). +// +// When Amazon S3 evaluates the PublicAccessBlock configuration for a bucket +// or an object, it checks the PublicAccessBlock configuration for both the +// bucket (or the bucket that contains the object) and the bucket owner's account. +// If the PublicAccessBlock configurations are different between the bucket +// and the account, Amazon S3 uses the most restrictive combination of the bucket-level +// and account-level settings. +// +// For more information about when Amazon S3 considers a bucket or an object +// public, see The Meaning of "Public" (https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html#access-control-block-public-access-policy-status). +// +// Related Resources +// +// * GetPublicAccessBlock +// +// * DeletePublicAccessBlock +// +// * GetBucketPolicyStatus +// +// * Using Amazon S3 Block Public Access (https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html) // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -6741,6 +9217,190 @@ func (c *S3) RestoreObjectRequest(input *RestoreObjectInput) (req *request.Reque // // Restores an archived copy of an object back into Amazon S3 // +// This operation performs the following types of requests: +// +// * select - Perform a select query on an archived object +// +// * restore an archive - Restore an archived object +// +// To use this operation, you must have permissions to perform the s3:RestoreObject +// action. The bucket owner has this permission by default and can grant this +// permission to others. For more information about permissions, see Permissions +// Related to Bucket Subresource Operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-bucket-subresources) +// and Managing Access Permissions to Your Amazon S3 Resources (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Querying Archives with Select Requests +// +// You use a select type of request to perform SQL queries on archived objects. +// The archived objects that are being queried by the select request must be +// formatted as uncompressed comma-separated values (CSV) files. You can run +// queries and custom analytics on your archived data without having to restore +// your data to a hotter Amazon S3 tier. For an overview about select requests, +// see Querying Archived Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/querying-glacier-archives.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// When making a select request, do the following: +// +// * Define an output location for the select query's output. This must be +// an Amazon S3 bucket in the same AWS Region as the bucket that contains +// the archive object that is being queried. The AWS account that initiates +// the job must have permissions to write to the S3 bucket. You can specify +// the storage class and encryption for the output objects stored in the +// bucket. For more information about output, see Querying Archived Objects +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/querying-glacier-archives.html) +// in the Amazon Simple Storage Service Developer Guide. For more information +// about the S3 structure in the request body, see the following: PutObject +// Managing Access with ACLs (https://docs.aws.amazon.com/AmazonS3/latest/dev/S3_ACLs_UsingACLs.html) +// in the Amazon Simple Storage Service Developer Guide Protecting Data Using +// Server-Side Encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html) +// in the Amazon Simple Storage Service Developer Guide +// +// * Define the SQL expression for the SELECT type of restoration for your +// query in the request body's SelectParameters structure. You can use expressions +// like the following examples. The following expression returns all records +// from the specified object. SELECT * FROM Object Assuming that you are +// not using any headers for data stored in the object, you can specify columns +// with positional headers. SELECT s._1, s._2 FROM Object s WHERE s._3 > +// 100 If you have headers and you set the fileHeaderInfo in the CSV structure +// in the request body to USE, you can specify headers in the query. (If +// you set the fileHeaderInfo field to IGNORE, the first row is skipped for +// the query.) You cannot mix ordinal positions with header column names. +// SELECT s.Id, s.FirstName, s.SSN FROM S3Object s +// +// For more information about using SQL with S3 Glacier Select restore, see +// SQL Reference for Amazon S3 Select and S3 Glacier Select (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-glacier-select-sql-reference.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// When making a select request, you can also do the following: +// +// * To expedite your queries, specify the Expedited tier. For more information +// about tiers, see "Restoring Archives," later in this topic. +// +// * Specify details about the data serialization format of both the input +// object that is being queried and the serialization of the CSV-encoded +// query results. +// +// The following are additional important facts about the select feature: +// +// * The output results are new Amazon S3 objects. Unlike archive retrievals, +// they are stored until explicitly deleted-manually or through a lifecycle +// policy. +// +// * You can issue more than one select request on the same Amazon S3 object. +// Amazon S3 doesn't deduplicate requests, so avoid issuing duplicate requests. +// +// * Amazon S3 accepts a select request even if the object has already been +// restored. A select request doesn’t return error response 409. +// +// Restoring Archives +// +// Objects in the GLACIER and DEEP_ARCHIVE storage classes are archived. To +// access an archived object, you must first initiate a restore request. This +// restores a temporary copy of the archived object. In a restore request, you +// specify the number of days that you want the restored copy to exist. After +// the specified period, Amazon S3 deletes the temporary copy but the object +// remains archived in the GLACIER or DEEP_ARCHIVE storage class that object +// was restored from. +// +// To restore a specific object version, you can provide a version ID. If you +// don't provide a version ID, Amazon S3 restores the current version. +// +// The time it takes restore jobs to finish depends on which storage class the +// object is being restored from and which data access tier you specify. +// +// When restoring an archived object (or using a select request), you can specify +// one of the following data access tier options in the Tier element of the +// request body: +// +// * Expedited - Expedited retrievals allow you to quickly access your data +// stored in the GLACIER storage class when occasional urgent requests for +// a subset of archives are required. For all but the largest archived objects +// (250 MB+), data accessed using Expedited retrievals are typically made +// available within 1–5 minutes. Provisioned capacity ensures that retrieval +// capacity for Expedited retrievals is available when you need it. Expedited +// retrievals and provisioned capacity are not available for the DEEP_ARCHIVE +// storage class. +// +// * Standard - S3 Standard retrievals allow you to access any of your archived +// objects within several hours. This is the default option for the GLACIER +// and DEEP_ARCHIVE retrieval requests that do not specify the retrieval +// option. S3 Standard retrievals typically complete within 3-5 hours from +// the GLACIER storage class and typically complete within 12 hours from +// the DEEP_ARCHIVE storage class. +// +// * Bulk - Bulk retrievals are Amazon S3 Glacier’s lowest-cost retrieval +// option, enabling you to retrieve large amounts, even petabytes, of data +// inexpensively in a day. Bulk retrievals typically complete within 5-12 +// hours from the GLACIER storage class and typically complete within 48 +// hours from the DEEP_ARCHIVE storage class. +// +// For more information about archive retrieval options and provisioned capacity +// for Expedited data access, see Restoring Archived Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/restoring-objects.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// You can use Amazon S3 restore speed upgrade to change the restore speed to +// a faster speed while it is in progress. You upgrade the speed of an in-progress +// restoration by issuing another restore request to the same object, setting +// a new Tier request element. When issuing a request to upgrade the restore +// tier, you must choose a tier that is faster than the tier that the in-progress +// restore is using. You must not change any other parameters, such as the Days +// request element. For more information, see Upgrading the Speed of an In-Progress +// Restore (https://docs.aws.amazon.com/AmazonS3/latest/dev/restoring-objects.html#restoring-objects-upgrade-tier.title.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// To get the status of object restoration, you can send a HEAD request. Operations +// return the x-amz-restore header, which provides information about the restoration +// status, in the response. You can use Amazon S3 event notifications to notify +// you when a restore is initiated or completed. For more information, see Configuring +// Amazon S3 Event Notifications (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// After restoring an archived object, you can update the restoration period +// by reissuing the request with a new period. Amazon S3 updates the restoration +// period relative to the current time and charges only for the request-there +// are no data transfer charges. You cannot update the restoration period when +// Amazon S3 is actively processing your current restore request for the object. +// +// If your bucket has a lifecycle configuration with a rule that includes an +// expiration action, the object expiration overrides the life span that you +// specify in a restore request. For example, if you restore an object copy +// for 10 days, but the object is scheduled to expire in 3 days, Amazon S3 deletes +// the object in 3 days. For more information about lifecycle configuration, +// see PutBucketLifecycleConfiguration and Object Lifecycle Management (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) +// in Amazon Simple Storage Service Developer Guide. +// +// Responses +// +// A successful operation returns either the 200 OK or 202 Accepted status code. +// +// * If the object copy is not previously restored, then Amazon S3 returns +// 202 Accepted in the response. +// +// * If the object copy is previously restored, Amazon S3 returns 200 OK +// in the response. +// +// Special Errors +// +// * Code: RestoreAlreadyInProgress Cause: Object restore is already in progress. +// (This error does not apply to SELECT type requests.) HTTP Status Code: +// 409 Conflict SOAP Fault Code Prefix: Client +// +// * Code: GlacierExpeditedRetrievalNotAvailable Cause: S3 Glacier expedited +// retrievals are currently not available. Try again later. (Returned if +// there is insufficient capacity to process the Expedited request. This +// error applies only to Expedited retrievals and not to S3 Standard or Bulk +// retrievals.) HTTP Status Code: 503 SOAP Fault Code Prefix: N/A +// +// Related Resources +// +// * PutBucketLifecycleConfiguration +// +// * GetBucketNotificationConfiguration +// +// * SQL Reference for Amazon S3 Select and S3 Glacier Select (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-glacier-select-sql-reference.html) +// in the Amazon Simple Storage Service Developer Guide +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -6750,7 +9410,7 @@ func (c *S3) RestoreObjectRequest(input *RestoreObjectInput) (req *request.Reque // // Returned Error Codes: // * ErrCodeObjectAlreadyInActiveTierError "ObjectAlreadyInActiveTierError" -// This operation is not allowed against this storage tier +// This operation is not allowed against this storage tier. // // See also, https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/RestoreObject func (c *S3) RestoreObject(input *RestoreObjectInput) (*RestoreObjectOutput, error) { @@ -6813,20 +9473,103 @@ func (c *S3) SelectObjectContentRequest(input *SelectObjectContentInput) (req *r output = &SelectObjectContentOutput{} req = c.newRequest(op, input, output) + + es := newSelectObjectContentEventStream() + req.Handlers.Unmarshal.PushBack(es.setStreamCloser) + output.EventStream = es + req.Handlers.Send.Swap(client.LogHTTPResponseHandler.Name, client.LogHTTPResponseHeaderHandler) req.Handlers.Unmarshal.Swap(restxml.UnmarshalHandler.Name, rest.UnmarshalHandler) - req.Handlers.Unmarshal.PushBack(output.runEventStreamLoop) + req.Handlers.Unmarshal.PushBack(es.runOutputStream) + req.Handlers.Unmarshal.PushBack(es.runOnStreamPartClose) return } // SelectObjectContent API operation for Amazon Simple Storage Service. // // This operation filters the contents of an Amazon S3 object based on a simple -// Structured Query Language (SQL) statement. In the request, along with the -// SQL expression, you must also specify a data serialization format (JSON or -// CSV) of the object. Amazon S3 uses this to parse object data into records, -// and returns only records that match the specified SQL expression. You must -// also specify the data serialization format for the response. +// structured query language (SQL) statement. In the request, along with the +// SQL expression, you must also specify a data serialization format (JSON, +// CSV, or Apache Parquet) of the object. Amazon S3 uses this format to parse +// object data into records, and returns only records that match the specified +// SQL expression. You must also specify the data serialization format for the +// response. +// +// For more information about Amazon S3 Select, see Selecting Content from Objects +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/selecting-content-from-objects.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// For more information about using SQL with Amazon S3 Select, see SQL Reference +// for Amazon S3 Select and S3 Glacier Select (https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-glacier-select-sql-reference.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Permissions +// +// You must have s3:GetObject permission for this operation. Amazon S3 Select +// does not support anonymous access. For more information about permissions, +// see Specifying Permissions in a Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Object Data Formats +// +// You can use Amazon S3 Select to query objects that have the following format +// properties: +// +// * CSV, JSON, and Parquet - Objects must be in CSV, JSON, or Parquet format. +// +// * UTF-8 - UTF-8 is the only encoding type Amazon S3 Select supports. +// +// * GZIP or BZIP2 - CSV and JSON files can be compressed using GZIP or BZIP2. +// GZIP and BZIP2 are the only compression formats that Amazon S3 Select +// supports for CSV and JSON files. Amazon S3 Select supports columnar compression +// for Parquet using GZIP or Snappy. Amazon S3 Select does not support whole-object +// compression for Parquet objects. +// +// * Server-side encryption - Amazon S3 Select supports querying objects +// that are protected with server-side encryption. For objects that are encrypted +// with customer-provided encryption keys (SSE-C), you must use HTTPS, and +// you must use the headers that are documented in the GetObject. For more +// information about SSE-C, see Server-Side Encryption (Using Customer-Provided +// Encryption Keys) (https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html) +// in the Amazon Simple Storage Service Developer Guide. For objects that +// are encrypted with Amazon S3 managed encryption keys (SSE-S3) and customer +// master keys (CMKs) stored in AWS Key Management Service (SSE-KMS), server-side +// encryption is handled transparently, so you don't need to specify anything. +// For more information about server-side encryption, including SSE-S3 and +// SSE-KMS, see Protecting Data Using Server-Side Encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Working with the Response Body +// +// Given the response size is unknown, Amazon S3 Select streams the response +// as a series of messages and includes a Transfer-Encoding header with chunked +// as its value in the response. For more information, see RESTSelectObjectAppendix . +// +// GetObject Support +// +// The SelectObjectContent operation does not support the following GetObject +// functionality. For more information, see GetObject. +// +// * Range: Although you can specify a scan range for an Amazon S3 Select +// request (see SelectObjectContentRequest$ScanRange in the request parameters), +// you cannot specify the range of bytes of an object to return. +// +// * GLACIER, DEEP_ARCHIVE and REDUCED_REDUNDANCY storage classes: You cannot +// specify the GLACIER, DEEP_ARCHIVE, or REDUCED_REDUNDANCY storage classes. +// For more information, about storage classes see Storage Classes (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html#storage-class-intro) +// in the Amazon Simple Storage Service Developer Guide. +// +// Special Errors +// +// For a list of special errors for this operation, see SelectObjectContentErrorCodeList +// +// Related Resources +// +// * GetObject +// +// * GetBucketLifecycleConfiguration +// +// * PutBucketLifecycleConfiguration // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -6856,36 +9599,180 @@ func (c *S3) SelectObjectContentWithContext(ctx aws.Context, input *SelectObject return out, req.Send() } -const opUploadPart = "UploadPart" +var _ awserr.Error -// UploadPartRequest generates a "aws/request.Request" representing the -// client's request for the UploadPart operation. The "output" return -// value will be populated with the request's response once the request completes -// successfully. -// -// Use "Send" method on the returned Request to send the API call to the service. -// the "output" return value is not valid until after Send returns without error. -// -// See UploadPart for more information on using the UploadPart -// API call, and error handling. -// -// This method is useful when you want to inject custom logic or configuration -// into the SDK's request lifecycle. Such as custom headers, or retry logic. -// -// -// // Example sending a request using the UploadPartRequest method. -// req, resp := client.UploadPartRequest(params) -// -// err := req.Send() -// if err == nil { // resp is now filled -// fmt.Println(resp) -// } -// -// See also, https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/UploadPart -func (c *S3) UploadPartRequest(input *UploadPartInput) (req *request.Request, output *UploadPartOutput) { - op := &request.Operation{ - Name: opUploadPart, - HTTPMethod: "PUT", +// SelectObjectContentEventStream provides the event stream handling for the SelectObjectContent. +type SelectObjectContentEventStream struct { + + // Reader is the EventStream reader for the SelectObjectContentEventStream + // events. This value is automatically set by the SDK when the API call is made + // Use this member when unit testing your code with the SDK to mock out the + // EventStream Reader. + // + // Must not be nil. + Reader SelectObjectContentEventStreamReader + + outputReader io.ReadCloser + + // StreamCloser is the io.Closer for the EventStream connection. For HTTP + // EventStream this is the response Body. The stream will be closed when + // the Close method of the EventStream is called. + StreamCloser io.Closer + + done chan struct{} + closeOnce sync.Once + err *eventstreamapi.OnceError +} + +func newSelectObjectContentEventStream() *SelectObjectContentEventStream { + return &SelectObjectContentEventStream{ + done: make(chan struct{}), + err: eventstreamapi.NewOnceError(), + } +} + +func (es *SelectObjectContentEventStream) setStreamCloser(r *request.Request) { + es.StreamCloser = r.HTTPResponse.Body +} + +func (es *SelectObjectContentEventStream) runOnStreamPartClose(r *request.Request) { + if es.done == nil { + return + } + go es.waitStreamPartClose() + +} + +func (es *SelectObjectContentEventStream) waitStreamPartClose() { + var outputErrCh <-chan struct{} + if v, ok := es.Reader.(interface{ ErrorSet() <-chan struct{} }); ok { + outputErrCh = v.ErrorSet() + } + var outputClosedCh <-chan struct{} + if v, ok := es.Reader.(interface{ Closed() <-chan struct{} }); ok { + outputClosedCh = v.Closed() + } + + select { + case <-es.done: + case <-outputErrCh: + es.err.SetError(es.Reader.Err()) + es.Close() + case <-outputClosedCh: + if err := es.Reader.Err(); err != nil { + es.err.SetError(es.Reader.Err()) + } + es.Close() + } +} + +// Events returns a channel to read events from. +// +// These events are: +// +// * ContinuationEvent +// * EndEvent +// * ProgressEvent +// * RecordsEvent +// * StatsEvent +// * SelectObjectContentEventStreamUnknownEvent +func (es *SelectObjectContentEventStream) Events() <-chan SelectObjectContentEventStreamEvent { + return es.Reader.Events() +} + +func (es *SelectObjectContentEventStream) runOutputStream(r *request.Request) { + var opts []func(*eventstream.Decoder) + if r.Config.Logger != nil && r.Config.LogLevel.Matches(aws.LogDebugWithEventStreamBody) { + opts = append(opts, eventstream.DecodeWithLogger(r.Config.Logger)) + } + + unmarshalerForEvent := unmarshalerForSelectObjectContentEventStreamEvent{ + metadata: protocol.ResponseMetadata{ + StatusCode: r.HTTPResponse.StatusCode, + RequestID: r.RequestID, + }, + }.UnmarshalerForEventName + + decoder := eventstream.NewDecoder(r.HTTPResponse.Body, opts...) + eventReader := eventstreamapi.NewEventReader(decoder, + protocol.HandlerPayloadUnmarshal{ + Unmarshalers: r.Handlers.UnmarshalStream, + }, + unmarshalerForEvent, + ) + + es.outputReader = r.HTTPResponse.Body + es.Reader = newReadSelectObjectContentEventStream(eventReader) +} + +// Close closes the stream. This will also cause the stream to be closed. +// Close must be called when done using the stream API. Not calling Close +// may result in resource leaks. +// +// You can use the closing of the Reader's Events channel to terminate your +// application's read from the API's stream. +// +func (es *SelectObjectContentEventStream) Close() (err error) { + es.closeOnce.Do(es.safeClose) + return es.Err() +} + +func (es *SelectObjectContentEventStream) safeClose() { + if es.done != nil { + close(es.done) + } + + es.Reader.Close() + if es.outputReader != nil { + es.outputReader.Close() + } + + es.StreamCloser.Close() +} + +// Err returns any error that occurred while reading or writing EventStream +// Events from the service API's response. Returns nil if there were no errors. +func (es *SelectObjectContentEventStream) Err() error { + if err := es.err.Err(); err != nil { + return err + } + if err := es.Reader.Err(); err != nil { + return err + } + + return nil +} + +const opUploadPart = "UploadPart" + +// UploadPartRequest generates a "aws/request.Request" representing the +// client's request for the UploadPart operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See UploadPart for more information on using the UploadPart +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the UploadPartRequest method. +// req, resp := client.UploadPartRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/s3-2006-03-01/UploadPart +func (c *S3) UploadPartRequest(input *UploadPartInput) (req *request.Request, output *UploadPartOutput) { + op := &request.Operation{ + Name: opUploadPart, + HTTPMethod: "PUT", HTTPPath: "/{Bucket}/{Key+}", } @@ -6902,12 +9789,87 @@ func (c *S3) UploadPartRequest(input *UploadPartInput) (req *request.Request, ou // // Uploads a part in a multipart upload. // +// In this operation, you provide part data in your request. However, you have +// an option to specify your existing Amazon S3 object as a data source for +// the part you are uploading. To upload a part from an existing object, you +// use the UploadPartCopy operation. +// +// You must initiate a multipart upload (see CreateMultipartUpload) before you +// can upload any part. In response to your initiate request, Amazon S3 returns +// an upload ID, a unique identifier, that you must include in your upload part +// request. +// +// Part numbers can be any number from 1 to 10,000, inclusive. A part number +// uniquely identifies a part and also defines its position within the object +// being created. If you upload a new part using the same part number that was +// used with a previous part, the previously uploaded part is overwritten. Each +// part must be at least 5 MB in size, except the last part. There is no size +// limit on the last part of your multipart upload. +// +// To ensure that data is not corrupted when traversing the network, specify +// the Content-MD5 header in the upload part request. Amazon S3 checks the part +// data against the provided MD5 value. If they do not match, Amazon S3 returns +// an error. +// // Note: After you initiate multipart upload and upload one or more parts, you // must either complete or abort multipart upload in order to stop getting charged // for storage of the uploaded parts. Only after you either complete or abort // multipart upload, Amazon S3 frees up the parts storage and stops charging // you for the parts storage. // +// For more information on multipart uploads, go to Multipart Upload Overview +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html) in the +// Amazon Simple Storage Service Developer Guide . +// +// For information on the permissions required to use the multipart upload API, +// go to Multipart Upload API and Permissions (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuAndPermissions.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// You can optionally request server-side encryption where Amazon S3 encrypts +// your data as it writes it to disks in its data centers and decrypts it for +// you when you access it. You have the option of providing your own encryption +// key, or you can use the AWS managed encryption keys. If you choose to provide +// your own encryption key, the request headers you provide in the request must +// match the headers you used in the request to initiate the upload by using +// CreateMultipartUpload. For more information, go to Using Server-Side Encryption +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Server-side encryption is supported by the S3 Multipart Upload actions. Unless +// you are using a customer-provided encryption key, you don't need to specify +// the encryption parameters in each UploadPart request. Instead, you only need +// to specify the server-side encryption parameters in the initial Initiate +// Multipart request. For more information, see CreateMultipartUpload. +// +// If you requested server-side encryption using a customer-provided encryption +// key in your initiate multipart upload request, you must provide identical +// encryption information in each part upload using the following headers. +// +// * x-amz-server-side​-encryption​-customer-algorithm +// +// * x-amz-server-side​-encryption​-customer-key +// +// * x-amz-server-side​-encryption​-customer-key-MD5 +// +// Special Errors +// +// * Code: NoSuchUpload Cause: The specified multipart upload does not exist. +// The upload ID might be invalid, or the multipart upload might have been +// aborted or completed. HTTP Status Code: 404 Not Found SOAP Fault Code +// Prefix: Client +// +// Related Resources +// +// * CreateMultipartUpload +// +// * CompleteMultipartUpload +// +// * AbortMultipartUpload +// +// * ListParts +// +// * ListMultipartUploads +// // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about // the error. @@ -6980,7 +9942,94 @@ func (c *S3) UploadPartCopyRequest(input *UploadPartCopyInput) (req *request.Req // UploadPartCopy API operation for Amazon Simple Storage Service. // -// Uploads a part by copying data from an existing object as data source. +// Uploads a part by copying data from an existing object as data source. You +// specify the data source by adding the request header x-amz-copy-source in +// your request and a byte range by adding the request header x-amz-copy-source-range +// in your request. +// +// The minimum allowable part size for a multipart upload is 5 MB. For more +// information about multipart upload limits, go to Quick Facts (https://docs.aws.amazon.com/AmazonS3/latest/dev/qfacts.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// Instead of using an existing object as part data, you might use the UploadPart +// operation and provide data in your request. +// +// You must initiate a multipart upload before you can upload any part. In response +// to your initiate request. Amazon S3 returns a unique identifier, the upload +// ID, that you must include in your upload part request. +// +// For more information about using the UploadPartCopy operation, see the following: +// +// * For conceptual information about multipart uploads, see Uploading Objects +// Using Multipart Upload (https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// * For information about permissions required to use the multipart upload +// API, see Multipart Upload API and Permissions (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuAndPermissions.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// * For information about copying objects using a single atomic operation +// vs. the multipart upload, see Operations on Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectOperations.html) +// in the Amazon Simple Storage Service Developer Guide. +// +// * For information about using server-side encryption with customer-provided +// encryption keys with the UploadPartCopy operation, see CopyObject and +// UploadPart. +// +// Note the following additional considerations about the request headers x-amz-copy-source-if-match, +// x-amz-copy-source-if-none-match, x-amz-copy-source-if-unmodified-since, and +// x-amz-copy-source-if-modified-since: +// +// * Consideration 1 - If both of the x-amz-copy-source-if-match and x-amz-copy-source-if-unmodified-since +// headers are present in the request as follows: x-amz-copy-source-if-match +// condition evaluates to true, and; x-amz-copy-source-if-unmodified-since +// condition evaluates to false; Amazon S3 returns 200 OK and copies the +// data. +// +// * Consideration 2 - If both of the x-amz-copy-source-if-none-match and +// x-amz-copy-source-if-modified-since headers are present in the request +// as follows: x-amz-copy-source-if-none-match condition evaluates to false, +// and; x-amz-copy-source-if-modified-since condition evaluates to true; +// Amazon S3 returns 412 Precondition Failed response code. +// +// Versioning +// +// If your bucket has versioning enabled, you could have multiple versions of +// the same object. By default, x-amz-copy-source identifies the current version +// of the object to copy. If the current version is a delete marker and you +// don't specify a versionId in the x-amz-copy-source, Amazon S3 returns a 404 +// error, because the object does not exist. If you specify versionId in the +// x-amz-copy-source and the versionId is a delete marker, Amazon S3 returns +// an HTTP 400 error, because you are not allowed to specify a delete marker +// as a version for the x-amz-copy-source. +// +// You can optionally specify a specific version of the source object to copy +// by adding the versionId subresource as shown in the following example: +// +// x-amz-copy-source: /bucket/object?versionId=version id +// +// Special Errors +// +// * Code: NoSuchUpload Cause: The specified multipart upload does not exist. +// The upload ID might be invalid, or the multipart upload might have been +// aborted or completed. HTTP Status Code: 404 Not Found +// +// * Code: InvalidRequest Cause: The specified copy source is not supported +// as a byte-range copy source. HTTP Status Code: 400 Bad Request +// +// Related Resources +// +// * CreateMultipartUpload +// +// * UploadPart +// +// * CompleteMultipartUpload +// +// * AbortMultipartUpload +// +// * ListParts +// +// * ListMultipartUploads // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -7010,13 +10059,16 @@ func (c *S3) UploadPartCopyWithContext(ctx aws.Context, input *UploadPartCopyInp return out, req.Send() } -// Specifies the days since the initiation of an Incomplete Multipart Upload -// that Lifecycle will wait before permanently removing all parts of the upload. +// Specifies the days since the initiation of an incomplete multipart upload +// that Amazon S3 will wait before permanently removing all parts of the upload. +// For more information, see Aborting Incomplete Multipart Uploads Using a Bucket +// Lifecycle Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-abort-incomplete-mpu-lifecycle-config) +// in the Amazon Simple Storage Service Developer Guide. type AbortIncompleteMultipartUpload struct { _ struct{} `type:"structure"` - // Indicates the number of days that must pass since initiation for Lifecycle - // to abort an Incomplete Multipart Upload. + // Specifies the number of days after which Amazon S3 aborts an incomplete multipart + // upload. DaysAfterInitiation *int64 `type:"integer"` } @@ -7037,20 +10089,34 @@ func (s *AbortIncompleteMultipartUpload) SetDaysAfterInitiation(v int64) *AbortI } type AbortMultipartUploadInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"AbortMultipartUploadRequest" type:"structure"` + // The bucket name to which the upload was taking place. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Key of the object for which the multipart upload was initiated. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` + // Upload ID that identifies the multipart upload. + // // UploadId is a required field UploadId *string `location:"querystring" locationName:"uploadId" type:"string" required:"true"` } @@ -7121,6 +10187,20 @@ func (s *AbortMultipartUploadInput) SetUploadId(v string) *AbortMultipartUploadI return s } +func (s *AbortMultipartUploadInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *AbortMultipartUploadInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type AbortMultipartUploadOutput struct { _ struct{} `type:"structure"` @@ -7145,10 +10225,13 @@ func (s *AbortMultipartUploadOutput) SetRequestCharged(v string) *AbortMultipart return s } +// Configures the transfer acceleration state for an Amazon S3 bucket. For more +// information, see Amazon S3 Transfer Acceleration (https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration.html) +// in the Amazon Simple Storage Service Developer Guide. type AccelerateConfiguration struct { _ struct{} `type:"structure"` - // The accelerate configuration of the bucket. + // Specifies the transfer acceleration status of the bucket. Status *string `type:"string" enum:"BucketAccelerateStatus"` } @@ -7168,12 +10251,14 @@ func (s *AccelerateConfiguration) SetStatus(v string) *AccelerateConfiguration { return s } +// Contains the elements that set the ACL permissions for an object per grantee. type AccessControlPolicy struct { _ struct{} `type:"structure"` // A list of grants. Grants []*Grant `locationName:"AccessControlList" locationNameList:"Grant" type:"list"` + // Container for the bucket owner's display name and ID. Owner *Owner `type:"structure"` } @@ -7223,7 +10308,9 @@ func (s *AccessControlPolicy) SetOwner(v *Owner) *AccessControlPolicy { type AccessControlTranslation struct { _ struct{} `type:"structure"` - // The override value for the owner of the replica object. + // Specifies the replica ownership. For default and valid values, see PUT bucket + // replication (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTreplication.html) + // in the Amazon Simple Storage Service API Reference. // // Owner is a required field Owner *string `type:"string" required:"true" enum:"OwnerOverride"` @@ -7258,10 +10345,14 @@ func (s *AccessControlTranslation) SetOwner(v string) *AccessControlTranslation return s } +// A conjunction (logical AND) of predicates, which is used in evaluating a +// metrics filter. The operator must have at least two predicates in any combination, +// and an object must match all of the predicates for the filter to apply. type AnalyticsAndOperator struct { _ struct{} `type:"structure"` - // The prefix to use when evaluating an AND predicate. + // The prefix to use when evaluating an AND predicate: The prefix that an object + // must have to be included in the metrics results. Prefix *string `type:"string"` // The list of tags to use when evaluating an AND predicate. @@ -7310,6 +10401,8 @@ func (s *AnalyticsAndOperator) SetTags(v []*Tag) *AnalyticsAndOperator { return s } +// Specifies the configuration and any analyses for the analytics filter of +// an Amazon S3 bucket. type AnalyticsConfiguration struct { _ struct{} `type:"structure"` @@ -7318,13 +10411,13 @@ type AnalyticsConfiguration struct { // If no filter is provided, all objects will be considered in any analysis. Filter *AnalyticsFilter `type:"structure"` - // The identifier used to represent an analytics configuration. + // The ID that identifies the analytics configuration. // // Id is a required field Id *string `type:"string" required:"true"` - // If present, it indicates that data related to access patterns will be collected - // and made available to analyze the tradeoffs between different storage classes. + // Contains data related to access patterns to be collected and made available + // to analyze the tradeoffs between different storage classes. // // StorageClassAnalysis is a required field StorageClassAnalysis *StorageClassAnalysis `type:"structure" required:"true"` @@ -7384,6 +10477,7 @@ func (s *AnalyticsConfiguration) SetStorageClassAnalysis(v *StorageClassAnalysis return s } +// Where to publish the analytics results. type AnalyticsExportDestination struct { _ struct{} `type:"structure"` @@ -7427,6 +10521,9 @@ func (s *AnalyticsExportDestination) SetS3BucketDestination(v *AnalyticsS3Bucket return s } +// The filter used to describe a set of objects for analyses. A filter must +// have exactly one prefix, one tag, or one conjunction (AnalyticsAndOperator). +// If no filter is provided, all objects will be considered in any analysis. type AnalyticsFilter struct { _ struct{} `type:"structure"` @@ -7489,25 +10586,28 @@ func (s *AnalyticsFilter) SetTag(v *Tag) *AnalyticsFilter { return s } +// Contains information about where to publish the analytics results. type AnalyticsS3BucketDestination struct { _ struct{} `type:"structure"` - // The Amazon resource name (ARN) of the bucket to which data is exported. + // The Amazon Resource Name (ARN) of the bucket to which data is exported. // // Bucket is a required field Bucket *string `type:"string" required:"true"` - // The account ID that owns the destination bucket. If no account ID is provided, - // the owner will not be validated prior to exporting data. + // The account ID that owns the destination S3 bucket. If no account ID is provided, + // the owner is not validated before exporting data. + // + // Although this value is optional, we strongly recommend that you set it to + // help prevent problems if the destination bucket ownership changes. BucketAccountId *string `type:"string"` - // The file format used when exporting data to Amazon S3. + // Specifies the file format used when exporting data to Amazon S3. // // Format is a required field Format *string `type:"string" required:"true" enum:"AnalyticsS3ExportFileFormat"` - // The prefix to use when exporting data. The exported data begins with this - // prefix. + // The prefix to use when exporting data. The prefix is prepended to all results. Prefix *string `type:"string"` } @@ -7568,6 +10668,8 @@ func (s *AnalyticsS3BucketDestination) SetPrefix(v string) *AnalyticsS3BucketDes return s } +// In terms of implementation, a Bucket is a resource. An Amazon S3 bucket name +// is globally unique, and the namespace is shared by all AWS accounts. type Bucket struct { _ struct{} `type:"structure"` @@ -7600,9 +10702,14 @@ func (s *Bucket) SetName(v string) *Bucket { return s } +// Specifies the lifecycle configuration for objects in an Amazon S3 bucket. +// For more information, see Object Lifecycle Management (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html) +// in the Amazon Simple Storage Service Developer Guide. type BucketLifecycleConfiguration struct { _ struct{} `type:"structure"` + // A lifecycle rule for individual objects in an Amazon S3 bucket. + // // Rules is a required field Rules []*LifecycleRule `locationName:"Rule" type:"list" flattened:"true" required:"true"` } @@ -7646,12 +10753,14 @@ func (s *BucketLifecycleConfiguration) SetRules(v []*LifecycleRule) *BucketLifec return s } +// Container for logging status information. type BucketLoggingStatus struct { _ struct{} `type:"structure"` - // Container for logging information. Presence of this element indicates that - // logging is enabled. Parameters TargetBucket and TargetPrefix are required - // in this case. + // Describes where logs are stored and the prefix that Amazon S3 assigns to + // all log object keys for a bucket. For more information, see PUT Bucket logging + // (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTlogging.html) + // in the Amazon Simple Storage Service API Reference. LoggingEnabled *LoggingEnabled `type:"structure"` } @@ -7686,9 +10795,16 @@ func (s *BucketLoggingStatus) SetLoggingEnabled(v *LoggingEnabled) *BucketLoggin return s } +// Describes the cross-origin access configuration for objects in an Amazon +// S3 bucket. For more information, see Enabling Cross-Origin Resource Sharing +// (https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html) in the Amazon +// Simple Storage Service Developer Guide. type CORSConfiguration struct { _ struct{} `type:"structure"` + // A set of origins and methods (cross-origin access that you want to allow). + // You can add up to 100 rules to the configuration. + // // CORSRules is a required field CORSRules []*CORSRule `locationName:"CORSRule" type:"list" flattened:"true" required:"true"` } @@ -7732,14 +10848,18 @@ func (s *CORSConfiguration) SetCORSRules(v []*CORSRule) *CORSConfiguration { return s } +// Specifies a cross-origin access rule for an Amazon S3 bucket. type CORSRule struct { _ struct{} `type:"structure"` - // Specifies which headers are allowed in a pre-flight OPTIONS request. + // Headers that are specified in the Access-Control-Request-Headers header. + // These headers are allowed in a preflight OPTIONS request. In response to + // any preflight OPTIONS request, Amazon S3 returns any requested headers that + // are allowed. AllowedHeaders []*string `locationName:"AllowedHeader" type:"list" flattened:"true"` - // Identifies HTTP methods that the domain/origin specified in the rule is allowed - // to execute. + // An HTTP method that you allow the origin to execute. Valid values are GET, + // PUT, HEAD, POST, and DELETE. // // AllowedMethods is a required field AllowedMethods []*string `locationName:"AllowedMethod" type:"list" flattened:"true" required:"true"` @@ -7815,7 +10935,8 @@ func (s *CORSRule) SetMaxAgeSeconds(v int64) *CORSRule { return s } -// Describes how a CSV-formatted input object is formatted. +// Describes how an uncompressed comma-separated values (CSV)-formatted input +// object is formatted. type CSVInput struct { _ struct{} `type:"structure"` @@ -7824,24 +10945,45 @@ type CSVInput struct { // to TRUE may lower performance. AllowQuotedRecordDelimiter *bool `type:"boolean"` - // The single character used to indicate a row should be ignored when present - // at the start of a row. + // A single character used to indicate that a row should be ignored when the + // character is present at the start of that row. You can specify any character + // to indicate a comment line. Comments *string `type:"string"` - // The value used to separate individual fields in a record. + // A single character used to separate individual fields in a record. You can + // specify an arbitrary delimiter. FieldDelimiter *string `type:"string"` - // Describes the first line of input. Valid values: None, Ignore, Use. + // Describes the first line of input. Valid values are: + // + // * NONE: First line is not a header. + // + // * IGNORE: First line is a header, but you can't use the header values + // to indicate the column in an expression. You can use column position (such + // as _1, _2, …) to indicate the column (SELECT s._1 FROM OBJECT s). + // + // * Use: First line is a header, and you can use the header value to identify + // a column in an expression (SELECT "name" FROM OBJECT). FileHeaderInfo *string `type:"string" enum:"FileHeaderInfo"` - // Value used for escaping where the field delimiter is part of the value. + // A single character used for escaping when the field delimiter is part of + // the value. For example, if the value is a, b, Amazon S3 wraps this field + // value in quotation marks, as follows: " a , b ". + // + // Type: String + // + // Default: " + // + // Ancestors: CSV QuoteCharacter *string `type:"string"` - // The single character used for escaping the quote character inside an already - // escaped value. + // A single character used for escaping the quotation mark character inside + // an already escaped value. For example, the value """ a , b """ is parsed + // as " a , b ". QuoteEscapeCharacter *string `type:"string"` - // The value used to separate individual records. + // A single character used to separate individual records in the input. Instead + // of the default value, you can specify an arbitrary delimiter. RecordDelimiter *string `type:"string"` } @@ -7897,24 +11039,33 @@ func (s *CSVInput) SetRecordDelimiter(v string) *CSVInput { return s } -// Describes how CSV-formatted results are formatted. +// Describes how uncompressed comma-separated values (CSV)-formatted results +// are formatted. type CSVOutput struct { _ struct{} `type:"structure"` - // The value used to separate individual fields in a record. + // The value used to separate individual fields in a record. You can specify + // an arbitrary delimiter. FieldDelimiter *string `type:"string"` - // The value used for escaping where the field delimiter is part of the value. + // A single character used for escaping when the field delimiter is part of + // the value. For example, if the value is a, b, Amazon S3 wraps this field + // value in quotation marks, as follows: " a , b ". QuoteCharacter *string `type:"string"` - // Th single character used for escaping the quote character inside an already + // The single character used for escaping the quote character inside an already // escaped value. QuoteEscapeCharacter *string `type:"string"` - // Indicates whether or not all output fields should be quoted. + // Indicates whether to use quotation marks around output fields. + // + // * ALWAYS: Always use quotation marks for output fields. + // + // * ASNEEDED: Use quotation marks for output fields when needed. QuoteFields *string `type:"string" enum:"QuoteFields"` - // The value used to separate individual records. + // A single character used to separate individual records in the output. Instead + // of the default value, you can specify an arbitrary delimiter. RecordDelimiter *string `type:"string"` } @@ -7958,9 +11109,12 @@ func (s *CSVOutput) SetRecordDelimiter(v string) *CSVOutput { return s } +// Container for specifying the AWS Lambda notification configuration. type CloudFunctionConfiguration struct { _ struct{} `type:"structure"` + // Lambda cloud function ARN that Amazon S3 can invoke when it detects events + // of the specified type. CloudFunction *string `type:"string"` // The bucket event for which to send notifications. @@ -7968,12 +11122,14 @@ type CloudFunctionConfiguration struct { // Deprecated: Event has been deprecated Event *string `deprecated:"true" type:"string" enum:"Event"` + // Bucket events for which to send notifications. Events []*string `locationName:"Event" type:"list" flattened:"true"` // An optional unique identifier for configurations in a notification configuration. // If you don't provide one, Amazon S3 will assign an ID. Id *string `type:"string"` + // The role supporting the invocation of the Lambda function InvocationRole *string `type:"string"` } @@ -8017,9 +11173,15 @@ func (s *CloudFunctionConfiguration) SetInvocationRole(v string) *CloudFunctionC return s } +// Container for all (if there are any) keys between Prefix and the next occurrence +// of the string specified by a delimiter. CommonPrefixes lists keys that act +// like subdirectories in the directory specified by Prefix. For example, if +// the prefix is notes/ and the delimiter is a slash (/) as in notes/summer/july, +// the common prefix is notes/summer/. type CommonPrefix struct { _ struct{} `type:"structure"` + // Container for the specified common prefix. Prefix *string `type:"string"` } @@ -8040,22 +11202,30 @@ func (s *CommonPrefix) SetPrefix(v string) *CommonPrefix { } type CompleteMultipartUploadInput struct { - _ struct{} `type:"structure" payload:"MultipartUpload"` + _ struct{} `locationName:"CompleteMultipartUploadRequest" type:"structure" payload:"MultipartUpload"` + // Name of the bucket to which the multipart upload was initiated. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Object key for which the multipart upload was initiated. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` + // The container for the multipart upload request information. MultipartUpload *CompletedMultipartUpload `locationName:"CompleteMultipartUpload" type:"structure" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` + // ID for the initiated multipart upload. + // // UploadId is a required field UploadId *string `location:"querystring" locationName:"uploadId" type:"string" required:"true"` } @@ -8132,35 +11302,61 @@ func (s *CompleteMultipartUploadInput) SetUploadId(v string) *CompleteMultipartU return s } +func (s *CompleteMultipartUploadInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *CompleteMultipartUploadInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type CompleteMultipartUploadOutput struct { _ struct{} `type:"structure"` + // The name of the bucket that contains the newly created object. Bucket *string `type:"string"` - // Entity tag of the object. + // Entity tag that identifies the newly created object's data. Objects with + // different object data will have different entity tags. The entity tag is + // an opaque string. The entity tag may or may not be an MD5 digest of the object + // data. If the entity tag is not an MD5 digest of the object data, it will + // contain one or more nonhexadecimal characters and/or will consist of less + // than 32 or more than 32 hexadecimal digits. ETag *string `type:"string"` // If the object expiration is configured, this will contain the expiration // date (expiry-date) and rule ID (rule-id). The value of rule-id is URL encoded. Expiration *string `location:"header" locationName:"x-amz-expiration" type:"string"` + // The object key of the newly created object. Key *string `min:"1" type:"string"` + // The URI that identifies the newly created object. Location *string `type:"string"` // If present, indicates that the requester was successfully charged for the // request. RequestCharged *string `location:"header" locationName:"x-amz-request-charged" type:"string" enum:"RequestCharged"` - // If present, specifies the ID of the AWS Key Management Service (KMS) master - // encryption key that was used for the object. + // If present, specifies the ID of the AWS Key Management Service (AWS KMS) + // symmetric customer managed customer master key (CMK) that was used for the + // object. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // If you specified server-side encryption either with an Amazon S3-managed + // encryption key or an AWS KMS customer master key (CMK) in your initiate multipart + // upload request, the response includes this header. It confirms the encryption + // algorithm that Amazon S3 used to encrypt the object. ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` - // Version of the object. + // Version ID of the newly created object, in case the bucket has versioning + // turned on. VersionId *string `location:"header" locationName:"x-amz-version-id" type:"string"` } @@ -8235,9 +11431,11 @@ func (s *CompleteMultipartUploadOutput) SetVersionId(v string) *CompleteMultipar return s } +// The container for the completed multipart upload details. type CompletedMultipartUpload struct { _ struct{} `type:"structure"` + // Array of CompletedPart data types. Parts []*CompletedPart `locationName:"Part" type:"list" flattened:"true"` } @@ -8257,6 +11455,7 @@ func (s *CompletedMultipartUpload) SetParts(v []*CompletedPart) *CompletedMultip return s } +// Details of the parts that were uploaded. type CompletedPart struct { _ struct{} `type:"structure"` @@ -8290,6 +11489,10 @@ func (s *CompletedPart) SetPartNumber(v int64) *CompletedPart { return s } +// A container for describing a condition that must be met for the specified +// redirect to apply. For example, 1. If request is for pages in the /docs folder, +// redirect to the /documents folder. 2. If request results in HTTP error 4xx, +// redirect request to another host where you might process the error. type Condition struct { _ struct{} `type:"structure"` @@ -8358,12 +11561,21 @@ func (s *ContinuationEvent) UnmarshalEvent( return nil } +// MarshalEvent marshals the type into an stream event value. This method +// should only used internally within the SDK's EventStream handling. +func (s *ContinuationEvent) MarshalEvent(pm protocol.PayloadMarshaler) (msg eventstream.Message, err error) { + msg.Headers.Set(eventstreamapi.MessageTypeHeader, eventstream.StringValue(eventstreamapi.EventMessageType)) + return msg, err +} + type CopyObjectInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"CopyObjectRequest" type:"structure"` // The canned ACL to apply to the object. ACL *string `location:"header" locationName:"x-amz-acl" type:"string" enum:"ObjectCannedACL"` + // The name of the destination bucket. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -8403,17 +11615,18 @@ type CopyObjectInput struct { // Copies the object if it hasn't been modified since the specified time. CopySourceIfUnmodifiedSince *time.Time `location:"header" locationName:"x-amz-copy-source-if-unmodified-since" type:"timestamp"` - // Specifies the algorithm to use when decrypting the source object (e.g., AES256). + // Specifies the algorithm to use when decrypting the source object (for example, + // AES256). CopySourceSSECustomerAlgorithm *string `location:"header" locationName:"x-amz-copy-source-server-side-encryption-customer-algorithm" type:"string"` // Specifies the customer-provided encryption key for Amazon S3 to use to decrypt // the source object. The encryption key provided in this header must be one // that was used when the source object was created. - CopySourceSSECustomerKey *string `location:"header" locationName:"x-amz-copy-source-server-side-encryption-customer-key" type:"string" sensitive:"true"` + CopySourceSSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-copy-source-server-side-encryption-customer-key" type:"string" sensitive:"true"` // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. - // Amazon S3 uses this header for a message integrity check to ensure the encryption - // key was transmitted without error. + // Amazon S3 uses this header for a message integrity check to ensure that the + // encryption key was transmitted without error. CopySourceSSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-copy-source-server-side-encryption-customer-key-MD5" type:"string"` // The date and time at which the object is no longer cacheable. @@ -8431,6 +11644,8 @@ type CopyObjectInput struct { // Allows grantee to write the ACL for the applicable object. GrantWriteACP *string `location:"header" locationName:"x-amz-grant-write-acp" type:"string"` + // The key of the destination object. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` @@ -8450,35 +11665,44 @@ type CopyObjectInput struct { // The date and time when you want the copied object's Object Lock to expire. ObjectLockRetainUntilDate *time.Time `location:"header" locationName:"x-amz-object-lock-retain-until-date" type:"timestamp" timestampFormat:"iso8601"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` - // Specifies the algorithm to use to when encrypting the object (e.g., AES256). + // Specifies the algorithm to use to when encrypting the object (for example, + // AES256). SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // Specifies the customer-provided encryption key for Amazon S3 to use in encrypting // data. This value is used to store the object and then it is discarded; Amazon - // does not store the encryption key. The key must be appropriate for use with - // the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm + // S3 does not store the encryption key. The key must be appropriate for use + // with the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm // header. - SSECustomerKey *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` + SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. - // Amazon S3 uses this header for a message integrity check to ensure the encryption - // key was transmitted without error. + // Amazon S3 uses this header for a message integrity check to ensure that the + // encryption key was transmitted without error. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` + // Specifies the AWS KMS Encryption Context to use for object encryption. The + // value of this header is a base64-encoded UTF-8 string holding JSON with the + // encryption context key-value pairs. + SSEKMSEncryptionContext *string `location:"header" locationName:"x-amz-server-side-encryption-context" type:"string" sensitive:"true"` + // Specifies the AWS KMS key ID to use for object encryption. All GET and PUT // requests for an object protected by AWS KMS will fail if not made via SSL - // or using SigV4. Documentation on configuring any of the officially supported - // AWS SDKs and CLI can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version + // or using SigV4. For information about configuring using any of the officially + // supported AWS SDKs and AWS CLI, see Specifying the Signature Version in Request + // Authentication (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version) + // in the Amazon S3 Developer Guide. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // The server-side encryption algorithm used when storing this object in Amazon + // S3 (for example, AES256, aws:kms). ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` // The type of storage to use for the object. Defaults to 'STANDARD'. @@ -8486,7 +11710,7 @@ type CopyObjectInput struct { // The tag-set for the object destination object this value must be used in // conjunction with the TaggingDirective. The tag-set must be encoded as URL - // Query parameters + // Query parameters. Tagging *string `location:"header" locationName:"x-amz-tagging" type:"string"` // Specifies whether the object tag-set are copied from the source object or @@ -8735,6 +11959,12 @@ func (s *CopyObjectInput) SetSSECustomerKeyMD5(v string) *CopyObjectInput { return s } +// SetSSEKMSEncryptionContext sets the SSEKMSEncryptionContext field's value. +func (s *CopyObjectInput) SetSSEKMSEncryptionContext(v string) *CopyObjectInput { + s.SSEKMSEncryptionContext = &v + return s +} + // SetSSEKMSKeyId sets the SSEKMSKeyId field's value. func (s *CopyObjectInput) SetSSEKMSKeyId(v string) *CopyObjectInput { s.SSEKMSKeyId = &v @@ -8771,11 +12001,27 @@ func (s *CopyObjectInput) SetWebsiteRedirectLocation(v string) *CopyObjectInput return s } +func (s *CopyObjectInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *CopyObjectInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type CopyObjectOutput struct { _ struct{} `type:"structure" payload:"CopyObjectResult"` + // Container for all response elements. CopyObjectResult *CopyObjectResult `type:"structure"` + // Version of the copied object in the destination bucket. CopySourceVersionId *string `location:"header" locationName:"x-amz-copy-source-version-id" type:"string"` // If the object expiration is configured, the response includes this header. @@ -8791,16 +12037,22 @@ type CopyObjectOutput struct { SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // If server-side encryption with a customer-provided encryption key was requested, - // the response will include this header to provide round trip message integrity + // the response will include this header to provide round-trip message integrity // verification of the customer-provided encryption key. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` - // If present, specifies the ID of the AWS Key Management Service (KMS) master - // encryption key that was used for the object. + // If present, specifies the AWS KMS Encryption Context to use for object encryption. + // The value of this header is a base64-encoded UTF-8 string holding JSON with + // the encryption context key-value pairs. + SSEKMSEncryptionContext *string `location:"header" locationName:"x-amz-server-side-encryption-context" type:"string" sensitive:"true"` + + // If present, specifies the ID of the AWS Key Management Service (AWS KMS) + // symmetric customer managed customer master key (CMK) that was used for the + // object. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // The server-side encryption algorithm used when storing this object in Amazon + // S3 (for example, AES256, aws:kms). ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` // Version ID of the newly created copy. @@ -8853,6 +12105,12 @@ func (s *CopyObjectOutput) SetSSECustomerKeyMD5(v string) *CopyObjectOutput { return s } +// SetSSEKMSEncryptionContext sets the SSEKMSEncryptionContext field's value. +func (s *CopyObjectOutput) SetSSEKMSEncryptionContext(v string) *CopyObjectOutput { + s.SSEKMSEncryptionContext = &v + return s +} + // SetSSEKMSKeyId sets the SSEKMSKeyId field's value. func (s *CopyObjectOutput) SetSSEKMSKeyId(v string) *CopyObjectOutput { s.SSEKMSKeyId = &v @@ -8871,11 +12129,16 @@ func (s *CopyObjectOutput) SetVersionId(v string) *CopyObjectOutput { return s } +// Container for all response elements. type CopyObjectResult struct { _ struct{} `type:"structure"` + // Returns the ETag of the new object. The ETag reflects only changes to the + // contents of an object, not its metadata. The source and destination ETag + // is identical for a successfully copied object. ETag *string `type:"string"` + // Returns the date that the object was last modified. LastModified *time.Time `type:"timestamp"` } @@ -8901,6 +12164,7 @@ func (s *CopyObjectResult) SetLastModified(v time.Time) *CopyObjectResult { return s } +// Container for all response elements. type CopyPartResult struct { _ struct{} `type:"structure"` @@ -8933,11 +12197,12 @@ func (s *CopyPartResult) SetLastModified(v time.Time) *CopyPartResult { return s } +// The configuration information for the bucket. type CreateBucketConfiguration struct { _ struct{} `type:"structure"` - // Specifies the region where the bucket will be created. If you don't specify - // a region, the bucket will be created in US Standard. + // Specifies the Region where the bucket will be created. If you don't specify + // a Region, the bucket is created in the US East (N. Virginia) Region (us-east-1). LocationConstraint *string `type:"string" enum:"BucketLocationConstraint"` } @@ -8958,14 +12223,17 @@ func (s *CreateBucketConfiguration) SetLocationConstraint(v string) *CreateBucke } type CreateBucketInput struct { - _ struct{} `type:"structure" payload:"CreateBucketConfiguration"` + _ struct{} `locationName:"CreateBucketRequest" type:"structure" payload:"CreateBucketConfiguration"` // The canned ACL to apply to the bucket. ACL *string `location:"header" locationName:"x-amz-acl" type:"string" enum:"BucketCannedACL"` + // The name of the bucket to create. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // The configuration information for the bucket. CreateBucketConfiguration *CreateBucketConfiguration `locationName:"CreateBucketConfiguration" type:"structure" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` // Allows grantee the read, write, read ACP, and write ACP permissions on the @@ -9078,6 +12346,9 @@ func (s *CreateBucketInput) SetObjectLockEnabledForBucket(v bool) *CreateBucketI type CreateBucketOutput struct { _ struct{} `type:"structure"` + // Specifies the Region where the bucket will be created. If you are creating + // a bucket on the US East (N. Virginia) Region (us-east-1), you do not need + // to specify the location. Location *string `location:"header" locationName:"Location" type:"string"` } @@ -9098,11 +12369,13 @@ func (s *CreateBucketOutput) SetLocation(v string) *CreateBucketOutput { } type CreateMultipartUploadInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"CreateMultipartUploadRequest" type:"structure"` // The canned ACL to apply to the object. ACL *string `location:"header" locationName:"x-amz-acl" type:"string" enum:"ObjectCannedACL"` + // The name of the bucket to which to initiate the upload + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -9138,6 +12411,8 @@ type CreateMultipartUploadInput struct { // Allows grantee to write the ACL for the applicable object. GrantWriteACP *string `location:"header" locationName:"x-amz-grant-write-acp" type:"string"` + // Object key for which the multipart upload is to be initiated. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` @@ -9153,41 +12428,50 @@ type CreateMultipartUploadInput struct { // Specifies the date and time when you want the Object Lock to expire. ObjectLockRetainUntilDate *time.Time `location:"header" locationName:"x-amz-object-lock-retain-until-date" type:"timestamp" timestampFormat:"iso8601"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` - // Specifies the algorithm to use to when encrypting the object (e.g., AES256). + // Specifies the algorithm to use to when encrypting the object (for example, + // AES256). SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // Specifies the customer-provided encryption key for Amazon S3 to use in encrypting // data. This value is used to store the object and then it is discarded; Amazon - // does not store the encryption key. The key must be appropriate for use with - // the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm + // S3 does not store the encryption key. The key must be appropriate for use + // with the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm // header. - SSECustomerKey *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` + SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. - // Amazon S3 uses this header for a message integrity check to ensure the encryption - // key was transmitted without error. + // Amazon S3 uses this header for a message integrity check to ensure that the + // encryption key was transmitted without error. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` - // Specifies the AWS KMS key ID to use for object encryption. All GET and PUT - // requests for an object protected by AWS KMS will fail if not made via SSL - // or using SigV4. Documentation on configuring any of the officially supported - // AWS SDKs and CLI can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version + // Specifies the AWS KMS Encryption Context to use for object encryption. The + // value of this header is a base64-encoded UTF-8 string holding JSON with the + // encryption context key-value pairs. + SSEKMSEncryptionContext *string `location:"header" locationName:"x-amz-server-side-encryption-context" type:"string" sensitive:"true"` + + // Specifies the ID of the symmetric customer managed AWS KMS CMK to use for + // object encryption. All GET and PUT requests for an object protected by AWS + // KMS will fail if not made via SSL or using SigV4. For information about configuring + // using any of the officially supported AWS SDKs and AWS CLI, see Specifying + // the Signature Version in Request Authentication (https://docs.aws.amazon.com/http:/docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version) + // in the Amazon S3 Developer Guide. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // The server-side encryption algorithm used when storing this object in Amazon + // S3 (for example, AES256, aws:kms). ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` // The type of storage to use for the object. Defaults to 'STANDARD'. StorageClass *string `location:"header" locationName:"x-amz-storage-class" type:"string" enum:"StorageClass"` - // The tag-set for the object. The tag-set must be encoded as URL Query parameters + // The tag-set for the object. The tag-set must be encoded as URL Query parameters. Tagging *string `location:"header" locationName:"x-amz-tagging" type:"string"` // If the bucket is configured as a website, redirects requests for this object @@ -9368,8 +12652,14 @@ func (s *CreateMultipartUploadInput) SetSSECustomerKeyMD5(v string) *CreateMulti return s } -// SetSSEKMSKeyId sets the SSEKMSKeyId field's value. -func (s *CreateMultipartUploadInput) SetSSEKMSKeyId(v string) *CreateMultipartUploadInput { +// SetSSEKMSEncryptionContext sets the SSEKMSEncryptionContext field's value. +func (s *CreateMultipartUploadInput) SetSSEKMSEncryptionContext(v string) *CreateMultipartUploadInput { + s.SSEKMSEncryptionContext = &v + return s +} + +// SetSSEKMSKeyId sets the SSEKMSKeyId field's value. +func (s *CreateMultipartUploadInput) SetSSEKMSKeyId(v string) *CreateMultipartUploadInput { s.SSEKMSKeyId = &v return s } @@ -9398,17 +12688,47 @@ func (s *CreateMultipartUploadInput) SetWebsiteRedirectLocation(v string) *Creat return s } +func (s *CreateMultipartUploadInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *CreateMultipartUploadInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type CreateMultipartUploadOutput struct { _ struct{} `type:"structure"` - // Date when multipart upload will become eligible for abort operation by lifecycle. + // If the bucket has a lifecycle rule configured with an action to abort incomplete + // multipart uploads and the prefix in the lifecycle rule matches the object + // name in the request, the response includes this header. The header indicates + // when the initiated multipart upload becomes eligible for an abort operation. + // For more information, see Aborting Incomplete Multipart Uploads Using a Bucket + // Lifecycle Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-abort-incomplete-mpu-lifecycle-config). + // + // The response also includes the x-amz-abort-rule-id header that provides the + // ID of the lifecycle configuration rule that defines this action. AbortDate *time.Time `location:"header" locationName:"x-amz-abort-date" type:"timestamp"` - // Id of the lifecycle rule that makes a multipart upload eligible for abort - // operation. + // This header is returned along with the x-amz-abort-date header. It identifies + // the applicable lifecycle configuration rule that defines the action to abort + // incomplete multipart uploads. AbortRuleId *string `location:"header" locationName:"x-amz-abort-rule-id" type:"string"` // Name of the bucket to which the multipart upload was initiated. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. Bucket *string `locationName:"Bucket" type:"string"` // Object key for which the multipart upload was initiated. @@ -9424,16 +12744,22 @@ type CreateMultipartUploadOutput struct { SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // If server-side encryption with a customer-provided encryption key was requested, - // the response will include this header to provide round trip message integrity + // the response will include this header to provide round-trip message integrity // verification of the customer-provided encryption key. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` - // If present, specifies the ID of the AWS Key Management Service (KMS) master - // encryption key that was used for the object. + // If present, specifies the AWS KMS Encryption Context to use for object encryption. + // The value of this header is a base64-encoded UTF-8 string holding JSON with + // the encryption context key-value pairs. + SSEKMSEncryptionContext *string `location:"header" locationName:"x-amz-server-side-encryption-context" type:"string" sensitive:"true"` + + // If present, specifies the ID of the AWS Key Management Service (AWS KMS) + // symmetric customer managed customer master key (CMK) that was used for the + // object. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // The server-side encryption algorithm used when storing this object in Amazon + // S3 (for example, AES256, aws:kms). ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` // ID for the initiated multipart upload. @@ -9499,6 +12825,12 @@ func (s *CreateMultipartUploadOutput) SetSSECustomerKeyMD5(v string) *CreateMult return s } +// SetSSEKMSEncryptionContext sets the SSEKMSEncryptionContext field's value. +func (s *CreateMultipartUploadOutput) SetSSEKMSEncryptionContext(v string) *CreateMultipartUploadOutput { + s.SSEKMSEncryptionContext = &v + return s +} + // SetSSEKMSKeyId sets the SSEKMSKeyId field's value. func (s *CreateMultipartUploadOutput) SetSSEKMSKeyId(v string) *CreateMultipartUploadOutput { s.SSEKMSKeyId = &v @@ -9561,9 +12893,12 @@ func (s *DefaultRetention) SetYears(v int64) *DefaultRetention { return s } +// Container for the objects to delete. type Delete struct { _ struct{} `type:"structure"` + // The objects to delete. + // // Objects is a required field Objects []*ObjectIdentifier `locationName:"Object" type:"list" flattened:"true" required:"true"` @@ -9618,14 +12953,14 @@ func (s *Delete) SetQuiet(v bool) *Delete { } type DeleteBucketAnalyticsConfigurationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketAnalyticsConfigurationRequest" type:"structure"` // The name of the bucket from which an analytics configuration is deleted. // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` - // The identifier used to represent an analytics configuration. + // The ID that identifies the analytics configuration. // // Id is a required field Id *string `location:"querystring" locationName:"id" type:"string" required:"true"` @@ -9679,6 +13014,20 @@ func (s *DeleteBucketAnalyticsConfigurationInput) SetId(v string) *DeleteBucketA return s } +func (s *DeleteBucketAnalyticsConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketAnalyticsConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketAnalyticsConfigurationOutput struct { _ struct{} `type:"structure"` } @@ -9694,8 +13043,10 @@ func (s DeleteBucketAnalyticsConfigurationOutput) GoString() string { } type DeleteBucketCorsInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketCorsRequest" type:"structure"` + // Specifies the bucket whose cors configuration is being deleted. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -9739,6 +13090,20 @@ func (s *DeleteBucketCorsInput) getBucket() (v string) { return *s.Bucket } +func (s *DeleteBucketCorsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketCorsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketCorsOutput struct { _ struct{} `type:"structure"` } @@ -9754,7 +13119,7 @@ func (s DeleteBucketCorsOutput) GoString() string { } type DeleteBucketEncryptionInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketEncryptionRequest" type:"structure"` // The name of the bucket containing the server-side encryption configuration // to delete. @@ -9802,6 +13167,20 @@ func (s *DeleteBucketEncryptionInput) getBucket() (v string) { return *s.Bucket } +func (s *DeleteBucketEncryptionInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketEncryptionInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketEncryptionOutput struct { _ struct{} `type:"structure"` } @@ -9817,8 +13196,10 @@ func (s DeleteBucketEncryptionOutput) GoString() string { } type DeleteBucketInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketRequest" type:"structure"` + // Specifies the bucket being deleted. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -9862,8 +13243,22 @@ func (s *DeleteBucketInput) getBucket() (v string) { return *s.Bucket } +func (s *DeleteBucketInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketInventoryConfigurationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketInventoryConfigurationRequest" type:"structure"` // The name of the bucket containing the inventory configuration to delete. // @@ -9924,6 +13319,20 @@ func (s *DeleteBucketInventoryConfigurationInput) SetId(v string) *DeleteBucketI return s } +func (s *DeleteBucketInventoryConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketInventoryConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketInventoryConfigurationOutput struct { _ struct{} `type:"structure"` } @@ -9939,8 +13348,10 @@ func (s DeleteBucketInventoryConfigurationOutput) GoString() string { } type DeleteBucketLifecycleInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketLifecycleRequest" type:"structure"` + // The bucket name of the lifecycle to delete. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -9984,6 +13395,20 @@ func (s *DeleteBucketLifecycleInput) getBucket() (v string) { return *s.Bucket } +func (s *DeleteBucketLifecycleInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketLifecycleInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketLifecycleOutput struct { _ struct{} `type:"structure"` } @@ -9999,7 +13424,7 @@ func (s DeleteBucketLifecycleOutput) GoString() string { } type DeleteBucketMetricsConfigurationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketMetricsConfigurationRequest" type:"structure"` // The name of the bucket containing the metrics configuration to delete. // @@ -10060,6 +13485,20 @@ func (s *DeleteBucketMetricsConfigurationInput) SetId(v string) *DeleteBucketMet return s } +func (s *DeleteBucketMetricsConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketMetricsConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketMetricsConfigurationOutput struct { _ struct{} `type:"structure"` } @@ -10089,8 +13528,10 @@ func (s DeleteBucketOutput) GoString() string { } type DeleteBucketPolicyInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketPolicyRequest" type:"structure"` + // The bucket name. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -10134,6 +13575,20 @@ func (s *DeleteBucketPolicyInput) getBucket() (v string) { return *s.Bucket } +func (s *DeleteBucketPolicyInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketPolicyInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketPolicyOutput struct { _ struct{} `type:"structure"` } @@ -10149,13 +13604,10 @@ func (s DeleteBucketPolicyOutput) GoString() string { } type DeleteBucketReplicationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketReplicationRequest" type:"structure"` // The bucket name. // - // It can take a while to propagate the deletion of a replication configuration - // to all Amazon S3 systems. - // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -10199,6 +13651,20 @@ func (s *DeleteBucketReplicationInput) getBucket() (v string) { return *s.Bucket } +func (s *DeleteBucketReplicationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketReplicationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketReplicationOutput struct { _ struct{} `type:"structure"` } @@ -10214,8 +13680,10 @@ func (s DeleteBucketReplicationOutput) GoString() string { } type DeleteBucketTaggingInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketTaggingRequest" type:"structure"` + // The bucket that has the tag set to be removed. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -10259,6 +13727,20 @@ func (s *DeleteBucketTaggingInput) getBucket() (v string) { return *s.Bucket } +func (s *DeleteBucketTaggingInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketTaggingInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketTaggingOutput struct { _ struct{} `type:"structure"` } @@ -10274,8 +13756,10 @@ func (s DeleteBucketTaggingOutput) GoString() string { } type DeleteBucketWebsiteInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteBucketWebsiteRequest" type:"structure"` + // The bucket name for which you want to remove the website configuration. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -10319,6 +13803,20 @@ func (s *DeleteBucketWebsiteInput) getBucket() (v string) { return *s.Bucket } +func (s *DeleteBucketWebsiteInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteBucketWebsiteInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteBucketWebsiteOutput struct { _ struct{} `type:"structure"` } @@ -10333,6 +13831,7 @@ func (s DeleteBucketWebsiteOutput) GoString() string { return s.String() } +// Information about the delete marker. type DeleteMarkerEntry struct { _ struct{} `type:"structure"` @@ -10346,6 +13845,7 @@ type DeleteMarkerEntry struct { // Date and time the object was last modified. LastModified *time.Time `type:"timestamp"` + // The account that created the delete marker.> Owner *Owner `type:"structure"` // Version ID of an object. @@ -10392,11 +13892,21 @@ func (s *DeleteMarkerEntry) SetVersionId(v string) *DeleteMarkerEntry { return s } -// Specifies whether Amazon S3 should replicate delete makers. +// Specifies whether Amazon S3 replicates the delete markers. If you specify +// a Filter, you must specify this element. However, in the latest version of +// replication configuration (when Filter is specified), Amazon S3 doesn't replicate +// delete markers. Therefore, the DeleteMarkerReplication element can contain +// only Disabled. For an example configuration, see Basic Rule +// Configuration (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html#replication-config-min-rule-config). +// +// If you don't specify the Filter element, Amazon S3 assumes that the replication +// configuration is the earlier version, V1. In the earlier version, Amazon +// S3 handled replication of delete markers differently. For more information, +// see Backward Compatibility (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html#replication-backward-compat-considerations). type DeleteMarkerReplication struct { _ struct{} `type:"structure"` - // The status of the delete marker replication. + // Indicates whether to replicate delete markers. // // In the current implementation, Amazon S3 doesn't replicate the delete markers. // The status must be Disabled. @@ -10420,8 +13930,17 @@ func (s *DeleteMarkerReplication) SetStatus(v string) *DeleteMarkerReplication { } type DeleteObjectInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteObjectRequest" type:"structure"` + // The bucket name of the bucket containing the object. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -10429,17 +13948,22 @@ type DeleteObjectInput struct { // to process this operation. BypassGovernanceRetention *bool `location:"header" locationName:"x-amz-bypass-governance-retention" type:"boolean"` + // Key name of the object to delete. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` // The concatenation of the authentication device's serial number, a space, - // and the value that is displayed on your authentication device. + // and the value that is displayed on your authentication device. Required to + // permanently delete a versioned object if versioning is configured with MFA + // delete enabled. MFA *string `location:"header" locationName:"x-amz-mfa" type:"string"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // VersionId used to reference a specific version of the object. @@ -10521,6 +14045,20 @@ func (s *DeleteObjectInput) SetVersionId(v string) *DeleteObjectInput { return s } +func (s *DeleteObjectInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteObjectInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteObjectOutput struct { _ struct{} `type:"structure"` @@ -10566,11 +14104,22 @@ func (s *DeleteObjectOutput) SetVersionId(v string) *DeleteObjectOutput { } type DeleteObjectTaggingInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeleteObjectTaggingRequest" type:"structure"` + // The bucket name containing the objects from which to remove the tags. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Name of the tag. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` @@ -10635,6 +14184,20 @@ func (s *DeleteObjectTaggingInput) SetVersionId(v string) *DeleteObjectTaggingIn return s } +func (s *DeleteObjectTaggingInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteObjectTaggingInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteObjectTaggingOutput struct { _ struct{} `type:"structure"` @@ -10659,8 +14222,17 @@ func (s *DeleteObjectTaggingOutput) SetVersionId(v string) *DeleteObjectTaggingO } type DeleteObjectsInput struct { - _ struct{} `type:"structure" payload:"Delete"` + _ struct{} `locationName:"DeleteObjectsRequest" type:"structure" payload:"Delete"` + // The bucket name containing the objects to delete. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -10669,17 +14241,22 @@ type DeleteObjectsInput struct { // operation. BypassGovernanceRetention *bool `location:"header" locationName:"x-amz-bypass-governance-retention" type:"boolean"` + // Container for the request. + // // Delete is a required field Delete *Delete `locationName:"Delete" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` // The concatenation of the authentication device's serial number, a space, - // and the value that is displayed on your authentication device. + // and the value that is displayed on your authentication device. Required to + // permanently delete a versioned object if versioning is configured with MFA + // delete enabled. MFA *string `location:"header" locationName:"x-amz-mfa" type:"string"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` } @@ -10754,11 +14331,29 @@ func (s *DeleteObjectsInput) SetRequestPayer(v string) *DeleteObjectsInput { return s } +func (s *DeleteObjectsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeleteObjectsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeleteObjectsOutput struct { _ struct{} `type:"structure"` + // Container element for a successful delete. It identifies the object that + // was successfully deleted. Deleted []*DeletedObject `type:"list" flattened:"true"` + // Container for a failed delete operation that describes the object that Amazon + // S3 attempted to delete and the error it encountered. Errors []*Error `locationName:"Error" type:"list" flattened:"true"` // If present, indicates that the requester was successfully charged for the @@ -10795,7 +14390,7 @@ func (s *DeleteObjectsOutput) SetRequestCharged(v string) *DeleteObjectsOutput { } type DeletePublicAccessBlockInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"DeletePublicAccessBlockRequest" type:"structure"` // The Amazon S3 bucket whose PublicAccessBlock configuration you want to delete. // @@ -10842,6 +14437,20 @@ func (s *DeletePublicAccessBlockInput) getBucket() (v string) { return *s.Bucket } +func (s *DeletePublicAccessBlockInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *DeletePublicAccessBlockInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type DeletePublicAccessBlockOutput struct { _ struct{} `type:"structure"` } @@ -10856,15 +14465,24 @@ func (s DeletePublicAccessBlockOutput) GoString() string { return s.String() } +// Information about the deleted object. type DeletedObject struct { _ struct{} `type:"structure"` + // Specifies whether the versioned object that was permanently deleted was (true) + // or was not (false) a delete marker. In a simple DELETE, this header indicates + // whether (true) or not (false) a delete marker was created. DeleteMarker *bool `type:"boolean"` + // The version ID of the delete marker created as a result of the DELETE operation. + // If you delete a specific object version, the value returned by this header + // is the version ID of the object version deleted. DeleteMarkerVersionId *string `type:"string"` + // The name of the deleted object. Key *string `min:"1" type:"string"` + // The version ID of the deleted object. VersionId *string `type:"string"` } @@ -10902,33 +14520,28 @@ func (s *DeletedObject) SetVersionId(v string) *DeletedObject { return s } -// A container for information about the replication destination. +// Specifies information about where to publish analysis or configuration results +// for an Amazon S3 bucket and S3 Replication Time Control (S3 RTC). type Destination struct { _ struct{} `type:"structure"` - // A container for information about access control for replicas. - // - // Use this element only in a cross-account scenario where source and destination - // bucket owners are not the same to change replica ownership to the AWS account - // that owns the destination bucket. If you don't add this element to the replication - // configuration, the replicas are owned by same AWS account that owns the source - // object. + // Specify this only in a cross-account scenario (where source and destination + // bucket owners are not the same), and you want to change replica ownership + // to the AWS account that owns the destination bucket. If this is not specified + // in the replication configuration, the replicas are owned by same AWS account + // that owns the source object. AccessControlTranslation *AccessControlTranslation `type:"structure"` - // The account ID of the destination bucket. Currently, Amazon S3 verifies this - // value only if Access Control Translation is enabled. - // - // In a cross-account scenario, if you change replica ownership to the AWS account - // that owns the destination bucket by adding the AccessControlTranslation element, - // this is the account ID of the owner of the destination bucket. + // Destination bucket owner account ID. In a cross-account scenario, if you + // direct Amazon S3 to change replica ownership to the AWS account that owns + // the destination bucket by specifying the AccessControlTranslation property, + // this is the account ID of the destination bucket owner. For more information, + // see Replication Additional Configuration: Changing the Replica Owner (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-change-owner.html) + // in the Amazon Simple Storage Service Developer Guide. Account *string `type:"string"` // The Amazon Resource Name (ARN) of the bucket where you want Amazon S3 to - // store replicas of the object identified by the rule. - // - // If there are multiple rules in your replication configuration, all rules - // must specify the same bucket as the destination. A replication configuration - // can replicate objects to only one destination bucket. + // store the results. // // Bucket is a required field Bucket *string `type:"string" required:"true"` @@ -10937,8 +14550,23 @@ type Destination struct { // is specified, you must specify this element. EncryptionConfiguration *EncryptionConfiguration `type:"structure"` - // The class of storage used to store the object. By default Amazon S3 uses - // storage class of the source object when creating a replica. + // A container specifying replication metrics-related settings enabling metrics + // and Amazon S3 events for S3 Replication Time Control (S3 RTC). Must be specified + // together with a ReplicationTime block. + Metrics *Metrics `type:"structure"` + + // A container specifying S3 Replication Time Control (S3 RTC), including whether + // S3 RTC is enabled and the time when all objects and operations on objects + // must be replicated. Must be specified together with a Metrics block. + ReplicationTime *ReplicationTime `type:"structure"` + + // The storage class to use when replicating objects, such as S3 Standard or + // reduced redundancy. By default, Amazon S3 uses the storage class of the source + // object to create the object replica. + // + // For valid values, see the StorageClass element of the PUT Bucket replication + // (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTreplication.html) + // action in the Amazon Simple Storage Service API Reference. StorageClass *string `type:"string" enum:"StorageClass"` } @@ -10963,6 +14591,16 @@ func (s *Destination) Validate() error { invalidParams.AddNested("AccessControlTranslation", err.(request.ErrInvalidParams)) } } + if s.Metrics != nil { + if err := s.Metrics.Validate(); err != nil { + invalidParams.AddNested("Metrics", err.(request.ErrInvalidParams)) + } + } + if s.ReplicationTime != nil { + if err := s.ReplicationTime.Validate(); err != nil { + invalidParams.AddNested("ReplicationTime", err.(request.ErrInvalidParams)) + } + } if invalidParams.Len() > 0 { return invalidParams @@ -11001,19 +14639,30 @@ func (s *Destination) SetEncryptionConfiguration(v *EncryptionConfiguration) *De return s } +// SetMetrics sets the Metrics field's value. +func (s *Destination) SetMetrics(v *Metrics) *Destination { + s.Metrics = v + return s +} + +// SetReplicationTime sets the ReplicationTime field's value. +func (s *Destination) SetReplicationTime(v *ReplicationTime) *Destination { + s.ReplicationTime = v + return s +} + // SetStorageClass sets the StorageClass field's value. func (s *Destination) SetStorageClass(v string) *Destination { s.StorageClass = &v return s } -// Describes the server-side encryption that will be applied to the restore -// results. +// Contains the type of server-side encryption used. type Encryption struct { _ struct{} `type:"structure"` // The server-side encryption algorithm used when storing job results in Amazon - // S3 (e.g., AES256, aws:kms). + // S3 (for example, AES256, aws:kms). // // EncryptionType is a required field EncryptionType *string `type:"string" required:"true" enum:"ServerSideEncryption"` @@ -11022,8 +14671,11 @@ type Encryption struct { // the encryption context for the restore results. KMSContext *string `type:"string"` - // If the encryption type is aws:kms, this optional value specifies the AWS - // KMS key ID to use for encryption of job results. + // If the encryption type is aws:kms, this optional value specifies the ID of + // the symmetric customer managed AWS KMS CMK to use for encryption of job results. + // Amazon S3 only supports symmetric CMKs. For more information, see Using Symmetric + // and Asymmetric Keys (https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html) + // in the AWS Key Management Service Developer Guide. KMSKeyId *string `type:"string" sensitive:"true"` } @@ -11068,13 +14720,17 @@ func (s *Encryption) SetKMSKeyId(v string) *Encryption { return s } -// A container for information about the encryption-based configuration for -// replicas. +// Specifies encryption-related information for an Amazon S3 bucket that is +// a destination for replicated objects. type EncryptionConfiguration struct { _ struct{} `type:"structure"` - // The ID of the AWS KMS key for the AWS Region where the destination bucket - // resides. Amazon S3 uses this key to encrypt the replica object. + // Specifies the ID (Key ARN or Alias ARN) of the customer managed customer + // master key (CMK) stored in AWS Key Management Service (KMS) for the destination + // bucket. Amazon S3 uses this key to encrypt replica objects. Amazon S3 only + // supports symmetric customer managed CMKs. For more information, see Using + // Symmetric and Asymmetric Keys (https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html) + // in the AWS Key Management Service Developer Guide. ReplicaKmsKeyID *string `type:"string"` } @@ -11094,6 +14750,9 @@ func (s *EncryptionConfiguration) SetReplicaKmsKeyID(v string) *EncryptionConfig return s } +// A message that indicates the request is complete and no more messages will +// be sent. You should not assume that the request is complete until the client +// receives an EndEvent. type EndEvent struct { _ struct{} `locationName:"EndEvent" type:"structure"` } @@ -11120,15 +14779,382 @@ func (s *EndEvent) UnmarshalEvent( return nil } +// MarshalEvent marshals the type into an stream event value. This method +// should only used internally within the SDK's EventStream handling. +func (s *EndEvent) MarshalEvent(pm protocol.PayloadMarshaler) (msg eventstream.Message, err error) { + msg.Headers.Set(eventstreamapi.MessageTypeHeader, eventstream.StringValue(eventstreamapi.EventMessageType)) + return msg, err +} + +// Container for all error elements. type Error struct { _ struct{} `type:"structure"` + // The error code is a string that uniquely identifies an error condition. It + // is meant to be read and understood by programs that detect and handle errors + // by type. + // + // Amazon S3 error codes + // + // * Code: AccessDenied Description: Access Denied HTTP Status Code: 403 + // Forbidden SOAP Fault Code Prefix: Client + // + // * Code: AccountProblem Description: There is a problem with your AWS account + // that prevents the operation from completing successfully. Contact AWS + // Support for further assistance. HTTP Status Code: 403 Forbidden SOAP Fault + // Code Prefix: Client + // + // * Code: AllAccessDisabled Description: All access to this Amazon S3 resource + // has been disabled. Contact AWS Support for further assistance. HTTP Status + // Code: 403 Forbidden SOAP Fault Code Prefix: Client + // + // * Code: AmbiguousGrantByEmailAddress Description: The email address you + // provided is associated with more than one account. HTTP Status Code: 400 + // Bad Request SOAP Fault Code Prefix: Client + // + // * Code: AuthorizationHeaderMalformed Description: The authorization header + // you provided is invalid. HTTP Status Code: 400 Bad Request HTTP Status + // Code: N/A + // + // * Code: BadDigest Description: The Content-MD5 you specified did not match + // what we received. HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: + // Client + // + // * Code: BucketAlreadyExists Description: The requested bucket name is + // not available. The bucket namespace is shared by all users of the system. + // Please select a different name and try again. HTTP Status Code: 409 Conflict + // SOAP Fault Code Prefix: Client + // + // * Code: BucketAlreadyOwnedByYou Description: The bucket you tried to create + // already exists, and you own it. Amazon S3 returns this error in all AWS + // Regions except in the North Virginia Region. For legacy compatibility, + // if you re-create an existing bucket that you already own in the North + // Virginia Region, Amazon S3 returns 200 OK and resets the bucket access + // control lists (ACLs). Code: 409 Conflict (in all Regions except the North + // Virginia Region) SOAP Fault Code Prefix: Client + // + // * Code: BucketNotEmpty Description: The bucket you tried to delete is + // not empty. HTTP Status Code: 409 Conflict SOAP Fault Code Prefix: Client + // + // * Code: CredentialsNotSupported Description: This request does not support + // credentials. HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: + // Client + // + // * Code: CrossLocationLoggingProhibited Description: Cross-location logging + // not allowed. Buckets in one geographic location cannot log information + // to a bucket in another location. HTTP Status Code: 403 Forbidden SOAP + // Fault Code Prefix: Client + // + // * Code: EntityTooSmall Description: Your proposed upload is smaller than + // the minimum allowed object size. HTTP Status Code: 400 Bad Request SOAP + // Fault Code Prefix: Client + // + // * Code: EntityTooLarge Description: Your proposed upload exceeds the maximum + // allowed object size. HTTP Status Code: 400 Bad Request SOAP Fault Code + // Prefix: Client + // + // * Code: ExpiredToken Description: The provided token has expired. HTTP + // Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: IllegalVersioningConfigurationException Description: Indicates + // that the versioning configuration specified in the request is invalid. + // HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: IncompleteBody Description: You did not provide the number of + // bytes specified by the Content-Length HTTP header HTTP Status Code: 400 + // Bad Request SOAP Fault Code Prefix: Client + // + // * Code: IncorrectNumberOfFilesInPostRequest Description: POST requires + // exactly one file upload per request. HTTP Status Code: 400 Bad Request + // SOAP Fault Code Prefix: Client + // + // * Code: InlineDataTooLarge Description: Inline data exceeds the maximum + // allowed size. HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: + // Client + // + // * Code: InternalError Description: We encountered an internal error. Please + // try again. HTTP Status Code: 500 Internal Server Error SOAP Fault Code + // Prefix: Server + // + // * Code: InvalidAccessKeyId Description: The AWS access key ID you provided + // does not exist in our records. HTTP Status Code: 403 Forbidden SOAP Fault + // Code Prefix: Client + // + // * Code: InvalidAddressingHeader Description: You must specify the Anonymous + // role. HTTP Status Code: N/A SOAP Fault Code Prefix: Client + // + // * Code: InvalidArgument Description: Invalid Argument HTTP Status Code: + // 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: InvalidBucketName Description: The specified bucket is not valid. + // HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: InvalidBucketState Description: The request is not valid with + // the current state of the bucket. HTTP Status Code: 409 Conflict SOAP Fault + // Code Prefix: Client + // + // * Code: InvalidDigest Description: The Content-MD5 you specified is not + // valid. HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: InvalidEncryptionAlgorithmError Description: The encryption request + // you specified is not valid. The valid value is AES256. HTTP Status Code: + // 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: InvalidLocationConstraint Description: The specified location + // constraint is not valid. For more information about Regions, see How to + // Select a Region for Your Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro). + // HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: InvalidObjectState Description: The operation is not valid for + // the current state of the object. HTTP Status Code: 403 Forbidden SOAP + // Fault Code Prefix: Client + // + // * Code: InvalidPart Description: One or more of the specified parts could + // not be found. The part might not have been uploaded, or the specified + // entity tag might not have matched the part's entity tag. HTTP Status Code: + // 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: InvalidPartOrder Description: The list of parts was not in ascending + // order. Parts list must be specified in order by part number. HTTP Status + // Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: InvalidPayer Description: All access to this object has been disabled. + // Please contact AWS Support for further assistance. HTTP Status Code: 403 + // Forbidden SOAP Fault Code Prefix: Client + // + // * Code: InvalidPolicyDocument Description: The content of the form does + // not meet the conditions specified in the policy document. HTTP Status + // Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: InvalidRange Description: The requested range cannot be satisfied. + // HTTP Status Code: 416 Requested Range Not Satisfiable SOAP Fault Code + // Prefix: Client + // + // * Code: InvalidRequest Description: Please use AWS4-HMAC-SHA256. HTTP + // Status Code: 400 Bad Request Code: N/A + // + // * Code: InvalidRequest Description: SOAP requests must be made over an + // HTTPS connection. HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: + // Client + // + // * Code: InvalidRequest Description: Amazon S3 Transfer Acceleration is + // not supported for buckets with non-DNS compliant names. HTTP Status Code: + // 400 Bad Request Code: N/A + // + // * Code: InvalidRequest Description: Amazon S3 Transfer Acceleration is + // not supported for buckets with periods (.) in their names. HTTP Status + // Code: 400 Bad Request Code: N/A + // + // * Code: InvalidRequest Description: Amazon S3 Transfer Accelerate endpoint + // only supports virtual style requests. HTTP Status Code: 400 Bad Request + // Code: N/A + // + // * Code: InvalidRequest Description: Amazon S3 Transfer Accelerate is not + // configured on this bucket. HTTP Status Code: 400 Bad Request Code: N/A + // + // * Code: InvalidRequest Description: Amazon S3 Transfer Accelerate is disabled + // on this bucket. HTTP Status Code: 400 Bad Request Code: N/A + // + // * Code: InvalidRequest Description: Amazon S3 Transfer Acceleration is + // not supported on this bucket. Contact AWS Support for more information. + // HTTP Status Code: 400 Bad Request Code: N/A + // + // * Code: InvalidRequest Description: Amazon S3 Transfer Acceleration cannot + // be enabled on this bucket. Contact AWS Support for more information. HTTP + // Status Code: 400 Bad Request Code: N/A + // + // * Code: InvalidSecurity Description: The provided security credentials + // are not valid. HTTP Status Code: 403 Forbidden SOAP Fault Code Prefix: + // Client + // + // * Code: InvalidSOAPRequest Description: The SOAP request body is invalid. + // HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: InvalidStorageClass Description: The storage class you specified + // is not valid. HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: + // Client + // + // * Code: InvalidTargetBucketForLogging Description: The target bucket for + // logging does not exist, is not owned by you, or does not have the appropriate + // grants for the log-delivery group. HTTP Status Code: 400 Bad Request SOAP + // Fault Code Prefix: Client + // + // * Code: InvalidToken Description: The provided token is malformed or otherwise + // invalid. HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: InvalidURI Description: Couldn't parse the specified URI. HTTP + // Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: KeyTooLongError Description: Your key is too long. HTTP Status + // Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: MalformedACLError Description: The XML you provided was not well-formed + // or did not validate against our published schema. HTTP Status Code: 400 + // Bad Request SOAP Fault Code Prefix: Client + // + // * Code: MalformedPOSTRequest Description: The body of your POST request + // is not well-formed multipart/form-data. HTTP Status Code: 400 Bad Request + // SOAP Fault Code Prefix: Client + // + // * Code: MalformedXML Description: This happens when the user sends malformed + // XML (XML that doesn't conform to the published XSD) for the configuration. + // The error message is, "The XML you provided was not well-formed or did + // not validate against our published schema." HTTP Status Code: 400 Bad + // Request SOAP Fault Code Prefix: Client + // + // * Code: MaxMessageLengthExceeded Description: Your request was too big. + // HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: MaxPostPreDataLengthExceededError Description: Your POST request + // fields preceding the upload file were too large. HTTP Status Code: 400 + // Bad Request SOAP Fault Code Prefix: Client + // + // * Code: MetadataTooLarge Description: Your metadata headers exceed the + // maximum allowed metadata size. HTTP Status Code: 400 Bad Request SOAP + // Fault Code Prefix: Client + // + // * Code: MethodNotAllowed Description: The specified method is not allowed + // against this resource. HTTP Status Code: 405 Method Not Allowed SOAP Fault + // Code Prefix: Client + // + // * Code: MissingAttachment Description: A SOAP attachment was expected, + // but none were found. HTTP Status Code: N/A SOAP Fault Code Prefix: Client + // + // * Code: MissingContentLength Description: You must provide the Content-Length + // HTTP header. HTTP Status Code: 411 Length Required SOAP Fault Code Prefix: + // Client + // + // * Code: MissingRequestBodyError Description: This happens when the user + // sends an empty XML document as a request. The error message is, "Request + // body is empty." HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: + // Client + // + // * Code: MissingSecurityElement Description: The SOAP 1.1 request is missing + // a security element. HTTP Status Code: 400 Bad Request SOAP Fault Code + // Prefix: Client + // + // * Code: MissingSecurityHeader Description: Your request is missing a required + // header. HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: NoLoggingStatusForKey Description: There is no such thing as a + // logging status subresource for a key. HTTP Status Code: 400 Bad Request + // SOAP Fault Code Prefix: Client + // + // * Code: NoSuchBucket Description: The specified bucket does not exist. + // HTTP Status Code: 404 Not Found SOAP Fault Code Prefix: Client + // + // * Code: NoSuchBucketPolicy Description: The specified bucket does not + // have a bucket policy. HTTP Status Code: 404 Not Found SOAP Fault Code + // Prefix: Client + // + // * Code: NoSuchKey Description: The specified key does not exist. HTTP + // Status Code: 404 Not Found SOAP Fault Code Prefix: Client + // + // * Code: NoSuchLifecycleConfiguration Description: The lifecycle configuration + // does not exist. HTTP Status Code: 404 Not Found SOAP Fault Code Prefix: + // Client + // + // * Code: NoSuchUpload Description: The specified multipart upload does + // not exist. The upload ID might be invalid, or the multipart upload might + // have been aborted or completed. HTTP Status Code: 404 Not Found SOAP Fault + // Code Prefix: Client + // + // * Code: NoSuchVersion Description: Indicates that the version ID specified + // in the request does not match an existing version. HTTP Status Code: 404 + // Not Found SOAP Fault Code Prefix: Client + // + // * Code: NotImplemented Description: A header you provided implies functionality + // that is not implemented. HTTP Status Code: 501 Not Implemented SOAP Fault + // Code Prefix: Server + // + // * Code: NotSignedUp Description: Your account is not signed up for the + // Amazon S3 service. You must sign up before you can use Amazon S3. You + // can sign up at the following URL: https://aws.amazon.com/s3 HTTP Status + // Code: 403 Forbidden SOAP Fault Code Prefix: Client + // + // * Code: OperationAborted Description: A conflicting conditional operation + // is currently in progress against this resource. Try again. HTTP Status + // Code: 409 Conflict SOAP Fault Code Prefix: Client + // + // * Code: PermanentRedirect Description: The bucket you are attempting to + // access must be addressed using the specified endpoint. Send all future + // requests to this endpoint. HTTP Status Code: 301 Moved Permanently SOAP + // Fault Code Prefix: Client + // + // * Code: PreconditionFailed Description: At least one of the preconditions + // you specified did not hold. HTTP Status Code: 412 Precondition Failed + // SOAP Fault Code Prefix: Client + // + // * Code: Redirect Description: Temporary redirect. HTTP Status Code: 307 + // Moved Temporarily SOAP Fault Code Prefix: Client + // + // * Code: RestoreAlreadyInProgress Description: Object restore is already + // in progress. HTTP Status Code: 409 Conflict SOAP Fault Code Prefix: Client + // + // * Code: RequestIsNotMultiPartContent Description: Bucket POST must be + // of the enclosure-type multipart/form-data. HTTP Status Code: 400 Bad Request + // SOAP Fault Code Prefix: Client + // + // * Code: RequestTimeout Description: Your socket connection to the server + // was not read from or written to within the timeout period. HTTP Status + // Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: RequestTimeTooSkewed Description: The difference between the request + // time and the server's time is too large. HTTP Status Code: 403 Forbidden + // SOAP Fault Code Prefix: Client + // + // * Code: RequestTorrentOfBucketError Description: Requesting the torrent + // file of a bucket is not permitted. HTTP Status Code: 400 Bad Request SOAP + // Fault Code Prefix: Client + // + // * Code: SignatureDoesNotMatch Description: The request signature we calculated + // does not match the signature you provided. Check your AWS secret access + // key and signing method. For more information, see REST Authentication + // (https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html) + // and SOAP Authentication (https://docs.aws.amazon.com/AmazonS3/latest/dev/SOAPAuthentication.html) + // for details. HTTP Status Code: 403 Forbidden SOAP Fault Code Prefix: Client + // + // * Code: ServiceUnavailable Description: Reduce your request rate. HTTP + // Status Code: 503 Service Unavailable SOAP Fault Code Prefix: Server + // + // * Code: SlowDown Description: Reduce your request rate. HTTP Status Code: + // 503 Slow Down SOAP Fault Code Prefix: Server + // + // * Code: TemporaryRedirect Description: You are being redirected to the + // bucket while DNS updates. HTTP Status Code: 307 Moved Temporarily SOAP + // Fault Code Prefix: Client + // + // * Code: TokenRefreshRequired Description: The provided token must be refreshed. + // HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: TooManyBuckets Description: You have attempted to create more + // buckets than allowed. HTTP Status Code: 400 Bad Request SOAP Fault Code + // Prefix: Client + // + // * Code: UnexpectedContent Description: This request does not support content. + // HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client + // + // * Code: UnresolvableGrantByEmailAddress Description: The email address + // you provided does not match any account on record. HTTP Status Code: 400 + // Bad Request SOAP Fault Code Prefix: Client + // + // * Code: UserKeyMustBeSpecified Description: The bucket POST must contain + // the specified field name. If it is specified, check the order of the fields. + // HTTP Status Code: 400 Bad Request SOAP Fault Code Prefix: Client Code *string `type:"string"` + // The error key. Key *string `min:"1" type:"string"` + // The error message contains a generic description of the error condition in + // English. It is intended for a human audience. Simple programs display the + // message directly to the end user if they encounter an error condition they + // don't know how or don't care to handle. Sophisticated programs with more + // exhaustive error handling and proper internationalization are more likely + // to ignore the error message. Message *string `type:"string"` + // The version ID of the error. VersionId *string `type:"string"` } @@ -11166,6 +15192,7 @@ func (s *Error) SetVersionId(v string) *Error { return s } +// The error information. type ErrorDocument struct { _ struct{} `type:"structure"` @@ -11207,18 +15234,58 @@ func (s *ErrorDocument) SetKey(v string) *ErrorDocument { return s } -// A container for a key value pair that defines the criteria for the filter -// rule. +// Optional configuration to replicate existing source bucket objects. For more +// information, see Replicating Existing Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-what-is-isnot-replicated.html#existing-object-replication) +// in the Amazon S3 Developer Guide. +type ExistingObjectReplication struct { + _ struct{} `type:"structure"` + + // Status is a required field + Status *string `type:"string" required:"true" enum:"ExistingObjectReplicationStatus"` +} + +// String returns the string representation +func (s ExistingObjectReplication) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ExistingObjectReplication) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ExistingObjectReplication) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ExistingObjectReplication"} + if s.Status == nil { + invalidParams.Add(request.NewErrParamRequired("Status")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetStatus sets the Status field's value. +func (s *ExistingObjectReplication) SetStatus(v string) *ExistingObjectReplication { + s.Status = &v + return s +} + +// Specifies the Amazon S3 object key name to filter on and whether to filter +// on the suffix or prefix of the key name. type FilterRule struct { _ struct{} `type:"structure"` // The object key name prefix or suffix identifying one or more objects to which - // the filtering rule applies. The maximum prefix length is 1,024 characters. - // Overlapping prefixes and suffixes are not supported. For more information, - // see Configuring Event Notifications (http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) + // the filtering rule applies. The maximum length is 1,024 characters. Overlapping + // prefixes and suffixes are not supported. For more information, see Configuring + // Event Notifications (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) // in the Amazon Simple Storage Service Developer Guide. Name *string `type:"string" enum:"FilterRuleName"` + // The value that the filter searches for in object key names. Value *string `type:"string"` } @@ -11245,7 +15312,7 @@ func (s *FilterRule) SetValue(v string) *FilterRule { } type GetBucketAccelerateConfigurationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketAccelerateConfigurationRequest" type:"structure"` // Name of the bucket for which the accelerate configuration is retrieved. // @@ -11292,6 +15359,20 @@ func (s *GetBucketAccelerateConfigurationInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketAccelerateConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketAccelerateConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketAccelerateConfigurationOutput struct { _ struct{} `type:"structure"` @@ -11316,8 +15397,10 @@ func (s *GetBucketAccelerateConfigurationOutput) SetStatus(v string) *GetBucketA } type GetBucketAclInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketAclRequest" type:"structure"` + // Specifies the S3 bucket whose ACL is being requested. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -11361,12 +15444,27 @@ func (s *GetBucketAclInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketAclInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketAclInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketAclOutput struct { _ struct{} `type:"structure"` // A list of grants. Grants []*Grant `locationName:"AccessControlList" locationNameList:"Grant" type:"list"` + // Container for the bucket owner's display name and ID. Owner *Owner `type:"structure"` } @@ -11393,14 +15491,14 @@ func (s *GetBucketAclOutput) SetOwner(v *Owner) *GetBucketAclOutput { } type GetBucketAnalyticsConfigurationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketAnalyticsConfigurationRequest" type:"structure"` // The name of the bucket from which an analytics configuration is retrieved. // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` - // The identifier used to represent an analytics configuration. + // The ID that identifies the analytics configuration. // // Id is a required field Id *string `location:"querystring" locationName:"id" type:"string" required:"true"` @@ -11454,6 +15552,20 @@ func (s *GetBucketAnalyticsConfigurationInput) SetId(v string) *GetBucketAnalyti return s } +func (s *GetBucketAnalyticsConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketAnalyticsConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketAnalyticsConfigurationOutput struct { _ struct{} `type:"structure" payload:"AnalyticsConfiguration"` @@ -11478,8 +15590,10 @@ func (s *GetBucketAnalyticsConfigurationOutput) SetAnalyticsConfiguration(v *Ana } type GetBucketCorsInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketCorsRequest" type:"structure"` + // The bucket name for which to get the cors configuration. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -11523,9 +15637,25 @@ func (s *GetBucketCorsInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketCorsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketCorsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketCorsOutput struct { _ struct{} `type:"structure"` + // A set of origins and methods (cross-origin access that you want to allow). + // You can add up to 100 rules to the configuration. CORSRules []*CORSRule `locationName:"CORSRule" type:"list" flattened:"true"` } @@ -11546,7 +15676,7 @@ func (s *GetBucketCorsOutput) SetCORSRules(v []*CORSRule) *GetBucketCorsOutput { } type GetBucketEncryptionInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketEncryptionRequest" type:"structure"` // The name of the bucket from which the server-side encryption configuration // is retrieved. @@ -11594,11 +15724,24 @@ func (s *GetBucketEncryptionInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketEncryptionInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketEncryptionInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketEncryptionOutput struct { _ struct{} `type:"structure" payload:"ServerSideEncryptionConfiguration"` - // Container for server-side encryption configuration rules. Currently S3 supports - // one rule only. + // Specifies the default server-side-encryption configuration. ServerSideEncryptionConfiguration *ServerSideEncryptionConfiguration `type:"structure"` } @@ -11619,7 +15762,7 @@ func (s *GetBucketEncryptionOutput) SetServerSideEncryptionConfiguration(v *Serv } type GetBucketInventoryConfigurationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketInventoryConfigurationRequest" type:"structure"` // The name of the bucket containing the inventory configuration to retrieve. // @@ -11680,6 +15823,20 @@ func (s *GetBucketInventoryConfigurationInput) SetId(v string) *GetBucketInvento return s } +func (s *GetBucketInventoryConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketInventoryConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketInventoryConfigurationOutput struct { _ struct{} `type:"structure" payload:"InventoryConfiguration"` @@ -11704,8 +15861,10 @@ func (s *GetBucketInventoryConfigurationOutput) SetInventoryConfiguration(v *Inv } type GetBucketLifecycleConfigurationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketLifecycleConfigurationRequest" type:"structure"` + // The name of the bucket for which to get the lifecycle information. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -11744,14 +15903,29 @@ func (s *GetBucketLifecycleConfigurationInput) SetBucket(v string) *GetBucketLif func (s *GetBucketLifecycleConfigurationInput) getBucket() (v string) { if s.Bucket == nil { - return v + return v + } + return *s.Bucket +} + +func (s *GetBucketLifecycleConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketLifecycleConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false } - return *s.Bucket + return arn.IsARN(*s.Bucket) } type GetBucketLifecycleConfigurationOutput struct { _ struct{} `type:"structure"` + // Container for a lifecycle rule. Rules []*LifecycleRule `locationName:"Rule" type:"list" flattened:"true"` } @@ -11772,8 +15946,10 @@ func (s *GetBucketLifecycleConfigurationOutput) SetRules(v []*LifecycleRule) *Ge } type GetBucketLifecycleInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketLifecycleRequest" type:"structure"` + // The name of the bucket for which to get the lifecycle information. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -11817,9 +15993,24 @@ func (s *GetBucketLifecycleInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketLifecycleInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketLifecycleInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketLifecycleOutput struct { _ struct{} `type:"structure"` + // Container for a lifecycle rule. Rules []*Rule `locationName:"Rule" type:"list" flattened:"true"` } @@ -11840,8 +16031,10 @@ func (s *GetBucketLifecycleOutput) SetRules(v []*Rule) *GetBucketLifecycleOutput } type GetBucketLocationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketLocationRequest" type:"structure"` + // The name of the bucket for which to get the location. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -11885,9 +16078,26 @@ func (s *GetBucketLocationInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketLocationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketLocationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketLocationOutput struct { _ struct{} `type:"structure"` + // Specifies the Region where the bucket resides. For a list of all the Amazon + // S3 supported location constraints by Region, see Regions and Endpoints (https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region). + // Buckets in Region us-east-1 have a LocationConstraint of null. LocationConstraint *string `type:"string" enum:"BucketLocationConstraint"` } @@ -11908,8 +16118,10 @@ func (s *GetBucketLocationOutput) SetLocationConstraint(v string) *GetBucketLoca } type GetBucketLoggingInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketLoggingRequest" type:"structure"` + // The bucket name for which to get the logging information. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -11953,12 +16165,27 @@ func (s *GetBucketLoggingInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketLoggingInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketLoggingInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketLoggingOutput struct { _ struct{} `type:"structure"` - // Container for logging information. Presence of this element indicates that - // logging is enabled. Parameters TargetBucket and TargetPrefix are required - // in this case. + // Describes where logs are stored and the prefix that Amazon S3 assigns to + // all log object keys for a bucket. For more information, see PUT Bucket logging + // (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTlogging.html) + // in the Amazon Simple Storage Service API Reference. LoggingEnabled *LoggingEnabled `type:"structure"` } @@ -11979,7 +16206,7 @@ func (s *GetBucketLoggingOutput) SetLoggingEnabled(v *LoggingEnabled) *GetBucket } type GetBucketMetricsConfigurationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketMetricsConfigurationRequest" type:"structure"` // The name of the bucket containing the metrics configuration to retrieve. // @@ -12040,6 +16267,20 @@ func (s *GetBucketMetricsConfigurationInput) SetId(v string) *GetBucketMetricsCo return s } +func (s *GetBucketMetricsConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketMetricsConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketMetricsConfigurationOutput struct { _ struct{} `type:"structure" payload:"MetricsConfiguration"` @@ -12064,9 +16305,9 @@ func (s *GetBucketMetricsConfigurationOutput) SetMetricsConfiguration(v *Metrics } type GetBucketNotificationConfigurationRequest struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketNotificationConfigurationRequest" type:"structure"` - // Name of the bucket to get the notification configuration for. + // Name of the bucket for which to get the notification configuration. // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -12111,9 +16352,25 @@ func (s *GetBucketNotificationConfigurationRequest) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketNotificationConfigurationRequest) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketNotificationConfigurationRequest) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketPolicyInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketPolicyRequest" type:"structure"` + // The bucket name for which to get the bucket policy. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -12157,6 +16414,20 @@ func (s *GetBucketPolicyInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketPolicyInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketPolicyInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketPolicyOutput struct { _ struct{} `type:"structure" payload:"Policy"` @@ -12181,7 +16452,7 @@ func (s *GetBucketPolicyOutput) SetPolicy(v string) *GetBucketPolicyOutput { } type GetBucketPolicyStatusInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketPolicyStatusRequest" type:"structure"` // The name of the Amazon S3 bucket whose policy status you want to retrieve. // @@ -12228,6 +16499,20 @@ func (s *GetBucketPolicyStatusInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketPolicyStatusInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketPolicyStatusInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketPolicyStatusOutput struct { _ struct{} `type:"structure" payload:"PolicyStatus"` @@ -12252,8 +16537,10 @@ func (s *GetBucketPolicyStatusOutput) SetPolicyStatus(v *PolicyStatus) *GetBucke } type GetBucketReplicationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketReplicationRequest" type:"structure"` + // The bucket name for which to get the replication information. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -12297,6 +16584,20 @@ func (s *GetBucketReplicationInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketReplicationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketReplicationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketReplicationOutput struct { _ struct{} `type:"structure" payload:"ReplicationConfiguration"` @@ -12322,8 +16623,10 @@ func (s *GetBucketReplicationOutput) SetReplicationConfiguration(v *ReplicationC } type GetBucketRequestPaymentInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketRequestPaymentRequest" type:"structure"` + // The name of the bucket for which to get the payment request configuration + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -12367,6 +16670,20 @@ func (s *GetBucketRequestPaymentInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketRequestPaymentInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketRequestPaymentInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketRequestPaymentOutput struct { _ struct{} `type:"structure"` @@ -12391,8 +16708,10 @@ func (s *GetBucketRequestPaymentOutput) SetPayer(v string) *GetBucketRequestPaym } type GetBucketTaggingInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketTaggingRequest" type:"structure"` + // The name of the bucket for which to get the tagging information. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -12436,9 +16755,25 @@ func (s *GetBucketTaggingInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketTaggingInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketTaggingInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketTaggingOutput struct { _ struct{} `type:"structure"` + // Contains the tag set. + // // TagSet is a required field TagSet []*Tag `locationNameList:"Tag" type:"list" required:"true"` } @@ -12460,8 +16795,10 @@ func (s *GetBucketTaggingOutput) SetTagSet(v []*Tag) *GetBucketTaggingOutput { } type GetBucketVersioningInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketVersioningRequest" type:"structure"` + // The name of the bucket for which to get the versioning information. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -12505,6 +16842,20 @@ func (s *GetBucketVersioningInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketVersioningInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketVersioningInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketVersioningOutput struct { _ struct{} `type:"structure"` @@ -12540,8 +16891,10 @@ func (s *GetBucketVersioningOutput) SetStatus(v string) *GetBucketVersioningOutp } type GetBucketWebsiteInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetBucketWebsiteRequest" type:"structure"` + // The bucket name for which to get the website configuration. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -12585,15 +16938,34 @@ func (s *GetBucketWebsiteInput) getBucket() (v string) { return *s.Bucket } +func (s *GetBucketWebsiteInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetBucketWebsiteInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetBucketWebsiteOutput struct { _ struct{} `type:"structure"` + // The object key name of the website error document to use for 4XX class errors. ErrorDocument *ErrorDocument `type:"structure"` + // The name of the index document for the website (for example index.html). IndexDocument *IndexDocument `type:"structure"` + // Specifies the redirect behavior of all requests to a website endpoint of + // an Amazon S3 bucket. RedirectAllRequestsTo *RedirectAllRequestsTo `type:"structure"` + // Rules that define when a redirect is applied and the redirect behavior. RoutingRules []*RoutingRule `locationNameList:"RoutingRule" type:"list"` } @@ -12632,18 +17004,30 @@ func (s *GetBucketWebsiteOutput) SetRoutingRules(v []*RoutingRule) *GetBucketWeb } type GetObjectAclInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetObjectAclRequest" type:"structure"` + // The bucket name that contains the object for which to get the ACL information. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // The key of the object for which to get the ACL information. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // VersionId used to reference a specific version of the object. @@ -12713,12 +17097,27 @@ func (s *GetObjectAclInput) SetVersionId(v string) *GetObjectAclInput { return s } +func (s *GetObjectAclInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetObjectAclInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetObjectAclOutput struct { _ struct{} `type:"structure"` // A list of grants. Grants []*Grant `locationName:"AccessControlList" locationNameList:"Grant" type:"list"` + // Container for the bucket owner's display name and ID. Owner *Owner `type:"structure"` // If present, indicates that the requester was successfully charged for the @@ -12755,8 +17154,17 @@ func (s *GetObjectAclOutput) SetRequestCharged(v string) *GetObjectAclOutput { } type GetObjectInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetObjectRequest" type:"structure"` + // The bucket name containing the object. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -12776,6 +17184,8 @@ type GetObjectInput struct { // otherwise return a 412 (precondition failed). IfUnmodifiedSince *time.Time `location:"header" locationName:"If-Unmodified-Since" type:"timestamp"` + // Key of the object to get. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` @@ -12785,13 +17195,17 @@ type GetObjectInput struct { PartNumber *int64 `location:"querystring" locationName:"partNumber" type:"integer"` // Downloads the specified range bytes of an object. For more information about - // the HTTP Range header, go to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35. + // the HTTP Range header, see https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 + // (https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35). + // + // Amazon S3 doesn't support retrieving multiple ranges of data per GET request. Range *string `location:"header" locationName:"Range" type:"string"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // Sets the Cache-Control header of the response. @@ -12812,19 +17226,20 @@ type GetObjectInput struct { // Sets the Expires header of the response. ResponseExpires *time.Time `location:"querystring" locationName:"response-expires" type:"timestamp"` - // Specifies the algorithm to use to when encrypting the object (e.g., AES256). + // Specifies the algorithm to use to when encrypting the object (for example, + // AES256). SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // Specifies the customer-provided encryption key for Amazon S3 to use in encrypting // data. This value is used to store the object and then it is discarded; Amazon - // does not store the encryption key. The key must be appropriate for use with - // the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm + // S3 does not store the encryption key. The key must be appropriate for use + // with the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm // header. - SSECustomerKey *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` + SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. - // Amazon S3 uses this header for a message integrity check to ensure the encryption - // key was transmitted without error. + // Amazon S3 uses this header for a message integrity check to ensure that the + // encryption key was transmitted without error. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` // VersionId used to reference a specific version of the object. @@ -12991,10 +17406,32 @@ func (s *GetObjectInput) SetVersionId(v string) *GetObjectInput { return s } +func (s *GetObjectInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetObjectInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetObjectLegalHoldInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetObjectLegalHoldRequest" type:"structure"` - // The bucket containing the object whose Legal Hold status you want to retrieve. + // The bucket name containing the object whose Legal Hold status you want to + // retrieve. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -13004,10 +17441,11 @@ type GetObjectLegalHoldInput struct { // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // The version ID of the object whose Legal Hold status you want to retrieve. @@ -13077,6 +17515,20 @@ func (s *GetObjectLegalHoldInput) SetVersionId(v string) *GetObjectLegalHoldInpu return s } +func (s *GetObjectLegalHoldInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetObjectLegalHoldInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetObjectLegalHoldOutput struct { _ struct{} `type:"structure" payload:"LegalHold"` @@ -13101,7 +17553,7 @@ func (s *GetObjectLegalHoldOutput) SetLegalHold(v *ObjectLockLegalHold) *GetObje } type GetObjectLockConfigurationInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetObjectLockConfigurationRequest" type:"structure"` // The bucket whose Object Lock configuration you want to retrieve. // @@ -13148,6 +17600,20 @@ func (s *GetObjectLockConfigurationInput) getBucket() (v string) { return *s.Bucket } +func (s *GetObjectLockConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetObjectLockConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetObjectLockConfigurationOutput struct { _ struct{} `type:"structure" payload:"ObjectLockConfiguration"` @@ -13174,6 +17640,7 @@ func (s *GetObjectLockConfigurationOutput) SetObjectLockConfiguration(v *ObjectL type GetObjectOutput struct { _ struct{} `type:"structure" payload:"Body"` + // Indicates that a range of bytes was specified. AcceptRanges *string `location:"header" locationName:"accept-ranges" type:"string"` // Object data. @@ -13207,11 +17674,11 @@ type GetObjectOutput struct { DeleteMarker *bool `location:"header" locationName:"x-amz-delete-marker" type:"boolean"` // An ETag is an opaque identifier assigned by a web server to a specific version - // of a resource found at a URL + // of a resource found at a URL. ETag *string `location:"header" locationName:"ETag" type:"string"` // If the object expiration is configured (see PUT Bucket lifecycle), the response - // includes this header. It includes the expiry-date and rule-id key value pairs + // includes this header. It includes the expiry-date and rule-id key-value pairs // providing object expiration information. The value of the rule-id is URL // encoded. Expiration *string `location:"header" locationName:"x-amz-expiration" type:"string"` @@ -13223,6 +17690,10 @@ type GetObjectOutput struct { LastModified *time.Time `location:"header" locationName:"Last-Modified" type:"timestamp"` // A map of metadata to store with the object in S3. + // + // By default unmarshaled keys are written as a map keys in following canonicalized format: + // the first letter and any letter following a hyphen will be capitalized, and the rest as lowercase. + // Set `aws.Config.LowerCaseHeaderMaps` to `true` to write unmarshaled keys to the map as lowercase. Metadata map[string]*string `location:"headers" locationName:"x-amz-meta-" type:"map"` // This is set to the number of metadata entries not returned in x-amz-meta @@ -13244,6 +17715,8 @@ type GetObjectOutput struct { // The count of parts this object has. PartsCount *int64 `location:"header" locationName:"x-amz-mp-parts-count" type:"integer"` + // Amazon S3 can return this if your request involves a bucket that is either + // a source or destination in a replication rule. ReplicationStatus *string `location:"header" locationName:"x-amz-replication-status" type:"string" enum:"ReplicationStatus"` // If present, indicates that the requester was successfully charged for the @@ -13260,18 +17733,21 @@ type GetObjectOutput struct { SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // If server-side encryption with a customer-provided encryption key was requested, - // the response will include this header to provide round trip message integrity + // the response will include this header to provide round-trip message integrity // verification of the customer-provided encryption key. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` - // If present, specifies the ID of the AWS Key Management Service (KMS) master - // encryption key that was used for the object. + // If present, specifies the ID of the AWS Key Management Service (AWS KMS) + // symmetric customer managed customer master key (CMK) that was used for the + // object. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // The server-side encryption algorithm used when storing this object in Amazon + // S3 (for example, AES256, aws:kms). ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` + // Provides storage class information of the object. Amazon S3 returns this + // header for all objects except for S3 Standard storage class objects. StorageClass *string `location:"header" locationName:"x-amz-storage-class" type:"string" enum:"StorageClass"` // The number of tags, if any, on the object. @@ -13483,9 +17959,17 @@ func (s *GetObjectOutput) SetWebsiteRedirectLocation(v string) *GetObjectOutput } type GetObjectRetentionInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetObjectRetentionRequest" type:"structure"` - // The bucket containing the object whose retention settings you want to retrieve. + // The bucket name containing the object whose retention settings you want to + // retrieve. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -13495,10 +17979,11 @@ type GetObjectRetentionInput struct { // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // The version ID for the object whose retention settings you want to retrieve. @@ -13568,6 +18053,20 @@ func (s *GetObjectRetentionInput) SetVersionId(v string) *GetObjectRetentionInpu return s } +func (s *GetObjectRetentionInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetObjectRetentionInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetObjectRetentionOutput struct { _ struct{} `type:"structure" payload:"Retention"` @@ -13592,14 +18091,26 @@ func (s *GetObjectRetentionOutput) SetRetention(v *ObjectLockRetention) *GetObje } type GetObjectTaggingInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetObjectTaggingRequest" type:"structure"` + // The bucket name containing the object for which to get the tagging information. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Object key for which to get the tagging information. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` + // The versionId of the object for which to get the tagging information. VersionId *string `location:"querystring" locationName:"versionId" type:"string"` } @@ -13660,12 +18171,29 @@ func (s *GetObjectTaggingInput) SetVersionId(v string) *GetObjectTaggingInput { return s } +func (s *GetObjectTaggingInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetObjectTaggingInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetObjectTaggingOutput struct { _ struct{} `type:"structure"` + // Contains the tag set. + // // TagSet is a required field TagSet []*Tag `locationNameList:"Tag" type:"list" required:"true"` + // The versionId of the object for which you got the tagging information. VersionId *string `location:"header" locationName:"x-amz-version-id" type:"string"` } @@ -13692,18 +18220,24 @@ func (s *GetObjectTaggingOutput) SetVersionId(v string) *GetObjectTaggingOutput } type GetObjectTorrentInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetObjectTorrentRequest" type:"structure"` + // The name of the bucket containing the object for which to get the torrent + // files. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // The object key for which to get the information. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` } @@ -13764,9 +18298,24 @@ func (s *GetObjectTorrentInput) SetRequestPayer(v string) *GetObjectTorrentInput return s } +func (s *GetObjectTorrentInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetObjectTorrentInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetObjectTorrentOutput struct { _ struct{} `type:"structure" payload:"Body"` + // A Bencoded dictionary as defined by the BitTorrent specification Body io.ReadCloser `type:"blob"` // If present, indicates that the requester was successfully charged for the @@ -13797,7 +18346,7 @@ func (s *GetObjectTorrentOutput) SetRequestCharged(v string) *GetObjectTorrentOu } type GetPublicAccessBlockInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"GetPublicAccessBlockRequest" type:"structure"` // The name of the Amazon S3 bucket whose PublicAccessBlock configuration you // want to retrieve. @@ -13845,6 +18394,20 @@ func (s *GetPublicAccessBlockInput) getBucket() (v string) { return *s.Bucket } +func (s *GetPublicAccessBlockInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *GetPublicAccessBlockInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type GetPublicAccessBlockOutput struct { _ struct{} `type:"structure" payload:"PublicAccessBlockConfiguration"` @@ -13869,10 +18432,11 @@ func (s *GetPublicAccessBlockOutput) SetPublicAccessBlockConfiguration(v *Public return s } +// Container for S3 Glacier job parameters. type GlacierJobParameters struct { _ struct{} `type:"structure"` - // Glacier retrieval tier at which the restore will be processed. + // S3 Glacier retrieval tier at which the restore will be processed. // // Tier is a required field Tier *string `type:"string" required:"true" enum:"Tier"` @@ -13907,9 +18471,11 @@ func (s *GlacierJobParameters) SetTier(v string) *GlacierJobParameters { return s } +// Container for grant information. type Grant struct { _ struct{} `type:"structure"` + // The person being granted permissions. Grantee *Grantee `type:"structure" xmlPrefix:"xsi" xmlURI:"http://www.w3.org/2001/XMLSchema-instance"` // Specifies the permission given to the grantee. @@ -13953,6 +18519,7 @@ func (s *Grant) SetPermission(v string) *Grant { return s } +// Container for the person being granted permissions. type Grantee struct { _ struct{} `type:"structure" xmlPrefix:"xsi" xmlURI:"http://www.w3.org/2001/XMLSchema-instance"` @@ -13960,6 +18527,29 @@ type Grantee struct { DisplayName *string `type:"string"` // Email address of the grantee. + // + // Using email addresses to specify a grantee is only supported in the following + // AWS Regions: + // + // * US East (N. Virginia) + // + // * US West (N. California) + // + // * US West (Oregon) + // + // * Asia Pacific (Singapore) + // + // * Asia Pacific (Sydney) + // + // * Asia Pacific (Tokyo) + // + // * Europe (Ireland) + // + // * South America (São Paulo) + // + // For a list of all the Amazon S3 supported Regions and endpoints, see Regions + // and Endpoints (https://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region) + // in the AWS General Reference. EmailAddress *string `type:"string"` // The canonical user ID of the grantee. @@ -14028,8 +18618,10 @@ func (s *Grantee) SetURI(v string) *Grantee { } type HeadBucketInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"HeadBucketRequest" type:"structure"` + // The bucket name. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` } @@ -14073,6 +18665,20 @@ func (s *HeadBucketInput) getBucket() (v string) { return *s.Bucket } +func (s *HeadBucketInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *HeadBucketInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type HeadBucketOutput struct { _ struct{} `type:"structure"` } @@ -14088,8 +18694,10 @@ func (s HeadBucketOutput) GoString() string { } type HeadObjectInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"HeadObjectRequest" type:"structure"` + // The name of the bucket containing the object. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -14109,6 +18717,8 @@ type HeadObjectInput struct { // otherwise return a 412 (precondition failed). IfUnmodifiedSince *time.Time `location:"header" locationName:"If-Unmodified-Since" type:"timestamp"` + // The object key. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` @@ -14119,28 +18729,32 @@ type HeadObjectInput struct { PartNumber *int64 `location:"querystring" locationName:"partNumber" type:"integer"` // Downloads the specified range bytes of an object. For more information about - // the HTTP Range header, go to http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35. + // the HTTP Range header, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35. + // + // Amazon S3 doesn't support retrieving multiple ranges of data per GET request. Range *string `location:"header" locationName:"Range" type:"string"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` - // Specifies the algorithm to use to when encrypting the object (e.g., AES256). + // Specifies the algorithm to use to when encrypting the object (for example, + // AES256). SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // Specifies the customer-provided encryption key for Amazon S3 to use in encrypting // data. This value is used to store the object and then it is discarded; Amazon - // does not store the encryption key. The key must be appropriate for use with - // the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm + // S3 does not store the encryption key. The key must be appropriate for use + // with the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm // header. - SSECustomerKey *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` + SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. - // Amazon S3 uses this header for a message integrity check to ensure the encryption - // key was transmitted without error. + // Amazon S3 uses this header for a message integrity check to ensure that the + // encryption key was transmitted without error. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` // VersionId used to reference a specific version of the object. @@ -14271,9 +18885,24 @@ func (s *HeadObjectInput) SetVersionId(v string) *HeadObjectInput { return s } +func (s *HeadObjectInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *HeadObjectInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type HeadObjectOutput struct { _ struct{} `type:"structure"` + // Indicates that a range of bytes was specified. AcceptRanges *string `location:"header" locationName:"accept-ranges" type:"string"` // Specifies caching behavior along the request/reply chain. @@ -14301,11 +18930,11 @@ type HeadObjectOutput struct { DeleteMarker *bool `location:"header" locationName:"x-amz-delete-marker" type:"boolean"` // An ETag is an opaque identifier assigned by a web server to a specific version - // of a resource found at a URL + // of a resource found at a URL. ETag *string `location:"header" locationName:"ETag" type:"string"` // If the object expiration is configured (see PUT Bucket lifecycle), the response - // includes this header. It includes the expiry-date and rule-id key value pairs + // includes this header. It includes the expiry-date and rule-id key-value pairs // providing object expiration information. The value of the rule-id is URL // encoded. Expiration *string `location:"header" locationName:"x-amz-expiration" type:"string"` @@ -14317,6 +18946,10 @@ type HeadObjectOutput struct { LastModified *time.Time `location:"header" locationName:"Last-Modified" type:"timestamp"` // A map of metadata to store with the object in S3. + // + // By default unmarshaled keys are written as a map keys in following canonicalized format: + // the first letter and any letter following a hyphen will be capitalized, and the rest as lowercase. + // Set `aws.Config.LowerCaseHeaderMaps` to `true` to write unmarshaled keys to the map as lowercase. Metadata map[string]*string `location:"headers" locationName:"x-amz-meta-" type:"map"` // This is set to the number of metadata entries not returned in x-amz-meta @@ -14325,26 +18958,69 @@ type HeadObjectOutput struct { // you can create metadata whose values are not legal HTTP headers. MissingMeta *int64 `location:"header" locationName:"x-amz-missing-meta" type:"integer"` - // The Legal Hold status for the specified object. + // Specifies whether a legal hold is in effect for this object. This header + // is only returned if the requester has the s3:GetObjectLegalHold permission. + // This header is not returned if the specified version of this object has never + // had a legal hold applied. For more information about S3 Object Lock, see + // Object Lock (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html). ObjectLockLegalHoldStatus *string `location:"header" locationName:"x-amz-object-lock-legal-hold" type:"string" enum:"ObjectLockLegalHoldStatus"` - // The Object Lock mode currently in place for this object. + // The Object Lock mode, if any, that's in effect for this object. This header + // is only returned if the requester has the s3:GetObjectRetention permission. + // For more information about S3 Object Lock, see Object Lock (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html). ObjectLockMode *string `location:"header" locationName:"x-amz-object-lock-mode" type:"string" enum:"ObjectLockMode"` - // The date and time when this object's Object Lock will expire. + // The date and time when the Object Lock retention period expires. This header + // is only returned if the requester has the s3:GetObjectRetention permission. ObjectLockRetainUntilDate *time.Time `location:"header" locationName:"x-amz-object-lock-retain-until-date" type:"timestamp" timestampFormat:"iso8601"` // The count of parts this object has. PartsCount *int64 `location:"header" locationName:"x-amz-mp-parts-count" type:"integer"` + // Amazon S3 can return this header if your request involves a bucket that is + // either a source or destination in a replication rule. + // + // In replication, you have a source bucket on which you configure replication + // and destination bucket where Amazon S3 stores object replicas. When you request + // an object (GetObject) or object metadata (HeadObject) from these buckets, + // Amazon S3 will return the x-amz-replication-status header in the response + // as follows: + // + // * If requesting an object from the source bucket — Amazon S3 will return + // the x-amz-replication-status header if the object in your request is eligible + // for replication. For example, suppose that in your replication configuration, + // you specify object prefix TaxDocs requesting Amazon S3 to replicate objects + // with key prefix TaxDocs. Any objects you upload with this key name prefix, + // for example TaxDocs/document1.pdf, are eligible for replication. For any + // object request with this key name prefix, Amazon S3 will return the x-amz-replication-status + // header with value PENDING, COMPLETED or FAILED indicating object replication + // status. + // + // * If requesting an object from the destination bucket — Amazon S3 will + // return the x-amz-replication-status header with value REPLICA if the object + // in your request is a replica that Amazon S3 created. + // + // For more information, see Replication (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html). ReplicationStatus *string `location:"header" locationName:"x-amz-replication-status" type:"string" enum:"ReplicationStatus"` // If present, indicates that the requester was successfully charged for the // request. RequestCharged *string `location:"header" locationName:"x-amz-request-charged" type:"string" enum:"RequestCharged"` - // Provides information about object restoration operation and expiration time - // of the restored object copy. + // If the object is an archived object (an object whose storage class is GLACIER), + // the response includes this header if either the archive restoration is in + // progress (see RestoreObject or an archive copy is already restored. + // + // If an archive copy is already restored, the header value indicates when Amazon + // S3 is scheduled to delete the object copy. For example: + // + // x-amz-restore: ongoing-request="false", expiry-date="Fri, 23 Dec 2012 00:00:00 + // GMT" + // + // If the object restoration is in progress, the header returns the value ongoing-request="true". + // + // For more information about archiving objects, see Transitioning Objects: + // General Considerations (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html#lifecycle-transition-general-considerations). Restore *string `location:"header" locationName:"x-amz-restore" type:"string"` // If server-side encryption with a customer-provided encryption key was requested, @@ -14353,18 +19029,25 @@ type HeadObjectOutput struct { SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // If server-side encryption with a customer-provided encryption key was requested, - // the response will include this header to provide round trip message integrity + // the response will include this header to provide round-trip message integrity // verification of the customer-provided encryption key. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` - // If present, specifies the ID of the AWS Key Management Service (KMS) master - // encryption key that was used for the object. + // If present, specifies the ID of the AWS Key Management Service (AWS KMS) + // symmetric customer managed customer master key (CMK) that was used for the + // object. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // If the object is stored using server-side encryption either with an AWS KMS + // customer master key (CMK) or an Amazon S3-managed encryption key, the response + // includes this header with the value of the server-side encryption algorithm + // used when storing this object in Amazon S3 (for example, AES256, aws:kms). ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` + // Provides storage class information of the object. Amazon S3 returns this + // header for all objects except for S3 Standard storage class objects. + // + // For more information, see Storage Classes (https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-class-intro.html). StorageClass *string `location:"header" locationName:"x-amz-storage-class" type:"string" enum:"StorageClass"` // Version of the object. @@ -14554,13 +19237,15 @@ func (s *HeadObjectOutput) SetWebsiteRedirectLocation(v string) *HeadObjectOutpu return s } +// Container for the Suffix element. type IndexDocument struct { _ struct{} `type:"structure"` // A suffix that is appended to a request that is for a directory on the website - // endpoint (e.g. if the suffix is index.html and you make a request to samplebucket/images/ - // the data that is returned will be for the object with the key name images/index.html) - // The suffix must not be empty and must not include a slash character. + // endpoint (for example,if the suffix is index.html and you make a request + // to samplebucket/images/ the data that is returned will be for the object + // with the key name images/index.html) The suffix must not be empty and must + // not include a slash character. // // Suffix is a required field Suffix *string `type:"string" required:"true"` @@ -14595,6 +19280,7 @@ func (s *IndexDocument) SetSuffix(v string) *IndexDocument { return s } +// Container element that identifies who initiated the multipart upload. type Initiator struct { _ struct{} `type:"structure"` @@ -14680,6 +19366,9 @@ func (s *InputSerialization) SetParquet(v *ParquetInput) *InputSerialization { return s } +// Specifies the inventory configuration for an Amazon S3 bucket. For more information, +// see GET Bucket inventory (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGETInventoryConfig.html) +// in the Amazon Simple Storage Service API Reference. type InventoryConfiguration struct { _ struct{} `type:"structure"` @@ -14697,12 +19386,16 @@ type InventoryConfiguration struct { // Id is a required field Id *string `type:"string" required:"true"` - // Specifies which object version(s) to included in the inventory results. + // Object versions to include in the inventory list. If set to All, the list + // includes all the object versions, which adds the version-related fields VersionId, + // IsLatest, and DeleteMarker to the list. If set to Current, the list does + // not contain these version-related fields. // // IncludedObjectVersions is a required field IncludedObjectVersions *string `type:"string" required:"true" enum:"InventoryIncludedObjectVersions"` - // Specifies whether the inventory is enabled or disabled. + // Specifies whether the inventory is enabled or disabled. If set to True, an + // inventory list is generated. If set to False, no inventory list is generated. // // IsEnabled is a required field IsEnabled *bool `type:"boolean" required:"true"` @@ -14808,6 +19501,7 @@ func (s *InventoryConfiguration) SetSchedule(v *InventorySchedule) *InventoryCon return s } +// Specifies the inventory configuration for an Amazon S3 bucket. type InventoryDestination struct { _ struct{} `type:"structure"` @@ -14857,10 +19551,10 @@ func (s *InventoryDestination) SetS3BucketDestination(v *InventoryS3BucketDestin type InventoryEncryption struct { _ struct{} `type:"structure"` - // Specifies the use of SSE-KMS to encrypt delivered Inventory reports. + // Specifies the use of SSE-KMS to encrypt delivered inventory reports. SSEKMS *SSEKMS `locationName:"SSE-KMS" type:"structure"` - // Specifies the use of SSE-S3 to encrypt delivered Inventory reports. + // Specifies the use of SSE-S3 to encrypt delivered inventory reports. SSES3 *SSES3 `locationName:"SSE-S3" type:"structure"` } @@ -14901,6 +19595,8 @@ func (s *InventoryEncryption) SetSSES3(v *SSES3) *InventoryEncryption { return s } +// Specifies an inventory filter. The inventory only includes objects that meet +// the filter's criteria. type InventoryFilter struct { _ struct{} `type:"structure"` @@ -14939,13 +19635,19 @@ func (s *InventoryFilter) SetPrefix(v string) *InventoryFilter { return s } +// Contains the bucket name, file format, bucket owner (optional), and prefix +// (optional) where inventory results are published. type InventoryS3BucketDestination struct { _ struct{} `type:"structure"` - // The ID of the account that owns the destination bucket. + // The account ID that owns the destination S3 bucket. If no account ID is provided, + // the owner is not validated before exporting data. + // + // Although this value is optional, we strongly recommend that you set it to + // help prevent problems if the destination bucket ownership changes. AccountId *string `type:"string"` - // The Amazon resource name (ARN) of the bucket where inventory results will + // The Amazon Resource Name (ARN) of the bucket where inventory results will // be published. // // Bucket is a required field @@ -15032,6 +19734,7 @@ func (s *InventoryS3BucketDestination) SetPrefix(v string) *InventoryS3BucketDes return s } +// Specifies the schedule for generating inventory results. type InventorySchedule struct { _ struct{} `type:"structure"` @@ -15070,6 +19773,7 @@ func (s *InventorySchedule) SetFrequency(v string) *InventorySchedule { return s } +// Specifies JSON as object's input serialization format. type JSONInput struct { _ struct{} `type:"structure"` @@ -15093,10 +19797,12 @@ func (s *JSONInput) SetType(v string) *JSONInput { return s } +// Specifies JSON as request's output serialization format. type JSONOutput struct { _ struct{} `type:"structure"` - // The value used to separate individual records in the output. + // The value used to separate individual records in the output. If no value + // is specified, Amazon S3 uses a newline character ('\n'). RecordDelimiter *string `type:"string"` } @@ -15120,7 +19826,7 @@ func (s *JSONOutput) SetRecordDelimiter(v string) *JSONOutput { type KeyFilter struct { _ struct{} `type:"structure"` - // A list of containers for the key value pair that defines the criteria for + // A list of containers for the key-value pair that defines the criteria for // the filter rule. FilterRules []*FilterRule `locationName:"FilterRule" type:"list" flattened:"true"` } @@ -15145,11 +19851,15 @@ func (s *KeyFilter) SetFilterRules(v []*FilterRule) *KeyFilter { type LambdaFunctionConfiguration struct { _ struct{} `type:"structure"` + // The Amazon S3 bucket event for which to invoke the AWS Lambda function. For + // more information, see Supported Event Types (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Events is a required field Events []*string `locationName:"Event" type:"list" flattened:"true" required:"true"` - // A container for object key name filtering rules. For information about key - // name filtering, see Configuring Event Notifications (http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) + // Specifies object key name filtering rules. For information about key name + // filtering, see Configuring Event Notifications (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) // in the Amazon Simple Storage Service Developer Guide. Filter *NotificationConfigurationFilter `type:"structure"` @@ -15157,8 +19867,8 @@ type LambdaFunctionConfiguration struct { // If you don't provide one, Amazon S3 will assign an ID. Id *string `type:"string"` - // The Amazon Resource Name (ARN) of the Lambda cloud function that Amazon S3 - // can invoke when it detects events of the specified type. + // The Amazon Resource Name (ARN) of the AWS Lambda function that Amazon S3 + // invokes when the specified event type occurs. // // LambdaFunctionArn is a required field LambdaFunctionArn *string `locationName:"CloudFunction" type:"string" required:"true"` @@ -15214,9 +19924,12 @@ func (s *LambdaFunctionConfiguration) SetLambdaFunctionArn(v string) *LambdaFunc return s } +// Container for lifecycle rules. You can add as many as 1000 rules. type LifecycleConfiguration struct { _ struct{} `type:"structure"` + // Specifies lifecycle configuration rules for an Amazon S3 bucket. + // // Rules is a required field Rules []*Rule `locationName:"Rule" type:"list" flattened:"true" required:"true"` } @@ -15260,6 +19973,7 @@ func (s *LifecycleConfiguration) SetRules(v []*Rule) *LifecycleConfiguration { return s } +// Container for the expiration for the lifecycle of the object. type LifecycleExpiration struct { _ struct{} `type:"structure"` @@ -15306,13 +20020,19 @@ func (s *LifecycleExpiration) SetExpiredObjectDeleteMarker(v bool) *LifecycleExp return s } +// A lifecycle rule for individual objects in an Amazon S3 bucket. type LifecycleRule struct { _ struct{} `type:"structure"` - // Specifies the days since the initiation of an Incomplete Multipart Upload - // that Lifecycle will wait before permanently removing all parts of the upload. + // Specifies the days since the initiation of an incomplete multipart upload + // that Amazon S3 will wait before permanently removing all parts of the upload. + // For more information, see Aborting Incomplete Multipart Uploads Using a Bucket + // Lifecycle Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-abort-incomplete-mpu-lifecycle-config) + // in the Amazon Simple Storage Service Developer Guide. AbortIncompleteMultipartUpload *AbortIncompleteMultipartUpload `type:"structure"` + // Specifies the expiration for the lifecycle of the object in the form of date, + // days and, whether the object has a delete marker. Expiration *LifecycleExpiration `type:"structure"` // The Filter is used to identify objects that a Lifecycle Rule applies to. @@ -15329,10 +20049,15 @@ type LifecycleRule struct { // period in the object's lifetime. NoncurrentVersionExpiration *NoncurrentVersionExpiration `type:"structure"` + // Specifies the transition rule for the lifecycle rule that describes when + // noncurrent objects transition to a specific storage class. If your bucket + // is versioning-enabled (or versioning is suspended), you can set this action + // to request that Amazon S3 transition noncurrent object versions to a specific + // storage class at a set period in the object's lifetime. NoncurrentVersionTransitions []*NoncurrentVersionTransition `locationName:"NoncurrentVersionTransition" type:"list" flattened:"true"` // Prefix identifying one or more objects to which the rule applies. This is - // deprecated; use Filter instead. + // No longer used; use Filter instead. // // Deprecated: Prefix has been deprecated Prefix *string `deprecated:"true" type:"string"` @@ -15343,6 +20068,7 @@ type LifecycleRule struct { // Status is a required field Status *string `type:"string" required:"true" enum:"ExpirationStatus"` + // Specifies when an Amazon S3 object transitions to a specified storage class. Transitions []*Transition `locationName:"Transition" type:"list" flattened:"true"` } @@ -15434,6 +20160,7 @@ func (s *LifecycleRule) SetTransitions(v []*Transition) *LifecycleRule { type LifecycleRuleAndOperator struct { _ struct{} `type:"structure"` + // Prefix identifying one or more objects to which the rule applies. Prefix *string `type:"string"` // All of these tags must exist in the object's tag set in order for the rule @@ -15549,7 +20276,7 @@ func (s *LifecycleRuleFilter) SetTag(v *Tag) *LifecycleRuleFilter { } type ListBucketAnalyticsConfigurationsInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"ListBucketAnalyticsConfigurationsRequest" type:"structure"` // The name of the bucket from which analytics configurations are retrieved. // @@ -15606,13 +20333,28 @@ func (s *ListBucketAnalyticsConfigurationsInput) SetContinuationToken(v string) return s } +func (s *ListBucketAnalyticsConfigurationsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *ListBucketAnalyticsConfigurationsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type ListBucketAnalyticsConfigurationsOutput struct { _ struct{} `type:"structure"` // The list of analytics configurations for a bucket. AnalyticsConfigurationList []*AnalyticsConfiguration `locationName:"AnalyticsConfiguration" type:"list" flattened:"true"` - // The ContinuationToken that represents where this request began. + // The marker that is used as a starting point for this analytics configuration + // list response. This value is present if it was sent in the request. ContinuationToken *string `type:"string"` // Indicates whether the returned list of analytics configurations is complete. @@ -15661,7 +20403,7 @@ func (s *ListBucketAnalyticsConfigurationsOutput) SetNextContinuationToken(v str } type ListBucketInventoryConfigurationsInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"ListBucketInventoryConfigurationsRequest" type:"structure"` // The name of the bucket containing the inventory configurations to retrieve. // @@ -15720,6 +20462,20 @@ func (s *ListBucketInventoryConfigurationsInput) SetContinuationToken(v string) return s } +func (s *ListBucketInventoryConfigurationsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *ListBucketInventoryConfigurationsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type ListBucketInventoryConfigurationsOutput struct { _ struct{} `type:"structure"` @@ -15730,8 +20486,9 @@ type ListBucketInventoryConfigurationsOutput struct { // The list of inventory configurations for a bucket. InventoryConfigurationList []*InventoryConfiguration `locationName:"InventoryConfiguration" type:"list" flattened:"true"` - // Indicates whether the returned list of inventory configurations is truncated - // in this response. A value of true indicates that the list is truncated. + // Tells whether the returned list of inventory configurations is complete. + // A value of true indicates that the list is not complete and the NextContinuationToken + // is provided for a subsequent request. IsTruncated *bool `type:"boolean"` // The marker used to continue this inventory configuration listing. Use the @@ -15775,7 +20532,7 @@ func (s *ListBucketInventoryConfigurationsOutput) SetNextContinuationToken(v str } type ListBucketMetricsConfigurationsInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"ListBucketMetricsConfigurationsRequest" type:"structure"` // The name of the bucket containing the metrics configurations to retrieve. // @@ -15834,6 +20591,20 @@ func (s *ListBucketMetricsConfigurationsInput) SetContinuationToken(v string) *L return s } +func (s *ListBucketMetricsConfigurationsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *ListBucketMetricsConfigurationsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type ListBucketMetricsConfigurationsOutput struct { _ struct{} `type:"structure"` @@ -15907,8 +20678,10 @@ func (s ListBucketsInput) GoString() string { type ListBucketsOutput struct { _ struct{} `type:"structure"` + // The list of buckets owned by the requestor. Buckets []*Bucket `locationNameList:"Bucket" type:"list"` + // The owner of the buckets listed. Owner *Owner `type:"structure"` } @@ -15935,12 +20708,28 @@ func (s *ListBucketsOutput) SetOwner(v *Owner) *ListBucketsOutput { } type ListMultipartUploadsInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"ListMultipartUploadsRequest" type:"structure"` + // Name of the bucket to which the multipart upload was initiated. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` // Character you use to group keys. + // + // All keys that contain the same string between the prefix, if specified, and + // the first occurrence of the delimiter after the prefix are grouped under + // a single result element, CommonPrefixes. If you don't specify the prefix + // parameter, then the substring starts at the beginning of the key. The keys + // that are grouped under CommonPrefixes result element are not returned elsewhere + // in the response. Delimiter *string `location:"querystring" locationName:"delimiter" type:"string"` // Requests Amazon S3 to encode the object keys in the response and specifies @@ -15953,6 +20742,13 @@ type ListMultipartUploadsInput struct { // Together with upload-id-marker, this parameter specifies the multipart upload // after which listing should begin. + // + // If upload-id-marker is not specified, only the keys lexicographically greater + // than the specified key-marker will be included in the list. + // + // If upload-id-marker is specified, any multipart uploads for a key equal to + // the key-marker might also be included, provided those multipart uploads have + // upload IDs lexicographically greater than the specified upload-id-marker. KeyMarker *string `location:"querystring" locationName:"key-marker" type:"string"` // Sets the maximum number of multipart uploads, from 1 to 1,000, to return @@ -15961,12 +20757,16 @@ type ListMultipartUploadsInput struct { MaxUploads *int64 `location:"querystring" locationName:"max-uploads" type:"integer"` // Lists in-progress uploads only for those keys that begin with the specified - // prefix. + // prefix. You can use prefixes to separate a bucket into different grouping + // of keys. (You can think of using prefix to make groups in the same way you'd + // use a folder in a file system.) Prefix *string `location:"querystring" locationName:"prefix" type:"string"` // Together with key-marker, specifies the multipart upload after which listing // should begin. If key-marker is not specified, the upload-id-marker parameter - // is ignored. + // is ignored. Otherwise, any multipart uploads for a key equal to the key-marker + // might be included in the list only if they have an upload ID lexicographically + // greater than the specified upload-id-marker. UploadIdMarker *string `location:"querystring" locationName:"upload-id-marker" type:"string"` } @@ -16045,17 +20845,42 @@ func (s *ListMultipartUploadsInput) SetUploadIdMarker(v string) *ListMultipartUp return s } +func (s *ListMultipartUploadsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *ListMultipartUploadsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type ListMultipartUploadsOutput struct { _ struct{} `type:"structure"` // Name of the bucket to which the multipart upload was initiated. Bucket *string `type:"string"` + // If you specify a delimiter in the request, then the result returns each distinct + // key prefix containing the delimiter in a CommonPrefixes element. The distinct + // key prefixes are returned in the Prefix child element. CommonPrefixes []*CommonPrefix `type:"list" flattened:"true"` + // Contains the delimiter you specified in the request. If you don't specify + // a delimiter in your request, this element is absent from the response. Delimiter *string `type:"string"` // Encoding type used by Amazon S3 to encode object keys in the response. + // + // If you specify encoding-type request parameter, Amazon S3 includes this element + // in the response, and returns encoded key name values in the following response + // elements: + // + // Delimiter, KeyMarker, Prefix, NextKeyMarker, Key. EncodingType *string `type:"string" enum:"EncodingType"` // Indicates whether the returned list of multipart uploads is truncated. A @@ -16086,6 +20911,8 @@ type ListMultipartUploadsOutput struct { // Upload ID after which listing began. UploadIdMarker *string `type:"string"` + // Container for elements related to a particular multipart upload. A response + // can contain zero or more Upload elements. Uploads []*MultipartUpload `locationName:"Upload" type:"list" flattened:"true"` } @@ -16179,12 +21006,25 @@ func (s *ListMultipartUploadsOutput) SetUploads(v []*MultipartUpload) *ListMulti } type ListObjectVersionsInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"ListObjectVersionsRequest" type:"structure"` + // The bucket name that contains the objects. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` - // A delimiter is a character you use to group keys. + // A delimiter is a character that you specify to group keys. All keys that + // contain the same string between the prefix and the first occurrence of the + // delimiter are grouped under a single result element in CommonPrefixes. These + // groups are counted as one result against the max-keys limitation. These keys + // are not returned elsewhere in the response. Delimiter *string `location:"querystring" locationName:"delimiter" type:"string"` // Requests Amazon S3 to encode the object keys in the response and specifies @@ -16198,11 +21038,19 @@ type ListObjectVersionsInput struct { // Specifies the key to start with when listing objects in a bucket. KeyMarker *string `location:"querystring" locationName:"key-marker" type:"string"` - // Sets the maximum number of keys returned in the response. The response might - // contain fewer keys but will never contain more. + // Sets the maximum number of keys returned in the response. By default the + // API returns up to 1,000 key names. The response might contain fewer keys + // but will never contain more. If additional keys satisfy the search criteria, + // but were not returned because max-keys was exceeded, the response contains + // true. To return the additional keys, see key-marker + // and version-id-marker. MaxKeys *int64 `location:"querystring" locationName:"max-keys" type:"integer"` - // Limits the response to keys that begin with the specified prefix. + // Use this parameter to select only those keys that begin with the specified + // prefix. You can use prefixes to separate a bucket into different groupings + // of keys. (You can think of using prefix to make groups in the same way you'd + // use a folder in a file system.) You can use prefix with delimiter to roll + // up numerous objects into a single result under CommonPrefixes. Prefix *string `location:"querystring" locationName:"prefix" type:"string"` // Specifies the object version you want to start listing from. @@ -16284,42 +21132,81 @@ func (s *ListObjectVersionsInput) SetVersionIdMarker(v string) *ListObjectVersio return s } +func (s *ListObjectVersionsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *ListObjectVersionsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type ListObjectVersionsOutput struct { _ struct{} `type:"structure"` + // All of the keys rolled up into a common prefix count as a single return when + // calculating the number of returns. CommonPrefixes []*CommonPrefix `type:"list" flattened:"true"` + // Container for an object that is a delete marker. DeleteMarkers []*DeleteMarkerEntry `locationName:"DeleteMarker" type:"list" flattened:"true"` + // The delimiter grouping the included keys. A delimiter is a character that + // you specify to group keys. All keys that contain the same string between + // the prefix and the first occurrence of the delimiter are grouped under a + // single result element in CommonPrefixes. These groups are counted as one + // result against the max-keys limitation. These keys are not returned elsewhere + // in the response. Delimiter *string `type:"string"` - // Encoding type used by Amazon S3 to encode object keys in the response. + // Encoding type used by Amazon S3 to encode object key names in the XML response. + // + // If you specify encoding-type request parameter, Amazon S3 includes this element + // in the response, and returns encoded key name values in the following response + // elements: + // + // KeyMarker, NextKeyMarker, Prefix, Key, and Delimiter. EncodingType *string `type:"string" enum:"EncodingType"` - // A flag that indicates whether or not Amazon S3 returned all of the results - // that satisfied the search criteria. If your results were truncated, you can - // make a follow-up paginated request using the NextKeyMarker and NextVersionIdMarker + // A flag that indicates whether Amazon S3 returned all of the results that + // satisfied the search criteria. If your results were truncated, you can make + // a follow-up paginated request using the NextKeyMarker and NextVersionIdMarker // response parameters as a starting place in another request to return the // rest of the results. IsTruncated *bool `type:"boolean"` - // Marks the last Key returned in a truncated response. + // Marks the last key returned in a truncated response. KeyMarker *string `type:"string"` + // Specifies the maximum number of objects to return. MaxKeys *int64 `type:"integer"` + // Bucket name. Name *string `type:"string"` - // Use this value for the key marker request parameter in a subsequent request. + // When the number of responses exceeds the value of MaxKeys, NextKeyMarker + // specifies the first key not returned that satisfies the search criteria. + // Use this value for the key-marker request parameter in a subsequent request. NextKeyMarker *string `type:"string"` - // Use this value for the next version id marker parameter in a subsequent request. + // When the number of responses exceeds the value of MaxKeys, NextVersionIdMarker + // specifies the first object version not returned that satisfies the search + // criteria. Use this value for the version-id-marker request parameter in a + // subsequent request. NextVersionIdMarker *string `type:"string"` + // Selects objects that start with the value supplied by this parameter. Prefix *string `type:"string"` + // Marks the last version of the key returned in a truncated response. VersionIdMarker *string `type:"string"` + // Container for version information. Versions []*ObjectVersion `locationName:"Version" type:"list" flattened:"true"` } @@ -16412,8 +21299,10 @@ func (s *ListObjectVersionsOutput) SetVersions(v []*ObjectVersion) *ListObjectVe } type ListObjectsInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"ListObjectsRequest" type:"structure"` + // The name of the bucket containing the objects. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -16431,8 +21320,9 @@ type ListObjectsInput struct { // Specifies the key to start with when listing objects in a bucket. Marker *string `location:"querystring" locationName:"marker" type:"string"` - // Sets the maximum number of keys returned in the response. The response might - // contain fewer keys but will never contain more. + // Sets the maximum number of keys returned in the response. By default the + // API returns up to 1,000 key names. The response might contain fewer keys + // but will never contain more. MaxKeys *int64 `location:"querystring" locationName:"max-keys" type:"integer"` // Limits the response to keys that begin with the specified prefix. @@ -16519,26 +21409,65 @@ func (s *ListObjectsInput) SetRequestPayer(v string) *ListObjectsInput { return s } +func (s *ListObjectsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *ListObjectsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type ListObjectsOutput struct { _ struct{} `type:"structure"` + // All of the keys rolled up in a common prefix count as a single return when + // calculating the number of returns. + // + // A response can contain CommonPrefixes only if you specify a delimiter. + // + // CommonPrefixes contains all (if there are any) keys between Prefix and the + // next occurrence of the string specified by the delimiter. + // + // CommonPrefixes lists keys that act like subdirectories in the directory specified + // by Prefix. + // + // For example, if the prefix is notes/ and the delimiter is a slash (/) as + // in notes/summer/july, the common prefix is notes/summer/. All of the keys + // that roll up into a common prefix count as a single return when calculating + // the number of returns. CommonPrefixes []*CommonPrefix `type:"list" flattened:"true"` + // Metadata about each object returned. Contents []*Object `type:"list" flattened:"true"` + // Causes keys that contain the same string between the prefix and the first + // occurrence of the delimiter to be rolled up into a single result element + // in the CommonPrefixes collection. These rolled-up keys are not returned elsewhere + // in the response. Each rolled-up result counts as only one return against + // the MaxKeys value. Delimiter *string `type:"string"` // Encoding type used by Amazon S3 to encode object keys in the response. EncodingType *string `type:"string" enum:"EncodingType"` - // A flag that indicates whether or not Amazon S3 returned all of the results - // that satisfied the search criteria. + // A flag that indicates whether Amazon S3 returned all of the results that + // satisfied the search criteria. IsTruncated *bool `type:"boolean"` + // Indicates where in the bucket listing begins. Marker is included in the response + // if it was sent with the request. Marker *string `type:"string"` + // The maximum number of keys returned in the response body. MaxKeys *int64 `type:"integer"` + // Bucket name. Name *string `type:"string"` // When response is truncated (the IsTruncated element value in the response @@ -16550,6 +21479,7 @@ type ListObjectsOutput struct { // subsequent request to get the next set of object keys. NextMarker *string `type:"string"` + // Keys that begin with the indicated prefix. Prefix *string `type:"string"` } @@ -16624,16 +21554,23 @@ func (s *ListObjectsOutput) SetPrefix(v string) *ListObjectsOutput { } type ListObjectsV2Input struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"ListObjectsV2Request" type:"structure"` - // Name of the bucket to list. + // Bucket name to list. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` // ContinuationToken indicates Amazon S3 that the list is being continued on // this bucket with a token. ContinuationToken is obfuscated and is not a real - // key + // key. ContinuationToken *string `location:"querystring" locationName:"continuation-token" type:"string"` // A delimiter is a character you use to group keys. @@ -16644,11 +21581,12 @@ type ListObjectsV2Input struct { // The owner field is not present in listV2 by default, if you want to return // owner field with each key in the result then set the fetch owner field to - // true + // true. FetchOwner *bool `location:"querystring" locationName:"fetch-owner" type:"boolean"` - // Sets the maximum number of keys returned in the response. The response might - // contain fewer keys but will never contain more. + // Sets the maximum number of keys returned in the response. By default the + // API returns up to 1,000 key names. The response might contain fewer keys + // but will never contain more. MaxKeys *int64 `location:"querystring" locationName:"max-keys" type:"integer"` // Limits the response to keys that begin with the specified prefix. @@ -16660,7 +21598,7 @@ type ListObjectsV2Input struct { RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // StartAfter is where you want Amazon S3 to start listing from. Amazon S3 starts - // listing after this specified key. StartAfter can be any key in the bucket + // listing after this specified key. StartAfter can be any key in the bucket. StartAfter *string `location:"querystring" locationName:"start-after" type:"string"` } @@ -16751,29 +21689,65 @@ func (s *ListObjectsV2Input) SetStartAfter(v string) *ListObjectsV2Input { return s } +func (s *ListObjectsV2Input) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *ListObjectsV2Input) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type ListObjectsV2Output struct { _ struct{} `type:"structure"` + // All of the keys rolled up into a common prefix count as a single return when + // calculating the number of returns. + // + // A response can contain CommonPrefixes only if you specify a delimiter. + // // CommonPrefixes contains all (if there are any) keys between Prefix and the - // next occurrence of the string specified by delimiter + // next occurrence of the string specified by a delimiter. + // + // CommonPrefixes lists keys that act like subdirectories in the directory specified + // by Prefix. + // + // For example, if the prefix is notes/ and the delimiter is a slash (/) as + // in notes/summer/july, the common prefix is notes/summer/. All of the keys + // that roll up into a common prefix count as a single return when calculating + // the number of returns. CommonPrefixes []*CommonPrefix `type:"list" flattened:"true"` // Metadata about each object returned. Contents []*Object `type:"list" flattened:"true"` - // ContinuationToken indicates Amazon S3 that the list is being continued on - // this bucket with a token. ContinuationToken is obfuscated and is not a real - // key + // If ContinuationToken was sent with the request, it is included in the response. ContinuationToken *string `type:"string"` - // A delimiter is a character you use to group keys. + // Causes keys that contain the same string between the prefix and the first + // occurrence of the delimiter to be rolled up into a single result element + // in the CommonPrefixes collection. These rolled-up keys are not returned elsewhere + // in the response. Each rolled-up result counts as only one return against + // the MaxKeys value. Delimiter *string `type:"string"` - // Encoding type used by Amazon S3 to encode object keys in the response. + // Encoding type used by Amazon S3 to encode object key names in the XML response. + // + // If you specify the encoding-type request parameter, Amazon S3 includes this + // element in the response, and returns encoded key name values in the following + // response elements: + // + // Delimiter, Prefix, Key, and StartAfter. EncodingType *string `type:"string" enum:"EncodingType"` - // A flag that indicates whether or not Amazon S3 returned all of the results - // that satisfied the search criteria. + // Set to false if all of the results were returned. Set to true if more keys + // are available to return. If the number of results exceeds that specified + // by MaxKeys, all of the results might not be returned. IsTruncated *bool `type:"boolean"` // KeyCount is the number of keys returned with this request. KeyCount will @@ -16781,24 +21755,31 @@ type ListObjectsV2Output struct { // result will include less than equals 50 keys KeyCount *int64 `type:"integer"` - // Sets the maximum number of keys returned in the response. The response might - // contain fewer keys but will never contain more. + // Sets the maximum number of keys returned in the response. By default the + // API returns up to 1,000 key names. The response might contain fewer keys + // but will never contain more. MaxKeys *int64 `type:"integer"` - // Name of the bucket to list. + // Bucket name. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. Name *string `type:"string"` - // NextContinuationToken is sent when isTruncated is true which means there + // NextContinuationToken is sent when isTruncated is true, which means there // are more keys in the bucket that can be listed. The next list requests to // Amazon S3 can be continued with this NextContinuationToken. NextContinuationToken // is obfuscated and is not a real key NextContinuationToken *string `type:"string"` - // Limits the response to keys that begin with the specified prefix. + // Keys that begin with the indicated prefix. Prefix *string `type:"string"` - // StartAfter is where you want Amazon S3 to start listing from. Amazon S3 starts - // listing after this specified key. StartAfter can be any key in the bucket + // If StartAfter was sent with the request, it is included in the response. StartAfter *string `type:"string"` } @@ -16885,11 +21866,22 @@ func (s *ListObjectsV2Output) SetStartAfter(v string) *ListObjectsV2Output { } type ListPartsInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"ListPartsRequest" type:"structure"` + // Name of the bucket to which the parts are being uploaded. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Object key for which the multipart upload was initiated. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` @@ -16900,10 +21892,11 @@ type ListPartsInput struct { // part numbers will be listed. PartNumberMarker *int64 `location:"querystring" locationName:"part-number-marker" type:"integer"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // Upload ID identifying the multipart upload whose parts are being listed. @@ -16990,23 +21983,51 @@ func (s *ListPartsInput) SetUploadId(v string) *ListPartsInput { return s } +func (s *ListPartsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *ListPartsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type ListPartsOutput struct { _ struct{} `type:"structure"` - // Date when multipart upload will become eligible for abort operation by lifecycle. + // If the bucket has a lifecycle rule configured with an action to abort incomplete + // multipart uploads and the prefix in the lifecycle rule matches the object + // name in the request, then the response includes this header indicating when + // the initiated multipart upload will become eligible for abort operation. + // For more information, see Aborting Incomplete Multipart Uploads Using a Bucket + // Lifecycle Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-abort-incomplete-mpu-lifecycle-config). + // + // The response will also include the x-amz-abort-rule-id header that will provide + // the ID of the lifecycle configuration rule that defines this action. AbortDate *time.Time `location:"header" locationName:"x-amz-abort-date" type:"timestamp"` - // Id of the lifecycle rule that makes a multipart upload eligible for abort - // operation. + // This header is returned along with the x-amz-abort-date header. It identifies + // applicable lifecycle configuration rule that defines the action to abort + // incomplete multipart uploads. AbortRuleId *string `location:"header" locationName:"x-amz-abort-rule-id" type:"string"` // Name of the bucket to which the multipart upload was initiated. Bucket *string `type:"string"` - // Identifies who initiated the multipart upload. + // Container element that identifies who initiated the multipart upload. If + // the initiator is an AWS account, this element provides the same information + // as the Owner element. If the initiator is an IAM User, this element provides + // the user ARN and display name. Initiator *Initiator `type:"structure"` - // Indicates whether the returned list of parts is truncated. + // Indicates whether the returned list of parts is truncated. A true value indicates + // that the list was truncated. A list can be truncated if the number of parts + // exceeds the limit returned in the MaxParts element. IsTruncated *bool `type:"boolean"` // Object key for which the multipart upload was initiated. @@ -17020,18 +22041,26 @@ type ListPartsOutput struct { // in a subsequent request. NextPartNumberMarker *int64 `type:"integer"` + // Container element that identifies the object owner, after the object is created. + // If multipart upload is initiated by an IAM user, this element provides the + // parent account ID and display name. Owner *Owner `type:"structure"` - // Part number after which listing begins. + // When a list is truncated, this element specifies the last part in the list, + // as well as the value to use for the part-number-marker request parameter + // in a subsequent request. PartNumberMarker *int64 `type:"integer"` + // Container for elements related to a particular part. A response can contain + // zero or more Part elements. Parts []*Part `locationName:"Part" type:"list" flattened:"true"` // If present, indicates that the requester was successfully charged for the // request. RequestCharged *string `location:"header" locationName:"x-amz-request-charged" type:"string" enum:"RequestCharged"` - // The class of storage used to store the object. + // Class of storage (STANDARD or REDUCED_REDUNDANCY) used to store the uploaded + // object. StorageClass *string `type:"string" enum:"StorageClass"` // Upload ID identifying the multipart upload whose parts are being listed. @@ -17139,7 +22168,8 @@ func (s *ListPartsOutput) SetUploadId(v string) *ListPartsOutput { return s } -// Describes an S3 location that will receive the results of the restore request. +// Describes an Amazon S3 location that will receive the results of the restore +// request. type Location struct { _ struct{} `type:"structure"` @@ -17154,8 +22184,7 @@ type Location struct { // The canned ACL to apply to the restore results. CannedACL *string `type:"string" enum:"ObjectCannedACL"` - // Describes the server-side encryption that will be applied to the restore - // results. + // Contains the type of server-side encryption used. Encryption *Encryption `type:"structure"` // The prefix that is prepended to the restore results for this request. @@ -17267,26 +22296,29 @@ func (s *Location) SetUserMetadata(v []*MetadataEntry) *Location { return s } -// Container for logging information. Presence of this element indicates that -// logging is enabled. Parameters TargetBucket and TargetPrefix are required -// in this case. +// Describes where logs are stored and the prefix that Amazon S3 assigns to +// all log object keys for a bucket. For more information, see PUT Bucket logging +// (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTlogging.html) +// in the Amazon Simple Storage Service API Reference. type LoggingEnabled struct { _ struct{} `type:"structure"` // Specifies the bucket where you want Amazon S3 to store server access logs. // You can have your logs delivered to any bucket that you own, including the // same bucket that is being logged. You can also configure multiple buckets - // to deliver their logs to the same target bucket. In this case you should + // to deliver their logs to the same target bucket. In this case, you should // choose a different TargetPrefix for each source bucket so that the delivered // log files can be distinguished by key. // // TargetBucket is a required field TargetBucket *string `type:"string" required:"true"` + // Container for granting information. TargetGrants []*TargetGrant `locationNameList:"Grant" type:"list"` - // This element lets you specify a prefix for the keys that the log files will - // be stored under. + // A prefix for all log object keys. If you store log files from multiple Amazon + // S3 buckets in a single bucket, you can use a prefix to distinguish which + // log files came from which bucket. // // TargetPrefix is a required field TargetPrefix *string `type:"string" required:"true"` @@ -17350,8 +22382,10 @@ func (s *LoggingEnabled) SetTargetPrefix(v string) *LoggingEnabled { type MetadataEntry struct { _ struct{} `type:"structure"` + // Name of the Object. Name *string `type:"string"` + // Value of the Object. Value *string `type:"string"` } @@ -17377,6 +22411,65 @@ func (s *MetadataEntry) SetValue(v string) *MetadataEntry { return s } +// A container specifying replication metrics-related settings enabling metrics +// and Amazon S3 events for S3 Replication Time Control (S3 RTC). Must be specified +// together with a ReplicationTime block. +type Metrics struct { + _ struct{} `type:"structure"` + + // A container specifying the time threshold for emitting the s3:Replication:OperationMissedThreshold + // event. + // + // EventThreshold is a required field + EventThreshold *ReplicationTimeValue `type:"structure" required:"true"` + + // Specifies whether the replication metrics are enabled. + // + // Status is a required field + Status *string `type:"string" required:"true" enum:"MetricsStatus"` +} + +// String returns the string representation +func (s Metrics) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s Metrics) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *Metrics) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "Metrics"} + if s.EventThreshold == nil { + invalidParams.Add(request.NewErrParamRequired("EventThreshold")) + } + if s.Status == nil { + invalidParams.Add(request.NewErrParamRequired("Status")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetEventThreshold sets the EventThreshold field's value. +func (s *Metrics) SetEventThreshold(v *ReplicationTimeValue) *Metrics { + s.EventThreshold = v + return s +} + +// SetStatus sets the Status field's value. +func (s *Metrics) SetStatus(v string) *Metrics { + s.Status = &v + return s +} + +// A conjunction (logical AND) of predicates, which is used in evaluating a +// metrics filter. The operator must have at least two predicates, and an object +// must match all of the predicates in order for the filter to apply. type MetricsAndOperator struct { _ struct{} `type:"structure"` @@ -17429,6 +22522,13 @@ func (s *MetricsAndOperator) SetTags(v []*Tag) *MetricsAndOperator { return s } +// Specifies a metrics configuration for the CloudWatch request metrics (specified +// by the metrics configuration ID) from an Amazon S3 bucket. If you're updating +// an existing metrics configuration, note that this is a full replacement of +// the existing metrics configuration. If you don't include the elements you +// want to keep, they are erased. For more information, see PUT Bucket metrics +// (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTMetricConfiguration.html) +// in the Amazon Simple Storage Service API Reference. type MetricsConfiguration struct { _ struct{} `type:"structure"` @@ -17483,6 +22583,9 @@ func (s *MetricsConfiguration) SetId(v string) *MetricsConfiguration { return s } +// Specifies a metrics configuration filter. The metrics configuration only +// includes objects that meet the filter's criteria. A filter must be a prefix, +// a tag, or a conjunction (MetricsAndOperator). type MetricsFilter struct { _ struct{} `type:"structure"` @@ -17546,6 +22649,7 @@ func (s *MetricsFilter) SetTag(v *Tag) *MetricsFilter { return s } +// Container for the MultipartUpload for the Amazon S3 object. type MultipartUpload struct { _ struct{} `type:"structure"` @@ -17558,6 +22662,7 @@ type MultipartUpload struct { // Key of the object for which the multipart upload was initiated. Key *string `min:"1" type:"string"` + // Specifies the owner of the object that is part of the multipart upload. Owner *Owner `type:"structure"` // The class of storage used to store the object. @@ -17624,8 +22729,8 @@ type NoncurrentVersionExpiration struct { // Specifies the number of days an object is noncurrent before Amazon S3 can // perform the associated action. For information about the noncurrent days // calculations, see How Amazon S3 Calculates When an Object Became Noncurrent - // (http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html) in - // the Amazon Simple Storage Service Developer Guide. + // (https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html#non-current-days-calculations) + // in the Amazon Simple Storage Service Developer Guide. NoncurrentDays *int64 `type:"integer"` } @@ -17646,19 +22751,20 @@ func (s *NoncurrentVersionExpiration) SetNoncurrentDays(v int64) *NoncurrentVers } // Container for the transition rule that describes when noncurrent objects -// transition to the STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING or GLACIER -// storage class. If your bucket is versioning-enabled (or versioning is suspended), -// you can set this action to request that Amazon S3 transition noncurrent object -// versions to the STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING or GLACIER storage -// class at a specific period in the object's lifetime. +// transition to the STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, +// or DEEP_ARCHIVE storage class. If your bucket is versioning-enabled (or versioning +// is suspended), you can set this action to request that Amazon S3 transition +// noncurrent object versions to the STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, +// GLACIER, or DEEP_ARCHIVE storage class at a specific period in the object's +// lifetime. type NoncurrentVersionTransition struct { _ struct{} `type:"structure"` // Specifies the number of days an object is noncurrent before Amazon S3 can // perform the associated action. For information about the noncurrent days - // calculations, see How Amazon S3 Calculates When an Object Became Noncurrent - // (http://docs.aws.amazon.com/AmazonS3/latest/dev/s3-access-control.html) in - // the Amazon Simple Storage Service Developer Guide. + // calculations, see How Amazon S3 Calculates How Long an Object Has Been Noncurrent + // (https://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html#non-current-days-calculations) + // in the Amazon Simple Storage Service Developer Guide. NoncurrentDays *int64 `type:"integer"` // The class of storage used to store the object. @@ -17692,10 +22798,16 @@ func (s *NoncurrentVersionTransition) SetStorageClass(v string) *NoncurrentVersi type NotificationConfiguration struct { _ struct{} `type:"structure"` + // Describes the AWS Lambda functions to invoke and the events for which to + // invoke them. LambdaFunctionConfigurations []*LambdaFunctionConfiguration `locationName:"CloudFunctionConfiguration" type:"list" flattened:"true"` + // The Amazon Simple Queue Service queues to publish messages to and the events + // for which to publish messages. QueueConfigurations []*QueueConfiguration `locationName:"QueueConfiguration" type:"list" flattened:"true"` + // The topic to which notifications are sent and the events for which notifications + // are generated. TopicConfigurations []*TopicConfiguration `locationName:"TopicConfiguration" type:"list" flattened:"true"` } @@ -17770,10 +22882,17 @@ func (s *NotificationConfiguration) SetTopicConfigurations(v []*TopicConfigurati type NotificationConfigurationDeprecated struct { _ struct{} `type:"structure"` + // Container for specifying the AWS Lambda notification configuration. CloudFunctionConfiguration *CloudFunctionConfiguration `type:"structure"` + // This data type is deprecated. This data type specifies the configuration + // for publishing messages to an Amazon Simple Queue Service (Amazon SQS) queue + // when Amazon S3 detects specified events. QueueConfiguration *QueueConfigurationDeprecated `type:"structure"` + // This data type is deprecated. A container for specifying the configuration + // for publication of messages to an Amazon Simple Notification Service (Amazon + // SNS) topic when Amazon S3 detects specified events. TopicConfiguration *TopicConfigurationDeprecated `type:"structure"` } @@ -17805,8 +22924,8 @@ func (s *NotificationConfigurationDeprecated) SetTopicConfiguration(v *TopicConf return s } -// A container for object key name filtering rules. For information about key -// name filtering, see Configuring Event Notifications (http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) +// Specifies object key name filtering rules. For information about key name +// filtering, see Configuring Event Notifications (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) // in the Amazon Simple Storage Service Developer Guide. type NotificationConfigurationFilter struct { _ struct{} `type:"structure"` @@ -17831,17 +22950,25 @@ func (s *NotificationConfigurationFilter) SetKey(v *KeyFilter) *NotificationConf return s } +// An object consists of data and its descriptive metadata. type Object struct { _ struct{} `type:"structure"` + // The entity tag is an MD5 hash of the object. ETag reflects only changes to + // the contents of an object, not its metadata. ETag *string `type:"string"` + // The name that you assign to an object. You use the object key to retrieve + // the object. Key *string `min:"1" type:"string"` + // The date the Object was Last Modified LastModified *time.Time `type:"timestamp"` + // The owner of the object Owner *Owner `type:"structure"` + // Size in bytes of the object Size *int64 `type:"integer"` // The class of storage used to store the object. @@ -17894,6 +23021,7 @@ func (s *Object) SetStorageClass(v string) *Object { return s } +// Object Identifier is unique value to identify objects. type ObjectIdentifier struct { _ struct{} `type:"structure"` @@ -18059,9 +23187,11 @@ func (s *ObjectLockRule) SetDefaultRetention(v *DefaultRetention) *ObjectLockRul return s } +// The version of an object. type ObjectVersion struct { _ struct{} `type:"structure"` + // The entity tag is an MD5 hash of that version of the object. ETag *string `type:"string"` // Specifies whether the object is (true) or is not (false) the latest version @@ -18074,6 +23204,7 @@ type ObjectVersion struct { // Date and time the object was last modified. LastModified *time.Time `type:"timestamp"` + // Specifies the owner of the object. Owner *Owner `type:"structure"` // Size in bytes of the object. @@ -18216,11 +23347,14 @@ func (s *OutputSerialization) SetJSON(v *JSONOutput) *OutputSerialization { return s } +// Container for the owner's display name and ID. type Owner struct { _ struct{} `type:"structure"` + // Container for the display name of the owner. DisplayName *string `type:"string"` + // Container for the ID of the owner. ID *string `type:"string"` } @@ -18246,6 +23380,7 @@ func (s *Owner) SetID(v string) *Owner { return s } +// Container for Parquet. type ParquetInput struct { _ struct{} `type:"structure"` } @@ -18260,6 +23395,7 @@ func (s ParquetInput) GoString() string { return s.String() } +// Container for elements related to a part. type Part struct { _ struct{} `type:"structure"` @@ -18336,6 +23472,7 @@ func (s *PolicyStatus) SetIsPublic(v bool) *PolicyStatus { return s } +// This data type contains information about progress of an operation. type Progress struct { _ struct{} `type:"structure"` @@ -18377,6 +23514,7 @@ func (s *Progress) SetBytesScanned(v int64) *Progress { return s } +// This data type contains information about the progress event of an operation. type ProgressEvent struct { _ struct{} `locationName:"ProgressEvent" type:"structure" payload:"Details"` @@ -18417,6 +23555,23 @@ func (s *ProgressEvent) UnmarshalEvent( return nil } +// MarshalEvent marshals the type into an stream event value. This method +// should only used internally within the SDK's EventStream handling. +func (s *ProgressEvent) MarshalEvent(pm protocol.PayloadMarshaler) (msg eventstream.Message, err error) { + msg.Headers.Set(eventstreamapi.MessageTypeHeader, eventstream.StringValue(eventstreamapi.EventMessageType)) + var buf bytes.Buffer + if err = pm.MarshalPayload(&buf, s); err != nil { + return eventstream.Message{}, err + } + msg.Payload = buf.Bytes() + return msg, err +} + +// The PublicAccessBlock configuration that you want to apply to this Amazon +// S3 bucket. You can enable the configuration options in any combination. For +// more information about when Amazon S3 considers a bucket or object public, +// see The Meaning of "Public" (https://docs.aws.amazon.com/AmazonS3/latest/dev/access-control-block-public-access.html#access-control-block-public-access-policy-status) +// in the Amazon Simple Storage Service Developer Guide. type PublicAccessBlockConfiguration struct { _ struct{} `type:"structure"` @@ -18429,6 +23584,8 @@ type PublicAccessBlockConfiguration struct { // // * PUT Object calls fail if the request includes a public ACL. // + // * PUT Bucket calls fail if the request includes a public ACL. + // // Enabling this setting doesn't affect existing policies or ACLs. BlockPublicAcls *bool `locationName:"BlockPublicAcls" type:"boolean"` @@ -18493,9 +23650,9 @@ func (s *PublicAccessBlockConfiguration) SetRestrictPublicBuckets(v bool) *Publi } type PutBucketAccelerateConfigurationInput struct { - _ struct{} `type:"structure" payload:"AccelerateConfiguration"` + _ struct{} `locationName:"PutBucketAccelerateConfigurationRequest" type:"structure" payload:"AccelerateConfiguration"` - // Specifies the Accelerate Configuration you want to set for the bucket. + // Container for setting the transfer acceleration state. // // AccelerateConfiguration is a required field AccelerateConfiguration *AccelerateConfiguration `locationName:"AccelerateConfiguration" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` @@ -18554,6 +23711,20 @@ func (s *PutBucketAccelerateConfigurationInput) getBucket() (v string) { return *s.Bucket } +func (s *PutBucketAccelerateConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketAccelerateConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketAccelerateConfigurationOutput struct { _ struct{} `type:"structure"` } @@ -18569,13 +23740,16 @@ func (s PutBucketAccelerateConfigurationOutput) GoString() string { } type PutBucketAclInput struct { - _ struct{} `type:"structure" payload:"AccessControlPolicy"` + _ struct{} `locationName:"PutBucketAclRequest" type:"structure" payload:"AccessControlPolicy"` // The canned ACL to apply to the bucket. ACL *string `location:"header" locationName:"x-amz-acl" type:"string" enum:"BucketCannedACL"` + // Contains the elements that set the ACL permissions for an object per grantee. AccessControlPolicy *AccessControlPolicy `locationName:"AccessControlPolicy" type:"structure" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` + // The bucket to which to apply the ACL. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -18682,6 +23856,20 @@ func (s *PutBucketAclInput) SetGrantWriteACP(v string) *PutBucketAclInput { return s } +func (s *PutBucketAclInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketAclInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketAclOutput struct { _ struct{} `type:"structure"` } @@ -18697,7 +23885,7 @@ func (s PutBucketAclOutput) GoString() string { } type PutBucketAnalyticsConfigurationInput struct { - _ struct{} `type:"structure" payload:"AnalyticsConfiguration"` + _ struct{} `locationName:"PutBucketAnalyticsConfigurationRequest" type:"structure" payload:"AnalyticsConfiguration"` // The configuration and any analyses for the analytics filter. // @@ -18709,7 +23897,7 @@ type PutBucketAnalyticsConfigurationInput struct { // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` - // The identifier used to represent an analytics configuration. + // The ID that identifies the analytics configuration. // // Id is a required field Id *string `location:"querystring" locationName:"id" type:"string" required:"true"` @@ -18777,6 +23965,20 @@ func (s *PutBucketAnalyticsConfigurationInput) SetId(v string) *PutBucketAnalyti return s } +func (s *PutBucketAnalyticsConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketAnalyticsConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketAnalyticsConfigurationOutput struct { _ struct{} `type:"structure"` } @@ -18792,11 +23994,18 @@ func (s PutBucketAnalyticsConfigurationOutput) GoString() string { } type PutBucketCorsInput struct { - _ struct{} `type:"structure" payload:"CORSConfiguration"` + _ struct{} `locationName:"PutBucketCorsRequest" type:"structure" payload:"CORSConfiguration"` + // Specifies the bucket impacted by the corsconfiguration. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Describes the cross-origin access configuration for objects in an Amazon + // S3 bucket. For more information, see Enabling Cross-Origin Resource Sharing + // (https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html) in the Amazon + // Simple Storage Service Developer Guide. + // // CORSConfiguration is a required field CORSConfiguration *CORSConfiguration `locationName:"CORSConfiguration" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` } @@ -18854,6 +24063,20 @@ func (s *PutBucketCorsInput) SetCORSConfiguration(v *CORSConfiguration) *PutBuck return s } +func (s *PutBucketCorsInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketCorsInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketCorsOutput struct { _ struct{} `type:"structure"` } @@ -18869,16 +24092,18 @@ func (s PutBucketCorsOutput) GoString() string { } type PutBucketEncryptionInput struct { - _ struct{} `type:"structure" payload:"ServerSideEncryptionConfiguration"` + _ struct{} `locationName:"PutBucketEncryptionRequest" type:"structure" payload:"ServerSideEncryptionConfiguration"` - // The name of the bucket for which the server-side encryption configuration - // is set. + // Specifies default encryption for a bucket using server-side encryption with + // Amazon S3-managed keys (SSE-S3) or customer master keys stored in AWS KMS + // (SSE-KMS). For information about the Amazon S3 default encryption feature, + // see Amazon S3 Default Bucket Encryption (https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html) + // in the Amazon Simple Storage Service Developer Guide. // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` - // Container for server-side encryption configuration rules. Currently S3 supports - // one rule only. + // Specifies the default server-side-encryption configuration. // // ServerSideEncryptionConfiguration is a required field ServerSideEncryptionConfiguration *ServerSideEncryptionConfiguration `locationName:"ServerSideEncryptionConfiguration" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` @@ -18937,6 +24162,20 @@ func (s *PutBucketEncryptionInput) SetServerSideEncryptionConfiguration(v *Serve return s } +func (s *PutBucketEncryptionInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketEncryptionInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketEncryptionOutput struct { _ struct{} `type:"structure"` } @@ -18952,7 +24191,7 @@ func (s PutBucketEncryptionOutput) GoString() string { } type PutBucketInventoryConfigurationInput struct { - _ struct{} `type:"structure" payload:"InventoryConfiguration"` + _ struct{} `locationName:"PutBucketInventoryConfigurationRequest" type:"structure" payload:"InventoryConfiguration"` // The name of the bucket where the inventory configuration will be stored. // @@ -19032,6 +24271,20 @@ func (s *PutBucketInventoryConfigurationInput) SetInventoryConfiguration(v *Inve return s } +func (s *PutBucketInventoryConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketInventoryConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketInventoryConfigurationOutput struct { _ struct{} `type:"structure"` } @@ -19047,11 +24300,14 @@ func (s PutBucketInventoryConfigurationOutput) GoString() string { } type PutBucketLifecycleConfigurationInput struct { - _ struct{} `type:"structure" payload:"LifecycleConfiguration"` + _ struct{} `locationName:"PutBucketLifecycleConfigurationRequest" type:"structure" payload:"LifecycleConfiguration"` + // The name of the bucket for which to set the configuration. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Container for lifecycle rules. You can add as many as 1,000 rules. LifecycleConfiguration *BucketLifecycleConfiguration `locationName:"LifecycleConfiguration" type:"structure" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` } @@ -19105,6 +24361,20 @@ func (s *PutBucketLifecycleConfigurationInput) SetLifecycleConfiguration(v *Buck return s } +func (s *PutBucketLifecycleConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketLifecycleConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketLifecycleConfigurationOutput struct { _ struct{} `type:"structure"` } @@ -19120,11 +24390,12 @@ func (s PutBucketLifecycleConfigurationOutput) GoString() string { } type PutBucketLifecycleInput struct { - _ struct{} `type:"structure" payload:"LifecycleConfiguration"` + _ struct{} `locationName:"PutBucketLifecycleRequest" type:"structure" payload:"LifecycleConfiguration"` // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Container for lifecycle rules. You can add as many as 1000 rules. LifecycleConfiguration *LifecycleConfiguration `locationName:"LifecycleConfiguration" type:"structure" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` } @@ -19178,6 +24449,20 @@ func (s *PutBucketLifecycleInput) SetLifecycleConfiguration(v *LifecycleConfigur return s } +func (s *PutBucketLifecycleInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketLifecycleInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketLifecycleOutput struct { _ struct{} `type:"structure"` } @@ -19193,11 +24478,15 @@ func (s PutBucketLifecycleOutput) GoString() string { } type PutBucketLoggingInput struct { - _ struct{} `type:"structure" payload:"BucketLoggingStatus"` + _ struct{} `locationName:"PutBucketLoggingRequest" type:"structure" payload:"BucketLoggingStatus"` + // The name of the bucket for which to set the logging parameters. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Container for logging status information. + // // BucketLoggingStatus is a required field BucketLoggingStatus *BucketLoggingStatus `locationName:"BucketLoggingStatus" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` } @@ -19255,6 +24544,20 @@ func (s *PutBucketLoggingInput) SetBucketLoggingStatus(v *BucketLoggingStatus) * return s } +func (s *PutBucketLoggingInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketLoggingInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketLoggingOutput struct { _ struct{} `type:"structure"` } @@ -19270,7 +24573,7 @@ func (s PutBucketLoggingOutput) GoString() string { } type PutBucketMetricsConfigurationInput struct { - _ struct{} `type:"structure" payload:"MetricsConfiguration"` + _ struct{} `locationName:"PutBucketMetricsConfigurationRequest" type:"structure" payload:"MetricsConfiguration"` // The name of the bucket for which the metrics configuration is set. // @@ -19350,6 +24653,20 @@ func (s *PutBucketMetricsConfigurationInput) SetMetricsConfiguration(v *MetricsC return s } +func (s *PutBucketMetricsConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketMetricsConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketMetricsConfigurationOutput struct { _ struct{} `type:"structure"` } @@ -19365,8 +24682,10 @@ func (s PutBucketMetricsConfigurationOutput) GoString() string { } type PutBucketNotificationConfigurationInput struct { - _ struct{} `type:"structure" payload:"NotificationConfiguration"` + _ struct{} `locationName:"PutBucketNotificationConfigurationRequest" type:"structure" payload:"NotificationConfiguration"` + // The name of the bucket. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -19430,6 +24749,20 @@ func (s *PutBucketNotificationConfigurationInput) SetNotificationConfiguration(v return s } +func (s *PutBucketNotificationConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketNotificationConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketNotificationConfigurationOutput struct { _ struct{} `type:"structure"` } @@ -19445,11 +24778,15 @@ func (s PutBucketNotificationConfigurationOutput) GoString() string { } type PutBucketNotificationInput struct { - _ struct{} `type:"structure" payload:"NotificationConfiguration"` + _ struct{} `locationName:"PutBucketNotificationRequest" type:"structure" payload:"NotificationConfiguration"` + // The name of the bucket. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // The container for the configuration. + // // NotificationConfiguration is a required field NotificationConfiguration *NotificationConfigurationDeprecated `locationName:"NotificationConfiguration" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` } @@ -19502,6 +24839,20 @@ func (s *PutBucketNotificationInput) SetNotificationConfiguration(v *Notificatio return s } +func (s *PutBucketNotificationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketNotificationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketNotificationOutput struct { _ struct{} `type:"structure"` } @@ -19517,8 +24868,10 @@ func (s PutBucketNotificationOutput) GoString() string { } type PutBucketPolicyInput struct { - _ struct{} `type:"structure" payload:"Policy"` + _ struct{} `locationName:"PutBucketPolicyRequest" type:"structure" payload:"Policy"` + // The name of the bucket. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -19586,6 +24939,20 @@ func (s *PutBucketPolicyInput) SetPolicy(v string) *PutBucketPolicyInput { return s } +func (s *PutBucketPolicyInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketPolicyInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketPolicyOutput struct { _ struct{} `type:"structure"` } @@ -19601,8 +24968,10 @@ func (s PutBucketPolicyOutput) GoString() string { } type PutBucketReplicationInput struct { - _ struct{} `type:"structure" payload:"ReplicationConfiguration"` + _ struct{} `locationName:"PutBucketReplicationRequest" type:"structure" payload:"ReplicationConfiguration"` + // The name of the bucket + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -19611,6 +24980,8 @@ type PutBucketReplicationInput struct { // // ReplicationConfiguration is a required field ReplicationConfiguration *ReplicationConfiguration `locationName:"ReplicationConfiguration" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` + + Token *string `location:"header" locationName:"x-amz-bucket-object-lock-token" type:"string"` } // String returns the string representation @@ -19666,6 +25037,26 @@ func (s *PutBucketReplicationInput) SetReplicationConfiguration(v *ReplicationCo return s } +// SetToken sets the Token field's value. +func (s *PutBucketReplicationInput) SetToken(v string) *PutBucketReplicationInput { + s.Token = &v + return s +} + +func (s *PutBucketReplicationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketReplicationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketReplicationOutput struct { _ struct{} `type:"structure"` } @@ -19681,11 +25072,15 @@ func (s PutBucketReplicationOutput) GoString() string { } type PutBucketRequestPaymentInput struct { - _ struct{} `type:"structure" payload:"RequestPaymentConfiguration"` + _ struct{} `locationName:"PutBucketRequestPaymentRequest" type:"structure" payload:"RequestPaymentConfiguration"` + // The bucket name. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Container for Payer. + // // RequestPaymentConfiguration is a required field RequestPaymentConfiguration *RequestPaymentConfiguration `locationName:"RequestPaymentConfiguration" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` } @@ -19743,6 +25138,20 @@ func (s *PutBucketRequestPaymentInput) SetRequestPaymentConfiguration(v *Request return s } +func (s *PutBucketRequestPaymentInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketRequestPaymentInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketRequestPaymentOutput struct { _ struct{} `type:"structure"` } @@ -19758,11 +25167,15 @@ func (s PutBucketRequestPaymentOutput) GoString() string { } type PutBucketTaggingInput struct { - _ struct{} `type:"structure" payload:"Tagging"` + _ struct{} `locationName:"PutBucketTaggingRequest" type:"structure" payload:"Tagging"` + // The bucket name. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Container for the TagSet and Tag elements. + // // Tagging is a required field Tagging *Tagging `locationName:"Tagging" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` } @@ -19820,6 +25233,20 @@ func (s *PutBucketTaggingInput) SetTagging(v *Tagging) *PutBucketTaggingInput { return s } +func (s *PutBucketTaggingInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketTaggingInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketTaggingOutput struct { _ struct{} `type:"structure"` } @@ -19835,8 +25262,10 @@ func (s PutBucketTaggingOutput) GoString() string { } type PutBucketVersioningInput struct { - _ struct{} `type:"structure" payload:"VersioningConfiguration"` + _ struct{} `locationName:"PutBucketVersioningRequest" type:"structure" payload:"VersioningConfiguration"` + // The bucket name. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -19844,6 +25273,8 @@ type PutBucketVersioningInput struct { // and the value that is displayed on your authentication device. MFA *string `location:"header" locationName:"x-amz-mfa" type:"string"` + // Container for setting the versioning state. + // // VersioningConfiguration is a required field VersioningConfiguration *VersioningConfiguration `locationName:"VersioningConfiguration" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` } @@ -19902,6 +25333,20 @@ func (s *PutBucketVersioningInput) SetVersioningConfiguration(v *VersioningConfi return s } +func (s *PutBucketVersioningInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketVersioningInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketVersioningOutput struct { _ struct{} `type:"structure"` } @@ -19917,11 +25362,15 @@ func (s PutBucketVersioningOutput) GoString() string { } type PutBucketWebsiteInput struct { - _ struct{} `type:"structure" payload:"WebsiteConfiguration"` + _ struct{} `locationName:"PutBucketWebsiteRequest" type:"structure" payload:"WebsiteConfiguration"` + // The bucket name. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Container for the request. + // // WebsiteConfiguration is a required field WebsiteConfiguration *WebsiteConfiguration `locationName:"WebsiteConfiguration" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` } @@ -19979,6 +25428,20 @@ func (s *PutBucketWebsiteInput) SetWebsiteConfiguration(v *WebsiteConfiguration) return s } +func (s *PutBucketWebsiteInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutBucketWebsiteInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutBucketWebsiteOutput struct { _ struct{} `type:"structure"` } @@ -19994,13 +25457,25 @@ func (s PutBucketWebsiteOutput) GoString() string { } type PutObjectAclInput struct { - _ struct{} `type:"structure" payload:"AccessControlPolicy"` + _ struct{} `locationName:"PutObjectAclRequest" type:"structure" payload:"AccessControlPolicy"` - // The canned ACL to apply to the object. + // The canned ACL to apply to the object. For more information, see Canned ACL + // (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL). ACL *string `location:"header" locationName:"x-amz-acl" type:"string" enum:"ObjectCannedACL"` + // Contains the elements that set the ACL permissions for an object per grantee. AccessControlPolicy *AccessControlPolicy `locationName:"AccessControlPolicy" type:"structure" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` + // The bucket name that contains the object to which you want to attach the + // ACL. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -20020,13 +25495,16 @@ type PutObjectAclInput struct { // Allows grantee to write the ACL for the applicable bucket. GrantWriteACP *string `location:"header" locationName:"x-amz-grant-write-acp" type:"string"` + // Key for which the PUT operation was initiated. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // VersionId used to reference a specific version of the object. @@ -20143,6 +25621,20 @@ func (s *PutObjectAclInput) SetVersionId(v string) *PutObjectAclInput { return s } +func (s *PutObjectAclInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutObjectAclInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutObjectAclOutput struct { _ struct{} `type:"structure"` @@ -20168,44 +25660,64 @@ func (s *PutObjectAclOutput) SetRequestCharged(v string) *PutObjectAclOutput { } type PutObjectInput struct { - _ struct{} `type:"structure" payload:"Body"` + _ struct{} `locationName:"PutObjectRequest" type:"structure" payload:"Body"` - // The canned ACL to apply to the object. + // The canned ACL to apply to the object. For more information, see Canned ACL + // (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL). ACL *string `location:"header" locationName:"x-amz-acl" type:"string" enum:"ObjectCannedACL"` // Object data. Body io.ReadSeeker `type:"blob"` - // Name of the bucket to which the PUT operation was initiated. + // Bucket name to which the PUT operation was initiated. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` - // Specifies caching behavior along the request/reply chain. + // Can be used to specify caching behavior along the request/reply chain. For + // more information, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9 + // (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9). CacheControl *string `location:"header" locationName:"Cache-Control" type:"string"` - // Specifies presentational information for the object. + // Specifies presentational information for the object. For more information, + // see http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1). ContentDisposition *string `location:"header" locationName:"Content-Disposition" type:"string"` // Specifies what content encodings have been applied to the object and thus // what decoding mechanisms must be applied to obtain the media-type referenced - // by the Content-Type header field. + // by the Content-Type header field. For more information, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11 + // (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11). ContentEncoding *string `location:"header" locationName:"Content-Encoding" type:"string"` // The language the content is in. ContentLanguage *string `location:"header" locationName:"Content-Language" type:"string"` // Size of the body in bytes. This parameter is useful when the size of the - // body cannot be determined automatically. + // body cannot be determined automatically. For more information, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13 + // (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.13). ContentLength *int64 `location:"header" locationName:"Content-Length" type:"long"` - // The base64-encoded 128-bit MD5 digest of the part data. + // The base64-encoded 128-bit MD5 digest of the message (without the headers) + // according to RFC 1864. This header can be used as a message integrity check + // to verify that the data is the same data that was originally sent. Although + // it is optional, we recommend using the Content-MD5 mechanism as an end-to-end + // integrity check. For more information about REST request authentication, + // see REST Authentication (https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html). ContentMD5 *string `location:"header" locationName:"Content-MD5" type:"string"` - // A standard MIME type describing the format of the object data. + // A standard MIME type describing the format of the contents. For more information, + // see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17). ContentType *string `location:"header" locationName:"Content-Type" type:"string"` - // The date and time at which the object is no longer cacheable. + // The date and time at which the object is no longer cacheable. For more information, + // see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.21 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.21). Expires *time.Time `location:"header" locationName:"Expires" type:"timestamp"` // Gives the grantee READ, READ_ACP, and WRITE_ACP permissions on the object. @@ -20228,7 +25740,8 @@ type PutObjectInput struct { // A map of metadata to store with the object in S3. Metadata map[string]*string `location:"headers" locationName:"x-amz-meta-" type:"map"` - // The Legal Hold status that you want to apply to the specified object. + // Specifies whether a legal hold will be applied to this object. For more information + // about S3 Object Lock, see Object Lock (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html). ObjectLockLegalHoldStatus *string `location:"header" locationName:"x-amz-object-lock-legal-hold" type:"string" enum:"ObjectLockLegalHoldStatus"` // The Object Lock mode that you want to apply to this object. @@ -20237,38 +25750,52 @@ type PutObjectInput struct { // The date and time when you want this object's Object Lock to expire. ObjectLockRetainUntilDate *time.Time `location:"header" locationName:"x-amz-object-lock-retain-until-date" type:"timestamp" timestampFormat:"iso8601"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` - // Specifies the algorithm to use to when encrypting the object (e.g., AES256). + // Specifies the algorithm to use to when encrypting the object (for example, + // AES256). SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // Specifies the customer-provided encryption key for Amazon S3 to use in encrypting // data. This value is used to store the object and then it is discarded; Amazon - // does not store the encryption key. The key must be appropriate for use with - // the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm + // S3 does not store the encryption key. The key must be appropriate for use + // with the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm // header. - SSECustomerKey *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` + SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. - // Amazon S3 uses this header for a message integrity check to ensure the encryption - // key was transmitted without error. + // Amazon S3 uses this header for a message integrity check to ensure that the + // encryption key was transmitted without error. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` - // Specifies the AWS KMS key ID to use for object encryption. All GET and PUT - // requests for an object protected by AWS KMS will fail if not made via SSL - // or using SigV4. Documentation on configuring any of the officially supported - // AWS SDKs and CLI can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version + // Specifies the AWS KMS Encryption Context to use for object encryption. The + // value of this header is a base64-encoded UTF-8 string holding JSON with the + // encryption context key-value pairs. + SSEKMSEncryptionContext *string `location:"header" locationName:"x-amz-server-side-encryption-context" type:"string" sensitive:"true"` + + // If x-amz-server-side-encryption is present and has the value of aws:kms, + // this header specifies the ID of the AWS Key Management Service (AWS KMS) + // symmetrical customer managed customer master key (CMK) that was used for + // the object. + // + // If the value of x-amz-server-side-encryption is aws:kms, this header specifies + // the ID of the symmetric customer managed AWS KMS CMK that will be used for + // the object. If you specify x-amz-server-side-encryption:aws:kms, but do not + // providex-amz-server-side-encryption-aws-kms-key-id, Amazon S3 uses the AWS + // managed CMK in AWS to protect the data. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // The server-side encryption algorithm used when storing this object in Amazon + // S3 (for example, AES256, aws:kms). ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` - // The type of storage to use for the object. Defaults to 'STANDARD'. + // If you don't specify, S3 Standard is the default storage class. Amazon S3 + // supports other storage classes. StorageClass *string `location:"header" locationName:"x-amz-storage-class" type:"string" enum:"StorageClass"` // The tag-set for the object. The tag-set must be encoded as URL Query parameters. @@ -20277,7 +25804,22 @@ type PutObjectInput struct { // If the bucket is configured as a website, redirects requests for this object // to another object in the same bucket or to an external URL. Amazon S3 stores - // the value of this header in the object metadata. + // the value of this header in the object metadata. For information about object + // metadata, see Object Key and Metadata (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html). + // + // In the following example, the request header sets the redirect to an object + // (anotherPage.html) in the same bucket: + // + // x-amz-website-redirect-location: /anotherPage.html + // + // In the following example, the request header sets the object redirect to + // another website: + // + // x-amz-website-redirect-location: http://www.example.com/ + // + // For more information about website hosting in Amazon S3, see Hosting Websites + // on Amazon S3 (https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html) + // and How to Configure Website Page Redirects (https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html). WebsiteRedirectLocation *string `location:"header" locationName:"x-amz-website-redirect-location" type:"string"` } @@ -20471,6 +26013,12 @@ func (s *PutObjectInput) SetSSECustomerKeyMD5(v string) *PutObjectInput { return s } +// SetSSEKMSEncryptionContext sets the SSEKMSEncryptionContext field's value. +func (s *PutObjectInput) SetSSEKMSEncryptionContext(v string) *PutObjectInput { + s.SSEKMSEncryptionContext = &v + return s +} + // SetSSEKMSKeyId sets the SSEKMSKeyId field's value. func (s *PutObjectInput) SetSSEKMSKeyId(v string) *PutObjectInput { s.SSEKMSKeyId = &v @@ -20501,10 +26049,32 @@ func (s *PutObjectInput) SetWebsiteRedirectLocation(v string) *PutObjectInput { return s } +func (s *PutObjectInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutObjectInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutObjectLegalHoldInput struct { - _ struct{} `type:"structure" payload:"LegalHold"` + _ struct{} `locationName:"PutObjectLegalHoldRequest" type:"structure" payload:"LegalHold"` - // The bucket containing the object that you want to place a Legal Hold on. + // The bucket name containing the object that you want to place a Legal Hold + // on. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -20518,10 +26088,11 @@ type PutObjectLegalHoldInput struct { // specified object. LegalHold *ObjectLockLegalHold `locationName:"LegalHold" type:"structure" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // The version ID of the object that you want to place a Legal Hold on. @@ -20597,6 +26168,20 @@ func (s *PutObjectLegalHoldInput) SetVersionId(v string) *PutObjectLegalHoldInpu return s } +func (s *PutObjectLegalHoldInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutObjectLegalHoldInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutObjectLegalHoldOutput struct { _ struct{} `type:"structure"` @@ -20622,7 +26207,7 @@ func (s *PutObjectLegalHoldOutput) SetRequestCharged(v string) *PutObjectLegalHo } type PutObjectLockConfigurationInput struct { - _ struct{} `type:"structure" payload:"ObjectLockConfiguration"` + _ struct{} `locationName:"PutObjectLockConfigurationRequest" type:"structure" payload:"ObjectLockConfiguration"` // The bucket whose Object Lock configuration you want to create or replace. // @@ -20632,12 +26217,14 @@ type PutObjectLockConfigurationInput struct { // The Object Lock configuration that you want to apply to the specified bucket. ObjectLockConfiguration *ObjectLockConfiguration `locationName:"ObjectLockConfiguration" type:"structure" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` + // A token to allow Object Lock to be enabled for an existing bucket. Token *string `location:"header" locationName:"x-amz-bucket-object-lock-token" type:"string"` } @@ -20698,6 +26285,20 @@ func (s *PutObjectLockConfigurationInput) SetToken(v string) *PutObjectLockConfi return s } +func (s *PutObjectLockConfigurationInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutObjectLockConfigurationInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutObjectLockConfigurationOutput struct { _ struct{} `type:"structure"` @@ -20728,8 +26329,10 @@ type PutObjectOutput struct { // Entity tag for the uploaded object. ETag *string `location:"header" locationName:"ETag" type:"string"` - // If the object expiration is configured, this will contain the expiration - // date (expiry-date) and rule ID (rule-id). The value of rule-id is URL encoded. + // If the expiration is configured for the object (see PutBucketLifecycleConfiguration), + // the response includes this header. It includes the expiry-date and rule-id + // key-value pairs that provide information about object expiration. The value + // of the rule-id is URL encoded. Expiration *string `location:"header" locationName:"x-amz-expiration" type:"string"` // If present, indicates that the requester was successfully charged for the @@ -20742,16 +26345,25 @@ type PutObjectOutput struct { SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // If server-side encryption with a customer-provided encryption key was requested, - // the response will include this header to provide round trip message integrity + // the response will include this header to provide round-trip message integrity // verification of the customer-provided encryption key. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` - // If present, specifies the ID of the AWS Key Management Service (KMS) master - // encryption key that was used for the object. + // If present, specifies the AWS KMS Encryption Context to use for object encryption. + // The value of this header is a base64-encoded UTF-8 string holding JSON with + // the encryption context key-value pairs. + SSEKMSEncryptionContext *string `location:"header" locationName:"x-amz-server-side-encryption-context" type:"string" sensitive:"true"` + + // If x-amz-server-side-encryption is present and has the value of aws:kms, + // this header specifies the ID of the AWS Key Management Service (AWS KMS) + // symmetric customer managed customer master key (CMK) that was used for the + // object. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // If you specified server-side encryption either with an AWS KMS customer master + // key (CMK) or Amazon S3-managed encryption key in your PUT request, the response + // includes this header. It confirms the encryption algorithm that Amazon S3 + // used to encrypt the object. ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` // Version of the object. @@ -20798,6 +26410,12 @@ func (s *PutObjectOutput) SetSSECustomerKeyMD5(v string) *PutObjectOutput { return s } +// SetSSEKMSEncryptionContext sets the SSEKMSEncryptionContext field's value. +func (s *PutObjectOutput) SetSSEKMSEncryptionContext(v string) *PutObjectOutput { + s.SSEKMSEncryptionContext = &v + return s +} + // SetSSEKMSKeyId sets the SSEKMSKeyId field's value. func (s *PutObjectOutput) SetSSEKMSKeyId(v string) *PutObjectOutput { s.SSEKMSKeyId = &v @@ -20817,15 +26435,22 @@ func (s *PutObjectOutput) SetVersionId(v string) *PutObjectOutput { } type PutObjectRetentionInput struct { - _ struct{} `type:"structure" payload:"Retention"` + _ struct{} `locationName:"PutObjectRetentionRequest" type:"structure" payload:"Retention"` - // The bucket that contains the object you want to apply this Object Retention + // The bucket name that contains the object you want to apply this Object Retention // configuration to. // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` - // Indicates whether this operation should bypass Governance-mode restrictions.j + // Indicates whether this operation should bypass Governance-mode restrictions. BypassGovernanceRetention *bool `location:"header" locationName:"x-amz-bypass-governance-retention" type:"boolean"` // The key name for the object that you want to apply this Object Retention @@ -20834,10 +26459,11 @@ type PutObjectRetentionInput struct { // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // The container element for the Object Retention configuration. @@ -20923,6 +26549,20 @@ func (s *PutObjectRetentionInput) SetVersionId(v string) *PutObjectRetentionInpu return s } +func (s *PutObjectRetentionInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutObjectRetentionInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutObjectRetentionOutput struct { _ struct{} `type:"structure"` @@ -20948,17 +26588,31 @@ func (s *PutObjectRetentionOutput) SetRequestCharged(v string) *PutObjectRetenti } type PutObjectTaggingInput struct { - _ struct{} `type:"structure" payload:"Tagging"` + _ struct{} `locationName:"PutObjectTaggingRequest" type:"structure" payload:"Tagging"` + // The bucket name containing the object. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Name of the tag. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` + // Container for the TagSet and Tag elements + // // Tagging is a required field Tagging *Tagging `locationName:"Tagging" type:"structure" required:"true" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` + // The versionId of the object that the tag-set will be added to. VersionId *string `location:"querystring" locationName:"versionId" type:"string"` } @@ -21033,9 +26687,24 @@ func (s *PutObjectTaggingInput) SetVersionId(v string) *PutObjectTaggingInput { return s } +func (s *PutObjectTaggingInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutObjectTaggingInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutObjectTaggingOutput struct { _ struct{} `type:"structure"` + // The versionId of the object the tag-set was added to. VersionId *string `location:"header" locationName:"x-amz-version-id" type:"string"` } @@ -21056,7 +26725,7 @@ func (s *PutObjectTaggingOutput) SetVersionId(v string) *PutObjectTaggingOutput } type PutPublicAccessBlockInput struct { - _ struct{} `type:"structure" payload:"PublicAccessBlockConfiguration"` + _ struct{} `locationName:"PutPublicAccessBlockRequest" type:"structure" payload:"PublicAccessBlockConfiguration"` // The name of the Amazon S3 bucket whose PublicAccessBlock configuration you // want to set. @@ -21122,6 +26791,20 @@ func (s *PutPublicAccessBlockInput) SetPublicAccessBlockConfiguration(v *PublicA return s } +func (s *PutPublicAccessBlockInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *PutPublicAccessBlockInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type PutPublicAccessBlockOutput struct { _ struct{} `type:"structure"` } @@ -21136,17 +26819,18 @@ func (s PutPublicAccessBlockOutput) GoString() string { return s.String() } -// A container for specifying the configuration for publication of messages -// to an Amazon Simple Queue Service (Amazon SQS) queue.when Amazon S3 detects -// specified events. +// Specifies the configuration for publishing messages to an Amazon Simple Queue +// Service (Amazon SQS) queue when Amazon S3 detects specified events. type QueueConfiguration struct { _ struct{} `type:"structure"` + // A collection of bucket events for which to send notifications + // // Events is a required field Events []*string `locationName:"Event" type:"list" flattened:"true" required:"true"` - // A container for object key name filtering rules. For information about key - // name filtering, see Configuring Event Notifications (http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) + // Specifies object key name filtering rules. For information about key name + // filtering, see Configuring Event Notifications (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) // in the Amazon Simple Storage Service Developer Guide. Filter *NotificationConfigurationFilter `type:"structure"` @@ -21155,7 +26839,7 @@ type QueueConfiguration struct { Id *string `type:"string"` // The Amazon Resource Name (ARN) of the Amazon SQS queue to which Amazon S3 - // will publish a message when it detects events of the specified type. + // publishes a message when it detects events of the specified type. // // QueueArn is a required field QueueArn *string `locationName:"Queue" type:"string" required:"true"` @@ -21211,6 +26895,10 @@ func (s *QueueConfiguration) SetQueueArn(v string) *QueueConfiguration { return s } +// This data type is deprecated. Use QueueConfiguration for the same purposes. +// This data type specifies the configuration for publishing messages to an +// Amazon Simple Queue Service (Amazon SQS) queue when Amazon S3 detects specified +// events. type QueueConfigurationDeprecated struct { _ struct{} `type:"structure"` @@ -21219,12 +26907,15 @@ type QueueConfigurationDeprecated struct { // Deprecated: Event has been deprecated Event *string `deprecated:"true" type:"string" enum:"Event"` + // A collection of bucket events for which to send notifications Events []*string `locationName:"Event" type:"list" flattened:"true"` // An optional unique identifier for configurations in a notification configuration. // If you don't provide one, Amazon S3 will assign an ID. Id *string `type:"string"` + // The Amazon Resource Name (ARN) of the Amazon SQS queue to which Amazon S3 + // publishes a message when it detects events of the specified type. Queue *string `type:"string"` } @@ -21262,6 +26953,7 @@ func (s *QueueConfigurationDeprecated) SetQueue(v string) *QueueConfigurationDep return s } +// The container for the records event. type RecordsEvent struct { _ struct{} `locationName:"RecordsEvent" type:"structure" payload:"Payload"` @@ -21301,6 +26993,17 @@ func (s *RecordsEvent) UnmarshalEvent( return nil } +// MarshalEvent marshals the type into an stream event value. This method +// should only used internally within the SDK's EventStream handling. +func (s *RecordsEvent) MarshalEvent(pm protocol.PayloadMarshaler) (msg eventstream.Message, err error) { + msg.Headers.Set(eventstreamapi.MessageTypeHeader, eventstream.StringValue(eventstreamapi.EventMessageType)) + msg.Headers.Set(":content-type", eventstream.StringValue("application/octet-stream")) + msg.Payload = s.Payload + return msg, err +} + +// Specifies how requests are redirected. In the event of an error, you can +// specify a different error code to return. type Redirect struct { _ struct{} `type:"structure"` @@ -21311,8 +27014,8 @@ type Redirect struct { // siblings is present. HttpRedirectCode *string `type:"string"` - // Protocol to use (http, https) when redirecting requests. The default is the - // protocol that is used in the original request. + // Protocol to use when redirecting requests. The default is the protocol that + // is used in the original request. Protocol *string `type:"string" enum:"Protocol"` // The object key prefix to use in the redirect request. For example, to redirect @@ -21324,7 +27027,7 @@ type Redirect struct { ReplaceKeyPrefixWith *string `type:"string"` // The specific object key to use in the redirect request. For example, redirect - // request to error.html. Not required if one of the sibling is present. Can + // request to error.html. Not required if one of the siblings is present. Can // be present only if ReplaceKeyPrefixWith is not provided. ReplaceKeyWith *string `type:"string"` } @@ -21369,16 +27072,18 @@ func (s *Redirect) SetReplaceKeyWith(v string) *Redirect { return s } +// Specifies the redirect behavior of all requests to a website endpoint of +// an Amazon S3 bucket. type RedirectAllRequestsTo struct { _ struct{} `type:"structure"` - // Name of the host where requests will be redirected. + // Name of the host where requests are redirected. // // HostName is a required field HostName *string `type:"string" required:"true"` - // Protocol to use (http, https) when redirecting requests. The default is the - // protocol that is used in the original request. + // Protocol to use when redirecting requests. The default is the protocol that + // is used in the original request. Protocol *string `type:"string" enum:"Protocol"` } @@ -21423,7 +27128,9 @@ type ReplicationConfiguration struct { _ struct{} `type:"structure"` // The Amazon Resource Name (ARN) of the AWS Identity and Access Management - // (IAM) role that Amazon S3 can assume when replicating the objects. + // (IAM) role that Amazon S3 assumes when replicating objects. For more information, + // see How to Set Up Replication (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-how-setup.html) + // in the Amazon Simple Storage Service Developer Guide. // // Role is a required field Role *string `type:"string" required:"true"` @@ -21483,18 +27190,34 @@ func (s *ReplicationConfiguration) SetRules(v []*ReplicationRule) *ReplicationCo return s } -// A container for information about a specific replication rule. +// Specifies which Amazon S3 objects to replicate and where to store the replicas. type ReplicationRule struct { _ struct{} `type:"structure"` - // Specifies whether Amazon S3 should replicate delete makers. + // Specifies whether Amazon S3 replicates the delete markers. If you specify + // a Filter, you must specify this element. However, in the latest version of + // replication configuration (when Filter is specified), Amazon S3 doesn't replicate + // delete markers. Therefore, the DeleteMarkerReplication element can contain + // only Disabled. For an example configuration, see Basic Rule + // Configuration (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html#replication-config-min-rule-config). + // + // If you don't specify the Filter element, Amazon S3 assumes that the replication + // configuration is the earlier version, V1. In the earlier version, Amazon + // S3 handled replication of delete markers differently. For more information, + // see Backward Compatibility (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-add-config.html#replication-backward-compat-considerations). DeleteMarkerReplication *DeleteMarkerReplication `type:"structure"` - // A container for information about the replication destination. + // A container for information about the replication destination and its configurations + // including enabling the S3 Replication Time Control (S3 RTC). // // Destination is a required field Destination *Destination `type:"structure" required:"true"` + // Optional configuration to replicate existing source bucket objects. For more + // information, see Replicating Existing Objects (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication-what-is-isnot-replicated.html#existing-object-replication) + // in the Amazon S3 Developer Guide. + ExistingObjectReplication *ExistingObjectReplication `type:"structure"` + // A filter that identifies the subset of objects to which the replication rule // applies. A Filter must specify exactly one Prefix, Tag, or an And child element. Filter *ReplicationRuleFilter `type:"structure"` @@ -21502,8 +27225,9 @@ type ReplicationRule struct { // A unique identifier for the rule. The maximum value is 255 characters. ID *string `type:"string"` - // An object keyname prefix that identifies the object or objects to which the - // rule applies. The maximum prefix length is 1,024 characters. + // An object key name prefix that identifies the object or objects to which + // the rule applies. The maximum prefix length is 1,024 characters. To include + // all objects in a bucket, specify an empty string. // // Deprecated: Prefix has been deprecated Prefix *string `deprecated:"true" type:"string"` @@ -21513,27 +27237,24 @@ type ReplicationRule struct { // when filtering. If two or more rules identify the same object based on a // specified filter, the rule with higher priority takes precedence. For example: // - // * Same object quality prefix based filter criteria If prefixes you specified + // * Same object quality prefix-based filter criteria if prefixes you specified // in multiple rules overlap // - // * Same object qualify tag based filter criteria specified in multiple + // * Same object qualify tag-based filter criteria specified in multiple // rules // - // For more information, see Cross-Region Replication (CRR) ( https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) - // in the Amazon S3 Developer Guide. + // For more information, see Replication (https://docs.aws.amazon.com/AmazonS3/latest/dev/replication.html) + // in the Amazon Simple Storage Service Developer Guide. Priority *int64 `type:"integer"` // A container that describes additional filters for identifying the source // objects that you want to replicate. You can choose to enable or disable the // replication of these objects. Currently, Amazon S3 supports only the filter // that you can specify for objects created with server-side encryption using - // an AWS KMS-Managed Key (SSE-KMS). - // - // If you want Amazon S3 to replicate objects created with server-side encryption - // using AWS KMS-Managed Keys. + // a customer master key (CMK) stored in AWS Key Management Service (SSE-KMS). SourceSelectionCriteria *SourceSelectionCriteria `type:"structure"` - // If status isn't enabled, the rule is ignored. + // Specifies whether the rule is enabled. // // Status is a required field Status *string `type:"string" required:"true" enum:"ReplicationRuleStatus"` @@ -21563,6 +27284,11 @@ func (s *ReplicationRule) Validate() error { invalidParams.AddNested("Destination", err.(request.ErrInvalidParams)) } } + if s.ExistingObjectReplication != nil { + if err := s.ExistingObjectReplication.Validate(); err != nil { + invalidParams.AddNested("ExistingObjectReplication", err.(request.ErrInvalidParams)) + } + } if s.Filter != nil { if err := s.Filter.Validate(); err != nil { invalidParams.AddNested("Filter", err.(request.ErrInvalidParams)) @@ -21592,6 +27318,12 @@ func (s *ReplicationRule) SetDestination(v *Destination) *ReplicationRule { return s } +// SetExistingObjectReplication sets the ExistingObjectReplication field's value. +func (s *ReplicationRule) SetExistingObjectReplication(v *ExistingObjectReplication) *ReplicationRule { + s.ExistingObjectReplication = v + return s +} + // SetFilter sets the Filter field's value. func (s *ReplicationRule) SetFilter(v *ReplicationRuleFilter) *ReplicationRule { s.Filter = v @@ -21628,11 +27360,25 @@ func (s *ReplicationRule) SetStatus(v string) *ReplicationRule { return s } +// A container for specifying rule filters. The filters determine the subset +// of objects to which the rule applies. This element is required only if you +// specify more than one filter. +// +// For example: +// +// * If you specify both a Prefix and a Tag filter, wrap these filters in +// an And tag. +// +// * If you specify a filter based on multiple tags, wrap the Tag elements +// in an And tag type ReplicationRuleAndOperator struct { _ struct{} `type:"structure"` + // An object key name prefix that identifies the subset of objects to which + // the rule applies. Prefix *string `type:"string"` + // An array of tags containing key and value pairs. Tags []*Tag `locationName:"Tag" locationNameList:"Tag" type:"list" flattened:"true"` } @@ -21694,8 +27440,8 @@ type ReplicationRuleFilter struct { // in an And tag. And *ReplicationRuleAndOperator `type:"structure"` - // An object keyname prefix that identifies the subset of objects to which the - // rule applies. + // An object key name prefix that identifies the subset of objects to which + // the rule applies. Prefix *string `type:"string"` // A container for specifying a tag key and value. @@ -21752,6 +27498,91 @@ func (s *ReplicationRuleFilter) SetTag(v *Tag) *ReplicationRuleFilter { return s } +// A container specifying S3 Replication Time Control (S3 RTC) related information, +// including whether S3 RTC is enabled and the time when all objects and operations +// on objects must be replicated. Must be specified together with a Metrics +// block. +type ReplicationTime struct { + _ struct{} `type:"structure"` + + // Specifies whether the replication time is enabled. + // + // Status is a required field + Status *string `type:"string" required:"true" enum:"ReplicationTimeStatus"` + + // A container specifying the time by which replication should be complete for + // all objects and operations on objects. + // + // Time is a required field + Time *ReplicationTimeValue `type:"structure" required:"true"` +} + +// String returns the string representation +func (s ReplicationTime) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ReplicationTime) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *ReplicationTime) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "ReplicationTime"} + if s.Status == nil { + invalidParams.Add(request.NewErrParamRequired("Status")) + } + if s.Time == nil { + invalidParams.Add(request.NewErrParamRequired("Time")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetStatus sets the Status field's value. +func (s *ReplicationTime) SetStatus(v string) *ReplicationTime { + s.Status = &v + return s +} + +// SetTime sets the Time field's value. +func (s *ReplicationTime) SetTime(v *ReplicationTimeValue) *ReplicationTime { + s.Time = v + return s +} + +// A container specifying the time value for S3 Replication Time Control (S3 +// RTC) and replication metrics EventThreshold. +type ReplicationTimeValue struct { + _ struct{} `type:"structure"` + + // Contains an integer specifying time in minutes. + // + // Valid values: 15 minutes. + Minutes *int64 `type:"integer"` +} + +// String returns the string representation +func (s ReplicationTimeValue) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s ReplicationTimeValue) GoString() string { + return s.String() +} + +// SetMinutes sets the Minutes field's value. +func (s *ReplicationTimeValue) SetMinutes(v int64) *ReplicationTimeValue { + s.Minutes = &v + return s +} + +// Container for Payer. type RequestPaymentConfiguration struct { _ struct{} `type:"structure"` @@ -21790,6 +27621,7 @@ func (s *RequestPaymentConfiguration) SetPayer(v string) *RequestPaymentConfigur return s } +// Container for specifying if periodic QueryProgress messages should be sent. type RequestProgress struct { _ struct{} `type:"structure"` @@ -21815,23 +27647,36 @@ func (s *RequestProgress) SetEnabled(v bool) *RequestProgress { } type RestoreObjectInput struct { - _ struct{} `type:"structure" payload:"RestoreRequest"` + _ struct{} `locationName:"RestoreObjectRequest" type:"structure" payload:"RestoreRequest"` + // The bucket name or containing the object to restore. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` + // Object key for which the operation was initiated. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` // Container for restore job parameters. RestoreRequest *RestoreRequest `locationName:"RestoreRequest" type:"structure" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` + // VersionId used to reference a specific version of the object. VersionId *string `location:"querystring" locationName:"versionId" type:"string"` } @@ -21909,6 +27754,20 @@ func (s *RestoreObjectInput) SetVersionId(v string) *RestoreObjectInput { return s } +func (s *RestoreObjectInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *RestoreObjectInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type RestoreObjectOutput struct { _ struct{} `type:"structure"` @@ -21954,7 +27813,7 @@ type RestoreRequest struct { // The optional description for the job. Description *string `type:"string"` - // Glacier related parameters pertaining to this job. Do not use with restores + // S3 Glacier related parameters pertaining to this job. Do not use with restores // that specify OutputLocation. GlacierJobParameters *GlacierJobParameters `type:"structure"` @@ -21964,7 +27823,7 @@ type RestoreRequest struct { // Describes the parameters for Select job types. SelectParameters *SelectParameters `type:"structure"` - // Glacier retrieval tier at which the restore will be processed. + // S3 Glacier retrieval tier at which the restore will be processed. Tier *string `type:"string" enum:"Tier"` // Type of restore request. @@ -22048,6 +27907,7 @@ func (s *RestoreRequest) SetType(v string) *RestoreRequest { return s } +// Specifies the redirect behavior and when a redirect is applied. type RoutingRule struct { _ struct{} `type:"structure"` @@ -22100,16 +27960,24 @@ func (s *RoutingRule) SetRedirect(v *Redirect) *RoutingRule { return s } +// Specifies lifecycle rules for an Amazon S3 bucket. For more information, +// see Put Bucket Lifecycle Configuration (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTlifecycle.html) +// in the Amazon Simple Storage Service API Reference. For examples, see Put +// Bucket Lifecycle Configuration Examples (https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html#API_PutBucketLifecycleConfiguration_Examples) type Rule struct { _ struct{} `type:"structure"` - // Specifies the days since the initiation of an Incomplete Multipart Upload - // that Lifecycle will wait before permanently removing all parts of the upload. + // Specifies the days since the initiation of an incomplete multipart upload + // that Amazon S3 will wait before permanently removing all parts of the upload. + // For more information, see Aborting Incomplete Multipart Uploads Using a Bucket + // Lifecycle Policy (https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-abort-incomplete-mpu-lifecycle-config) + // in the Amazon Simple Storage Service Developer Guide. AbortIncompleteMultipartUpload *AbortIncompleteMultipartUpload `type:"structure"` + // Specifies the expiration for the lifecycle of the object. Expiration *LifecycleExpiration `type:"structure"` - // Unique identifier for the rule. The value cannot be longer than 255 characters. + // Unique identifier for the rule. The value can't be longer than 255 characters. ID *string `type:"string"` // Specifies when noncurrent object versions expire. Upon expiration, Amazon @@ -22120,24 +27988,30 @@ type Rule struct { NoncurrentVersionExpiration *NoncurrentVersionExpiration `type:"structure"` // Container for the transition rule that describes when noncurrent objects - // transition to the STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING or GLACIER - // storage class. If your bucket is versioning-enabled (or versioning is suspended), - // you can set this action to request that Amazon S3 transition noncurrent object - // versions to the STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING or GLACIER storage - // class at a specific period in the object's lifetime. + // transition to the STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, + // or DEEP_ARCHIVE storage class. If your bucket is versioning-enabled (or versioning + // is suspended), you can set this action to request that Amazon S3 transition + // noncurrent object versions to the STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, + // GLACIER, or DEEP_ARCHIVE storage class at a specific period in the object's + // lifetime. NoncurrentVersionTransition *NoncurrentVersionTransition `type:"structure"` - // Prefix identifying one or more objects to which the rule applies. + // Object key prefix that identifies one or more objects to which this rule + // applies. // // Prefix is a required field Prefix *string `type:"string" required:"true"` - // If 'Enabled', the rule is currently being applied. If 'Disabled', the rule - // is not currently being applied. + // If Enabled, the rule is currently being applied. If Disabled, the rule is + // not currently being applied. // // Status is a required field Status *string `type:"string" required:"true" enum:"ExpirationStatus"` + // Specifies when an object transitions to a specified storage class. For more + // information about Amazon S3 lifecycle configuration rules, see Transitioning + // Objects Using Amazon S3 Lifecycle (https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html) + // in the Amazon Simple Storage Service Developer Guide. Transition *Transition `type:"structure"` } @@ -22215,12 +28089,12 @@ func (s *Rule) SetTransition(v *Transition) *Rule { return s } -// Specifies the use of SSE-KMS to encrypt delivered Inventory reports. +// Specifies the use of SSE-KMS to encrypt delivered inventory reports. type SSEKMS struct { _ struct{} `locationName:"SSE-KMS" type:"structure"` - // Specifies the ID of the AWS Key Management Service (KMS) master encryption - // key to use for encrypting Inventory reports. + // Specifies the ID of the AWS Key Management Service (AWS KMS) symmetric customer + // managed customer master key (CMK) to use for encrypting inventory reports. // // KeyId is a required field KeyId *string `type:"string" required:"true" sensitive:"true"` @@ -22255,7 +28129,7 @@ func (s *SSEKMS) SetKeyId(v string) *SSEKMS { return s } -// Specifies the use of SSE-S3 to encrypt delivered Inventory reports. +// Specifies the use of SSE-S3 to encrypt delivered inventory reports. type SSES3 struct { _ struct{} `locationName:"SSE-S3" type:"structure"` } @@ -22270,75 +28144,51 @@ func (s SSES3) GoString() string { return s.String() } -// SelectObjectContentEventStream provides handling of EventStreams for -// the SelectObjectContent API. -// -// Use this type to receive SelectObjectContentEventStream events. The events -// can be read from the Events channel member. -// -// The events that can be received are: -// -// * ContinuationEvent -// * EndEvent -// * ProgressEvent -// * RecordsEvent -// * StatsEvent -type SelectObjectContentEventStream struct { - // Reader is the EventStream reader for the SelectObjectContentEventStream - // events. This value is automatically set by the SDK when the API call is made - // Use this member when unit testing your code with the SDK to mock out the - // EventStream Reader. - // - // Must not be nil. - Reader SelectObjectContentEventStreamReader +// Specifies the byte range of the object to get the records from. A record +// is processed when its first byte is contained by the range. This parameter +// is optional, but when specified, it must not be empty. See RFC 2616, Section +// 14.35.1 about how to specify the start and end of the range. +type ScanRange struct { + _ struct{} `type:"structure"` - // StreamCloser is the io.Closer for the EventStream connection. For HTTP - // EventStream this is the response Body. The stream will be closed when - // the Close method of the EventStream is called. - StreamCloser io.Closer + // Specifies the end of the byte range. This parameter is optional. Valid values: + // non-negative integers. The default value is one less than the size of the + // object being queried. If only the End parameter is supplied, it is interpreted + // to mean scan the last N bytes of the file. For example, 50 + // means scan the last 50 bytes. + End *int64 `type:"long"` + + // Specifies the start of the byte range. This parameter is optional. Valid + // values: non-negative integers. The default value is 0. If only start is supplied, + // it means scan from that point to the end of the file.For example; 50 + // means scan from byte 50 until the end of the file. + Start *int64 `type:"long"` } -// Close closes the EventStream. This will also cause the Events channel to be -// closed. You can use the closing of the Events channel to terminate your -// application's read from the API's EventStream. -// -// Will close the underlying EventStream reader. For EventStream over HTTP -// connection this will also close the HTTP connection. -// -// Close must be called when done using the EventStream API. Not calling Close -// may result in resource leaks. -func (es *SelectObjectContentEventStream) Close() (err error) { - es.Reader.Close() - return es.Err() +// String returns the string representation +func (s ScanRange) String() string { + return awsutil.Prettify(s) } -// Err returns any error that occurred while reading EventStream Events from -// the service API's response. Returns nil if there were no errors. -func (es *SelectObjectContentEventStream) Err() error { - if err := es.Reader.Err(); err != nil { - return err - } - es.StreamCloser.Close() +// GoString returns the string representation +func (s ScanRange) GoString() string { + return s.String() +} - return nil +// SetEnd sets the End field's value. +func (s *ScanRange) SetEnd(v int64) *ScanRange { + s.End = &v + return s } -// Events returns a channel to read EventStream Events from the -// SelectObjectContent API. -// -// These events are: -// -// * ContinuationEvent -// * EndEvent -// * ProgressEvent -// * RecordsEvent -// * StatsEvent -func (es *SelectObjectContentEventStream) Events() <-chan SelectObjectContentEventStreamEvent { - return es.Reader.Events() +// SetStart sets the Start field's value. +func (s *ScanRange) SetStart(v int64) *ScanRange { + s.Start = &v + return s } // SelectObjectContentEventStreamEvent groups together all EventStream -// events read from the SelectObjectContent API. +// events writes for SelectObjectContentEventStream. // // These events are: // @@ -22349,11 +28199,12 @@ func (es *SelectObjectContentEventStream) Events() <-chan SelectObjectContentEve // * StatsEvent type SelectObjectContentEventStreamEvent interface { eventSelectObjectContentEventStream() + eventstreamapi.Marshaler + eventstreamapi.Unmarshaler } -// SelectObjectContentEventStreamReader provides the interface for reading EventStream -// Events from the SelectObjectContent API. The -// default implementation for this interface will be SelectObjectContentEventStream. +// SelectObjectContentEventStreamReader provides the interface for reading to the stream. The +// default implementation for this interface will be SelectObjectContentEventStreamData. // // The reader's Close method must allow multiple concurrent calls. // @@ -22364,12 +28215,12 @@ type SelectObjectContentEventStreamEvent interface { // * ProgressEvent // * RecordsEvent // * StatsEvent +// * SelectObjectContentEventStreamUnknownEvent type SelectObjectContentEventStreamReader interface { // Returns a channel of events as they are read from the event stream. Events() <-chan SelectObjectContentEventStreamEvent - // Close will close the underlying event stream reader. For event stream over - // HTTP this will also close the HTTP connection. + // Close will stop the reader reading events from the stream. Close() error // Returns any error that has occurred while reading from the event stream. @@ -22379,57 +28230,44 @@ type SelectObjectContentEventStreamReader interface { type readSelectObjectContentEventStream struct { eventReader *eventstreamapi.EventReader stream chan SelectObjectContentEventStreamEvent - errVal atomic.Value + err *eventstreamapi.OnceError done chan struct{} closeOnce sync.Once } -func newReadSelectObjectContentEventStream( - reader io.ReadCloser, - unmarshalers request.HandlerList, - logger aws.Logger, - logLevel aws.LogLevelType, -) *readSelectObjectContentEventStream { +func newReadSelectObjectContentEventStream(eventReader *eventstreamapi.EventReader) *readSelectObjectContentEventStream { r := &readSelectObjectContentEventStream{ - stream: make(chan SelectObjectContentEventStreamEvent), - done: make(chan struct{}), + eventReader: eventReader, + stream: make(chan SelectObjectContentEventStreamEvent), + done: make(chan struct{}), + err: eventstreamapi.NewOnceError(), } - - r.eventReader = eventstreamapi.NewEventReader( - reader, - protocol.HandlerPayloadUnmarshal{ - Unmarshalers: unmarshalers, - }, - r.unmarshalerForEventType, - ) - r.eventReader.UseLogger(logger, logLevel) + go r.readEventStream() return r } -// Close will close the underlying event stream reader. For EventStream over -// HTTP this will also close the HTTP connection. +// Close will close the underlying event stream reader. func (r *readSelectObjectContentEventStream) Close() error { r.closeOnce.Do(r.safeClose) - return r.Err() } +func (r *readSelectObjectContentEventStream) ErrorSet() <-chan struct{} { + return r.err.ErrorSet() +} + +func (r *readSelectObjectContentEventStream) Closed() <-chan struct{} { + return r.done +} + func (r *readSelectObjectContentEventStream) safeClose() { close(r.done) - err := r.eventReader.Close() - if err != nil { - r.errVal.Store(err) - } } func (r *readSelectObjectContentEventStream) Err() error { - if v := r.errVal.Load(); v != nil { - return v.(error) - } - - return nil + return r.err.Err() } func (r *readSelectObjectContentEventStream) Events() <-chan SelectObjectContentEventStreamEvent { @@ -22437,6 +28275,7 @@ func (r *readSelectObjectContentEventStream) Events() <-chan SelectObjectContent } func (r *readSelectObjectContentEventStream) readEventStream() { + defer r.Close() defer close(r.stream) for { @@ -22451,7 +28290,10 @@ func (r *readSelectObjectContentEventStream) readEventStream() { return default: } - r.errVal.Store(err) + if _, ok := err.(*eventstreamapi.UnknownMessageTypeError); ok { + continue + } + r.err.SetError(err) return } @@ -22463,40 +28305,63 @@ func (r *readSelectObjectContentEventStream) readEventStream() { } } -func (r *readSelectObjectContentEventStream) unmarshalerForEventType( - eventType string, -) (eventstreamapi.Unmarshaler, error) { +type unmarshalerForSelectObjectContentEventStreamEvent struct { + metadata protocol.ResponseMetadata +} + +func (u unmarshalerForSelectObjectContentEventStreamEvent) UnmarshalerForEventName(eventType string) (eventstreamapi.Unmarshaler, error) { switch eventType { case "Cont": return &ContinuationEvent{}, nil - case "End": return &EndEvent{}, nil - case "Progress": return &ProgressEvent{}, nil - case "Records": return &RecordsEvent{}, nil - case "Stats": return &StatsEvent{}, nil default: - return nil, awserr.New( - request.ErrCodeSerialization, - fmt.Sprintf("unknown event type name, %s, for SelectObjectContentEventStream", eventType), - nil, - ) + return &SelectObjectContentEventStreamUnknownEvent{Type: eventType}, nil } } +// SelectObjectContentEventStreamUnknownEvent provides a failsafe event for the +// SelectObjectContentEventStream group of events when an unknown event is received. +type SelectObjectContentEventStreamUnknownEvent struct { + Type string + Message eventstream.Message +} + +// The SelectObjectContentEventStreamUnknownEvent is and event in the SelectObjectContentEventStream +// group of events. +func (s *SelectObjectContentEventStreamUnknownEvent) eventSelectObjectContentEventStream() {} + +// MarshalEvent marshals the type into an stream event value. This method +// should only used internally within the SDK's EventStream handling. +func (e *SelectObjectContentEventStreamUnknownEvent) MarshalEvent(pm protocol.PayloadMarshaler) ( + msg eventstream.Message, err error, +) { + return e.Message.Clone(), nil +} + +// UnmarshalEvent unmarshals the EventStream Message into the SelectObjectContentEventStreamData value. +// This method is only used internally within the SDK's EventStream handling. +func (e *SelectObjectContentEventStreamUnknownEvent) UnmarshalEvent( + payloadUnmarshaler protocol.PayloadUnmarshaler, + msg eventstream.Message, +) error { + e.Message = msg.Clone() + return nil +} + // Request to filter the contents of an Amazon S3 object based on a simple Structured // Query Language (SQL) statement. In the request, along with the SQL expression, // you must specify a data serialization format (JSON or CSV) of the object. // Amazon S3 uses this to parse object data into records. It returns only records // that match the specified SQL expression. You must also specify the data serialization // format for the response. For more information, see S3Select API Documentation -// (http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html). +// (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectSELECTContent.html). type SelectObjectContentInput struct { _ struct{} `locationName:"SelectObjectContentRequest" type:"structure" xmlURI:"http://s3.amazonaws.com/doc/2006-03-01/"` @@ -22510,7 +28375,7 @@ type SelectObjectContentInput struct { // Expression is a required field Expression *string `type:"string" required:"true"` - // The type of the provided expression (for example., SQL). + // The type of the provided expression (for example, SQL). // // ExpressionType is a required field ExpressionType *string `type:"string" required:"true" enum:"ExpressionType"` @@ -22533,17 +28398,35 @@ type SelectObjectContentInput struct { // Specifies if periodic request progress information should be enabled. RequestProgress *RequestProgress `type:"structure"` - // The SSE Algorithm used to encrypt the object. For more information, see - // Server-Side Encryption (Using Customer-Provided Encryption Keys (http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html). + // The SSE Algorithm used to encrypt the object. For more information, see Server-Side + // Encryption (Using Customer-Provided Encryption Keys (https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html). SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` - // The SSE Customer Key. For more information, see Server-Side Encryption (Using - // Customer-Provided Encryption Keys (http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html). - SSECustomerKey *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` + // The SSE Customer Key. For more information, see Server-Side Encryption (Using + // Customer-Provided Encryption Keys (https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html). + SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` - // The SSE Customer Key MD5. For more information, see Server-Side Encryption - // (Using Customer-Provided Encryption Keys (http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html). + // The SSE Customer Key MD5. For more information, see Server-Side Encryption + // (Using Customer-Provided Encryption Keys (https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html). SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` + + // Specifies the byte range of the object to get the records from. A record + // is processed when its first byte is contained by the range. This parameter + // is optional, but when specified, it must not be empty. See RFC 2616, Section + // 14.35.1 about how to specify the start and end of the range. + // + // ScanRangemay be used in the following ways: + // + // * 50100 - process only + // the records starting between the bytes 50 and 100 (inclusive, counting + // from zero) + // + // * 50 - process only the records + // starting after the byte 50 + // + // * 50 - process only the records within + // the last 50 bytes of the file. + ScanRange *ScanRange `type:"structure"` } // String returns the string representation @@ -22664,11 +28547,30 @@ func (s *SelectObjectContentInput) SetSSECustomerKeyMD5(v string) *SelectObjectC return s } +// SetScanRange sets the ScanRange field's value. +func (s *SelectObjectContentInput) SetScanRange(v *ScanRange) *SelectObjectContentInput { + s.ScanRange = v + return s +} + +func (s *SelectObjectContentInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *SelectObjectContentInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type SelectObjectContentOutput struct { _ struct{} `type:"structure" payload:"Payload"` - // Use EventStream to use the API's stream. - EventStream *SelectObjectContentEventStream `type:"structure"` + EventStream *SelectObjectContentEventStream } // String returns the string representation @@ -22681,29 +28583,17 @@ func (s SelectObjectContentOutput) GoString() string { return s.String() } -// SetEventStream sets the EventStream field's value. func (s *SelectObjectContentOutput) SetEventStream(v *SelectObjectContentEventStream) *SelectObjectContentOutput { s.EventStream = v return s } +func (s *SelectObjectContentOutput) GetEventStream() *SelectObjectContentEventStream { + return s.EventStream +} -func (s *SelectObjectContentOutput) runEventStreamLoop(r *request.Request) { - if r.Error != nil { - return - } - reader := newReadSelectObjectContentEventStream( - r.HTTPResponse.Body, - r.Handlers.UnmarshalStream, - r.Config.Logger, - r.Config.LogLevel.Value(), - ) - go reader.readEventStream() - - eventStream := &SelectObjectContentEventStream{ - StreamCloser: r.HTTPResponse.Body, - Reader: reader, - } - s.EventStream = eventStream +// GetStream returns the type to interact with the event stream. +func (s *SelectObjectContentOutput) GetStream() *SelectObjectContentEventStream { + return s.EventStream } // Describes the parameters for Select job types. @@ -22715,7 +28605,7 @@ type SelectParameters struct { // Expression is a required field Expression *string `type:"string" required:"true"` - // The type of the provided expression (e.g., SQL). + // The type of the provided expression (for example, SQL). // // ExpressionType is a required field ExpressionType *string `type:"string" required:"true" enum:"ExpressionType"` @@ -22788,13 +28678,31 @@ func (s *SelectParameters) SetOutputSerialization(v *OutputSerialization) *Selec } // Describes the default server-side encryption to apply to new objects in the -// bucket. If Put Object request does not specify any server-side encryption, -// this default encryption will be applied. +// bucket. If a PUT Object request doesn't specify any server-side encryption, +// this default encryption will be applied. For more information, see PUT Bucket +// encryption (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTencryption.html) +// in the Amazon Simple Storage Service API Reference. type ServerSideEncryptionByDefault struct { _ struct{} `type:"structure"` - // KMS master key ID to use for the default encryption. This parameter is allowed - // if SSEAlgorithm is aws:kms. + // AWS Key Management Service (KMS) customer master key ID to use for the default + // encryption. This parameter is allowed if and only if SSEAlgorithm is set + // to aws:kms. + // + // You can specify the key ID or the Amazon Resource Name (ARN) of the CMK. + // However, if you are using encryption with cross-account operations, you must + // use a fully qualified CMK ARN. For more information, see Using encryption + // for cross-account operations (https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html#bucket-encryption-update-bucket-policy). + // + // For example: + // + // * Key ID: 1234abcd-12ab-34cd-56ef-1234567890ab + // + // * Key ARN: arn:aws:kms:us-east-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab + // + // Amazon S3 only supports symmetric CMKs and not asymmetric CMKs. For more + // information, see Using Symmetric and Asymmetric Keys (https://docs.aws.amazon.com/kms/latest/developerguide/symmetric-asymmetric.html) + // in the AWS Key Management Service Developer Guide. KMSMasterKeyID *string `type:"string" sensitive:"true"` // Server-side encryption algorithm to use for the default encryption. @@ -22838,8 +28746,7 @@ func (s *ServerSideEncryptionByDefault) SetSSEAlgorithm(v string) *ServerSideEnc return s } -// Container for server-side encryption configuration rules. Currently S3 supports -// one rule only. +// Specifies the default server-side-encryption configuration. type ServerSideEncryptionConfiguration struct { _ struct{} `type:"structure"` @@ -22889,13 +28796,12 @@ func (s *ServerSideEncryptionConfiguration) SetRules(v []*ServerSideEncryptionRu return s } -// Container for information about a particular server-side encryption configuration -// rule. +// Specifies the default server-side encryption configuration. type ServerSideEncryptionRule struct { _ struct{} `type:"structure"` - // Describes the default server-side encryption to apply to new objects in the - // bucket. If Put Object request does not specify any server-side encryption, + // Specifies the default server-side encryption to apply to new objects in the + // bucket. If a PUT Object request doesn't specify any server-side encryption, // this default encryption will be applied. ApplyServerSideEncryptionByDefault *ServerSideEncryptionByDefault `type:"structure"` } @@ -22931,13 +28837,17 @@ func (s *ServerSideEncryptionRule) SetApplyServerSideEncryptionByDefault(v *Serv return s } -// A container for filters that define which source objects should be replicated. +// A container that describes additional filters for identifying the source +// objects that you want to replicate. You can choose to enable or disable the +// replication of these objects. Currently, Amazon S3 supports only the filter +// that you can specify for objects created with server-side encryption using +// a customer master key (CMK) stored in AWS Key Management Service (SSE-KMS). type SourceSelectionCriteria struct { _ struct{} `type:"structure"` - // A container for filter information for the selection of S3 objects encrypted - // with AWS KMS. If you include SourceSelectionCriteria in the replication configuration, - // this element is required. + // A container for filter information for the selection of Amazon S3 objects + // encrypted with AWS KMS. If you include SourceSelectionCriteria in the replication + // configuration, this element is required. SseKmsEncryptedObjects *SseKmsEncryptedObjects `type:"structure"` } @@ -22977,8 +28887,8 @@ func (s *SourceSelectionCriteria) SetSseKmsEncryptedObjects(v *SseKmsEncryptedOb type SseKmsEncryptedObjects struct { _ struct{} `type:"structure"` - // If the status is not Enabled, replication for S3 objects encrypted with AWS - // KMS is disabled. + // Specifies whether Amazon S3 replicates objects created with server-side encryption + // using a customer master key (CMK) stored in AWS Key Management Service. // // Status is a required field Status *string `type:"string" required:"true" enum:"SseKmsEncryptedObjectsStatus"` @@ -23013,6 +28923,7 @@ func (s *SseKmsEncryptedObjects) SetStatus(v string) *SseKmsEncryptedObjects { return s } +// Container for the stats details. type Stats struct { _ struct{} `type:"structure"` @@ -23054,6 +28965,7 @@ func (s *Stats) SetBytesScanned(v int64) *Stats { return s } +// Container for the Stats Event. type StatsEvent struct { _ struct{} `locationName:"StatsEvent" type:"structure" payload:"Details"` @@ -23094,11 +29006,26 @@ func (s *StatsEvent) UnmarshalEvent( return nil } +// MarshalEvent marshals the type into an stream event value. This method +// should only used internally within the SDK's EventStream handling. +func (s *StatsEvent) MarshalEvent(pm protocol.PayloadMarshaler) (msg eventstream.Message, err error) { + msg.Headers.Set(eventstreamapi.MessageTypeHeader, eventstream.StringValue(eventstreamapi.EventMessageType)) + var buf bytes.Buffer + if err = pm.MarshalPayload(&buf, s); err != nil { + return eventstream.Message{}, err + } + msg.Payload = buf.Bytes() + return msg, err +} + +// Specifies data related to access patterns to be collected and made available +// to analyze the tradeoffs between different storage classes for an Amazon +// S3 bucket. type StorageClassAnalysis struct { _ struct{} `type:"structure"` - // A container used to describe how data related to the storage class analysis - // should be exported. + // Specifies how data related to the storage class analysis for an Amazon S3 + // bucket should be exported. DataExport *StorageClassAnalysisDataExport `type:"structure"` } @@ -23133,6 +29060,8 @@ func (s *StorageClassAnalysis) SetDataExport(v *StorageClassAnalysisDataExport) return s } +// Container for data related to the storage class analysis for an Amazon S3 +// bucket for export. type StorageClassAnalysisDataExport struct { _ struct{} `type:"structure"` @@ -23190,6 +29119,7 @@ func (s *StorageClassAnalysisDataExport) SetOutputSchemaVersion(v string) *Stora return s } +// A container of a key value name pair. type Tag struct { _ struct{} `type:"structure"` @@ -23245,9 +29175,12 @@ func (s *Tag) SetValue(v string) *Tag { return s } +// Container for TagSet elements. type Tagging struct { _ struct{} `type:"structure"` + // A collection for a set of tags + // // TagSet is a required field TagSet []*Tag `locationNameList:"Tag" type:"list" required:"true"` } @@ -23291,9 +29224,11 @@ func (s *Tagging) SetTagSet(v []*Tag) *Tagging { return s } +// Container for granting information. type TargetGrant struct { _ struct{} `type:"structure"` + // Container for the person being granted permissions. Grantee *Grantee `type:"structure" xmlPrefix:"xsi" xmlURI:"http://www.w3.org/2001/XMLSchema-instance"` // Logging permissions assigned to the Grantee for the bucket. @@ -23338,16 +29273,20 @@ func (s *TargetGrant) SetPermission(v string) *TargetGrant { } // A container for specifying the configuration for publication of messages -// to an Amazon Simple Notification Service (Amazon SNS) topic.when Amazon S3 +// to an Amazon Simple Notification Service (Amazon SNS) topic when Amazon S3 // detects specified events. type TopicConfiguration struct { _ struct{} `type:"structure"` + // The Amazon S3 bucket event about which to send notifications. For more information, + // see Supported Event Types (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) + // in the Amazon Simple Storage Service Developer Guide. + // // Events is a required field Events []*string `locationName:"Event" type:"list" flattened:"true" required:"true"` - // A container for object key name filtering rules. For information about key - // name filtering, see Configuring Event Notifications (http://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) + // Specifies object key name filtering rules. For information about key name + // filtering, see Configuring Event Notifications (https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html) // in the Amazon Simple Storage Service Developer Guide. Filter *NotificationConfigurationFilter `type:"structure"` @@ -23356,7 +29295,7 @@ type TopicConfiguration struct { Id *string `type:"string"` // The Amazon Resource Name (ARN) of the Amazon SNS topic to which Amazon S3 - // will publish a message when it detects events of the specified type. + // publishes a message when it detects events of the specified type. // // TopicArn is a required field TopicArn *string `locationName:"Topic" type:"string" required:"true"` @@ -23412,6 +29351,10 @@ func (s *TopicConfiguration) SetTopicArn(v string) *TopicConfiguration { return s } +// A container for specifying the configuration for publication of messages +// to an Amazon Simple Notification Service (Amazon SNS) topic when Amazon S3 +// detects specified events. This data type is deprecated. Use TopicConfiguration +// instead. type TopicConfigurationDeprecated struct { _ struct{} `type:"structure"` @@ -23420,6 +29363,7 @@ type TopicConfigurationDeprecated struct { // Deprecated: Event has been deprecated Event *string `deprecated:"true" type:"string" enum:"Event"` + // A collection of events related to objects Events []*string `locationName:"Event" type:"list" flattened:"true"` // An optional unique identifier for configurations in a notification configuration. @@ -23465,18 +29409,22 @@ func (s *TopicConfigurationDeprecated) SetTopic(v string) *TopicConfigurationDep return s } +// Specifies when an object transitions to a specified storage class. For more +// information about Amazon S3 lifecycle configuration rules, see Transitioning +// Objects Using Amazon S3 Lifecycle (https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html) +// in the Amazon Simple Storage Service Developer Guide. type Transition struct { _ struct{} `type:"structure"` - // Indicates at what date the object is to be moved or deleted. Should be in - // GMT ISO 8601 Format. + // Indicates when objects are transitioned to the specified storage class. The + // date value must be in ISO 8601 format. The time is always midnight UTC. Date *time.Time `type:"timestamp" timestampFormat:"iso8601"` - // Indicates the lifetime, in days, of the objects that are subject to the rule. - // The value must be a non-zero positive integer. + // Indicates the number of days after creation when objects are transitioned + // to the specified storage class. The value must be a positive integer. Days *int64 `type:"integer"` - // The class of storage used to store the object. + // The storage class to which you want the object to transition. StorageClass *string `type:"string" enum:"TransitionStorageClass"` } @@ -23509,8 +29457,10 @@ func (s *Transition) SetStorageClass(v string) *Transition { } type UploadPartCopyInput struct { - _ struct{} `type:"structure"` + _ struct{} `locationName:"UploadPartCopyRequest" type:"structure"` + // The bucket name. + // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` @@ -23536,23 +29486,26 @@ type UploadPartCopyInput struct { // The range of bytes to copy from the source object. The range value must use // the form bytes=first-last, where the first and last are the zero-based byte // offsets to copy. For example, bytes=0-9 indicates that you want to copy the - // first ten bytes of the source. You can copy a range only if the source object - // is greater than 5 GB. + // first 10 bytes of the source. You can copy a range only if the source object + // is greater than 5 MB. CopySourceRange *string `location:"header" locationName:"x-amz-copy-source-range" type:"string"` - // Specifies the algorithm to use when decrypting the source object (e.g., AES256). + // Specifies the algorithm to use when decrypting the source object (for example, + // AES256). CopySourceSSECustomerAlgorithm *string `location:"header" locationName:"x-amz-copy-source-server-side-encryption-customer-algorithm" type:"string"` // Specifies the customer-provided encryption key for Amazon S3 to use to decrypt // the source object. The encryption key provided in this header must be one // that was used when the source object was created. - CopySourceSSECustomerKey *string `location:"header" locationName:"x-amz-copy-source-server-side-encryption-customer-key" type:"string" sensitive:"true"` + CopySourceSSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-copy-source-server-side-encryption-customer-key" type:"string" sensitive:"true"` // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. - // Amazon S3 uses this header for a message integrity check to ensure the encryption - // key was transmitted without error. + // Amazon S3 uses this header for a message integrity check to ensure that the + // encryption key was transmitted without error. CopySourceSSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-copy-source-server-side-encryption-customer-key-MD5" type:"string"` + // Object key for which the multipart upload was initiated. + // // Key is a required field Key *string `location:"uri" locationName:"Key" min:"1" type:"string" required:"true"` @@ -23562,26 +29515,28 @@ type UploadPartCopyInput struct { // PartNumber is a required field PartNumber *int64 `location:"querystring" locationName:"partNumber" type:"integer" required:"true"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` - // Specifies the algorithm to use to when encrypting the object (e.g., AES256). + // Specifies the algorithm to use to when encrypting the object (for example, + // AES256). SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // Specifies the customer-provided encryption key for Amazon S3 to use in encrypting // data. This value is used to store the object and then it is discarded; Amazon - // does not store the encryption key. The key must be appropriate for use with - // the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm + // S3 does not store the encryption key. The key must be appropriate for use + // with the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm // header. This must be the same encryption key specified in the initiate multipart // upload request. - SSECustomerKey *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` + SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. - // Amazon S3 uses this header for a message integrity check to ensure the encryption - // key was transmitted without error. + // Amazon S3 uses this header for a message integrity check to ensure that the + // encryption key was transmitted without error. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` // Upload ID identifying the multipart upload whose part is being copied. @@ -23754,9 +29709,24 @@ func (s *UploadPartCopyInput) SetUploadId(v string) *UploadPartCopyInput { return s } +func (s *UploadPartCopyInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *UploadPartCopyInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type UploadPartCopyOutput struct { _ struct{} `type:"structure" payload:"CopyPartResult"` + // Container for all response elements. CopyPartResult *CopyPartResult `type:"structure"` // The version of the source object that was copied, if you have enabled versioning @@ -23773,16 +29743,17 @@ type UploadPartCopyOutput struct { SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // If server-side encryption with a customer-provided encryption key was requested, - // the response will include this header to provide round trip message integrity + // the response will include this header to provide round-trip message integrity // verification of the customer-provided encryption key. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` - // If present, specifies the ID of the AWS Key Management Service (KMS) master - // encryption key that was used for the object. + // If present, specifies the ID of the AWS Key Management Service (AWS KMS) + // symmetric customer managed customer master key (CMK) that was used for the + // object. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // The server-side encryption algorithm used when storing this object in Amazon + // S3 (for example, AES256, aws:kms). ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` } @@ -23839,7 +29810,7 @@ func (s *UploadPartCopyOutput) SetServerSideEncryption(v string) *UploadPartCopy } type UploadPartInput struct { - _ struct{} `type:"structure" payload:"Body"` + _ struct{} `locationName:"UploadPartRequest" type:"structure" payload:"Body"` // Object data. Body io.ReadSeeker `type:"blob"` @@ -23853,7 +29824,9 @@ type UploadPartInput struct { // body cannot be determined automatically. ContentLength *int64 `location:"header" locationName:"Content-Length" type:"long"` - // The base64-encoded 128-bit MD5 digest of the part data. + // The base64-encoded 128-bit MD5 digest of the part data. This parameter is + // auto-populated when using the command from the CLI. This parameter is required + // if object lock parameters are specified. ContentMD5 *string `location:"header" locationName:"Content-MD5" type:"string"` // Object key for which the multipart upload was initiated. @@ -23867,26 +29840,28 @@ type UploadPartInput struct { // PartNumber is a required field PartNumber *int64 `location:"querystring" locationName:"partNumber" type:"integer" required:"true"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` - // Specifies the algorithm to use to when encrypting the object (e.g., AES256). + // Specifies the algorithm to use to when encrypting the object (for example, + // AES256). SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // Specifies the customer-provided encryption key for Amazon S3 to use in encrypting // data. This value is used to store the object and then it is discarded; Amazon - // does not store the encryption key. The key must be appropriate for use with - // the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm + // S3 does not store the encryption key. The key must be appropriate for use + // with the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm // header. This must be the same encryption key specified in the initiate multipart // upload request. - SSECustomerKey *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` + SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. - // Amazon S3 uses this header for a message integrity check to ensure the encryption - // key was transmitted without error. + // Amazon S3 uses this header for a message integrity check to ensure that the + // encryption key was transmitted without error. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` // Upload ID identifying the multipart upload whose part is being uploaded. @@ -24013,6 +29988,20 @@ func (s *UploadPartInput) SetUploadId(v string) *UploadPartInput { return s } +func (s *UploadPartInput) getEndpointARN() (arn.Resource, error) { + if s.Bucket == nil { + return nil, fmt.Errorf("member Bucket is nil") + } + return parseEndpointARN(*s.Bucket) +} + +func (s *UploadPartInput) hasEndpointARN() bool { + if s.Bucket == nil { + return false + } + return arn.IsARN(*s.Bucket) +} + type UploadPartOutput struct { _ struct{} `type:"structure"` @@ -24029,16 +30018,16 @@ type UploadPartOutput struct { SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // If server-side encryption with a customer-provided encryption key was requested, - // the response will include this header to provide round trip message integrity + // the response will include this header to provide round-trip message integrity // verification of the customer-provided encryption key. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` - // If present, specifies the ID of the AWS Key Management Service (KMS) master - // encryption key that was used for the object. + // If present, specifies the ID of the AWS Key Management Service (AWS KMS) + // symmetric customer managed customer master key (CMK) was used for the object. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // The server-side encryption algorithm used when storing this object in Amazon + // S3 (for example, AES256, aws:kms). ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` } @@ -24088,6 +30077,9 @@ func (s *UploadPartOutput) SetServerSideEncryption(v string) *UploadPartOutput { return s } +// Describes the versioning state of an Amazon S3 bucket. For more information, +// see PUT Bucket versioning (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUTVersioningStatus.html) +// in the Amazon Simple Storage Service API Reference. type VersioningConfiguration struct { _ struct{} `type:"structure"` @@ -24122,15 +30114,22 @@ func (s *VersioningConfiguration) SetStatus(v string) *VersioningConfiguration { return s } +// Specifies website configuration parameters for an Amazon S3 bucket. type WebsiteConfiguration struct { _ struct{} `type:"structure"` + // The name of the error document for the website. ErrorDocument *ErrorDocument `type:"structure"` + // The name of the index document for the website. IndexDocument *IndexDocument `type:"structure"` + // The redirect behavior for every request to this bucket's website endpoint. + // + // If you specify this property, you can't specify any other property. RedirectAllRequestsTo *RedirectAllRequestsTo `type:"structure"` + // Rules that define when a redirect is applied and the redirect behavior. RoutingRules []*RoutingRule `locationNameList:"RoutingRule" type:"list"` } @@ -24343,11 +30342,37 @@ const ( // EventS3ObjectRemovedDeleteMarkerCreated is a Event enum value EventS3ObjectRemovedDeleteMarkerCreated = "s3:ObjectRemoved:DeleteMarkerCreated" + // EventS3ObjectRestore is a Event enum value + EventS3ObjectRestore = "s3:ObjectRestore:*" + // EventS3ObjectRestorePost is a Event enum value EventS3ObjectRestorePost = "s3:ObjectRestore:Post" // EventS3ObjectRestoreCompleted is a Event enum value EventS3ObjectRestoreCompleted = "s3:ObjectRestore:Completed" + + // EventS3Replication is a Event enum value + EventS3Replication = "s3:Replication:*" + + // EventS3ReplicationOperationFailedReplication is a Event enum value + EventS3ReplicationOperationFailedReplication = "s3:Replication:OperationFailedReplication" + + // EventS3ReplicationOperationNotTracked is a Event enum value + EventS3ReplicationOperationNotTracked = "s3:Replication:OperationNotTracked" + + // EventS3ReplicationOperationMissedThreshold is a Event enum value + EventS3ReplicationOperationMissedThreshold = "s3:Replication:OperationMissedThreshold" + + // EventS3ReplicationOperationReplicatedAfterThreshold is a Event enum value + EventS3ReplicationOperationReplicatedAfterThreshold = "s3:Replication:OperationReplicatedAfterThreshold" +) + +const ( + // ExistingObjectReplicationStatusEnabled is a ExistingObjectReplicationStatus enum value + ExistingObjectReplicationStatusEnabled = "Enabled" + + // ExistingObjectReplicationStatusDisabled is a ExistingObjectReplicationStatus enum value + ExistingObjectReplicationStatusDisabled = "Disabled" ) const ( @@ -24439,6 +30464,9 @@ const ( // InventoryOptionalFieldObjectLockLegalHoldStatus is a InventoryOptionalField enum value InventoryOptionalFieldObjectLockLegalHoldStatus = "ObjectLockLegalHoldStatus" + + // InventoryOptionalFieldIntelligentTieringAccessTier is a InventoryOptionalField enum value + InventoryOptionalFieldIntelligentTieringAccessTier = "IntelligentTieringAccessTier" ) const ( @@ -24473,6 +30501,14 @@ const ( MetadataDirectiveReplace = "REPLACE" ) +const ( + // MetricsStatusEnabled is a MetricsStatus enum value + MetricsStatusEnabled = "Enabled" + + // MetricsStatusDisabled is a MetricsStatus enum value + MetricsStatusDisabled = "Disabled" +) + const ( // ObjectCannedACLPrivate is a ObjectCannedACL enum value ObjectCannedACLPrivate = "private" @@ -24543,6 +30579,9 @@ const ( // ObjectStorageClassIntelligentTiering is a ObjectStorageClass enum value ObjectStorageClassIntelligentTiering = "INTELLIGENT_TIERING" + + // ObjectStorageClassDeepArchive is a ObjectStorageClass enum value + ObjectStorageClassDeepArchive = "DEEP_ARCHIVE" ) const ( @@ -24618,6 +30657,14 @@ const ( ReplicationStatusReplica = "REPLICA" ) +const ( + // ReplicationTimeStatusEnabled is a ReplicationTimeStatus enum value + ReplicationTimeStatusEnabled = "Enabled" + + // ReplicationTimeStatusDisabled is a ReplicationTimeStatus enum value + ReplicationTimeStatusDisabled = "Disabled" +) + // If present, indicates that the requester was successfully charged for the // request. const ( @@ -24625,10 +30672,11 @@ const ( RequestChargedRequester = "requester" ) -// Confirms that the requester knows that she or he will be charged for the -// request. Bucket owners need not specify this parameter in their requests. -// Documentation on downloading objects from requester pays buckets can be found -// at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html +// Confirms that the requester knows that they will be charged for the request. +// Bucket owners need not specify this parameter in their requests. For information +// about downloading objects from requester pays buckets, see Downloading Objects +// in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) +// in the Amazon S3 Developer Guide. const ( // RequestPayerRequester is a RequestPayer enum value RequestPayerRequester = "requester" @@ -24673,6 +30721,9 @@ const ( // StorageClassGlacier is a StorageClass enum value StorageClassGlacier = "GLACIER" + + // StorageClassDeepArchive is a StorageClass enum value + StorageClassDeepArchive = "DEEP_ARCHIVE" ) const ( @@ -24711,6 +30762,9 @@ const ( // TransitionStorageClassIntelligentTiering is a TransitionStorageClass enum value TransitionStorageClassIntelligentTiering = "INTELLIGENT_TIERING" + + // TransitionStorageClassDeepArchive is a TransitionStorageClass enum value + TransitionStorageClassDeepArchive = "DEEP_ARCHIVE" ) const ( diff --git a/github.com/aws/aws-sdk-go/service/s3/body_hash.go b/github.com/aws/aws-sdk-go/service/s3/body_hash.go index 5c8ce5cc8a..407f06b6ed 100644 --- a/github.com/aws/aws-sdk-go/service/s3/body_hash.go +++ b/github.com/aws/aws-sdk-go/service/s3/body_hash.go @@ -13,7 +13,6 @@ import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/request" - "github.com/aws/aws-sdk-go/internal/sdkio" ) const ( @@ -25,30 +24,6 @@ const ( appendMD5TxEncoding = "append-md5" ) -// contentMD5 computes and sets the HTTP Content-MD5 header for requests that -// require it. -func contentMD5(r *request.Request) { - h := md5.New() - - if !aws.IsReaderSeekable(r.Body) { - if r.Config.Logger != nil { - r.Config.Logger.Log(fmt.Sprintf( - "Unable to compute Content-MD5 for unseekable body, S3.%s", - r.Operation.Name)) - } - return - } - - if _, err := copySeekableBody(h, r.Body); err != nil { - r.Error = awserr.New("ContentMD5", "failed to compute body MD5", err) - return - } - - // encode the md5 checksum in base64 and set the request header. - v := base64.StdEncoding.EncodeToString(h.Sum(nil)) - r.HTTPRequest.Header.Set(contentMD5Header, v) -} - // computeBodyHashes will add Content MD5 and Content Sha256 hashes to the // request. If the body is not seekable or S3DisableContentMD5Validation set // this handler will be ignored. @@ -90,7 +65,7 @@ func computeBodyHashes(r *request.Request) { dst = io.MultiWriter(hashers...) } - if _, err := copySeekableBody(dst, r.Body); err != nil { + if _, err := aws.CopySeekableBody(dst, r.Body); err != nil { r.Error = awserr.New("BodyHashError", "failed to compute body hashes", err) return } @@ -119,28 +94,6 @@ const ( sha256HexEncLen = sha256.Size * 2 // hex.EncodedLen ) -func copySeekableBody(dst io.Writer, src io.ReadSeeker) (int64, error) { - curPos, err := src.Seek(0, sdkio.SeekCurrent) - if err != nil { - return 0, err - } - - // hash the body. seek back to the first position after reading to reset - // the body for transmission. copy errors may be assumed to be from the - // body. - n, err := io.Copy(dst, src) - if err != nil { - return n, err - } - - _, err = src.Seek(curPos, sdkio.SeekStart) - if err != nil { - return n, err - } - - return n, nil -} - // Adds the x-amz-te: append_md5 header to the request. This requests the service // responds with a trailing MD5 checksum. // diff --git a/github.com/aws/aws-sdk-go/service/s3/bucket_location.go b/github.com/aws/aws-sdk-go/service/s3/bucket_location.go index bc68a46acf..9ba8a78872 100644 --- a/github.com/aws/aws-sdk-go/service/s3/bucket_location.go +++ b/github.com/aws/aws-sdk-go/service/s3/bucket_location.go @@ -80,7 +80,8 @@ func buildGetBucketLocation(r *request.Request) { out := r.Data.(*GetBucketLocationOutput) b, err := ioutil.ReadAll(r.HTTPResponse.Body) if err != nil { - r.Error = awserr.New("SerializationError", "failed reading response body", err) + r.Error = awserr.New(request.ErrCodeSerialization, + "failed reading response body", err) return } diff --git a/github.com/aws/aws-sdk-go/service/s3/customizations.go b/github.com/aws/aws-sdk-go/service/s3/customizations.go index 95f2456363..a7698d5eb9 100644 --- a/github.com/aws/aws-sdk-go/service/s3/customizations.go +++ b/github.com/aws/aws-sdk-go/service/s3/customizations.go @@ -4,6 +4,7 @@ import ( "github.com/aws/aws-sdk-go/aws/client" "github.com/aws/aws-sdk-go/aws/request" "github.com/aws/aws-sdk-go/internal/s3err" + "github.com/aws/aws-sdk-go/service/s3/internal/arn" ) func init() { @@ -13,11 +14,12 @@ func init() { func defaultInitClientFn(c *client.Client) { // Support building custom endpoints based on config - c.Handlers.Build.PushFront(updateEndpointForS3Config) + c.Handlers.Build.PushFront(endpointHandler) // Require SSL when using SSE keys c.Handlers.Validate.PushBack(validateSSERequiresSSL) - c.Handlers.Build.PushBack(computeSSEKeys) + c.Handlers.Build.PushBack(computeSSEKeyMD5) + c.Handlers.Build.PushBack(computeCopySourceSSEKeyMD5) // S3 uses custom error unmarshaling logic c.Handlers.UnmarshalError.Clear() @@ -26,17 +28,11 @@ func defaultInitClientFn(c *client.Client) { } func defaultInitRequestFn(r *request.Request) { - // Add reuest handlers for specific platforms. + // Add request handlers for specific platforms. // e.g. 100-continue support for PUT requests using Go 1.6 platformRequestHandlers(r) switch r.Operation.Name { - case opPutBucketCors, opPutBucketLifecycle, opPutBucketPolicy, - opPutBucketTagging, opDeleteObjects, opPutBucketLifecycleConfiguration, - opPutObjectLegalHold, opPutObjectRetention, opPutObjectLockConfiguration, - opPutBucketReplication: - // These S3 operations require Content-MD5 to be set - r.Handlers.Build.PushBack(contentMD5) case opGetBucketLocation: // GetBucketLocation has custom parsing logic r.Handlers.Unmarshal.PushFront(buildGetBucketLocation) @@ -72,3 +68,8 @@ type sseCustomerKeyGetter interface { type copySourceSSECustomerKeyGetter interface { getCopySourceSSECustomerKey() string } + +type endpointARNGetter interface { + getEndpointARN() (arn.Resource, error) + hasEndpointARN() bool +} diff --git a/github.com/aws/aws-sdk-go/service/s3/doc_custom.go b/github.com/aws/aws-sdk-go/service/s3/doc_custom.go index 39b912c260..4b65f71531 100644 --- a/github.com/aws/aws-sdk-go/service/s3/doc_custom.go +++ b/github.com/aws/aws-sdk-go/service/s3/doc_custom.go @@ -63,6 +63,20 @@ // See the s3manager package's Downloader type documentation for more information. // https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#Downloader // +// Automatic URI cleaning +// +// Interacting with objects whose keys contain adjacent slashes (e.g. bucketname/foo//bar/objectname) +// requires setting DisableRestProtocolURICleaning to true in the aws.Config struct +// used by the service client. +// +// svc := s3.New(sess, &aws.Config{ +// DisableRestProtocolURICleaning: aws.Bool(true), +// }) +// out, err := svc.GetObject(&s3.GetObjectInput { +// Bucket: aws.String("bucketname"), +// Key: aws.String("//foo//bar//moo"), +// }) +// // Get Bucket Region // // GetBucketRegion will attempt to get the region for a bucket using a region diff --git a/github.com/aws/aws-sdk-go/service/s3/endpoint.go b/github.com/aws/aws-sdk-go/service/s3/endpoint.go new file mode 100644 index 0000000000..c4048fbfb6 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/endpoint.go @@ -0,0 +1,233 @@ +package s3 + +import ( + "net/url" + "strings" + + "github.com/aws/aws-sdk-go/aws" + awsarn "github.com/aws/aws-sdk-go/aws/arn" + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/aws/endpoints" + "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/private/protocol" + "github.com/aws/aws-sdk-go/service/s3/internal/arn" +) + +// Used by shapes with members decorated as endpoint ARN. +func parseEndpointARN(v string) (arn.Resource, error) { + return arn.ParseResource(v, accessPointResourceParser) +} + +func accessPointResourceParser(a awsarn.ARN) (arn.Resource, error) { + resParts := arn.SplitResource(a.Resource) + switch resParts[0] { + case "accesspoint": + return arn.ParseAccessPointResource(a, resParts[1:]) + default: + return nil, arn.InvalidARNError{ARN: a, Reason: "unknown resource type"} + } +} + +func endpointHandler(req *request.Request) { + endpoint, ok := req.Params.(endpointARNGetter) + if !ok || !endpoint.hasEndpointARN() { + updateBucketEndpointFromParams(req) + return + } + + resource, err := endpoint.getEndpointARN() + if err != nil { + req.Error = newInvalidARNError(nil, err) + return + } + + resReq := resourceRequest{ + Resource: resource, + Request: req, + } + + if resReq.IsCrossPartition() { + req.Error = newClientPartitionMismatchError(resource, + req.ClientInfo.PartitionID, aws.StringValue(req.Config.Region), nil) + return + } + + if !resReq.AllowCrossRegion() && resReq.IsCrossRegion() { + req.Error = newClientRegionMismatchError(resource, + req.ClientInfo.PartitionID, aws.StringValue(req.Config.Region), nil) + return + } + + if resReq.HasCustomEndpoint() { + req.Error = newInvalidARNWithCustomEndpointError(resource, nil) + return + } + + switch tv := resource.(type) { + case arn.AccessPointARN: + err = updateRequestAccessPointEndpoint(req, tv) + if err != nil { + req.Error = err + } + default: + req.Error = newInvalidARNError(resource, nil) + } +} + +type resourceRequest struct { + Resource arn.Resource + Request *request.Request +} + +func (r resourceRequest) ARN() awsarn.ARN { + return r.Resource.GetARN() +} + +func (r resourceRequest) AllowCrossRegion() bool { + return aws.BoolValue(r.Request.Config.S3UseARNRegion) +} + +func (r resourceRequest) UseFIPS() bool { + return isFIPS(aws.StringValue(r.Request.Config.Region)) +} + +func (r resourceRequest) IsCrossPartition() bool { + return r.Request.ClientInfo.PartitionID != r.Resource.GetARN().Partition +} + +func (r resourceRequest) IsCrossRegion() bool { + return isCrossRegion(r.Request, r.Resource.GetARN().Region) +} + +func (r resourceRequest) HasCustomEndpoint() bool { + return len(aws.StringValue(r.Request.Config.Endpoint)) > 0 +} + +func isFIPS(clientRegion string) bool { + return strings.HasPrefix(clientRegion, "fips-") || strings.HasSuffix(clientRegion, "-fips") +} +func isCrossRegion(req *request.Request, otherRegion string) bool { + return req.ClientInfo.SigningRegion != otherRegion +} + +func updateBucketEndpointFromParams(r *request.Request) { + bucket, ok := bucketNameFromReqParams(r.Params) + if !ok { + // Ignore operation requests if the bucket name was not provided + // if this is an input validation error the validation handler + // will report it. + return + } + updateEndpointForS3Config(r, bucket) +} + +func updateRequestAccessPointEndpoint(req *request.Request, accessPoint arn.AccessPointARN) error { + // Accelerate not supported + if aws.BoolValue(req.Config.S3UseAccelerate) { + return newClientConfiguredForAccelerateError(accessPoint, + req.ClientInfo.PartitionID, aws.StringValue(req.Config.Region), nil) + } + + // Ignore the disable host prefix for access points since custom endpoints + // are not supported. + req.Config.DisableEndpointHostPrefix = aws.Bool(false) + + if err := accessPointEndpointBuilder(accessPoint).Build(req); err != nil { + return err + } + + removeBucketFromPath(req.HTTPRequest.URL) + + return nil +} + +func removeBucketFromPath(u *url.URL) { + u.Path = strings.Replace(u.Path, "/{Bucket}", "", -1) + if u.Path == "" { + u.Path = "/" + } +} + +type accessPointEndpointBuilder arn.AccessPointARN + +const ( + accessPointPrefixLabel = "accesspoint" + accountIDPrefixLabel = "accountID" + accesPointPrefixTemplate = "{" + accessPointPrefixLabel + "}-{" + accountIDPrefixLabel + "}." +) + +func (a accessPointEndpointBuilder) Build(req *request.Request) error { + resolveRegion := arn.AccessPointARN(a).Region + cfgRegion := aws.StringValue(req.Config.Region) + + if isFIPS(cfgRegion) { + if aws.BoolValue(req.Config.S3UseARNRegion) && isCrossRegion(req, resolveRegion) { + // FIPS with cross region is not supported, the SDK must fail + // because there is no well defined method for SDK to construct a + // correct FIPS endpoint. + return newClientConfiguredForCrossRegionFIPSError(arn.AccessPointARN(a), + req.ClientInfo.PartitionID, cfgRegion, nil) + } + resolveRegion = cfgRegion + } + + endpoint, err := resolveRegionalEndpoint(req, resolveRegion) + if err != nil { + return newFailedToResolveEndpointError(arn.AccessPointARN(a), + req.ClientInfo.PartitionID, cfgRegion, err) + } + + if err = updateRequestEndpoint(req, endpoint.URL); err != nil { + return err + } + + const serviceEndpointLabel = "s3-accesspoint" + + // dualstack provided by endpoint resolver + cfgHost := req.HTTPRequest.URL.Host + if strings.HasPrefix(cfgHost, "s3") { + req.HTTPRequest.URL.Host = serviceEndpointLabel + cfgHost[2:] + } + + protocol.HostPrefixBuilder{ + Prefix: accesPointPrefixTemplate, + LabelsFn: a.hostPrefixLabelValues, + }.Build(req) + + req.ClientInfo.SigningName = endpoint.SigningName + req.ClientInfo.SigningRegion = endpoint.SigningRegion + + err = protocol.ValidateEndpointHost(req.Operation.Name, req.HTTPRequest.URL.Host) + if err != nil { + return newInvalidARNError(arn.AccessPointARN(a), err) + } + + return nil +} + +func (a accessPointEndpointBuilder) hostPrefixLabelValues() map[string]string { + return map[string]string{ + accessPointPrefixLabel: arn.AccessPointARN(a).AccessPointName, + accountIDPrefixLabel: arn.AccessPointARN(a).AccountID, + } +} + +func resolveRegionalEndpoint(r *request.Request, region string) (endpoints.ResolvedEndpoint, error) { + return r.Config.EndpointResolver.EndpointFor(EndpointsID, region, func(opts *endpoints.Options) { + opts.DisableSSL = aws.BoolValue(r.Config.DisableSSL) + opts.UseDualStack = aws.BoolValue(r.Config.UseDualStack) + opts.S3UsEast1RegionalEndpoint = endpoints.RegionalS3UsEast1Endpoint + }) +} + +func updateRequestEndpoint(r *request.Request, endpoint string) (err error) { + endpoint = endpoints.AddScheme(endpoint, aws.BoolValue(r.Config.DisableSSL)) + + r.HTTPRequest.URL, err = url.Parse(endpoint + r.Operation.HTTPPath) + if err != nil { + return awserr.New(request.ErrCodeSerialization, + "failed to parse endpoint URL", err) + } + + return nil +} diff --git a/github.com/aws/aws-sdk-go/service/s3/endpoint_errors.go b/github.com/aws/aws-sdk-go/service/s3/endpoint_errors.go new file mode 100644 index 0000000000..9df03e78d3 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/endpoint_errors.go @@ -0,0 +1,151 @@ +package s3 + +import ( + "fmt" + + "github.com/aws/aws-sdk-go/aws/awserr" + "github.com/aws/aws-sdk-go/service/s3/internal/arn" +) + +const ( + invalidARNErrorErrCode = "InvalidARNError" + configurationErrorErrCode = "ConfigurationError" +) + +type invalidARNError struct { + message string + resource arn.Resource + origErr error +} + +func (e invalidARNError) Error() string { + var extra string + if e.resource != nil { + extra = "ARN: " + e.resource.String() + } + return awserr.SprintError(e.Code(), e.Message(), extra, e.origErr) +} + +func (e invalidARNError) Code() string { + return invalidARNErrorErrCode +} + +func (e invalidARNError) Message() string { + return e.message +} + +func (e invalidARNError) OrigErr() error { + return e.origErr +} + +func newInvalidARNError(resource arn.Resource, err error) invalidARNError { + return invalidARNError{ + message: "invalid ARN", + origErr: err, + resource: resource, + } +} + +func newInvalidARNWithCustomEndpointError(resource arn.Resource, err error) invalidARNError { + return invalidARNError{ + message: "resource ARN not supported with custom client endpoints", + origErr: err, + resource: resource, + } +} + +// ARN not supported for the target partition +func newInvalidARNWithUnsupportedPartitionError(resource arn.Resource, err error) invalidARNError { + return invalidARNError{ + message: "resource ARN not supported for the target ARN partition", + origErr: err, + resource: resource, + } +} + +type configurationError struct { + message string + resource arn.Resource + clientPartitionID string + clientRegion string + origErr error +} + +func (e configurationError) Error() string { + extra := fmt.Sprintf("ARN: %s, client partition: %s, client region: %s", + e.resource, e.clientPartitionID, e.clientRegion) + + return awserr.SprintError(e.Code(), e.Message(), extra, e.origErr) +} + +func (e configurationError) Code() string { + return configurationErrorErrCode +} + +func (e configurationError) Message() string { + return e.message +} + +func (e configurationError) OrigErr() error { + return e.origErr +} + +func newClientPartitionMismatchError(resource arn.Resource, clientPartitionID, clientRegion string, err error) configurationError { + return configurationError{ + message: "client partition does not match provided ARN partition", + origErr: err, + resource: resource, + clientPartitionID: clientPartitionID, + clientRegion: clientRegion, + } +} + +func newClientRegionMismatchError(resource arn.Resource, clientPartitionID, clientRegion string, err error) configurationError { + return configurationError{ + message: "client region does not match provided ARN region", + origErr: err, + resource: resource, + clientPartitionID: clientPartitionID, + clientRegion: clientRegion, + } +} + +func newFailedToResolveEndpointError(resource arn.Resource, clientPartitionID, clientRegion string, err error) configurationError { + return configurationError{ + message: "endpoint resolver failed to find an endpoint for the provided ARN region", + origErr: err, + resource: resource, + clientPartitionID: clientPartitionID, + clientRegion: clientRegion, + } +} + +func newClientConfiguredForFIPSError(resource arn.Resource, clientPartitionID, clientRegion string, err error) configurationError { + return configurationError{ + message: "client configured for fips but cross-region resource ARN provided", + origErr: err, + resource: resource, + clientPartitionID: clientPartitionID, + clientRegion: clientRegion, + } +} + +func newClientConfiguredForAccelerateError(resource arn.Resource, clientPartitionID, clientRegion string, err error) configurationError { + return configurationError{ + message: "client configured for S3 Accelerate but is supported with resource ARN", + origErr: err, + resource: resource, + clientPartitionID: clientPartitionID, + clientRegion: clientRegion, + } +} + +func newClientConfiguredForCrossRegionFIPSError(resource arn.Resource, clientPartitionID, clientRegion string, err error) configurationError { + return configurationError{ + message: "client configured for FIPS with cross-region enabled but is supported with cross-region resource ARN", + origErr: err, + resource: resource, + clientPartitionID: clientPartitionID, + clientRegion: clientRegion, + } +} diff --git a/github.com/aws/aws-sdk-go/service/s3/errors.go b/github.com/aws/aws-sdk-go/service/s3/errors.go index 931cb17bb0..49aeff16f2 100644 --- a/github.com/aws/aws-sdk-go/service/s3/errors.go +++ b/github.com/aws/aws-sdk-go/service/s3/errors.go @@ -13,6 +13,12 @@ const ( // ErrCodeBucketAlreadyOwnedByYou for service response error code // "BucketAlreadyOwnedByYou". + // + // The bucket you tried to create already exists, and you own it. Amazon S3 + // returns this error in all AWS Regions except in the North Virginia Region. + // For legacy compatibility, if you re-create an existing bucket that you already + // own in the North Virginia Region, Amazon S3 returns 200 OK and resets the + // bucket access control lists (ACLs). ErrCodeBucketAlreadyOwnedByYou = "BucketAlreadyOwnedByYou" // ErrCodeNoSuchBucket for service response error code @@ -36,13 +42,13 @@ const ( // ErrCodeObjectAlreadyInActiveTierError for service response error code // "ObjectAlreadyInActiveTierError". // - // This operation is not allowed against this storage tier + // This operation is not allowed against this storage tier. ErrCodeObjectAlreadyInActiveTierError = "ObjectAlreadyInActiveTierError" // ErrCodeObjectNotInActiveTierError for service response error code // "ObjectNotInActiveTierError". // // The source object of the COPY operation is not in the active tier and is - // only stored in Amazon Glacier. + // only stored in Amazon S3 Glacier. ErrCodeObjectNotInActiveTierError = "ObjectNotInActiveTierError" ) diff --git a/github.com/aws/aws-sdk-go/service/s3/host_style_bucket.go b/github.com/aws/aws-sdk-go/service/s3/host_style_bucket.go index a7fbc2de2f..81cdec1ae7 100644 --- a/github.com/aws/aws-sdk-go/service/s3/host_style_bucket.go +++ b/github.com/aws/aws-sdk-go/service/s3/host_style_bucket.go @@ -30,10 +30,10 @@ var accelerateOpBlacklist = operationBlacklist{ opListBuckets, opCreateBucket, opDeleteBucket, } -// Request handler to automatically add the bucket name to the endpoint domain +// Automatically add the bucket name to the endpoint domain // if possible. This style of bucket is valid for all bucket names which are // DNS compatible and do not contain "." -func updateEndpointForS3Config(r *request.Request) { +func updateEndpointForS3Config(r *request.Request, bucketName string) { forceHostStyle := aws.BoolValue(r.Config.S3ForcePathStyle) accelerate := aws.BoolValue(r.Config.S3UseAccelerate) @@ -43,45 +43,29 @@ func updateEndpointForS3Config(r *request.Request) { r.Config.Logger.Log("ERROR: aws.Config.S3UseAccelerate is not compatible with aws.Config.S3ForcePathStyle, ignoring S3ForcePathStyle.") } } - updateEndpointForAccelerate(r) + updateEndpointForAccelerate(r, bucketName) } else if !forceHostStyle && r.Operation.Name != opGetBucketLocation { - updateEndpointForHostStyle(r) + updateEndpointForHostStyle(r, bucketName) } } -func updateEndpointForHostStyle(r *request.Request) { - bucket, ok := bucketNameFromReqParams(r.Params) - if !ok { - // Ignore operation requests if the bucketname was not provided - // if this is an input validation error the validation handler - // will report it. - return - } - - if !hostCompatibleBucketName(r.HTTPRequest.URL, bucket) { +func updateEndpointForHostStyle(r *request.Request, bucketName string) { + if !hostCompatibleBucketName(r.HTTPRequest.URL, bucketName) { // bucket name must be valid to put into the host return } - moveBucketToHost(r.HTTPRequest.URL, bucket) + moveBucketToHost(r.HTTPRequest.URL, bucketName) } var ( accelElem = []byte("s3-accelerate.dualstack.") ) -func updateEndpointForAccelerate(r *request.Request) { - bucket, ok := bucketNameFromReqParams(r.Params) - if !ok { - // Ignore operation requests if the bucketname was not provided - // if this is an input validation error the validation handler - // will report it. - return - } - - if !hostCompatibleBucketName(r.HTTPRequest.URL, bucket) { +func updateEndpointForAccelerate(r *request.Request, bucketName string) { + if !hostCompatibleBucketName(r.HTTPRequest.URL, bucketName) { r.Error = awserr.New("InvalidParameterException", - fmt.Sprintf("bucket name %s is not compatible with S3 Accelerate", bucket), + fmt.Sprintf("bucket name %s is not compatible with S3 Accelerate", bucketName), nil) return } @@ -106,7 +90,7 @@ func updateEndpointForAccelerate(r *request.Request) { r.HTTPRequest.URL.Host = strings.Join(parts, ".") - moveBucketToHost(r.HTTPRequest.URL, bucket) + moveBucketToHost(r.HTTPRequest.URL, bucketName) } // Attempts to retrieve the bucket name from the request input parameters. @@ -148,8 +132,5 @@ func dnsCompatibleBucketName(bucket string) bool { // moveBucketToHost moves the bucket name from the URI path to URL host. func moveBucketToHost(u *url.URL, bucket string) { u.Host = bucket + "." + u.Host - u.Path = strings.Replace(u.Path, "/{Bucket}", "", -1) - if u.Path == "" { - u.Path = "/" - } + removeBucketFromPath(u) } diff --git a/github.com/aws/aws-sdk-go/service/s3/internal/arn/accesspoint_arn.go b/github.com/aws/aws-sdk-go/service/s3/internal/arn/accesspoint_arn.go new file mode 100644 index 0000000000..2f93f96fd5 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/internal/arn/accesspoint_arn.go @@ -0,0 +1,45 @@ +package arn + +import ( + "strings" + + "github.com/aws/aws-sdk-go/aws/arn" +) + +// AccessPointARN provides representation +type AccessPointARN struct { + arn.ARN + AccessPointName string +} + +// GetARN returns the base ARN for the Access Point resource +func (a AccessPointARN) GetARN() arn.ARN { + return a.ARN +} + +// ParseAccessPointResource attempts to parse the ARN's resource as an +// AccessPoint resource. +func ParseAccessPointResource(a arn.ARN, resParts []string) (AccessPointARN, error) { + if len(a.Region) == 0 { + return AccessPointARN{}, InvalidARNError{a, "region not set"} + } + if len(a.AccountID) == 0 { + return AccessPointARN{}, InvalidARNError{a, "account-id not set"} + } + if len(resParts) == 0 { + return AccessPointARN{}, InvalidARNError{a, "resource-id not set"} + } + if len(resParts) > 1 { + return AccessPointARN{}, InvalidARNError{a, "sub resource not supported"} + } + + resID := resParts[0] + if len(strings.TrimSpace(resID)) == 0 { + return AccessPointARN{}, InvalidARNError{a, "resource-id not set"} + } + + return AccessPointARN{ + ARN: a, + AccessPointName: resID, + }, nil +} diff --git a/github.com/aws/aws-sdk-go/service/s3/internal/arn/arn.go b/github.com/aws/aws-sdk-go/service/s3/internal/arn/arn.go new file mode 100644 index 0000000000..a942d887f7 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/internal/arn/arn.go @@ -0,0 +1,71 @@ +package arn + +import ( + "strings" + + "github.com/aws/aws-sdk-go/aws/arn" +) + +// Resource provides the interfaces abstracting ARNs of specific resource +// types. +type Resource interface { + GetARN() arn.ARN + String() string +} + +// ResourceParser provides the function for parsing an ARN's resource +// component into a typed resource. +type ResourceParser func(arn.ARN) (Resource, error) + +// ParseResource parses an AWS ARN into a typed resource for the S3 API. +func ParseResource(s string, resParser ResourceParser) (resARN Resource, err error) { + a, err := arn.Parse(s) + if err != nil { + return nil, err + } + + if len(a.Partition) == 0 { + return nil, InvalidARNError{a, "partition not set"} + } + if a.Service != "s3" { + return nil, InvalidARNError{a, "service is not S3"} + } + if len(a.Resource) == 0 { + return nil, InvalidARNError{a, "resource not set"} + } + + return resParser(a) +} + +// SplitResource splits the resource components by the ARN resource delimiters. +func SplitResource(v string) []string { + var parts []string + var offset int + + for offset <= len(v) { + idx := strings.IndexAny(v[offset:], "/:") + if idx < 0 { + parts = append(parts, v[offset:]) + break + } + parts = append(parts, v[offset:idx+offset]) + offset += idx + 1 + } + + return parts +} + +// IsARN returns whether the given string is an ARN +func IsARN(s string) bool { + return arn.IsARN(s) +} + +// InvalidARNError provides the error for an invalid ARN error. +type InvalidARNError struct { + ARN arn.ARN + Reason string +} + +func (e InvalidARNError) Error() string { + return "invalid Amazon S3 ARN, " + e.Reason + ", " + e.ARN.String() +} diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/batch.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/batch.go index 18215574bf..22bd0b7ce5 100644 --- a/github.com/aws/aws-sdk-go/service/s3/s3manager/batch.go +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/batch.go @@ -273,7 +273,7 @@ type DeleteObjectsIterator struct { inc bool } -// Next will increment the default iterator's index and and ensure that there +// Next will increment the default iterator's index and ensure that there // is another object to iterator to. func (iter *DeleteObjectsIterator) Next() bool { if iter.inc { @@ -458,7 +458,7 @@ type DownloadObjectsIterator struct { inc bool } -// Next will increment the default iterator's index and and ensure that there +// Next will increment the default iterator's index and ensure that there // is another object to iterator to. func (batcher *DownloadObjectsIterator) Next() bool { if batcher.inc { @@ -497,7 +497,7 @@ type UploadObjectsIterator struct { inc bool } -// Next will increment the default iterator's index and and ensure that there +// Next will increment the default iterator's index and ensure that there // is another object to iterator to. func (batcher *UploadObjectsIterator) Next() bool { if batcher.inc { diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/bucket_region.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/bucket_region.go index f61665a58a..9cc1e5970c 100644 --- a/github.com/aws/aws-sdk-go/service/s3/s3manager/bucket_region.go +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/bucket_region.go @@ -3,6 +3,7 @@ package s3manager import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/client" + "github.com/aws/aws-sdk-go/aws/corehandlers" "github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/request" "github.com/aws/aws-sdk-go/service/s3" @@ -35,6 +36,30 @@ import ( // } // fmt.Printf("Bucket %s is in %s region\n", bucket, region) // +// By default the request will be made to the Amazon S3 endpoint using the Path +// style addressing. +// +// s3.us-west-2.amazonaws.com/bucketname +// +// This is not compatible with Amazon S3's FIPS endpoints. To override this +// behavior to use Virtual Host style addressing, provide a functional option +// that will set the Request's Config.S3ForcePathStyle to aws.Bool(false). +// +// region, err := s3manager.GetBucketRegion(ctx, sess, "bucketname", "us-west-2", func(r *request.Request) { +// r.S3ForcePathStyle = aws.Bool(false) +// }) +// +// To configure the GetBucketRegion to make a request via the Amazon +// S3 FIPS endpoints directly when a FIPS region name is not available, (e.g. +// fips-us-gov-west-1) set the Config.Endpoint on the Session, or client the +// utility is called with. The hint region will be ignored if an endpoint URL +// is configured on the session or client. +// +// sess, err := session.NewSession(&aws.Config{ +// Endpoint: aws.String("https://s3-fips.us-west-2.amazonaws.com"), +// }) +// +// region, err := s3manager.GetBucketRegion(context.Background(), sess, "bucketname", "") func GetBucketRegion(ctx aws.Context, c client.ConfigProvider, bucket, regionHint string, opts ...request.Option) (string, error) { var cfg aws.Config if len(regionHint) != 0 { @@ -50,12 +75,38 @@ const bucketRegionHeader = "X-Amz-Bucket-Region" // that it takes a S3 service client instead of a Session. The regionHint is // derived from the region the S3 service client was created in. // +// By default the request will be made to the Amazon S3 endpoint using the Path +// style addressing. +// +// s3.us-west-2.amazonaws.com/bucketname +// +// This is not compatible with Amazon S3's FIPS endpoints. To override this +// behavior to use Virtual Host style addressing, provide a functional option +// that will set the Request's Config.S3ForcePathStyle to aws.Bool(false). +// +// region, err := s3manager.GetBucketRegionWithClient(ctx, client, "bucketname", func(r *request.Request) { +// r.S3ForcePathStyle = aws.Bool(false) +// }) +// +// To configure the GetBucketRegion to make a request via the Amazon +// S3 FIPS endpoints directly when a FIPS region name is not available, (e.g. +// fips-us-gov-west-1) set the Config.Endpoint on the Session, or client the +// utility is called with. The hint region will be ignored if an endpoint URL +// is configured on the session or client. +// +// region, err := s3manager.GetBucketRegionWithClient(context.Background(), +// s3.New(sess, &aws.Config{ +// Endpoint: aws.String("https://s3-fips.us-west-2.amazonaws.com"), +// }), +// "bucketname") +// // See GetBucketRegion for more information. func GetBucketRegionWithClient(ctx aws.Context, svc s3iface.S3API, bucket string, opts ...request.Option) (string, error) { req, _ := svc.HeadBucketRequest(&s3.HeadBucketInput{ Bucket: aws.String(bucket), }) req.Config.S3ForcePathStyle = aws.Bool(true) + req.Config.Credentials = credentials.AnonymousCredentials req.SetContext(ctx) @@ -75,6 +126,16 @@ func GetBucketRegionWithClient(ctx aws.Context, svc s3iface.S3API, bucket string r.HTTPResponse.Status = "OK" r.Error = nil }) + // Replace the endpoint validation handler to not require a region if an + // endpoint URL was specified. Since these requests are not authenticated, + // requiring a region is not needed when an endpoint URL is provided. + req.Handlers.Validate.Swap( + corehandlers.ValidateEndpointHandler.Name, + request.NamedHandler{ + Name: "validateEndpointWithoutRegion", + Fn: validateEndpointWithoutRegion, + }, + ) req.ApplyOptions(opts...) @@ -86,3 +147,13 @@ func GetBucketRegionWithClient(ctx aws.Context, svc s3iface.S3API, bucket string return bucketRegion, nil } + +func validateEndpointWithoutRegion(r *request.Request) { + // Check if the caller provided an explicit URL instead of one derived by + // the SDK's endpoint resolver. For GetBucketRegion, with an explicit + // endpoint URL, a region is not needed. If no endpoint URL is provided, + // fallback the SDK's standard endpoint validation handler. + if len(aws.StringValue(r.Config.Endpoint)) == 0 { + corehandlers.ValidateEndpointHandler.Fn(r) + } +} diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/buffered_read_seeker.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/buffered_read_seeker.go new file mode 100644 index 0000000000..f1d9e85c7b --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/buffered_read_seeker.go @@ -0,0 +1,81 @@ +package s3manager + +import ( + "io" + + "github.com/aws/aws-sdk-go/internal/sdkio" +) + +// BufferedReadSeeker is buffered io.ReadSeeker +type BufferedReadSeeker struct { + r io.ReadSeeker + buffer []byte + readIdx, writeIdx int +} + +// NewBufferedReadSeeker returns a new BufferedReadSeeker +// if len(b) == 0 then the buffer will be initialized to 64 KiB. +func NewBufferedReadSeeker(r io.ReadSeeker, b []byte) *BufferedReadSeeker { + if len(b) == 0 { + b = make([]byte, 64*1024) + } + return &BufferedReadSeeker{r: r, buffer: b} +} + +func (b *BufferedReadSeeker) reset(r io.ReadSeeker) { + b.r = r + b.readIdx, b.writeIdx = 0, 0 +} + +// Read will read up len(p) bytes into p and will return +// the number of bytes read and any error that occurred. +// If the len(p) > the buffer size then a single read request +// will be issued to the underlying io.ReadSeeker for len(p) bytes. +// A Read request will at most perform a single Read to the underlying +// io.ReadSeeker, and may return < len(p) if serviced from the buffer. +func (b *BufferedReadSeeker) Read(p []byte) (n int, err error) { + if len(p) == 0 { + return n, err + } + + if b.readIdx == b.writeIdx { + if len(p) >= len(b.buffer) { + n, err = b.r.Read(p) + return n, err + } + b.readIdx, b.writeIdx = 0, 0 + + n, err = b.r.Read(b.buffer) + if n == 0 { + return n, err + } + + b.writeIdx += n + } + + n = copy(p, b.buffer[b.readIdx:b.writeIdx]) + b.readIdx += n + + return n, err +} + +// Seek will position then underlying io.ReadSeeker to the given offset +// and will clear the buffer. +func (b *BufferedReadSeeker) Seek(offset int64, whence int) (int64, error) { + n, err := b.r.Seek(offset, whence) + + b.reset(b.r) + + return n, err +} + +// ReadAt will read up to len(p) bytes at the given file offset. +// This will result in the buffer being cleared. +func (b *BufferedReadSeeker) ReadAt(p []byte, off int64) (int, error) { + _, err := b.Seek(off, sdkio.SeekStart) + if err != nil { + return 0, err + } + + return b.Read(p) +} diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/default_read_seeker_write_to.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/default_read_seeker_write_to.go new file mode 100644 index 0000000000..42276530a8 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/default_read_seeker_write_to.go @@ -0,0 +1,7 @@ +// +build !windows + +package s3manager + +func defaultUploadBufferProvider() ReadSeekerWriteToProvider { + return nil +} diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/default_read_seeker_write_to_windows.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/default_read_seeker_write_to_windows.go new file mode 100644 index 0000000000..687082c306 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/default_read_seeker_write_to_windows.go @@ -0,0 +1,5 @@ +package s3manager + +func defaultUploadBufferProvider() ReadSeekerWriteToProvider { + return NewBufferedReadSeekerWriteToPool(1024 * 1024) +} diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/default_writer_read_from.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/default_writer_read_from.go new file mode 100644 index 0000000000..ada50c2435 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/default_writer_read_from.go @@ -0,0 +1,7 @@ +// +build !windows + +package s3manager + +func defaultDownloadBufferProvider() WriterReadFromProvider { + return nil +} diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/default_writer_read_from_windows.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/default_writer_read_from_windows.go new file mode 100644 index 0000000000..7e9d9579f6 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/default_writer_read_from_windows.go @@ -0,0 +1,5 @@ +package s3manager + +func defaultDownloadBufferProvider() WriterReadFromProvider { + return NewPooledBufferedWriterReadFromProvider(1024 * 1024) +} diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/download.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/download.go index 4a9ad65e49..4b54b7c033 100644 --- a/github.com/aws/aws-sdk-go/service/s3/s3manager/download.go +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/download.go @@ -25,13 +25,25 @@ const DefaultDownloadPartSize = 1024 * 1024 * 5 // when using Download(). const DefaultDownloadConcurrency = 5 +type errReadingBody struct { + err error +} + +func (e *errReadingBody) Error() string { + return fmt.Sprintf("failed to read part body: %v", e.err) +} + +func (e *errReadingBody) Unwrap() error { + return e.err +} + // The Downloader structure that calls Download(). It is safe to call Download() // on this structure for multiple objects and across concurrent goroutines. // Mutating the Downloader's properties is not safe to be done concurrently. type Downloader struct { - // The buffer size (in bytes) to use when buffering data into chunks and - // sending them as parts to S3. The minimum allowed part size is 5MB, and - // if this value is set to zero, the DefaultDownloadPartSize value will be used. + // The size (in bytes) to request from S3 for each part. + // The minimum allowed part size is 5MB, and if this value is set to zero, + // the DefaultDownloadPartSize value will be used. // // PartSize is ignored if the Range input parameter is provided. PartSize int64 @@ -50,6 +62,14 @@ type Downloader struct { // List of request options that will be passed down to individual API // operation requests made by the downloader. RequestOptions []request.Option + + // Defines the buffer strategy used when downloading a part. + // + // If a WriterReadFromProvider is given the Download manager + // will pass the io.WriterAt of the Download request to the provider + // and will use the returned WriterReadFrom from the provider as the + // destination writer when copying from http response body. + BufferProvider WriterReadFromProvider } // WithDownloaderRequestOptions appends to the Downloader's API request options. @@ -77,10 +97,15 @@ func WithDownloaderRequestOptions(opts ...request.Option) func(*Downloader) { // d.PartSize = 64 * 1024 * 1024 // 64MB per part // }) func NewDownloader(c client.ConfigProvider, options ...func(*Downloader)) *Downloader { + return newDownloader(s3.New(c), options...) +} + +func newDownloader(client s3iface.S3API, options ...func(*Downloader)) *Downloader { d := &Downloader{ - S3: s3.New(c), - PartSize: DefaultDownloadPartSize, - Concurrency: DefaultDownloadConcurrency, + S3: client, + PartSize: DefaultDownloadPartSize, + Concurrency: DefaultDownloadConcurrency, + BufferProvider: defaultDownloadBufferProvider(), } for _, option := range options { option(d) @@ -99,7 +124,7 @@ func NewDownloader(c client.ConfigProvider, options ...func(*Downloader)) *Downl // sess := session.Must(session.NewSession()) // // // The S3 client the S3 Downloader will use -// s3Svc := s3.new(sess) +// s3Svc := s3.New(sess) // // // Create a downloader with the s3 client and default options // downloader := s3manager.NewDownloaderWithClient(s3Svc) @@ -109,16 +134,7 @@ func NewDownloader(c client.ConfigProvider, options ...func(*Downloader)) *Downl // d.PartSize = 64 * 1024 * 1024 // 64MB per part // }) func NewDownloaderWithClient(svc s3iface.S3API, options ...func(*Downloader)) *Downloader { - d := &Downloader{ - S3: svc, - PartSize: DefaultDownloadPartSize, - Concurrency: DefaultDownloadConcurrency, - } - for _, option := range options { - option(d) - } - - return d + return newDownloader(svc, options...) } type maxRetrier interface { @@ -126,7 +142,8 @@ type maxRetrier interface { } // Download downloads an object in S3 and writes the payload into w using -// concurrent GET requests. +// concurrent GET requests. The n int64 returned is the size of the object downloaded +// in bytes. // // Additional functional options can be provided to configure the individual // download. These options are copies of the Downloader instance Download is called from. @@ -148,7 +165,8 @@ func (d Downloader) Download(w io.WriterAt, input *s3.GetObjectInput, options .. } // DownloadWithContext downloads an object in S3 and writes the payload into w -// using concurrent GET requests. +// using concurrent GET requests. The n int64 returned is the size of the object downloaded +// in bytes. // // DownloadWithContext is the same as Download with the additional support for // Context input parameters. The Context must not be nil. A nil Context will @@ -403,18 +421,20 @@ func (d *downloader) downloadChunk(chunk dlchunk) error { var n int64 var err error for retry := 0; retry <= d.partBodyMaxRetries; retry++ { - var resp *s3.GetObjectOutput - resp, err = d.cfg.S3.GetObjectWithContext(d.ctx, in, d.cfg.RequestOptions...) - if err != nil { - return err - } - d.setTotalBytes(resp) // Set total if not yet set. - - n, err = io.Copy(&chunk, resp.Body) - resp.Body.Close() + n, err = d.tryDownloadChunk(in, &chunk) if err == nil { break } + // Check if the returned error is an errReadingBody. + // If err is errReadingBody this indicates that an error + // occurred while copying the http response body. + // If this occurs we unwrap the err to set the underlying error + // and attempt any remaining retries. + if bodyErr, ok := err.(*errReadingBody); ok { + err = bodyErr.Unwrap() + } else { + return err + } chunk.cur = 0 logMessage(d.cfg.S3, aws.LogDebugWithRequestRetries, @@ -427,6 +447,28 @@ func (d *downloader) downloadChunk(chunk dlchunk) error { return err } +func (d *downloader) tryDownloadChunk(in *s3.GetObjectInput, w io.Writer) (int64, error) { + cleanup := func() {} + if d.cfg.BufferProvider != nil { + w, cleanup = d.cfg.BufferProvider.GetReadFrom(w) + } + defer cleanup() + + resp, err := d.cfg.S3.GetObjectWithContext(d.ctx, in, d.cfg.RequestOptions...) + if err != nil { + return 0, err + } + d.setTotalBytes(resp) // Set total if not yet set. + + n, err := io.Copy(w, resp.Body) + resp.Body.Close() + if err != nil { + return n, &errReadingBody{err: err} + } + + return n, nil +} + func logMessage(svc s3iface.S3API, level aws.LogLevelType, msg string) { s, ok := svc.(*s3.S3) if !ok { diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/pool.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/pool.go new file mode 100644 index 0000000000..05113286d3 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/pool.go @@ -0,0 +1,244 @@ +package s3manager + +import ( + "fmt" + "sync" + + "github.com/aws/aws-sdk-go/aws" +) + +type byteSlicePool interface { + Get(aws.Context) (*[]byte, error) + Put(*[]byte) + ModifyCapacity(int) + SliceSize() int64 + Close() +} + +type maxSlicePool struct { + // allocator is defined as a function pointer to allow + // for test cases to instrument custom tracers when allocations + // occur. + allocator sliceAllocator + + slices chan *[]byte + allocations chan struct{} + capacityChange chan struct{} + + max int + sliceSize int64 + + mtx sync.RWMutex +} + +func newMaxSlicePool(sliceSize int64) *maxSlicePool { + p := &maxSlicePool{sliceSize: sliceSize} + p.allocator = p.newSlice + + return p +} + +var errZeroCapacity = fmt.Errorf("get called on zero capacity pool") + +func (p *maxSlicePool) Get(ctx aws.Context) (*[]byte, error) { + // check if context is canceled before attempting to get a slice + // this ensures priority is given to the cancel case first + select { + case <-ctx.Done(): + return nil, ctx.Err() + default: + } + + p.mtx.RLock() + + for { + select { + case bs, ok := <-p.slices: + p.mtx.RUnlock() + if !ok { + // attempt to get on a zero capacity pool + return nil, errZeroCapacity + } + return bs, nil + case _, ok := <-p.allocations: + p.mtx.RUnlock() + if !ok { + // attempt to get on a zero capacity pool + return nil, errZeroCapacity + } + return p.allocator(), nil + case <-ctx.Done(): + p.mtx.RUnlock() + return nil, ctx.Err() + default: + // In the event that there are no slices or allocations available + // This prevents some deadlock situations that can occur around sync.RWMutex + // When a lock request occurs on ModifyCapacity, no new readers are allowed to acquire a read lock. + // By releasing the read lock here and waiting for a notification, we prevent a deadlock situation where + // Get could hold the read lock indefinitely waiting for capacity, ModifyCapacity is waiting for a write lock, + // and a Put is blocked trying to get a read-lock which is blocked by ModifyCapacity. + + // Short-circuit if the pool capacity is zero. + if p.max == 0 { + p.mtx.RUnlock() + return nil, errZeroCapacity + } + + // Since we will be releasing the read-lock we need to take the reference to the channel. + // Since channels are references we will still get notified if slices are added, or if + // the channel is closed due to a capacity modification. This specifically avoids a data race condition + // where ModifyCapacity both closes a channel and initializes a new one while we don't have a read-lock. + c := p.capacityChange + + p.mtx.RUnlock() + + select { + case _ = <-c: + p.mtx.RLock() + case <-ctx.Done(): + return nil, ctx.Err() + } + } + } +} + +func (p *maxSlicePool) Put(bs *[]byte) { + p.mtx.RLock() + defer p.mtx.RUnlock() + + if p.max == 0 { + return + } + + select { + case p.slices <- bs: + p.notifyCapacity() + default: + // If the new channel when attempting to add the slice then we drop the slice. + // The logic here is to prevent a deadlock situation if channel is already at max capacity. + // Allows us to reap allocations that are returned and are no longer needed. + } +} + +func (p *maxSlicePool) ModifyCapacity(delta int) { + if delta == 0 { + return + } + + p.mtx.Lock() + defer p.mtx.Unlock() + + p.max += delta + + if p.max == 0 { + p.empty() + return + } + + if p.capacityChange != nil { + close(p.capacityChange) + } + p.capacityChange = make(chan struct{}, p.max) + + origAllocations := p.allocations + p.allocations = make(chan struct{}, p.max) + + newAllocs := len(origAllocations) + delta + for i := 0; i < newAllocs; i++ { + p.allocations <- struct{}{} + } + + if origAllocations != nil { + close(origAllocations) + } + + origSlices := p.slices + p.slices = make(chan *[]byte, p.max) + if origSlices == nil { + return + } + + close(origSlices) + for bs := range origSlices { + select { + case p.slices <- bs: + default: + // If the new channel blocks while adding slices from the old channel + // then we drop the slice. The logic here is to prevent a deadlock situation + // if the new channel has a smaller capacity then the old. + } + } +} + +func (p *maxSlicePool) notifyCapacity() { + select { + case p.capacityChange <- struct{}{}: + default: + // This *shouldn't* happen as the channel is both buffered to the max pool capacity size and is resized + // on capacity modifications. This is just a safety to ensure that a blocking situation can't occur. + } +} + +func (p *maxSlicePool) SliceSize() int64 { + return p.sliceSize +} + +func (p *maxSlicePool) Close() { + p.mtx.Lock() + defer p.mtx.Unlock() + p.empty() +} + +func (p *maxSlicePool) empty() { + p.max = 0 + + if p.capacityChange != nil { + close(p.capacityChange) + p.capacityChange = nil + } + + if p.allocations != nil { + close(p.allocations) + for range p.allocations { + // drain channel + } + p.allocations = nil + } + + if p.slices != nil { + close(p.slices) + for range p.slices { + // drain channel + } + p.slices = nil + } +} + +func (p *maxSlicePool) newSlice() *[]byte { + bs := make([]byte, p.sliceSize) + return &bs +} + +type returnCapacityPoolCloser struct { + byteSlicePool + returnCapacity int +} + +func (n *returnCapacityPoolCloser) ModifyCapacity(delta int) { + if delta > 0 { + n.returnCapacity = -1 * delta + } + n.byteSlicePool.ModifyCapacity(delta) +} + +func (n *returnCapacityPoolCloser) Close() { + if n.returnCapacity < 0 { + n.byteSlicePool.ModifyCapacity(n.returnCapacity) + } +} + +type sliceAllocator func() *[]byte + +var newByteSlicePool = func(sliceSize int64) byteSlicePool { + return newMaxSlicePool(sliceSize) +} diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/read_seeker_write_to.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/read_seeker_write_to.go new file mode 100644 index 0000000000..f62e1a45ee --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/read_seeker_write_to.go @@ -0,0 +1,65 @@ +package s3manager + +import ( + "io" + "sync" +) + +// ReadSeekerWriteTo defines an interface implementing io.WriteTo and io.ReadSeeker +type ReadSeekerWriteTo interface { + io.ReadSeeker + io.WriterTo +} + +// BufferedReadSeekerWriteTo wraps a BufferedReadSeeker with an io.WriteAt +// implementation. +type BufferedReadSeekerWriteTo struct { + *BufferedReadSeeker +} + +// WriteTo writes to the given io.Writer from BufferedReadSeeker until there's no more data to write or +// an error occurs. Returns the number of bytes written and any error encountered during the write. +func (b *BufferedReadSeekerWriteTo) WriteTo(writer io.Writer) (int64, error) { + return io.Copy(writer, b.BufferedReadSeeker) +} + +// ReadSeekerWriteToProvider provides an implementation of io.WriteTo for an io.ReadSeeker +type ReadSeekerWriteToProvider interface { + GetWriteTo(seeker io.ReadSeeker) (r ReadSeekerWriteTo, cleanup func()) +} + +// BufferedReadSeekerWriteToPool uses a sync.Pool to create and reuse +// []byte slices for buffering parts in memory +type BufferedReadSeekerWriteToPool struct { + pool sync.Pool +} + +// NewBufferedReadSeekerWriteToPool will return a new BufferedReadSeekerWriteToPool that will create +// a pool of reusable buffers . If size is less then < 64 KiB then the buffer +// will default to 64 KiB. Reason: io.Copy from writers or readers that don't support io.WriteTo or io.ReadFrom +// respectively will default to copying 32 KiB. +func NewBufferedReadSeekerWriteToPool(size int) *BufferedReadSeekerWriteToPool { + if size < 65536 { + size = 65536 + } + + return &BufferedReadSeekerWriteToPool{ + pool: sync.Pool{New: func() interface{} { + return make([]byte, size) + }}, + } +} + +// GetWriteTo will wrap the provided io.ReadSeeker with a BufferedReadSeekerWriteTo. +// The provided cleanup must be called after operations have been completed on the +// returned io.ReadSeekerWriteTo in order to signal the return of resources to the pool. +func (p *BufferedReadSeekerWriteToPool) GetWriteTo(seeker io.ReadSeeker) (r ReadSeekerWriteTo, cleanup func()) { + buffer := p.pool.Get().([]byte) + + r = &BufferedReadSeekerWriteTo{BufferedReadSeeker: NewBufferedReadSeeker(seeker, buffer)} + cleanup = func() { + p.pool.Put(buffer) + } + + return r, cleanup +} diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go index a9a005f0ac..8770d40411 100644 --- a/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/upload.go @@ -11,6 +11,7 @@ import ( "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/awsutil" "github.com/aws/aws-sdk-go/aws/client" + "github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/request" "github.com/aws/aws-sdk-go/service/s3" "github.com/aws/aws-sdk-go/service/s3/s3iface" @@ -144,8 +145,13 @@ type Uploader struct { // MaxUploadParts is the max number of parts which will be uploaded to S3. // Will be used to calculate the partsize of the object to be uploaded. // E.g: 5GB file, with MaxUploadParts set to 100, will upload the file - // as 100, 50MB parts. - // With a limited of s3.MaxUploadParts (10,000 parts). + // as 100, 50MB parts. With a limited of s3.MaxUploadParts (10,000 parts). + // + // MaxUploadParts must not be used to limit the total number of bytes uploaded. + // Use a type like to io.LimitReader (https://golang.org/pkg/io/#LimitedReader) + // instead. An io.LimitReader is helpful when uploading an unbounded reader + // to S3, and you know its maximum size. Otherwise the reader's io.EOF returned + // error must be used to signal end of stream. // // Defaults to package const's MaxUploadParts value. MaxUploadParts int @@ -156,6 +162,12 @@ type Uploader struct { // List of request options that will be passed down to individual API // operation requests made by the uploader. RequestOptions []request.Option + + // Defines the buffer strategy used when uploading a part + BufferProvider ReadSeekerWriteToProvider + + // partPool allows for the re-usage of streaming payload part buffers between upload calls + partPool byteSlicePool } // NewUploader creates a new Uploader instance to upload objects to S3. Pass In @@ -175,18 +187,25 @@ type Uploader struct { // u.PartSize = 64 * 1024 * 1024 // 64MB per part // }) func NewUploader(c client.ConfigProvider, options ...func(*Uploader)) *Uploader { + return newUploader(s3.New(c), options...) +} + +func newUploader(client s3iface.S3API, options ...func(*Uploader)) *Uploader { u := &Uploader{ - S3: s3.New(c), + S3: client, PartSize: DefaultUploadPartSize, Concurrency: DefaultUploadConcurrency, LeavePartsOnError: false, MaxUploadParts: MaxUploadParts, + BufferProvider: defaultUploadBufferProvider(), } for _, option := range options { option(u) } + u.partPool = newByteSlicePool(u.PartSize) + return u } @@ -209,19 +228,7 @@ func NewUploader(c client.ConfigProvider, options ...func(*Uploader)) *Uploader // u.PartSize = 64 * 1024 * 1024 // 64MB per part // }) func NewUploaderWithClient(svc s3iface.S3API, options ...func(*Uploader)) *Uploader { - u := &Uploader{ - S3: svc, - PartSize: DefaultUploadPartSize, - Concurrency: DefaultUploadConcurrency, - LeavePartsOnError: false, - MaxUploadParts: MaxUploadParts, - } - - for _, option := range options { - option(u) - } - - return u + return newUploader(svc, options...) } // Upload uploads an object to S3, intelligently buffering large files into @@ -281,6 +288,7 @@ func (u Uploader) UploadWithContext(ctx aws.Context, input *UploadInput, opts .. for _, opt := range opts { opt(&i.cfg) } + i.cfg.RequestOptions = append(i.cfg.RequestOptions, request.WithAppendUserAgent("S3Manager")) return i.upload() @@ -350,14 +358,15 @@ type uploader struct { readerPos int64 // current reader position totalSize int64 // set to -1 if the size is not known - - bufferPool sync.Pool } // internal logic for deciding whether to upload a single part or use a // multipart upload. func (u *uploader) upload() (*UploadOutput, error) { - u.init() + if err := u.init(); err != nil { + return nil, awserr.New("ReadRequestBody", "unable to initialize upload", err) + } + defer u.cfg.partPool.Close() if u.cfg.PartSize < MinUploadPartSize { msg := fmt.Sprintf("part size must be at least %d bytes", MinUploadPartSize) @@ -365,19 +374,20 @@ func (u *uploader) upload() (*UploadOutput, error) { } // Do one read to determine if we have more than one part - reader, _, part, err := u.nextReader() + reader, _, cleanup, err := u.nextReader() if err == io.EOF { // single part - return u.singlePart(reader) + return u.singlePart(reader, cleanup) } else if err != nil { + cleanup() return nil, awserr.New("ReadRequestBody", "read upload data failed", err) } mu := multiuploader{uploader: u} - return mu.upload(reader, part) + return mu.upload(reader, cleanup) } // init will initialize all default options. -func (u *uploader) init() { +func (u *uploader) init() error { if u.cfg.Concurrency == 0 { u.cfg.Concurrency = DefaultUploadConcurrency } @@ -388,24 +398,35 @@ func (u *uploader) init() { u.cfg.MaxUploadParts = MaxUploadParts } - u.bufferPool = sync.Pool{ - New: func() interface{} { return make([]byte, u.cfg.PartSize) }, + // Try to get the total size for some optimizations + if err := u.initSize(); err != nil { + return err } - // Try to get the total size for some optimizations - u.initSize() + // If PartSize was changed or partPool was never setup then we need to allocated a new pool + // so that we return []byte slices of the correct size + poolCap := u.cfg.Concurrency + 1 + if u.cfg.partPool == nil || u.cfg.partPool.SliceSize() != u.cfg.PartSize { + u.cfg.partPool = newByteSlicePool(u.cfg.PartSize) + u.cfg.partPool.ModifyCapacity(poolCap) + } else { + u.cfg.partPool = &returnCapacityPoolCloser{byteSlicePool: u.cfg.partPool} + u.cfg.partPool.ModifyCapacity(poolCap) + } + + return nil } // initSize tries to detect the total stream size, setting u.totalSize. If // the size is not known, totalSize is set to -1. -func (u *uploader) initSize() { +func (u *uploader) initSize() error { u.totalSize = -1 switch r := u.in.Body.(type) { case io.Seeker: n, err := aws.SeekerLen(r) if err != nil { - return + return err } u.totalSize = n @@ -417,17 +438,15 @@ func (u *uploader) initSize() { u.cfg.PartSize = (u.totalSize / int64(u.cfg.MaxUploadParts)) + 1 } } + + return nil } // nextReader returns a seekable reader representing the next packet of data. // This operation increases the shared u.readerPos counter, but note that it // does not need to be wrapped in a mutex because nextReader is only called // from the main thread. -func (u *uploader) nextReader() (io.ReadSeeker, int, []byte, error) { - type readerAtSeeker interface { - io.ReaderAt - io.ReadSeeker - } +func (u *uploader) nextReader() (io.ReadSeeker, int, func(), error) { switch r := u.in.Body.(type) { case readerAtSeeker: var err error @@ -442,17 +461,36 @@ func (u *uploader) nextReader() (io.ReadSeeker, int, []byte, error) { } } - reader := io.NewSectionReader(r, u.readerPos, n) + var ( + reader io.ReadSeeker + cleanup func() + ) + + reader = io.NewSectionReader(r, u.readerPos, n) + if u.cfg.BufferProvider != nil { + reader, cleanup = u.cfg.BufferProvider.GetWriteTo(reader) + } else { + cleanup = func() {} + } + u.readerPos += n - return reader, int(n), nil, err + return reader, int(n), cleanup, err default: - part := u.bufferPool.Get().([]byte) - n, err := readFillBuf(r, part) + part, err := u.cfg.partPool.Get(u.ctx) + if err != nil { + return nil, 0, func() {}, err + } + + n, err := readFillBuf(r, *part) u.readerPos += int64(n) - return bytes.NewReader(part[0:n]), n, part, err + cleanup := func() { + u.cfg.partPool.Put(part) + } + + return bytes.NewReader((*part)[0:n]), n, cleanup, err } } @@ -469,10 +507,12 @@ func readFillBuf(r io.Reader, b []byte) (offset int, err error) { // singlePart contains upload logic for uploading a single chunk via // a regular PutObject request. Multipart requests require at least two // parts, or at least 5MB of data. -func (u *uploader) singlePart(buf io.ReadSeeker) (*UploadOutput, error) { +func (u *uploader) singlePart(r io.ReadSeeker, cleanup func()) (*UploadOutput, error) { + defer cleanup() + params := &s3.PutObjectInput{} awsutil.Copy(params, u.in) - params.Body = buf + params.Body = r // Need to use request form because URL generated in request is // used in return. @@ -502,9 +542,9 @@ type multiuploader struct { // keeps track of a single chunk of data being sent to S3. type chunk struct { - buf io.ReadSeeker - part []byte - num int64 + buf io.ReadSeeker + num int64 + cleanup func() } // completedParts is a wrapper to make parts sortable by their part number, @@ -517,13 +557,14 @@ func (a completedParts) Less(i, j int) bool { return *a[i].PartNumber < *a[j].Pa // upload will perform a multipart upload using the firstBuf buffer containing // the first chunk of data. -func (u *multiuploader) upload(firstBuf io.ReadSeeker, firstPart []byte) (*UploadOutput, error) { +func (u *multiuploader) upload(firstBuf io.ReadSeeker, cleanup func()) (*UploadOutput, error) { params := &s3.CreateMultipartUploadInput{} awsutil.Copy(params, u.in) // Create the multipart resp, err := u.cfg.S3.CreateMultipartUploadWithContext(u.ctx, params, u.cfg.RequestOptions...) if err != nil { + cleanup() return nil, err } u.uploadID = *resp.UploadId @@ -537,46 +578,29 @@ func (u *multiuploader) upload(firstBuf io.ReadSeeker, firstPart []byte) (*Uploa // Send part 1 to the workers var num int64 = 1 - ch <- chunk{buf: firstBuf, part: firstPart, num: num} + ch <- chunk{buf: firstBuf, num: num, cleanup: cleanup} // Read and queue the rest of the parts for u.geterr() == nil && err == nil { - num++ - // This upload exceeded maximum number of supported parts, error now. - if num > int64(u.cfg.MaxUploadParts) || num > int64(MaxUploadParts) { - var msg string - if num > int64(u.cfg.MaxUploadParts) { - msg = fmt.Sprintf("exceeded total allowed configured MaxUploadParts (%d). Adjust PartSize to fit in this limit", - u.cfg.MaxUploadParts) - } else { - msg = fmt.Sprintf("exceeded total allowed S3 limit MaxUploadParts (%d). Adjust PartSize to fit in this limit", - MaxUploadParts) + var ( + reader io.ReadSeeker + nextChunkLen int + ok bool + ) + + reader, nextChunkLen, cleanup, err = u.nextReader() + ok, err = u.shouldContinue(num, nextChunkLen, err) + if !ok { + cleanup() + if err != nil { + u.seterr(err) } - u.seterr(awserr.New("TotalPartsExceeded", msg, nil)) - break - } - - var reader io.ReadSeeker - var nextChunkLen int - var part []byte - reader, nextChunkLen, part, err = u.nextReader() - - if err != nil && err != io.EOF { - u.seterr(awserr.New( - "ReadRequestBody", - "read multipart upload data failed", - err)) break } - if nextChunkLen == 0 { - // No need to upload empty part, if file was empty to start - // with empty single part would of been created and never - // started multipart upload. - break - } + num++ - ch <- chunk{buf: reader, part: part, num: num} + ch <- chunk{buf: reader, num: num, cleanup: cleanup} } // Close the channel, wait for workers, and complete upload @@ -593,13 +617,53 @@ func (u *multiuploader) upload(firstBuf io.ReadSeeker, firstPart []byte) (*Uploa uploadID: u.uploadID, } } + + // Create a presigned URL of the S3 Get Object in order to have parity with + // single part upload. + getReq, _ := u.cfg.S3.GetObjectRequest(&s3.GetObjectInput{ + Bucket: u.in.Bucket, + Key: u.in.Key, + }) + getReq.Config.Credentials = credentials.AnonymousCredentials + getReq.SetContext(u.ctx) + uploadLocation, _, _ := getReq.PresignRequest(1) + return &UploadOutput{ - Location: aws.StringValue(complete.Location), + Location: uploadLocation, VersionID: complete.VersionId, UploadID: u.uploadID, }, nil } +func (u *multiuploader) shouldContinue(part int64, nextChunkLen int, err error) (bool, error) { + if err != nil && err != io.EOF { + return false, awserr.New("ReadRequestBody", "read multipart upload data failed", err) + } + + if nextChunkLen == 0 { + // No need to upload empty part, if file was empty to start + // with empty single part would of been created and never + // started multipart upload. + return false, nil + } + + part++ + // This upload exceeded maximum number of supported parts, error now. + if part > int64(u.cfg.MaxUploadParts) || part > int64(MaxUploadParts) { + var msg string + if part > int64(u.cfg.MaxUploadParts) { + msg = fmt.Sprintf("exceeded total allowed configured MaxUploadParts (%d). Adjust PartSize to fit in this limit", + u.cfg.MaxUploadParts) + } else { + msg = fmt.Sprintf("exceeded total allowed S3 limit MaxUploadParts (%d). Adjust PartSize to fit in this limit", + MaxUploadParts) + } + return false, awserr.New("TotalPartsExceeded", msg, nil) + } + + return true, err +} + // readChunk runs in worker goroutines to pull chunks off of the ch channel // and send() them as UploadPart requests. func (u *multiuploader) readChunk(ch chan chunk) { @@ -616,6 +680,8 @@ func (u *multiuploader) readChunk(ch chan chunk) { u.seterr(err) } } + + data.cleanup() } } @@ -631,9 +697,8 @@ func (u *multiuploader) send(c chunk) error { SSECustomerKey: u.in.SSECustomerKey, PartNumber: &c.num, } + resp, err := u.cfg.S3.UploadPartWithContext(u.ctx, params, u.cfg.RequestOptions...) - // put the byte array back into the pool to conserve memory - u.bufferPool.Put(c.part) if err != nil { return err } @@ -705,3 +770,8 @@ func (u *multiuploader) complete() *s3.CompleteMultipartUploadOutput { return resp } + +type readerAtSeeker interface { + io.ReaderAt + io.ReadSeeker +} diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/upload_input.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/upload_input.go index ebf48204c1..c8810c11bb 100644 --- a/github.com/aws/aws-sdk-go/service/s3/s3manager/upload_input.go +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/upload_input.go @@ -12,40 +12,59 @@ import ( // package's PutObjectInput with the exception that the Body member is an // io.Reader instead of an io.ReadSeeker. type UploadInput struct { - _ struct{} `type:"structure" payload:"Body"` + _ struct{} `locationName:"PutObjectRequest" type:"structure" payload:"Body"` - // The canned ACL to apply to the object. + // The canned ACL to apply to the object. For more information, see Canned ACL + // (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#CannedACL). ACL *string `location:"header" locationName:"x-amz-acl" type:"string" enum:"ObjectCannedACL"` // The readable body payload to send to S3. Body io.Reader - // Name of the bucket to which the PUT operation was initiated. + // Bucket name to which the PUT operation was initiated. + // + // When using this API with an access point, you must direct requests to the + // access point hostname. The access point hostname takes the form AccessPointName-AccountId.s3-accesspoint.Region.amazonaws.com. + // When using this operation using an access point through the AWS SDKs, you + // provide the access point ARN in place of the bucket name. For more information + // about access point ARNs, see Using Access Points (https://docs.aws.amazon.com/AmazonS3/latest/dev/using-access-points.html) + // in the Amazon Simple Storage Service Developer Guide. // // Bucket is a required field Bucket *string `location:"uri" locationName:"Bucket" type:"string" required:"true"` - // Specifies caching behavior along the request/reply chain. + // Can be used to specify caching behavior along the request/reply chain. For + // more information, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9 + // (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9). CacheControl *string `location:"header" locationName:"Cache-Control" type:"string"` - // Specifies presentational information for the object. + // Specifies presentational information for the object. For more information, + // see http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1). ContentDisposition *string `location:"header" locationName:"Content-Disposition" type:"string"` // Specifies what content encodings have been applied to the object and thus // what decoding mechanisms must be applied to obtain the media-type referenced - // by the Content-Type header field. + // by the Content-Type header field. For more information, see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11 + // (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11). ContentEncoding *string `location:"header" locationName:"Content-Encoding" type:"string"` // The language the content is in. ContentLanguage *string `location:"header" locationName:"Content-Language" type:"string"` - // The base64-encoded 128-bit MD5 digest of the part data. + // The base64-encoded 128-bit MD5 digest of the message (without the headers) + // according to RFC 1864. This header can be used as a message integrity check + // to verify that the data is the same data that was originally sent. Although + // it is optional, we recommend using the Content-MD5 mechanism as an end-to-end + // integrity check. For more information about REST request authentication, + // see REST Authentication (https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html). ContentMD5 *string `location:"header" locationName:"Content-MD5" type:"string"` - // A standard MIME type describing the format of the object data. + // A standard MIME type describing the format of the contents. For more information, + // see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.17). ContentType *string `location:"header" locationName:"Content-Type" type:"string"` - // The date and time at which the object is no longer cacheable. + // The date and time at which the object is no longer cacheable. For more information, + // see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.21 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.21). Expires *time.Time `location:"header" locationName:"Expires" type:"timestamp"` // Gives the grantee READ, READ_ACP, and WRITE_ACP permissions on the object. @@ -68,7 +87,8 @@ type UploadInput struct { // A map of metadata to store with the object in S3. Metadata map[string]*string `location:"headers" locationName:"x-amz-meta-" type:"map"` - // The Legal Hold status that you want to apply to the specified object. + // Specifies whether a legal hold will be applied to this object. For more information + // about S3 Object Lock, see Object Lock (https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lock.html). ObjectLockLegalHoldStatus *string `location:"header" locationName:"x-amz-object-lock-legal-hold" type:"string" enum:"ObjectLockLegalHoldStatus"` // The Object Lock mode that you want to apply to this object. @@ -77,38 +97,52 @@ type UploadInput struct { // The date and time when you want this object's Object Lock to expire. ObjectLockRetainUntilDate *time.Time `location:"header" locationName:"x-amz-object-lock-retain-until-date" type:"timestamp" timestampFormat:"iso8601"` - // Confirms that the requester knows that she or he will be charged for the - // request. Bucket owners need not specify this parameter in their requests. - // Documentation on downloading objects from requester pays buckets can be found - // at http://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html + // Confirms that the requester knows that they will be charged for the request. + // Bucket owners need not specify this parameter in their requests. For information + // about downloading objects from requester pays buckets, see Downloading Objects + // in Requestor Pays Buckets (https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectsinRequesterPaysBuckets.html) + // in the Amazon S3 Developer Guide. RequestPayer *string `location:"header" locationName:"x-amz-request-payer" type:"string" enum:"RequestPayer"` - // Specifies the algorithm to use to when encrypting the object (e.g., AES256). + // Specifies the algorithm to use to when encrypting the object (for example, + // AES256). SSECustomerAlgorithm *string `location:"header" locationName:"x-amz-server-side-encryption-customer-algorithm" type:"string"` // Specifies the customer-provided encryption key for Amazon S3 to use in encrypting // data. This value is used to store the object and then it is discarded; Amazon - // does not store the encryption key. The key must be appropriate for use with - // the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm + // S3 does not store the encryption key. The key must be appropriate for use + // with the algorithm specified in the x-amz-server-side​-encryption​-customer-algorithm // header. - SSECustomerKey *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` + SSECustomerKey *string `marshal-as:"blob" location:"header" locationName:"x-amz-server-side-encryption-customer-key" type:"string" sensitive:"true"` // Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321. - // Amazon S3 uses this header for a message integrity check to ensure the encryption - // key was transmitted without error. + // Amazon S3 uses this header for a message integrity check to ensure that the + // encryption key was transmitted without error. SSECustomerKeyMD5 *string `location:"header" locationName:"x-amz-server-side-encryption-customer-key-MD5" type:"string"` - // Specifies the AWS KMS key ID to use for object encryption. All GET and PUT - // requests for an object protected by AWS KMS will fail if not made via SSL - // or using SigV4. Documentation on configuring any of the officially supported - // AWS SDKs and CLI can be found at http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingAWSSDK.html#specify-signature-version + // Specifies the AWS KMS Encryption Context to use for object encryption. The + // value of this header is a base64-encoded UTF-8 string holding JSON with the + // encryption context key-value pairs. + SSEKMSEncryptionContext *string `location:"header" locationName:"x-amz-server-side-encryption-context" type:"string" sensitive:"true"` + + // If x-amz-server-side-encryption is present and has the value of aws:kms, + // this header specifies the ID of the AWS Key Management Service (AWS KMS) + // symmetrical customer managed customer master key (CMK) that was used for + // the object. + // + // If the value of x-amz-server-side-encryption is aws:kms, this header specifies + // the ID of the symmetric customer managed AWS KMS CMK that will be used for + // the object. If you specify x-amz-server-side-encryption:aws:kms, but do not + // providex-amz-server-side-encryption-aws-kms-key-id, Amazon S3 uses the AWS + // managed CMK in AWS to protect the data. SSEKMSKeyId *string `location:"header" locationName:"x-amz-server-side-encryption-aws-kms-key-id" type:"string" sensitive:"true"` - // The Server-side encryption algorithm used when storing this object in S3 - // (e.g., AES256, aws:kms). + // The server-side encryption algorithm used when storing this object in Amazon + // S3 (for example, AES256, aws:kms). ServerSideEncryption *string `location:"header" locationName:"x-amz-server-side-encryption" type:"string" enum:"ServerSideEncryption"` - // The type of storage to use for the object. Defaults to 'STANDARD'. + // If you don't specify, S3 Standard is the default storage class. Amazon S3 + // supports other storage classes. StorageClass *string `location:"header" locationName:"x-amz-storage-class" type:"string" enum:"StorageClass"` // The tag-set for the object. The tag-set must be encoded as URL Query parameters. @@ -117,6 +151,21 @@ type UploadInput struct { // If the bucket is configured as a website, redirects requests for this object // to another object in the same bucket or to an external URL. Amazon S3 stores - // the value of this header in the object metadata. + // the value of this header in the object metadata. For information about object + // metadata, see Object Key and Metadata (https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html). + // + // In the following example, the request header sets the redirect to an object + // (anotherPage.html) in the same bucket: + // + // x-amz-website-redirect-location: /anotherPage.html + // + // In the following example, the request header sets the object redirect to + // another website: + // + // x-amz-website-redirect-location: http://www.example.com/ + // + // For more information about website hosting in Amazon S3, see Hosting Websites + // on Amazon S3 (https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteHosting.html) + // and How to Configure Website Page Redirects (https://docs.aws.amazon.com/AmazonS3/latest/dev/how-to-page-redirect.html). WebsiteRedirectLocation *string `location:"header" locationName:"x-amz-website-redirect-location" type:"string"` } diff --git a/github.com/aws/aws-sdk-go/service/s3/s3manager/writer_read_from.go b/github.com/aws/aws-sdk-go/service/s3/s3manager/writer_read_from.go new file mode 100644 index 0000000000..765dc07ca3 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/s3/s3manager/writer_read_from.go @@ -0,0 +1,75 @@ +package s3manager + +import ( + "bufio" + "io" + "sync" + + "github.com/aws/aws-sdk-go/internal/sdkio" +) + +// WriterReadFrom defines an interface implementing io.Writer and io.ReaderFrom +type WriterReadFrom interface { + io.Writer + io.ReaderFrom +} + +// WriterReadFromProvider provides an implementation of io.ReadFrom for the given io.Writer +type WriterReadFromProvider interface { + GetReadFrom(writer io.Writer) (w WriterReadFrom, cleanup func()) +} + +type bufferedWriter interface { + WriterReadFrom + Flush() error + Reset(io.Writer) +} + +type bufferedReadFrom struct { + bufferedWriter +} + +func (b *bufferedReadFrom) ReadFrom(r io.Reader) (int64, error) { + n, err := b.bufferedWriter.ReadFrom(r) + if flushErr := b.Flush(); flushErr != nil && err == nil { + err = flushErr + } + return n, err +} + +// PooledBufferedReadFromProvider is a WriterReadFromProvider that uses a sync.Pool +// to manage allocation and reuse of *bufio.Writer structures. +type PooledBufferedReadFromProvider struct { + pool sync.Pool +} + +// NewPooledBufferedWriterReadFromProvider returns a new PooledBufferedReadFromProvider +// Size is used to control the size of the underlying *bufio.Writer created for +// calls to GetReadFrom. +func NewPooledBufferedWriterReadFromProvider(size int) *PooledBufferedReadFromProvider { + if size < int(32*sdkio.KibiByte) { + size = int(64 * sdkio.KibiByte) + } + + return &PooledBufferedReadFromProvider{ + pool: sync.Pool{ + New: func() interface{} { + return &bufferedReadFrom{bufferedWriter: bufio.NewWriterSize(nil, size)} + }, + }, + } +} + +// GetReadFrom takes an io.Writer and wraps it with a type which satisfies the WriterReadFrom +// interface/ Additionally a cleanup function is provided which must be called after usage of the WriterReadFrom +// has been completed in order to allow the reuse of the *bufio.Writer +func (p *PooledBufferedReadFromProvider) GetReadFrom(writer io.Writer) (r WriterReadFrom, cleanup func()) { + buffer := p.pool.Get().(*bufferedReadFrom) + buffer.Reset(writer) + r = buffer + cleanup = func() { + buffer.Reset(nil) // Reset to nil writer to release reference + p.pool.Put(buffer) + } + return r, cleanup +} diff --git a/github.com/aws/aws-sdk-go/service/s3/service.go b/github.com/aws/aws-sdk-go/service/s3/service.go index d17dcc9dad..b4c07b4d47 100644 --- a/github.com/aws/aws-sdk-go/service/s3/service.go +++ b/github.com/aws/aws-sdk-go/service/s3/service.go @@ -31,7 +31,7 @@ var initRequest func(*request.Request) const ( ServiceName = "s3" // Name of service. EndpointsID = ServiceName // ID to lookup a service endpoint with. - ServiceID = "S3" // ServiceID is a unique identifer of a specific service. + ServiceID = "S3" // ServiceID is a unique identifier of a specific service. ) // New creates a new instance of the S3 client with a session. @@ -39,6 +39,8 @@ const ( // aws.Config parameter to add your extra config. // // Example: +// mySession := session.Must(session.NewSession()) +// // // Create a S3 client from just a session. // svc := s3.New(mySession) // @@ -46,11 +48,11 @@ const ( // svc := s3.New(mySession, aws.NewConfig().WithRegion("us-west-2")) func New(p client.ConfigProvider, cfgs ...*aws.Config) *S3 { c := p.ClientConfig(EndpointsID, cfgs...) - return newClient(*c.Config, c.Handlers, c.Endpoint, c.SigningRegion, c.SigningName) + return newClient(*c.Config, c.Handlers, c.PartitionID, c.Endpoint, c.SigningRegion, c.SigningName) } // newClient creates, initializes and returns a new service client instance. -func newClient(cfg aws.Config, handlers request.Handlers, endpoint, signingRegion, signingName string) *S3 { +func newClient(cfg aws.Config, handlers request.Handlers, partitionID, endpoint, signingRegion, signingName string) *S3 { svc := &S3{ Client: client.New( cfg, @@ -59,6 +61,7 @@ func newClient(cfg aws.Config, handlers request.Handlers, endpoint, signingRegio ServiceID: ServiceID, SigningName: signingName, SigningRegion: signingRegion, + PartitionID: partitionID, Endpoint: endpoint, APIVersion: "2006-03-01", }, @@ -75,6 +78,7 @@ func newClient(cfg aws.Config, handlers request.Handlers, endpoint, signingRegio svc.Handlers.UnmarshalMeta.PushBackNamed(restxml.UnmarshalMetaHandler) svc.Handlers.UnmarshalError.PushBackNamed(restxml.UnmarshalErrorHandler) + svc.Handlers.BuildStream.PushBackNamed(restxml.BuildHandler) svc.Handlers.UnmarshalStream.PushBackNamed(restxml.UnmarshalHandler) // Run custom client initialization if present diff --git a/github.com/aws/aws-sdk-go/service/s3/sse.go b/github.com/aws/aws-sdk-go/service/s3/sse.go index 8010c4fa19..b71c835dee 100644 --- a/github.com/aws/aws-sdk-go/service/s3/sse.go +++ b/github.com/aws/aws-sdk-go/service/s3/sse.go @@ -3,6 +3,7 @@ package s3 import ( "crypto/md5" "encoding/base64" + "net/http" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/request" @@ -30,25 +31,54 @@ func validateSSERequiresSSL(r *request.Request) { } } -func computeSSEKeys(r *request.Request) { - headers := []string{ - "x-amz-server-side-encryption-customer-key", - "x-amz-copy-source-server-side-encryption-customer-key", +const ( + sseKeyHeader = "x-amz-server-side-encryption-customer-key" + sseKeyMD5Header = sseKeyHeader + "-md5" +) + +func computeSSEKeyMD5(r *request.Request) { + var key string + if g, ok := r.Params.(sseCustomerKeyGetter); ok { + key = g.getSSECustomerKey() + } + + computeKeyMD5(sseKeyHeader, sseKeyMD5Header, key, r.HTTPRequest) +} + +const ( + copySrcSSEKeyHeader = "x-amz-copy-source-server-side-encryption-customer-key" + copySrcSSEKeyMD5Header = copySrcSSEKeyHeader + "-md5" +) + +func computeCopySourceSSEKeyMD5(r *request.Request) { + var key string + if g, ok := r.Params.(copySourceSSECustomerKeyGetter); ok { + key = g.getCopySourceSSECustomerKey() } - for _, h := range headers { - md5h := h + "-md5" - if key := r.HTTPRequest.Header.Get(h); key != "" { - // Base64-encode the value - b64v := base64.StdEncoding.EncodeToString([]byte(key)) - r.HTTPRequest.Header.Set(h, b64v) - - // Add MD5 if it wasn't computed - if r.HTTPRequest.Header.Get(md5h) == "" { - sum := md5.Sum([]byte(key)) - b64sum := base64.StdEncoding.EncodeToString(sum[:]) - r.HTTPRequest.Header.Set(md5h, b64sum) - } + computeKeyMD5(copySrcSSEKeyHeader, copySrcSSEKeyMD5Header, key, r.HTTPRequest) +} + +func computeKeyMD5(keyHeader, keyMD5Header, key string, r *http.Request) { + if len(key) == 0 { + // Backwards compatiablity where user just set the header value instead + // of using the API parameter, or setting the header value for an + // operation without the parameters modeled. + key = r.Header.Get(keyHeader) + if len(key) == 0 { + return } + + // In backwards compatiable, the header's value is not base64 encoded, + // and needs to be encoded and updated by the SDK's customizations. + b64Key := base64.StdEncoding.EncodeToString([]byte(key)) + r.Header.Set(keyHeader, b64Key) + } + + // Only update Key's MD5 if not already set. + if len(r.Header.Get(keyMD5Header)) == 0 { + sum := md5.Sum([]byte(key)) + keyMD5 := base64.StdEncoding.EncodeToString(sum[:]) + r.Header.Set(keyMD5Header, keyMD5) } } diff --git a/github.com/aws/aws-sdk-go/service/s3/statusok_error.go b/github.com/aws/aws-sdk-go/service/s3/statusok_error.go index fde3050f95..247770e4c8 100644 --- a/github.com/aws/aws-sdk-go/service/s3/statusok_error.go +++ b/github.com/aws/aws-sdk-go/service/s3/statusok_error.go @@ -2,6 +2,7 @@ package s3 import ( "bytes" + "io" "io/ioutil" "net/http" @@ -14,7 +15,7 @@ func copyMultipartStatusOKUnmarhsalError(r *request.Request) { b, err := ioutil.ReadAll(r.HTTPResponse.Body) if err != nil { r.Error = awserr.NewRequestFailure( - awserr.New("SerializationError", "unable to read response body", err), + awserr.New(request.ErrCodeSerialization, "unable to read response body", err), r.HTTPResponse.StatusCode, r.RequestID, ) @@ -24,17 +25,18 @@ func copyMultipartStatusOKUnmarhsalError(r *request.Request) { r.HTTPResponse.Body = ioutil.NopCloser(body) defer body.Seek(0, sdkio.SeekStart) - if body.Len() == 0 { - // If there is no body don't attempt to parse the body. - return - } - unmarshalError(r) if err, ok := r.Error.(awserr.Error); ok && err != nil { - if err.Code() == "SerializationError" { + if err.Code() == request.ErrCodeSerialization && + err.OrigErr() != io.EOF { r.Error = nil return } - r.HTTPResponse.StatusCode = http.StatusServiceUnavailable + // if empty payload + if err.OrigErr() == io.EOF { + r.HTTPResponse.StatusCode = http.StatusInternalServerError + } else { + r.HTTPResponse.StatusCode = http.StatusServiceUnavailable + } } } diff --git a/github.com/aws/aws-sdk-go/service/s3/unmarshal_error.go b/github.com/aws/aws-sdk-go/service/s3/unmarshal_error.go index 12c0612c8d..6eecf66910 100644 --- a/github.com/aws/aws-sdk-go/service/s3/unmarshal_error.go +++ b/github.com/aws/aws-sdk-go/service/s3/unmarshal_error.go @@ -1,6 +1,7 @@ package s3 import ( + "bytes" "encoding/xml" "fmt" "io" @@ -11,6 +12,7 @@ import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/private/protocol/xml/xmlutil" ) type xmlErrorResponse struct { @@ -26,40 +28,57 @@ func unmarshalError(r *request.Request) { // Bucket exists in a different region, and request needs // to be made to the correct region. if r.HTTPResponse.StatusCode == http.StatusMovedPermanently { + msg := fmt.Sprintf( + "incorrect region, the bucket is not in '%s' region at endpoint '%s'", + aws.StringValue(r.Config.Region), + aws.StringValue(r.Config.Endpoint), + ) + if v := r.HTTPResponse.Header.Get("x-amz-bucket-region"); len(v) != 0 { + msg += fmt.Sprintf(", bucket is in '%s' region", v) + } r.Error = awserr.NewRequestFailure( - awserr.New("BucketRegionError", - fmt.Sprintf("incorrect region, the bucket is not in '%s' region", - aws.StringValue(r.Config.Region)), - nil), + awserr.New("BucketRegionError", msg, nil), r.HTTPResponse.StatusCode, r.RequestID, ) return } - var errCode, errMsg string - // Attempt to parse error from body if it is known - resp := &xmlErrorResponse{} - err := xml.NewDecoder(r.HTTPResponse.Body).Decode(resp) - if err != nil && err != io.EOF { - errCode = "SerializationError" - errMsg = "failed to decode S3 XML error response" + var errResp xmlErrorResponse + var err error + if r.HTTPResponse.StatusCode >= 200 && r.HTTPResponse.StatusCode < 300 { + err = s3unmarshalXMLError(&errResp, r.HTTPResponse.Body) } else { - errCode = resp.Code - errMsg = resp.Message - err = nil + err = xmlutil.UnmarshalXMLError(&errResp, r.HTTPResponse.Body) + } + + if err != nil { + var errorMsg string + if err == io.EOF { + errorMsg = "empty response payload" + } else { + errorMsg = "failed to unmarshal error message" + } + + r.Error = awserr.NewRequestFailure( + awserr.New(request.ErrCodeSerialization, + errorMsg, err), + r.HTTPResponse.StatusCode, + r.RequestID, + ) + return } // Fallback to status code converted to message if still no error code - if len(errCode) == 0 { + if len(errResp.Code) == 0 { statusText := http.StatusText(r.HTTPResponse.StatusCode) - errCode = strings.Replace(statusText, " ", "", -1) - errMsg = statusText + errResp.Code = strings.Replace(statusText, " ", "", -1) + errResp.Message = statusText } r.Error = awserr.NewRequestFailure( - awserr.New(errCode, errMsg, err), + awserr.New(errResp.Code, errResp.Message, err), r.HTTPResponse.StatusCode, r.RequestID, ) @@ -75,3 +94,21 @@ type RequestFailure interface { // Host ID is the S3 Host ID needed for debug, and contacting support HostID() string } + +// s3unmarshalXMLError is s3 specific xml error unmarshaler +// for 200 OK errors and response payloads. +// This function differs from the xmlUtil.UnmarshalXMLError +// func. It does not ignore the EOF error and passes it up. +// Related to bug fix for `s3 200 OK response with empty payload` +func s3unmarshalXMLError(v interface{}, stream io.Reader) error { + var errBuf bytes.Buffer + body := io.TeeReader(stream, &errBuf) + + err := xml.NewDecoder(body).Decode(v) + if err != nil && err != io.EOF { + return awserr.NewUnmarshalError(err, + "failed to unmarshal error message", errBuf.Bytes()) + } + + return err +} diff --git a/github.com/aws/aws-sdk-go/service/sts/api.go b/github.com/aws/aws-sdk-go/service/sts/api.go index ee908f9167..550b5f687f 100644 --- a/github.com/aws/aws-sdk-go/service/sts/api.go +++ b/github.com/aws/aws-sdk-go/service/sts/api.go @@ -3,10 +3,12 @@ package sts import ( + "fmt" "time" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/awsutil" + "github.com/aws/aws-sdk-go/aws/credentials" "github.com/aws/aws-sdk-go/aws/request" ) @@ -54,39 +56,29 @@ func (c *STS) AssumeRoleRequest(input *AssumeRoleInput) (req *request.Request, o // AssumeRole API operation for AWS Security Token Service. // -// Returns a set of temporary security credentials (consisting of an access -// key ID, a secret access key, and a security token) that you can use to access -// AWS resources that you might not normally have access to. Typically, you -// use AssumeRole for cross-account access or federation. For a comparison of -// AssumeRole with the other APIs that produce temporary credentials, see Requesting -// Temporary Security Credentials (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) -// and Comparing the AWS STS APIs (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) +// Returns a set of temporary security credentials that you can use to access +// AWS resources that you might not normally have access to. These temporary +// credentials consist of an access key ID, a secret access key, and a security +// token. Typically, you use AssumeRole within your account or for cross-account +// access. For a comparison of AssumeRole with other API operations that produce +// temporary credentials, see Requesting Temporary Security Credentials (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) +// and Comparing the AWS STS API operations (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) // in the IAM User Guide. // -// Important: You cannot call AssumeRole by using AWS root account credentials; -// access is denied. You must use credentials for an IAM user or an IAM role -// to call AssumeRole. +// You cannot use AWS account root user credentials to call AssumeRole. You +// must use credentials for an IAM user or an IAM role to call AssumeRole. // // For cross-account access, imagine that you own multiple accounts and need // to access resources in each account. You could create long-term credentials // in each account to access those resources. However, managing all those credentials // and remembering which one can access which account can be time consuming. -// Instead, you can create one set of long-term credentials in one account and -// then use temporary security credentials to access all the other accounts +// Instead, you can create one set of long-term credentials in one account. +// Then use temporary security credentials to access all the other accounts // by assuming roles in those accounts. For more information about roles, see -// IAM Roles (Delegation and Federation) (http://docs.aws.amazon.com/IAM/latest/UserGuide/roles-toplevel.html) +// IAM Roles (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) // in the IAM User Guide. // -// For federation, you can, for example, grant single sign-on access to the -// AWS Management Console. If you already have an identity and authentication -// system in your corporate network, you don't have to recreate user identities -// in AWS in order to grant those user identities access to AWS. Instead, after -// a user has been authenticated, you call AssumeRole (and specify the role -// with the appropriate permissions) to get temporary security credentials for -// that user. With those temporary security credentials, you construct a sign-in -// URL that users can use to access the console. For more information, see Common -// Scenarios for Temporary Credentials (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html#sts-introduction) -// in the IAM User Guide. +// Session Duration // // By default, the temporary security credentials created by AssumeRole last // for one hour. However, you can use the optional DurationSeconds parameter @@ -94,69 +86,93 @@ func (c *STS) AssumeRoleRequest(input *AssumeRoleInput) (req *request.Request, o // seconds (15 minutes) up to the maximum session duration setting for the role. // This setting can have a value from 1 hour to 12 hours. To learn how to view // the maximum value for your role, see View the Maximum Session Duration Setting -// for a Role (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) +// for a Role (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) // in the IAM User Guide. The maximum session duration limit applies when you -// use the AssumeRole* API operations or the assume-role* CLI operations but -// does not apply when you use those operations to create a console URL. For -// more information, see Using IAM Roles (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) +// use the AssumeRole* API operations or the assume-role* CLI commands. However +// the limit does not apply when you use those operations to create a console +// URL. For more information, see Using IAM Roles (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) // in the IAM User Guide. // +// Permissions +// // The temporary security credentials created by AssumeRole can be used to make -// API calls to any AWS service with the following exception: you cannot call -// the STS service's GetFederationToken or GetSessionToken APIs. -// -// Optionally, you can pass an IAM access policy to this operation. If you choose -// not to pass a policy, the temporary security credentials that are returned -// by the operation have the permissions that are defined in the access policy -// of the role that is being assumed. If you pass a policy to this operation, -// the temporary security credentials that are returned by the operation have -// the permissions that are allowed by both the access policy of the role that -// is being assumed, and the policy that you pass. This gives you a way to further -// restrict the permissions for the resulting temporary security credentials. -// You cannot use the passed policy to grant permissions that are in excess -// of those allowed by the access policy of the role that is being assumed. -// For more information, see Permissions for AssumeRole, AssumeRoleWithSAML, -// and AssumeRoleWithWebIdentity (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_assumerole.html) +// API calls to any AWS service with the following exception: You cannot call +// the AWS STS GetFederationToken or GetSessionToken API operations. +// +// (Optional) You can pass inline or managed session policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) +// to this operation. You can pass a single JSON policy document to use as an +// inline session policy. You can also specify up to 10 managed policies to +// use as managed session policies. The plain text that you use for both inline +// and managed session policies can't exceed 2,048 characters. Passing policies +// to this operation returns new temporary credentials. The resulting session's +// permissions are the intersection of the role's identity-based policy and +// the session policies. You can use the role's temporary credentials in subsequent +// AWS API calls to access resources in the account that owns the role. You +// cannot use session policies to grant more permissions than those allowed +// by the identity-based policy of the role that is being assumed. For more +// information, see Session Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) // in the IAM User Guide. // -// To assume a role, your AWS account must be trusted by the role. The trust -// relationship is defined in the role's trust policy when the role is created. -// That trust policy states which accounts are allowed to delegate access to -// this account's role. -// -// The user who wants to access the role must also have permissions delegated -// from the role's administrator. If the user is in a different account than -// the role, then the user's administrator must attach a policy that allows -// the user to call AssumeRole on the ARN of the role in the other account. -// If the user is in the same account as the role, then you can either attach -// a policy to the user (identical to the previous different account user), -// or you can add the user as a principal directly in the role's trust policy. -// In this case, the trust policy acts as the only resource-based policy in -// IAM, and users in the same account as the role do not need explicit permission -// to assume the role. For more information about trust policies and resource-based -// policies, see IAM Policies (http://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html) +// To assume a role from a different account, your AWS account must be trusted +// by the role. The trust relationship is defined in the role's trust policy +// when the role is created. That trust policy states which accounts are allowed +// to delegate that access to users in the account. +// +// A user who wants to access a role in a different account must also have permissions +// that are delegated from the user account administrator. The administrator +// must attach a policy that allows the user to call AssumeRole for the ARN +// of the role in the other account. If the user is in the same account as the +// role, then you can do either of the following: +// +// * Attach a policy to the user (identical to the previous user in a different +// account). +// +// * Add the user as a principal directly in the role's trust policy. +// +// In this case, the trust policy acts as an IAM resource-based policy. Users +// in the same account as the role do not need explicit permission to assume +// the role. For more information about trust policies and resource-based policies, +// see IAM Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html) +// in the IAM User Guide. +// +// Tags +// +// (Optional) You can pass tag key-value pairs to your session. These tags are +// called session tags. For more information about session tags, see Passing +// Session Tags in STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) +// in the IAM User Guide. +// +// An administrator must grant you the permissions necessary to pass session +// tags. The administrator can also create granular permissions to allow you +// to pass only specific session tags. For more information, see Tutorial: Using +// Tags for Attribute-Based Access Control (https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_attribute-based-access-control.html) +// in the IAM User Guide. +// +// You can set the session tags as transitive. Transitive tags persist during +// role chaining. For more information, see Chaining Roles with Session Tags +// (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html#id_session-tags_role-chaining) // in the IAM User Guide. // // Using MFA with AssumeRole // -// You can optionally include multi-factor authentication (MFA) information -// when you call AssumeRole. This is useful for cross-account scenarios in which -// you want to make sure that the user who is assuming the role has been authenticated -// using an AWS MFA device. In that scenario, the trust policy of the role being -// assumed includes a condition that tests for MFA authentication; if the caller -// does not include valid MFA information, the request to assume the role is -// denied. The condition in a trust policy that tests for MFA authentication -// might look like the following example. +// (Optional) You can include multi-factor authentication (MFA) information +// when you call AssumeRole. This is useful for cross-account scenarios to ensure +// that the user that assumes the role has been authenticated with an AWS MFA +// device. In that scenario, the trust policy of the role being assumed includes +// a condition that tests for MFA authentication. If the caller does not include +// valid MFA information, the request to assume the role is denied. The condition +// in a trust policy that tests for MFA authentication might look like the following +// example. // // "Condition": {"Bool": {"aws:MultiFactorAuthPresent": true}} // -// For more information, see Configuring MFA-Protected API Access (http://docs.aws.amazon.com/IAM/latest/UserGuide/MFAProtectedAPI.html) +// For more information, see Configuring MFA-Protected API Access (https://docs.aws.amazon.com/IAM/latest/UserGuide/MFAProtectedAPI.html) // in the IAM User Guide guide. // // To use MFA with AssumeRole, you pass values for the SerialNumber and TokenCode // parameters. The SerialNumber value identifies the user's hardware or virtual // MFA device. The TokenCode is the time-based one-time password (TOTP) that -// the MFA devices produces. +// the MFA device produces. // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -171,15 +187,24 @@ func (c *STS) AssumeRoleRequest(input *AssumeRoleInput) (req *request.Request, o // message describes the specific error. // // * ErrCodePackedPolicyTooLargeException "PackedPolicyTooLarge" -// The request was rejected because the policy document was too large. The error -// message describes how big the policy document is, in packed form, as a percentage -// of what the API allows. +// The request was rejected because the total packed size of the session policies +// and session tags combined was too large. An AWS conversion compresses the +// session policy document, session policy ARNs, and session tags into a packed +// binary format that has a separate limit. The error message indicates by percentage +// how close the policies and tags are to the upper size limit. For more information, +// see Passing Session Tags in STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) +// in the IAM User Guide. +// +// You could receive this error even though you meet other defined session policy +// and session tag limits. For more information, see IAM and STS Entity Character +// Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// in the IAM User Guide. // // * ErrCodeRegionDisabledException "RegionDisabledException" // STS is not activated in the requested region for the account that is being // asked to generate credentials. The account administrator must use the IAM // console to activate STS in that region. For more information, see Activating -// and Deactivating AWS STS in an AWS Region (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// and Deactivating AWS STS in an AWS Region (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) // in the IAM User Guide. // // See also, https://docs.aws.amazon.com/goto/WebAPI/sts-2011-06-15/AssumeRole @@ -243,6 +268,7 @@ func (c *STS) AssumeRoleWithSAMLRequest(input *AssumeRoleWithSAMLInput) (req *re output = &AssumeRoleWithSAMLOutput{} req = c.newRequest(op, input, output) + req.Config.Credentials = credentials.AnonymousCredentials return } @@ -252,15 +278,17 @@ func (c *STS) AssumeRoleWithSAMLRequest(input *AssumeRoleWithSAMLInput) (req *re // via a SAML authentication response. This operation provides a mechanism for // tying an enterprise identity store or directory to role-based AWS access // without user-specific credentials or configuration. For a comparison of AssumeRoleWithSAML -// with the other APIs that produce temporary credentials, see Requesting Temporary -// Security Credentials (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) -// and Comparing the AWS STS APIs (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) +// with the other API operations that produce temporary credentials, see Requesting +// Temporary Security Credentials (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) +// and Comparing the AWS STS API operations (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) // in the IAM User Guide. // // The temporary security credentials returned by this operation consist of // an access key ID, a secret access key, and a security token. Applications // can use these temporary security credentials to sign calls to AWS services. // +// Session Duration +// // By default, the temporary security credentials created by AssumeRoleWithSAML // last for one hour. However, you can use the optional DurationSeconds parameter // to specify the duration of your session. Your role session lasts for the @@ -269,38 +297,33 @@ func (c *STS) AssumeRoleWithSAMLRequest(input *AssumeRoleWithSAMLInput) (req *re // a DurationSeconds value from 900 seconds (15 minutes) up to the maximum session // duration setting for the role. This setting can have a value from 1 hour // to 12 hours. To learn how to view the maximum value for your role, see View -// the Maximum Session Duration Setting for a Role (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) +// the Maximum Session Duration Setting for a Role (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) // in the IAM User Guide. The maximum session duration limit applies when you -// use the AssumeRole* API operations or the assume-role* CLI operations but -// does not apply when you use those operations to create a console URL. For -// more information, see Using IAM Roles (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) +// use the AssumeRole* API operations or the assume-role* CLI commands. However +// the limit does not apply when you use those operations to create a console +// URL. For more information, see Using IAM Roles (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) // in the IAM User Guide. // +// Permissions +// // The temporary security credentials created by AssumeRoleWithSAML can be used // to make API calls to any AWS service with the following exception: you cannot -// call the STS service's GetFederationToken or GetSessionToken APIs. -// -// Optionally, you can pass an IAM access policy to this operation. If you choose -// not to pass a policy, the temporary security credentials that are returned -// by the operation have the permissions that are defined in the access policy -// of the role that is being assumed. If you pass a policy to this operation, -// the temporary security credentials that are returned by the operation have -// the permissions that are allowed by the intersection of both the access policy -// of the role that is being assumed, and the policy that you pass. This means -// that both policies must grant the permission for the action to be allowed. -// This gives you a way to further restrict the permissions for the resulting -// temporary security credentials. You cannot use the passed policy to grant -// permissions that are in excess of those allowed by the access policy of the -// role that is being assumed. For more information, see Permissions for AssumeRole, -// AssumeRoleWithSAML, and AssumeRoleWithWebIdentity (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_assumerole.html) +// call the STS GetFederationToken or GetSessionToken API operations. +// +// (Optional) You can pass inline or managed session policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) +// to this operation. You can pass a single JSON policy document to use as an +// inline session policy. You can also specify up to 10 managed policies to +// use as managed session policies. The plain text that you use for both inline +// and managed session policies can't exceed 2,048 characters. Passing policies +// to this operation returns new temporary credentials. The resulting session's +// permissions are the intersection of the role's identity-based policy and +// the session policies. You can use the role's temporary credentials in subsequent +// AWS API calls to access resources in the account that owns the role. You +// cannot use session policies to grant more permissions than those allowed +// by the identity-based policy of the role that is being assumed. For more +// information, see Session Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) // in the IAM User Guide. // -// Before your application can call AssumeRoleWithSAML, you must configure your -// SAML identity provider (IdP) to issue the claims required by AWS. Additionally, -// you must use AWS Identity and Access Management (IAM) to create a SAML provider -// entity in your AWS account that represents your identity provider, and create -// an IAM role that specifies this SAML provider in its trust policy. -// // Calling AssumeRoleWithSAML does not require the use of AWS security credentials. // The identity of the caller is validated by using keys in the metadata document // that is uploaded for the SAML provider entity for your identity provider. @@ -308,21 +331,63 @@ func (c *STS) AssumeRoleWithSAMLRequest(input *AssumeRoleWithSAMLInput) (req *re // Calling AssumeRoleWithSAML can result in an entry in your AWS CloudTrail // logs. The entry includes the value in the NameID element of the SAML assertion. // We recommend that you use a NameIDType that is not associated with any personally -// identifiable information (PII). For example, you could instead use the Persistent -// Identifier (urn:oasis:names:tc:SAML:2.0:nameid-format:persistent). +// identifiable information (PII). For example, you could instead use the persistent +// identifier (urn:oasis:names:tc:SAML:2.0:nameid-format:persistent). +// +// Tags +// +// (Optional) You can configure your IdP to pass attributes into your SAML assertion +// as session tags. Each session tag consists of a key name and an associated +// value. For more information about session tags, see Passing Session Tags +// in STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) +// in the IAM User Guide. +// +// You can pass up to 50 session tags. The plain text session tag keys can’t +// exceed 128 characters and the values can’t exceed 256 characters. For these +// and additional limits, see IAM and STS Character Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-limits.html#reference_iam-limits-entity-length) +// in the IAM User Guide. +// +// An AWS conversion compresses the passed session policies and session tags +// into a packed binary format that has a separate limit. Your request can fail +// for this limit even if your plain text meets the other requirements. The +// PackedPolicySize response element indicates by percentage how close the policies +// and tags for your request are to the upper size limit. +// +// You can pass a session tag with the same key as a tag that is attached to +// the role. When you do, session tags override the role's tags with the same +// key. +// +// An administrator must grant you the permissions necessary to pass session +// tags. The administrator can also create granular permissions to allow you +// to pass only specific session tags. For more information, see Tutorial: Using +// Tags for Attribute-Based Access Control (https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_attribute-based-access-control.html) +// in the IAM User Guide. +// +// You can set the session tags as transitive. Transitive tags persist during +// role chaining. For more information, see Chaining Roles with Session Tags +// (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html#id_session-tags_role-chaining) +// in the IAM User Guide. +// +// SAML Configuration +// +// Before your application can call AssumeRoleWithSAML, you must configure your +// SAML identity provider (IdP) to issue the claims required by AWS. Additionally, +// you must use AWS Identity and Access Management (IAM) to create a SAML provider +// entity in your AWS account that represents your identity provider. You must +// also create an IAM role that specifies this SAML provider in its trust policy. // // For more information, see the following resources: // -// * About SAML 2.0-based Federation (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_saml.html) +// * About SAML 2.0-based Federation (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_saml.html) // in the IAM User Guide. // -// * Creating SAML Identity Providers (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml.html) +// * Creating SAML Identity Providers (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml.html) // in the IAM User Guide. // -// * Configuring a Relying Party and Claims (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml_relying-party.html) +// * Configuring a Relying Party and Claims (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml_relying-party.html) // in the IAM User Guide. // -// * Creating a Role for SAML 2.0 Federation (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_saml.html) +// * Creating a Role for SAML 2.0 Federation (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_saml.html) // in the IAM User Guide. // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions @@ -338,9 +403,18 @@ func (c *STS) AssumeRoleWithSAMLRequest(input *AssumeRoleWithSAMLInput) (req *re // message describes the specific error. // // * ErrCodePackedPolicyTooLargeException "PackedPolicyTooLarge" -// The request was rejected because the policy document was too large. The error -// message describes how big the policy document is, in packed form, as a percentage -// of what the API allows. +// The request was rejected because the total packed size of the session policies +// and session tags combined was too large. An AWS conversion compresses the +// session policy document, session policy ARNs, and session tags into a packed +// binary format that has a separate limit. The error message indicates by percentage +// how close the policies and tags are to the upper size limit. For more information, +// see Passing Session Tags in STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) +// in the IAM User Guide. +// +// You could receive this error even though you meet other defined session policy +// and session tag limits. For more information, see IAM and STS Entity Character +// Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// in the IAM User Guide. // // * ErrCodeIDPRejectedClaimException "IDPRejectedClaim" // The identity provider (IdP) reported that authentication failed. This might @@ -361,7 +435,7 @@ func (c *STS) AssumeRoleWithSAMLRequest(input *AssumeRoleWithSAMLInput) (req *re // STS is not activated in the requested region for the account that is being // asked to generate credentials. The account administrator must use the IAM // console to activate STS in that region. For more information, see Activating -// and Deactivating AWS STS in an AWS Region (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// and Deactivating AWS STS in an AWS Region (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) // in the IAM User Guide. // // See also, https://docs.aws.amazon.com/goto/WebAPI/sts-2011-06-15/AssumeRoleWithSAML @@ -425,41 +499,44 @@ func (c *STS) AssumeRoleWithWebIdentityRequest(input *AssumeRoleWithWebIdentityI output = &AssumeRoleWithWebIdentityOutput{} req = c.newRequest(op, input, output) + req.Config.Credentials = credentials.AnonymousCredentials return } // AssumeRoleWithWebIdentity API operation for AWS Security Token Service. // // Returns a set of temporary security credentials for users who have been authenticated -// in a mobile or web application with a web identity provider, such as Amazon -// Cognito, Login with Amazon, Facebook, Google, or any OpenID Connect-compatible -// identity provider. +// in a mobile or web application with a web identity provider. Example providers +// include Amazon Cognito, Login with Amazon, Facebook, Google, or any OpenID +// Connect-compatible identity provider. // // For mobile applications, we recommend that you use Amazon Cognito. You can -// use Amazon Cognito with the AWS SDK for iOS (http://aws.amazon.com/sdkforios/) -// and the AWS SDK for Android (http://aws.amazon.com/sdkforandroid/) to uniquely -// identify a user and supply the user with a consistent identity throughout -// the lifetime of an application. -// -// To learn more about Amazon Cognito, see Amazon Cognito Overview (http://docs.aws.amazon.com/mobile/sdkforandroid/developerguide/cognito-auth.html#d0e840) -// in the AWS SDK for Android Developer Guide guide and Amazon Cognito Overview -// (http://docs.aws.amazon.com/mobile/sdkforios/developerguide/cognito-auth.html#d0e664) +// use Amazon Cognito with the AWS SDK for iOS Developer Guide (http://aws.amazon.com/sdkforios/) +// and the AWS SDK for Android Developer Guide (http://aws.amazon.com/sdkforandroid/) +// to uniquely identify a user. You can also supply the user with a consistent +// identity throughout the lifetime of an application. +// +// To learn more about Amazon Cognito, see Amazon Cognito Overview (https://docs.aws.amazon.com/mobile/sdkforandroid/developerguide/cognito-auth.html#d0e840) +// in AWS SDK for Android Developer Guide and Amazon Cognito Overview (https://docs.aws.amazon.com/mobile/sdkforios/developerguide/cognito-auth.html#d0e664) // in the AWS SDK for iOS Developer Guide. // // Calling AssumeRoleWithWebIdentity does not require the use of AWS security // credentials. Therefore, you can distribute an application (for example, on // mobile devices) that requests temporary security credentials without including -// long-term AWS credentials in the application, and without deploying server-based -// proxy services that use long-term AWS credentials. Instead, the identity -// of the caller is validated by using a token from the web identity provider. -// For a comparison of AssumeRoleWithWebIdentity with the other APIs that produce -// temporary credentials, see Requesting Temporary Security Credentials (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) -// and Comparing the AWS STS APIs (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) +// long-term AWS credentials in the application. You also don't need to deploy +// server-based proxy services that use long-term AWS credentials. Instead, +// the identity of the caller is validated by using a token from the web identity +// provider. For a comparison of AssumeRoleWithWebIdentity with the other API +// operations that produce temporary credentials, see Requesting Temporary Security +// Credentials (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) +// and Comparing the AWS STS API operations (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) // in the IAM User Guide. // // The temporary security credentials returned by this API consist of an access // key ID, a secret access key, and a security token. Applications can use these -// temporary security credentials to sign calls to AWS service APIs. +// temporary security credentials to sign calls to AWS service API operations. +// +// Session Duration // // By default, the temporary security credentials created by AssumeRoleWithWebIdentity // last for one hour. However, you can use the optional DurationSeconds parameter @@ -467,31 +544,69 @@ func (c *STS) AssumeRoleWithWebIdentityRequest(input *AssumeRoleWithWebIdentityI // seconds (15 minutes) up to the maximum session duration setting for the role. // This setting can have a value from 1 hour to 12 hours. To learn how to view // the maximum value for your role, see View the Maximum Session Duration Setting -// for a Role (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) +// for a Role (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) // in the IAM User Guide. The maximum session duration limit applies when you -// use the AssumeRole* API operations or the assume-role* CLI operations but -// does not apply when you use those operations to create a console URL. For -// more information, see Using IAM Roles (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) +// use the AssumeRole* API operations or the assume-role* CLI commands. However +// the limit does not apply when you use those operations to create a console +// URL. For more information, see Using IAM Roles (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) // in the IAM User Guide. // +// Permissions +// // The temporary security credentials created by AssumeRoleWithWebIdentity can // be used to make API calls to any AWS service with the following exception: -// you cannot call the STS service's GetFederationToken or GetSessionToken APIs. -// -// Optionally, you can pass an IAM access policy to this operation. If you choose -// not to pass a policy, the temporary security credentials that are returned -// by the operation have the permissions that are defined in the access policy -// of the role that is being assumed. If you pass a policy to this operation, -// the temporary security credentials that are returned by the operation have -// the permissions that are allowed by both the access policy of the role that -// is being assumed, and the policy that you pass. This gives you a way to further -// restrict the permissions for the resulting temporary security credentials. -// You cannot use the passed policy to grant permissions that are in excess -// of those allowed by the access policy of the role that is being assumed. -// For more information, see Permissions for AssumeRole, AssumeRoleWithSAML, -// and AssumeRoleWithWebIdentity (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_assumerole.html) +// you cannot call the STS GetFederationToken or GetSessionToken API operations. +// +// (Optional) You can pass inline or managed session policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) +// to this operation. You can pass a single JSON policy document to use as an +// inline session policy. You can also specify up to 10 managed policies to +// use as managed session policies. The plain text that you use for both inline +// and managed session policies can't exceed 2,048 characters. Passing policies +// to this operation returns new temporary credentials. The resulting session's +// permissions are the intersection of the role's identity-based policy and +// the session policies. You can use the role's temporary credentials in subsequent +// AWS API calls to access resources in the account that owns the role. You +// cannot use session policies to grant more permissions than those allowed +// by the identity-based policy of the role that is being assumed. For more +// information, see Session Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) // in the IAM User Guide. // +// Tags +// +// (Optional) You can configure your IdP to pass attributes into your web identity +// token as session tags. Each session tag consists of a key name and an associated +// value. For more information about session tags, see Passing Session Tags +// in STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) +// in the IAM User Guide. +// +// You can pass up to 50 session tags. The plain text session tag keys can’t +// exceed 128 characters and the values can’t exceed 256 characters. For these +// and additional limits, see IAM and STS Character Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-limits.html#reference_iam-limits-entity-length) +// in the IAM User Guide. +// +// An AWS conversion compresses the passed session policies and session tags +// into a packed binary format that has a separate limit. Your request can fail +// for this limit even if your plain text meets the other requirements. The +// PackedPolicySize response element indicates by percentage how close the policies +// and tags for your request are to the upper size limit. +// +// You can pass a session tag with the same key as a tag that is attached to +// the role. When you do, the session tag overrides the role tag with the same +// key. +// +// An administrator must grant you the permissions necessary to pass session +// tags. The administrator can also create granular permissions to allow you +// to pass only specific session tags. For more information, see Tutorial: Using +// Tags for Attribute-Based Access Control (https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_attribute-based-access-control.html) +// in the IAM User Guide. +// +// You can set the session tags as transitive. Transitive tags persist during +// role chaining. For more information, see Chaining Roles with Session Tags +// (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html#id_session-tags_role-chaining) +// in the IAM User Guide. +// +// Identities +// // Before your application can call AssumeRoleWithWebIdentity, you must have // an identity token from a supported identity provider and create a role that // the application can assume. The role that your application assumes must trust @@ -508,21 +623,19 @@ func (c *STS) AssumeRoleWithWebIdentityRequest(input *AssumeRoleWithWebIdentityI // For more information about how to use web identity federation and the AssumeRoleWithWebIdentity // API, see the following resources: // -// * Using Web Identity Federation APIs for Mobile Apps (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc_manual.html) -// and Federation Through a Web-based Identity Provider (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_assumerolewithwebidentity). -// -// -// * Web Identity Federation Playground (https://web-identity-federation-playground.s3.amazonaws.com/index.html). -// This interactive website lets you walk through the process of authenticating -// via Login with Amazon, Facebook, or Google, getting temporary security -// credentials, and then using those credentials to make a request to AWS. +// * Using Web Identity Federation API Operations for Mobile Apps (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_oidc_manual.html) +// and Federation Through a Web-based Identity Provider (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_assumerolewithwebidentity). // +// * Web Identity Federation Playground (https://web-identity-federation-playground.s3.amazonaws.com/index.html). +// Walk through the process of authenticating through Login with Amazon, +// Facebook, or Google, getting temporary security credentials, and then +// using those credentials to make a request to AWS. // -// * AWS SDK for iOS (http://aws.amazon.com/sdkforios/) and AWS SDK for Android -// (http://aws.amazon.com/sdkforandroid/). These toolkits contain sample -// apps that show how to invoke the identity providers, and then how to use -// the information from these providers to get and use temporary security -// credentials. +// * AWS SDK for iOS Developer Guide (http://aws.amazon.com/sdkforios/) and +// AWS SDK for Android Developer Guide (http://aws.amazon.com/sdkforandroid/). +// These toolkits contain sample apps that show how to invoke the identity +// providers. The toolkits then show how to use the information from these +// providers to get and use temporary security credentials. // // * Web Identity Federation with Mobile Applications (http://aws.amazon.com/articles/web-identity-federation-with-mobile-applications). // This article discusses web identity federation and shows an example of @@ -542,9 +655,18 @@ func (c *STS) AssumeRoleWithWebIdentityRequest(input *AssumeRoleWithWebIdentityI // message describes the specific error. // // * ErrCodePackedPolicyTooLargeException "PackedPolicyTooLarge" -// The request was rejected because the policy document was too large. The error -// message describes how big the policy document is, in packed form, as a percentage -// of what the API allows. +// The request was rejected because the total packed size of the session policies +// and session tags combined was too large. An AWS conversion compresses the +// session policy document, session policy ARNs, and session tags into a packed +// binary format that has a separate limit. The error message indicates by percentage +// how close the policies and tags are to the upper size limit. For more information, +// see Passing Session Tags in STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) +// in the IAM User Guide. +// +// You could receive this error even though you meet other defined session policy +// and session tag limits. For more information, see IAM and STS Entity Character +// Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// in the IAM User Guide. // // * ErrCodeIDPRejectedClaimException "IDPRejectedClaim" // The identity provider (IdP) reported that authentication failed. This might @@ -554,11 +676,11 @@ func (c *STS) AssumeRoleWithWebIdentityRequest(input *AssumeRoleWithWebIdentityI // can also mean that the claim has expired or has been explicitly revoked. // // * ErrCodeIDPCommunicationErrorException "IDPCommunicationError" -// The request could not be fulfilled because the non-AWS identity provider -// (IDP) that was asked to verify the incoming identity token could not be reached. -// This is often a transient error caused by network conditions. Retry the request +// The request could not be fulfilled because the identity provider (IDP) that +// was asked to verify the incoming identity token could not be reached. This +// is often a transient error caused by network conditions. Retry the request // a limited number of times so that you don't exceed the request rate. If the -// error persists, the non-AWS identity provider might be down or not responding. +// error persists, the identity provider might be down or not responding. // // * ErrCodeInvalidIdentityTokenException "InvalidIdentityToken" // The web identity token that was passed could not be validated by AWS. Get @@ -572,7 +694,7 @@ func (c *STS) AssumeRoleWithWebIdentityRequest(input *AssumeRoleWithWebIdentityI // STS is not activated in the requested region for the account that is being // asked to generate credentials. The account administrator must use the IAM // console to activate STS in that region. For more information, see Activating -// and Deactivating AWS STS in an AWS Region (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// and Deactivating AWS STS in an AWS Region (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) // in the IAM User Guide. // // See also, https://docs.aws.amazon.com/goto/WebAPI/sts-2011-06-15/AssumeRoleWithWebIdentity @@ -644,17 +766,17 @@ func (c *STS) DecodeAuthorizationMessageRequest(input *DecodeAuthorizationMessag // Decodes additional information about the authorization status of a request // from an encoded message returned in response to an AWS request. // -// For example, if a user is not authorized to perform an action that he or -// she has requested, the request returns a Client.UnauthorizedOperation response -// (an HTTP 403 response). Some AWS actions additionally return an encoded message -// that can provide details about this authorization failure. +// For example, if a user is not authorized to perform an operation that he +// or she has requested, the request returns a Client.UnauthorizedOperation +// response (an HTTP 403 response). Some AWS operations additionally return +// an encoded message that can provide details about this authorization failure. // -// Only certain AWS actions return an encoded authorization message. The documentation -// for an individual action indicates whether that action returns an encoded -// message in addition to returning an HTTP code. +// Only certain AWS operations return an encoded authorization message. The +// documentation for an individual operation indicates whether that operation +// returns an encoded message in addition to returning an HTTP code. // // The message is encoded because the details of the authorization status can -// constitute privileged information that the user who requested the action +// constitute privileged information that the user who requested the operation // should not see. To decode an authorization status message, a user must be // granted permissions via an IAM policy to request the DecodeAuthorizationMessage // (sts:DecodeAuthorizationMessage) action. @@ -663,7 +785,7 @@ func (c *STS) DecodeAuthorizationMessageRequest(input *DecodeAuthorizationMessag // // * Whether the request was denied due to an explicit deny or due to the // absence of an explicit allow. For more information, see Determining Whether -// a Request is Allowed or Denied (http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html#policy-eval-denyallow) +// a Request is Allowed or Denied (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_evaluation-logic.html#policy-eval-denyallow) // in the IAM User Guide. // // * The principal who made the request. @@ -709,6 +831,103 @@ func (c *STS) DecodeAuthorizationMessageWithContext(ctx aws.Context, input *Deco return out, req.Send() } +const opGetAccessKeyInfo = "GetAccessKeyInfo" + +// GetAccessKeyInfoRequest generates a "aws/request.Request" representing the +// client's request for the GetAccessKeyInfo operation. The "output" return +// value will be populated with the request's response once the request completes +// successfully. +// +// Use "Send" method on the returned Request to send the API call to the service. +// the "output" return value is not valid until after Send returns without error. +// +// See GetAccessKeyInfo for more information on using the GetAccessKeyInfo +// API call, and error handling. +// +// This method is useful when you want to inject custom logic or configuration +// into the SDK's request lifecycle. Such as custom headers, or retry logic. +// +// +// // Example sending a request using the GetAccessKeyInfoRequest method. +// req, resp := client.GetAccessKeyInfoRequest(params) +// +// err := req.Send() +// if err == nil { // resp is now filled +// fmt.Println(resp) +// } +// +// See also, https://docs.aws.amazon.com/goto/WebAPI/sts-2011-06-15/GetAccessKeyInfo +func (c *STS) GetAccessKeyInfoRequest(input *GetAccessKeyInfoInput) (req *request.Request, output *GetAccessKeyInfoOutput) { + op := &request.Operation{ + Name: opGetAccessKeyInfo, + HTTPMethod: "POST", + HTTPPath: "/", + } + + if input == nil { + input = &GetAccessKeyInfoInput{} + } + + output = &GetAccessKeyInfoOutput{} + req = c.newRequest(op, input, output) + return +} + +// GetAccessKeyInfo API operation for AWS Security Token Service. +// +// Returns the account identifier for the specified access key ID. +// +// Access keys consist of two parts: an access key ID (for example, AKIAIOSFODNN7EXAMPLE) +// and a secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY). +// For more information about access keys, see Managing Access Keys for IAM +// Users (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) +// in the IAM User Guide. +// +// When you pass an access key ID to this operation, it returns the ID of the +// AWS account to which the keys belong. Access key IDs beginning with AKIA +// are long-term credentials for an IAM user or the AWS account root user. Access +// key IDs beginning with ASIA are temporary credentials that are created using +// STS operations. If the account in the response belongs to you, you can sign +// in as the root user and review your root user access keys. Then, you can +// pull a credentials report (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_getting-report.html) +// to learn which IAM user owns the keys. To learn who requested the temporary +// credentials for an ASIA access key, view the STS events in your CloudTrail +// logs (https://docs.aws.amazon.com/IAM/latest/UserGuide/cloudtrail-integration.html) +// in the IAM User Guide. +// +// This operation does not indicate the state of the access key. The key might +// be active, inactive, or deleted. Active keys might not have permissions to +// perform an operation. Providing a deleted access key might return an error +// that the key doesn't exist. +// +// Returns awserr.Error for service API and SDK errors. Use runtime type assertions +// with awserr.Error's Code and Message methods to get detailed information about +// the error. +// +// See the AWS API reference guide for AWS Security Token Service's +// API operation GetAccessKeyInfo for usage and error information. +// See also, https://docs.aws.amazon.com/goto/WebAPI/sts-2011-06-15/GetAccessKeyInfo +func (c *STS) GetAccessKeyInfo(input *GetAccessKeyInfoInput) (*GetAccessKeyInfoOutput, error) { + req, out := c.GetAccessKeyInfoRequest(input) + return out, req.Send() +} + +// GetAccessKeyInfoWithContext is the same as GetAccessKeyInfo with the addition of +// the ability to pass a context and additional request options. +// +// See GetAccessKeyInfo for details on how to use this API operation. +// +// The context must be non-nil and will be used for request cancellation. If +// the context is nil a panic will occur. In the future the SDK may create +// sub-contexts for http.Requests. See https://golang.org/pkg/context/ +// for more information on using Contexts. +func (c *STS) GetAccessKeyInfoWithContext(ctx aws.Context, input *GetAccessKeyInfoInput, opts ...request.Option) (*GetAccessKeyInfoOutput, error) { + req, out := c.GetAccessKeyInfoRequest(input) + req.SetContext(ctx) + req.ApplyOptions(opts...) + return out, req.Send() +} + const opGetCallerIdentity = "GetCallerIdentity" // GetCallerIdentityRequest generates a "aws/request.Request" representing the @@ -753,8 +972,16 @@ func (c *STS) GetCallerIdentityRequest(input *GetCallerIdentityInput) (req *requ // GetCallerIdentity API operation for AWS Security Token Service. // -// Returns details about the IAM identity whose credentials are used to call -// the API. +// Returns details about the IAM user or role whose credentials are used to +// call the operation. +// +// No permissions are required to perform this operation. If an administrator +// adds a policy to your IAM user or role that explicitly denies access to the +// sts:GetCallerIdentity action, you can still perform this operation. Permissions +// are not required because the same information is returned when an IAM user +// or role is denied access. To view an example response, see I Am Not Authorized +// to Perform: iam:DeleteVirtualMFADevice (https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoot_general.html#troubleshoot_general_access-denied-delete-mfa) +// in the IAM User Guide. // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -831,81 +1058,92 @@ func (c *STS) GetFederationTokenRequest(input *GetFederationTokenInput) (req *re // Returns a set of temporary security credentials (consisting of an access // key ID, a secret access key, and a security token) for a federated user. // A typical use is in a proxy application that gets temporary security credentials -// on behalf of distributed applications inside a corporate network. Because -// you must call the GetFederationToken action using the long-term security -// credentials of an IAM user, this call is appropriate in contexts where those +// on behalf of distributed applications inside a corporate network. You must +// call the GetFederationToken operation using the long-term security credentials +// of an IAM user. As a result, this call is appropriate in contexts where those // credentials can be safely stored, usually in a server-based application. -// For a comparison of GetFederationToken with the other APIs that produce temporary -// credentials, see Requesting Temporary Security Credentials (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) -// and Comparing the AWS STS APIs (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) +// For a comparison of GetFederationToken with the other API operations that +// produce temporary credentials, see Requesting Temporary Security Credentials +// (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) +// and Comparing the AWS STS API operations (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) // in the IAM User Guide. // -// If you are creating a mobile-based or browser-based app that can authenticate +// You can create a mobile-based or browser-based app that can authenticate // users using a web identity provider like Login with Amazon, Facebook, Google, -// or an OpenID Connect-compatible identity provider, we recommend that you -// use Amazon Cognito (http://aws.amazon.com/cognito/) or AssumeRoleWithWebIdentity. +// or an OpenID Connect-compatible identity provider. In this case, we recommend +// that you use Amazon Cognito (http://aws.amazon.com/cognito/) or AssumeRoleWithWebIdentity. // For more information, see Federation Through a Web-based Identity Provider -// (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_assumerolewithwebidentity). -// -// The GetFederationToken action must be called by using the long-term AWS security -// credentials of an IAM user. You can also call GetFederationToken using the -// security credentials of an AWS root account, but we do not recommended it. -// Instead, we recommend that you create an IAM user for the purpose of the -// proxy application and then attach a policy to the IAM user that limits federated -// users to only the actions and resources that they need access to. For more -// information, see IAM Best Practices (http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) +// (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_assumerolewithwebidentity) // in the IAM User Guide. // -// The temporary security credentials that are obtained by using the long-term -// credentials of an IAM user are valid for the specified duration, from 900 -// seconds (15 minutes) up to a maximium of 129600 seconds (36 hours). The default -// is 43200 seconds (12 hours). Temporary credentials that are obtained by using -// AWS root account credentials have a maximum duration of 3600 seconds (1 hour). -// -// The temporary security credentials created by GetFederationToken can be used -// to make API calls to any AWS service with the following exceptions: +// You can also call GetFederationToken using the security credentials of an +// AWS account root user, but we do not recommend it. Instead, we recommend +// that you create an IAM user for the purpose of the proxy application. Then +// attach a policy to the IAM user that limits federated users to only the actions +// and resources that they need to access. For more information, see IAM Best +// Practices (https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html) +// in the IAM User Guide. // -// * You cannot use these credentials to call any IAM APIs. +// Session duration // -// * You cannot call any STS APIs except GetCallerIdentity. +// The temporary credentials are valid for the specified duration, from 900 +// seconds (15 minutes) up to a maximum of 129,600 seconds (36 hours). The default +// session duration is 43,200 seconds (12 hours). Temporary credentials that +// are obtained by using AWS account root user credentials have a maximum duration +// of 3,600 seconds (1 hour). // // Permissions // -// The permissions for the temporary security credentials returned by GetFederationToken -// are determined by a combination of the following: -// -// * The policy or policies that are attached to the IAM user whose credentials -// are used to call GetFederationToken. -// -// * The policy that is passed as a parameter in the call. -// -// The passed policy is attached to the temporary security credentials that -// result from the GetFederationToken API call--that is, to the federated user. -// When the federated user makes an AWS request, AWS evaluates the policy attached -// to the federated user in combination with the policy or policies attached -// to the IAM user whose credentials were used to call GetFederationToken. AWS -// allows the federated user's request only when both the federated user and -// the IAM user are explicitly allowed to perform the requested action. The -// passed policy cannot grant more permissions than those that are defined in -// the IAM user policy. -// -// A typical use case is that the permissions of the IAM user whose credentials -// are used to call GetFederationToken are designed to allow access to all the -// actions and resources that any federated user will need. Then, for individual -// users, you pass a policy to the operation that scopes down the permissions -// to a level that's appropriate to that individual user, using a policy that -// allows only a subset of permissions that are granted to the IAM user. -// -// If you do not pass a policy, the resulting temporary security credentials -// have no effective permissions. The only exception is when the temporary security -// credentials are used to access a resource that has a resource-based policy -// that specifically allows the federated user to access the resource. -// -// For more information about how permissions work, see Permissions for GetFederationToken -// (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_getfederationtoken.html). -// For information about using GetFederationToken to create temporary security -// credentials, see GetFederationToken—Federation Through a Custom Identity -// Broker (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getfederationtoken). +// You can use the temporary credentials created by GetFederationToken in any +// AWS service except the following: +// +// * You cannot call any IAM operations using the AWS CLI or the AWS API. +// +// * You cannot call any STS operations except GetCallerIdentity. +// +// You must pass an inline or managed session policy (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) +// to this operation. You can pass a single JSON policy document to use as an +// inline session policy. You can also specify up to 10 managed policies to +// use as managed session policies. The plain text that you use for both inline +// and managed session policies can't exceed 2,048 characters. +// +// Though the session policy parameters are optional, if you do not pass a policy, +// then the resulting federated user session has no permissions. When you pass +// session policies, the session permissions are the intersection of the IAM +// user policies and the session policies that you pass. This gives you a way +// to further restrict the permissions for a federated user. You cannot use +// session policies to grant more permissions than those that are defined in +// the permissions policy of the IAM user. For more information, see Session +// Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) +// in the IAM User Guide. For information about using GetFederationToken to +// create temporary security credentials, see GetFederationToken—Federation +// Through a Custom Identity Broker (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getfederationtoken). +// +// You can use the credentials to access a resource that has a resource-based +// policy. If that policy specifically references the federated user session +// in the Principal element of the policy, the session has the permissions allowed +// by the policy. These permissions are granted in addition to the permissions +// granted by the session policies. +// +// Tags +// +// (Optional) You can pass tag key-value pairs to your session. These are called +// session tags. For more information about session tags, see Passing Session +// Tags in STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) +// in the IAM User Guide. +// +// An administrator must grant you the permissions necessary to pass session +// tags. The administrator can also create granular permissions to allow you +// to pass only specific session tags. For more information, see Tutorial: Using +// Tags for Attribute-Based Access Control (https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_attribute-based-access-control.html) +// in the IAM User Guide. +// +// Tag key–value pairs are not case sensitive, but case is preserved. This +// means that you cannot have separate Department and department tag keys. Assume +// that the user that you are federating has the Department=Marketing tag and +// you pass the department=engineering session tag. Department and department +// are not saved as separate tags, and the session tag passed in the request +// takes precedence over the user tag. // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions // with awserr.Error's Code and Message methods to get detailed information about @@ -920,15 +1158,24 @@ func (c *STS) GetFederationTokenRequest(input *GetFederationTokenInput) (req *re // message describes the specific error. // // * ErrCodePackedPolicyTooLargeException "PackedPolicyTooLarge" -// The request was rejected because the policy document was too large. The error -// message describes how big the policy document is, in packed form, as a percentage -// of what the API allows. +// The request was rejected because the total packed size of the session policies +// and session tags combined was too large. An AWS conversion compresses the +// session policy document, session policy ARNs, and session tags into a packed +// binary format that has a separate limit. The error message indicates by percentage +// how close the policies and tags are to the upper size limit. For more information, +// see Passing Session Tags in STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) +// in the IAM User Guide. +// +// You could receive this error even though you meet other defined session policy +// and session tag limits. For more information, see IAM and STS Entity Character +// Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// in the IAM User Guide. // // * ErrCodeRegionDisabledException "RegionDisabledException" // STS is not activated in the requested region for the account that is being // asked to generate credentials. The account administrator must use the IAM // console to activate STS in that region. For more information, see Activating -// and Deactivating AWS STS in an AWS Region (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// and Deactivating AWS STS in an AWS Region (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) // in the IAM User Guide. // // See also, https://docs.aws.amazon.com/goto/WebAPI/sts-2011-06-15/GetFederationToken @@ -1000,48 +1247,51 @@ func (c *STS) GetSessionTokenRequest(input *GetSessionTokenInput) (req *request. // Returns a set of temporary credentials for an AWS account or IAM user. The // credentials consist of an access key ID, a secret access key, and a security // token. Typically, you use GetSessionToken if you want to use MFA to protect -// programmatic calls to specific AWS APIs like Amazon EC2 StopInstances. MFA-enabled -// IAM users would need to call GetSessionToken and submit an MFA code that -// is associated with their MFA device. Using the temporary security credentials -// that are returned from the call, IAM users can then make programmatic calls -// to APIs that require MFA authentication. If you do not supply a correct MFA -// code, then the API returns an access denied error. For a comparison of GetSessionToken -// with the other APIs that produce temporary credentials, see Requesting Temporary -// Security Credentials (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) -// and Comparing the AWS STS APIs (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) +// programmatic calls to specific AWS API operations like Amazon EC2 StopInstances. +// MFA-enabled IAM users would need to call GetSessionToken and submit an MFA +// code that is associated with their MFA device. Using the temporary security +// credentials that are returned from the call, IAM users can then make programmatic +// calls to API operations that require MFA authentication. If you do not supply +// a correct MFA code, then the API returns an access denied error. For a comparison +// of GetSessionToken with the other API operations that produce temporary credentials, +// see Requesting Temporary Security Credentials (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html) +// and Comparing the AWS STS API operations (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#stsapi_comparison) // in the IAM User Guide. // -// The GetSessionToken action must be called by using the long-term AWS security -// credentials of the AWS account or an IAM user. Credentials that are created -// by IAM users are valid for the duration that you specify, from 900 seconds -// (15 minutes) up to a maximum of 129600 seconds (36 hours), with a default -// of 43200 seconds (12 hours); credentials that are created by using account -// credentials can range from 900 seconds (15 minutes) up to a maximum of 3600 -// seconds (1 hour), with a default of 1 hour. +// Session Duration +// +// The GetSessionToken operation must be called by using the long-term AWS security +// credentials of the AWS account root user or an IAM user. Credentials that +// are created by IAM users are valid for the duration that you specify. This +// duration can range from 900 seconds (15 minutes) up to a maximum of 129,600 +// seconds (36 hours), with a default of 43,200 seconds (12 hours). Credentials +// based on account credentials can range from 900 seconds (15 minutes) up to +// 3,600 seconds (1 hour), with a default of 1 hour. +// +// Permissions // // The temporary security credentials created by GetSessionToken can be used // to make API calls to any AWS service with the following exceptions: // -// * You cannot call any IAM APIs unless MFA authentication information is -// included in the request. +// * You cannot call any IAM API operations unless MFA authentication information +// is included in the request. // -// * You cannot call any STS API exceptAssumeRole or GetCallerIdentity. +// * You cannot call any STS API except AssumeRole or GetCallerIdentity. // -// We recommend that you do not call GetSessionToken with root account credentials. -// Instead, follow our best practices (http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#create-iam-users) +// We recommend that you do not call GetSessionToken with AWS account root user +// credentials. Instead, follow our best practices (https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#create-iam-users) // by creating one or more IAM users, giving them the necessary permissions, // and using IAM users for everyday interaction with AWS. // -// The permissions associated with the temporary security credentials returned -// by GetSessionToken are based on the permissions associated with account or -// IAM user whose credentials are used to call the action. If GetSessionToken -// is called using root account credentials, the temporary credentials have -// root account permissions. Similarly, if GetSessionToken is called using the -// credentials of an IAM user, the temporary credentials have the same permissions -// as the IAM user. +// The credentials that are returned by GetSessionToken are based on permissions +// associated with the user whose credentials were used to call the operation. +// If GetSessionToken is called using AWS account root user credentials, the +// temporary credentials have root user permissions. Similarly, if GetSessionToken +// is called using the credentials of an IAM user, the temporary credentials +// have the same permissions as the IAM user. // // For more information about using GetSessionToken to create temporary credentials, -// go to Temporary Credentials for Users in Untrusted Environments (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken) +// go to Temporary Credentials for Users in Untrusted Environments (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_request.html#api_getsessiontoken) // in the IAM User Guide. // // Returns awserr.Error for service API and SDK errors. Use runtime type assertions @@ -1056,7 +1306,7 @@ func (c *STS) GetSessionTokenRequest(input *GetSessionTokenInput) (req *request. // STS is not activated in the requested region for the account that is being // asked to generate credentials. The account administrator must use the IAM // console to activate STS in that region. For more information, see Activating -// and Deactivating AWS STS in an AWS Region (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// and Deactivating AWS STS in an AWS Region (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) // in the IAM User Guide. // // See also, https://docs.aws.amazon.com/goto/WebAPI/sts-2011-06-15/GetSessionToken @@ -1091,7 +1341,7 @@ type AssumeRoleInput struct { // a session duration of 12 hours, but your administrator set the maximum session // duration to 6 hours, your operation fails. To learn how to view the maximum // value for your role, see View the Maximum Session Duration Setting for a - // Role (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) + // Role (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) // in the IAM User Guide. // // By default, the value is set to 3600 seconds. @@ -1101,51 +1351,77 @@ type AssumeRoleInput struct { // to the federation endpoint for a console sign-in token takes a SessionDuration // parameter that specifies the maximum length of the console session. For more // information, see Creating a URL that Enables Federated Users to Access the - // AWS Management Console (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_enable-console-custom-url.html) + // AWS Management Console (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_enable-console-custom-url.html) // in the IAM User Guide. DurationSeconds *int64 `min:"900" type:"integer"` - // A unique identifier that is used by third parties when assuming roles in - // their customers' accounts. For each role that the third party can assume, - // they should instruct their customers to ensure the role's trust policy checks - // for the external ID that the third party generated. Each time the third party - // assumes the role, they should pass the customer's external ID. The external - // ID is useful in order to help third parties bind a role to the customer who - // created it. For more information about the external ID, see How to Use an - // External ID When Granting Access to Your AWS Resources to a Third Party (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html) + // A unique identifier that might be required when you assume a role in another + // account. If the administrator of the account to which the role belongs provided + // you with an external ID, then provide that value in the ExternalId parameter. + // This value can be any string, such as a passphrase or account number. A cross-account + // role is usually set up to trust everyone in an account. Therefore, the administrator + // of the trusting account might send an external ID to the administrator of + // the trusted account. That way, only someone with the ID can assume the role, + // rather than everyone in the account. For more information about the external + // ID, see How to Use an External ID When Granting Access to Your AWS Resources + // to a Third Party (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-user_externalid.html) // in the IAM User Guide. // - // The regex used to validated this parameter is a string of characters consisting + // The regex used to validate this parameter is a string of characters consisting // of upper- and lower-case alphanumeric characters with no spaces. You can // also include underscores or any of the following characters: =,.@:/- ExternalId *string `min:"2" type:"string"` - // An IAM policy in JSON format. - // - // This parameter is optional. If you pass a policy, the temporary security - // credentials that are returned by the operation have the permissions that - // are allowed by both (the intersection of) the access policy of the role that - // is being assumed, and the policy that you pass. This gives you a way to further - // restrict the permissions for the resulting temporary security credentials. - // You cannot use the passed policy to grant permissions that are in excess - // of those allowed by the access policy of the role that is being assumed. - // For more information, see Permissions for AssumeRole, AssumeRoleWithSAML, - // and AssumeRoleWithWebIdentity (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_assumerole.html) + // An IAM policy in JSON format that you want to use as an inline session policy. + // + // This parameter is optional. Passing policies to this operation returns new + // temporary credentials. The resulting session's permissions are the intersection + // of the role's identity-based policy and the session policies. You can use + // the role's temporary credentials in subsequent AWS API calls to access resources + // in the account that owns the role. You cannot use session policies to grant + // more permissions than those allowed by the identity-based policy of the role + // that is being assumed. For more information, see Session Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) // in the IAM User Guide. // - // The format for this parameter, as described by its regex pattern, is a string - // of characters up to 2048 characters in length. The characters can be any - // ASCII character from the space character to the end of the valid character - // list (\u0020-\u00FF). It can also include the tab (\u0009), linefeed (\u000A), + // The plain text that you use for both inline and managed session policies + // can't exceed 2,048 characters. The JSON policy characters can be any ASCII + // character from the space character to the end of the valid character list + // (\u0020 through \u00FF). It can also include the tab (\u0009), linefeed (\u000A), // and carriage return (\u000D) characters. // - // The policy plain text must be 2048 bytes or shorter. However, an internal - // conversion compresses it into a packed binary format with a separate limit. - // The PackedPolicySize response element indicates by percentage how close to - // the upper size limit the policy is, with 100% equaling the maximum allowed - // size. + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. Policy *string `min:"1" type:"string"` + // The Amazon Resource Names (ARNs) of the IAM managed policies that you want + // to use as managed session policies. The policies must exist in the same account + // as the role. + // + // This parameter is optional. You can provide up to 10 managed policy ARNs. + // However, the plain text that you use for both inline and managed session + // policies can't exceed 2,048 characters. For more information about ARNs, + // see Amazon Resource Names (ARNs) and AWS Service Namespaces (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) + // in the AWS General Reference. + // + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. + // + // Passing policies to this operation returns new temporary credentials. The + // resulting session's permissions are the intersection of the role's identity-based + // policy and the session policies. You can use the role's temporary credentials + // in subsequent AWS API calls to access resources in the account that owns + // the role. You cannot use session policies to grant more permissions than + // those allowed by the identity-based policy of the role that is being assumed. + // For more information, see Session Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) + // in the IAM User Guide. + PolicyArns []*PolicyDescriptorType `type:"list"` + // The Amazon Resource Name (ARN) of the role to assume. // // RoleArn is a required field @@ -1158,8 +1434,8 @@ type AssumeRoleInput struct { // scenarios, the role session name is visible to, and can be logged by the // account that owns the role. The role session name is also used in the ARN // of the assumed role principal. This means that subsequent cross-account API - // requests using the temporary security credentials will expose the role session - // name to the external account in their CloudTrail logs. + // requests that use the temporary security credentials will expose the role + // session name to the external account in their AWS CloudTrail logs. // // The regex used to validate this parameter is a string of characters consisting // of upper- and lower-case alphanumeric characters with no spaces. You can @@ -1179,6 +1455,41 @@ type AssumeRoleInput struct { // also include underscores or any of the following characters: =,.@- SerialNumber *string `min:"9" type:"string"` + // A list of session tags that you want to pass. Each session tag consists of + // a key name and an associated value. For more information about session tags, + // see Tagging AWS STS Sessions (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) + // in the IAM User Guide. + // + // This parameter is optional. You can pass up to 50 session tags. The plain + // text session tag keys can’t exceed 128 characters, and the values can’t + // exceed 256 characters. For these and additional limits, see IAM and STS Character + // Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-limits.html#reference_iam-limits-entity-length) + // in the IAM User Guide. + // + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. + // + // You can pass a session tag with the same key as a tag that is already attached + // to the role. When you do, session tags override a role tag with the same + // key. + // + // Tag key–value pairs are not case sensitive, but case is preserved. This + // means that you cannot have separate Department and department tag keys. Assume + // that the role has the Department=Marketing tag and you pass the department=engineering + // session tag. Department and department are not saved as separate tags, and + // the session tag passed in the request takes precedence over the role tag. + // + // Additionally, if you used temporary credentials to perform this operation, + // the new session inherits any transitive session tags from the calling session. + // If you pass a session tag with the same key as an inherited tag, the operation + // fails. To view the inherited tags for a session, see the AWS CloudTrail logs. + // For more information, see Viewing Session Tags in CloudTrail (https://docs.aws.amazon.com/IAM/latest/UserGuide/session-tags.html#id_session-tags_ctlogs) + // in the IAM User Guide. + Tags []*Tag `type:"list"` + // The value provided by the MFA device, if the trust policy of the role being // assumed requires MFA (that is, if the policy includes a condition that tests // for MFA). If the role being assumed requires MFA and if the TokenCode value @@ -1187,6 +1498,19 @@ type AssumeRoleInput struct { // The format for this parameter, as described by its regex pattern, is a sequence // of six numeric digits. TokenCode *string `min:"6" type:"string"` + + // A list of keys for session tags that you want to set as transitive. If you + // set a tag key as transitive, the corresponding key and value passes to subsequent + // sessions in a role chain. For more information, see Chaining Roles with Session + // Tags (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html#id_session-tags_role-chaining) + // in the IAM User Guide. + // + // This parameter is optional. When you set session tags as transitive, the + // session policy and session tags packed binary limit is not affected. + // + // If you choose not to specify a transitive tag key, then no tags are passed + // from this session to any subsequent sessions. + TransitiveTagKeys []*string `type:"list"` } // String returns the string representation @@ -1229,6 +1553,26 @@ func (s *AssumeRoleInput) Validate() error { if s.TokenCode != nil && len(*s.TokenCode) < 6 { invalidParams.Add(request.NewErrParamMinLen("TokenCode", 6)) } + if s.PolicyArns != nil { + for i, v := range s.PolicyArns { + if v == nil { + continue + } + if err := v.Validate(); err != nil { + invalidParams.AddNested(fmt.Sprintf("%s[%v]", "PolicyArns", i), err.(request.ErrInvalidParams)) + } + } + } + if s.Tags != nil { + for i, v := range s.Tags { + if v == nil { + continue + } + if err := v.Validate(); err != nil { + invalidParams.AddNested(fmt.Sprintf("%s[%v]", "Tags", i), err.(request.ErrInvalidParams)) + } + } + } if invalidParams.Len() > 0 { return invalidParams @@ -1254,6 +1598,12 @@ func (s *AssumeRoleInput) SetPolicy(v string) *AssumeRoleInput { return s } +// SetPolicyArns sets the PolicyArns field's value. +func (s *AssumeRoleInput) SetPolicyArns(v []*PolicyDescriptorType) *AssumeRoleInput { + s.PolicyArns = v + return s +} + // SetRoleArn sets the RoleArn field's value. func (s *AssumeRoleInput) SetRoleArn(v string) *AssumeRoleInput { s.RoleArn = &v @@ -1272,12 +1622,24 @@ func (s *AssumeRoleInput) SetSerialNumber(v string) *AssumeRoleInput { return s } +// SetTags sets the Tags field's value. +func (s *AssumeRoleInput) SetTags(v []*Tag) *AssumeRoleInput { + s.Tags = v + return s +} + // SetTokenCode sets the TokenCode field's value. func (s *AssumeRoleInput) SetTokenCode(v string) *AssumeRoleInput { s.TokenCode = &v return s } +// SetTransitiveTagKeys sets the TransitiveTagKeys field's value. +func (s *AssumeRoleInput) SetTransitiveTagKeys(v []*string) *AssumeRoleInput { + s.TransitiveTagKeys = v + return s +} + // Contains the response to a successful AssumeRole request, including temporary // AWS credentials that can be used to make AWS requests. type AssumeRoleOutput struct { @@ -1293,15 +1655,14 @@ type AssumeRoleOutput struct { // The temporary security credentials, which include an access key ID, a secret // access key, and a security (or session) token. // - // Note: The size of the security token that STS APIs return is not fixed. We - // strongly recommend that you make no assumptions about the maximum size. As - // of this writing, the typical size is less than 4096 bytes, but that can vary. - // Also, future updates to AWS might require larger sizes. + // The size of the security token that STS API operations return is not fixed. + // We strongly recommend that you make no assumptions about the maximum size. Credentials *Credentials `type:"structure"` - // A percentage value that indicates the size of the policy in packed form. - // The service rejects any policy with a packed size greater than 100 percent, - // which means the policy exceeded the allowed space. + // A percentage value that indicates the packed size of the session policies + // and session tags combined passed in the request. The request fails if the + // packed size is greater than 100 percent, which means the policies and tags + // exceeded the allowed space. PackedPolicySize *int64 `type:"integer"` } @@ -1346,7 +1707,7 @@ type AssumeRoleWithSAMLInput struct { // specify a session duration of 12 hours, but your administrator set the maximum // session duration to 6 hours, your operation fails. To learn how to view the // maximum value for your role, see View the Maximum Session Duration Setting - // for a Role (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) + // for a Role (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) // in the IAM User Guide. // // By default, the value is set to 3600 seconds. @@ -1356,36 +1717,60 @@ type AssumeRoleWithSAMLInput struct { // to the federation endpoint for a console sign-in token takes a SessionDuration // parameter that specifies the maximum length of the console session. For more // information, see Creating a URL that Enables Federated Users to Access the - // AWS Management Console (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_enable-console-custom-url.html) + // AWS Management Console (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_enable-console-custom-url.html) // in the IAM User Guide. DurationSeconds *int64 `min:"900" type:"integer"` - // An IAM policy in JSON format. - // - // The policy parameter is optional. If you pass a policy, the temporary security - // credentials that are returned by the operation have the permissions that - // are allowed by both the access policy of the role that is being assumed, - // and the policy that you pass. This gives you a way to further restrict the - // permissions for the resulting temporary security credentials. You cannot - // use the passed policy to grant permissions that are in excess of those allowed - // by the access policy of the role that is being assumed. For more information, - // Permissions for AssumeRole, AssumeRoleWithSAML, and AssumeRoleWithWebIdentity - // (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_assumerole.html) + // An IAM policy in JSON format that you want to use as an inline session policy. + // + // This parameter is optional. Passing policies to this operation returns new + // temporary credentials. The resulting session's permissions are the intersection + // of the role's identity-based policy and the session policies. You can use + // the role's temporary credentials in subsequent AWS API calls to access resources + // in the account that owns the role. You cannot use session policies to grant + // more permissions than those allowed by the identity-based policy of the role + // that is being assumed. For more information, see Session Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) // in the IAM User Guide. // - // The format for this parameter, as described by its regex pattern, is a string - // of characters up to 2048 characters in length. The characters can be any - // ASCII character from the space character to the end of the valid character - // list (\u0020-\u00FF). It can also include the tab (\u0009), linefeed (\u000A), + // The plain text that you use for both inline and managed session policies + // can't exceed 2,048 characters. The JSON policy characters can be any ASCII + // character from the space character to the end of the valid character list + // (\u0020 through \u00FF). It can also include the tab (\u0009), linefeed (\u000A), // and carriage return (\u000D) characters. // - // The policy plain text must be 2048 bytes or shorter. However, an internal - // conversion compresses it into a packed binary format with a separate limit. - // The PackedPolicySize response element indicates by percentage how close to - // the upper size limit the policy is, with 100% equaling the maximum allowed - // size. + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. Policy *string `min:"1" type:"string"` + // The Amazon Resource Names (ARNs) of the IAM managed policies that you want + // to use as managed session policies. The policies must exist in the same account + // as the role. + // + // This parameter is optional. You can provide up to 10 managed policy ARNs. + // However, the plain text that you use for both inline and managed session + // policies can't exceed 2,048 characters. For more information about ARNs, + // see Amazon Resource Names (ARNs) and AWS Service Namespaces (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) + // in the AWS General Reference. + // + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. + // + // Passing policies to this operation returns new temporary credentials. The + // resulting session's permissions are the intersection of the role's identity-based + // policy and the session policies. You can use the role's temporary credentials + // in subsequent AWS API calls to access resources in the account that owns + // the role. You cannot use session policies to grant more permissions than + // those allowed by the identity-based policy of the role that is being assumed. + // For more information, see Session Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) + // in the IAM User Guide. + PolicyArns []*PolicyDescriptorType `type:"list"` + // The Amazon Resource Name (ARN) of the SAML provider in IAM that describes // the IdP. // @@ -1399,11 +1784,11 @@ type AssumeRoleWithSAMLInput struct { // The base-64 encoded SAML authentication response provided by the IdP. // - // For more information, see Configuring a Relying Party and Adding Claims (http://docs.aws.amazon.com/IAM/latest/UserGuide/create-role-saml-IdP-tasks.html) - // in the Using IAM guide. + // For more information, see Configuring a Relying Party and Adding Claims (https://docs.aws.amazon.com/IAM/latest/UserGuide/create-role-saml-IdP-tasks.html) + // in the IAM User Guide. // // SAMLAssertion is a required field - SAMLAssertion *string `min:"4" type:"string" required:"true"` + SAMLAssertion *string `min:"4" type:"string" required:"true" sensitive:"true"` } // String returns the string representation @@ -1443,6 +1828,16 @@ func (s *AssumeRoleWithSAMLInput) Validate() error { if s.SAMLAssertion != nil && len(*s.SAMLAssertion) < 4 { invalidParams.Add(request.NewErrParamMinLen("SAMLAssertion", 4)) } + if s.PolicyArns != nil { + for i, v := range s.PolicyArns { + if v == nil { + continue + } + if err := v.Validate(); err != nil { + invalidParams.AddNested(fmt.Sprintf("%s[%v]", "PolicyArns", i), err.(request.ErrInvalidParams)) + } + } + } if invalidParams.Len() > 0 { return invalidParams @@ -1462,6 +1857,12 @@ func (s *AssumeRoleWithSAMLInput) SetPolicy(v string) *AssumeRoleWithSAMLInput { return s } +// SetPolicyArns sets the PolicyArns field's value. +func (s *AssumeRoleWithSAMLInput) SetPolicyArns(v []*PolicyDescriptorType) *AssumeRoleWithSAMLInput { + s.PolicyArns = v + return s +} + // SetPrincipalArn sets the PrincipalArn field's value. func (s *AssumeRoleWithSAMLInput) SetPrincipalArn(v string) *AssumeRoleWithSAMLInput { s.PrincipalArn = &v @@ -1496,10 +1897,8 @@ type AssumeRoleWithSAMLOutput struct { // The temporary security credentials, which include an access key ID, a secret // access key, and a security (or session) token. // - // Note: The size of the security token that STS APIs return is not fixed. We - // strongly recommend that you make no assumptions about the maximum size. As - // of this writing, the typical size is less than 4096 bytes, but that can vary. - // Also, future updates to AWS might require larger sizes. + // The size of the security token that STS API operations return is not fixed. + // We strongly recommend that you make no assumptions about the maximum size. Credentials *Credentials `type:"structure"` // The value of the Issuer element of the SAML assertion. @@ -1516,9 +1915,10 @@ type AssumeRoleWithSAMLOutput struct { // ) ) NameQualifier *string `type:"string"` - // A percentage value that indicates the size of the policy in packed form. - // The service rejects any policy with a packed size greater than 100 percent, - // which means the policy exceeded the allowed space. + // A percentage value that indicates the packed size of the session policies + // and session tags combined passed in the request. The request fails if the + // packed size is greater than 100 percent, which means the policies and tags + // exceeded the allowed space. PackedPolicySize *int64 `type:"integer"` // The value of the NameID element in the Subject element of the SAML assertion. @@ -1603,7 +2003,7 @@ type AssumeRoleWithWebIdentityInput struct { // a session duration of 12 hours, but your administrator set the maximum session // duration to 6 hours, your operation fails. To learn how to view the maximum // value for your role, see View the Maximum Session Duration Setting for a - // Role (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) + // Role (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html#id_roles_use_view-role-max-session) // in the IAM User Guide. // // By default, the value is set to 3600 seconds. @@ -1613,35 +2013,60 @@ type AssumeRoleWithWebIdentityInput struct { // to the federation endpoint for a console sign-in token takes a SessionDuration // parameter that specifies the maximum length of the console session. For more // information, see Creating a URL that Enables Federated Users to Access the - // AWS Management Console (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_enable-console-custom-url.html) + // AWS Management Console (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_enable-console-custom-url.html) // in the IAM User Guide. DurationSeconds *int64 `min:"900" type:"integer"` - // An IAM policy in JSON format. + // An IAM policy in JSON format that you want to use as an inline session policy. // - // The policy parameter is optional. If you pass a policy, the temporary security - // credentials that are returned by the operation have the permissions that - // are allowed by both the access policy of the role that is being assumed, - // and the policy that you pass. This gives you a way to further restrict the - // permissions for the resulting temporary security credentials. You cannot - // use the passed policy to grant permissions that are in excess of those allowed - // by the access policy of the role that is being assumed. For more information, - // see Permissions for AssumeRoleWithWebIdentity (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_assumerole.html) + // This parameter is optional. Passing policies to this operation returns new + // temporary credentials. The resulting session's permissions are the intersection + // of the role's identity-based policy and the session policies. You can use + // the role's temporary credentials in subsequent AWS API calls to access resources + // in the account that owns the role. You cannot use session policies to grant + // more permissions than those allowed by the identity-based policy of the role + // that is being assumed. For more information, see Session Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) // in the IAM User Guide. // - // The format for this parameter, as described by its regex pattern, is a string - // of characters up to 2048 characters in length. The characters can be any - // ASCII character from the space character to the end of the valid character - // list (\u0020-\u00FF). It can also include the tab (\u0009), linefeed (\u000A), + // The plain text that you use for both inline and managed session policies + // can't exceed 2,048 characters. The JSON policy characters can be any ASCII + // character from the space character to the end of the valid character list + // (\u0020 through \u00FF). It can also include the tab (\u0009), linefeed (\u000A), // and carriage return (\u000D) characters. // - // The policy plain text must be 2048 bytes or shorter. However, an internal - // conversion compresses it into a packed binary format with a separate limit. - // The PackedPolicySize response element indicates by percentage how close to - // the upper size limit the policy is, with 100% equaling the maximum allowed - // size. + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. Policy *string `min:"1" type:"string"` + // The Amazon Resource Names (ARNs) of the IAM managed policies that you want + // to use as managed session policies. The policies must exist in the same account + // as the role. + // + // This parameter is optional. You can provide up to 10 managed policy ARNs. + // However, the plain text that you use for both inline and managed session + // policies can't exceed 2,048 characters. For more information about ARNs, + // see Amazon Resource Names (ARNs) and AWS Service Namespaces (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) + // in the AWS General Reference. + // + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. + // + // Passing policies to this operation returns new temporary credentials. The + // resulting session's permissions are the intersection of the role's identity-based + // policy and the session policies. You can use the role's temporary credentials + // in subsequent AWS API calls to access resources in the account that owns + // the role. You cannot use session policies to grant more permissions than + // those allowed by the identity-based policy of the role that is being assumed. + // For more information, see Session Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) + // in the IAM User Guide. + PolicyArns []*PolicyDescriptorType `type:"list"` + // The fully qualified host component of the domain name of the identity provider. // // Specify this value only for OAuth 2.0 access tokens. Currently www.amazon.com @@ -1675,7 +2100,7 @@ type AssumeRoleWithWebIdentityInput struct { // the application makes an AssumeRoleWithWebIdentity call. // // WebIdentityToken is a required field - WebIdentityToken *string `min:"4" type:"string" required:"true"` + WebIdentityToken *string `min:"4" type:"string" required:"true" sensitive:"true"` } // String returns the string representation @@ -1718,6 +2143,16 @@ func (s *AssumeRoleWithWebIdentityInput) Validate() error { if s.WebIdentityToken != nil && len(*s.WebIdentityToken) < 4 { invalidParams.Add(request.NewErrParamMinLen("WebIdentityToken", 4)) } + if s.PolicyArns != nil { + for i, v := range s.PolicyArns { + if v == nil { + continue + } + if err := v.Validate(); err != nil { + invalidParams.AddNested(fmt.Sprintf("%s[%v]", "PolicyArns", i), err.(request.ErrInvalidParams)) + } + } + } if invalidParams.Len() > 0 { return invalidParams @@ -1737,6 +2172,12 @@ func (s *AssumeRoleWithWebIdentityInput) SetPolicy(v string) *AssumeRoleWithWebI return s } +// SetPolicyArns sets the PolicyArns field's value. +func (s *AssumeRoleWithWebIdentityInput) SetPolicyArns(v []*PolicyDescriptorType) *AssumeRoleWithWebIdentityInput { + s.PolicyArns = v + return s +} + // SetProviderId sets the ProviderId field's value. func (s *AssumeRoleWithWebIdentityInput) SetProviderId(v string) *AssumeRoleWithWebIdentityInput { s.ProviderId = &v @@ -1781,19 +2222,18 @@ type AssumeRoleWithWebIdentityOutput struct { // The temporary security credentials, which include an access key ID, a secret // access key, and a security token. // - // Note: The size of the security token that STS APIs return is not fixed. We - // strongly recommend that you make no assumptions about the maximum size. As - // of this writing, the typical size is less than 4096 bytes, but that can vary. - // Also, future updates to AWS might require larger sizes. + // The size of the security token that STS API operations return is not fixed. + // We strongly recommend that you make no assumptions about the maximum size. Credentials *Credentials `type:"structure"` - // A percentage value that indicates the size of the policy in packed form. - // The service rejects any policy with a packed size greater than 100 percent, - // which means the policy exceeded the allowed space. + // A percentage value that indicates the packed size of the session policies + // and session tags combined passed in the request. The request fails if the + // packed size is greater than 100 percent, which means the policies and tags + // exceeded the allowed space. PackedPolicySize *int64 `type:"integer"` // The issuing authority of the web identity token presented. For OpenID Connect - // ID Tokens this contains the value of the iss field. For OAuth 2.0 access + // ID tokens, this contains the value of the iss field. For OAuth 2.0 access // tokens, this contains the value of the ProviderId parameter that was passed // in the AssumeRoleWithWebIdentity request. Provider *string `type:"string"` @@ -1860,8 +2300,8 @@ type AssumedRoleUser struct { // The ARN of the temporary security credentials that are returned from the // AssumeRole action. For more information about ARNs and how to use them in - // policies, see IAM Identifiers (http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html) - // in Using IAM. + // policies, see IAM Identifiers (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html) + // in the IAM User Guide. // // Arn is a required field Arn *string `min:"20" type:"string" required:"true"` @@ -2028,8 +2468,8 @@ type FederatedUser struct { // The ARN that specifies the federated user that is associated with the credentials. // For more information about ARNs and how to use them in policies, see IAM - // Identifiers (http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html) - // in Using IAM. + // Identifiers (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_identifiers.html) + // in the IAM User Guide. // // Arn is a required field Arn *string `min:"20" type:"string" required:"true"` @@ -2063,6 +2503,73 @@ func (s *FederatedUser) SetFederatedUserId(v string) *FederatedUser { return s } +type GetAccessKeyInfoInput struct { + _ struct{} `type:"structure"` + + // The identifier of an access key. + // + // This parameter allows (through its regex pattern) a string of characters + // that can consist of any upper- or lowercase letter or digit. + // + // AccessKeyId is a required field + AccessKeyId *string `min:"16" type:"string" required:"true"` +} + +// String returns the string representation +func (s GetAccessKeyInfoInput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GetAccessKeyInfoInput) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *GetAccessKeyInfoInput) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "GetAccessKeyInfoInput"} + if s.AccessKeyId == nil { + invalidParams.Add(request.NewErrParamRequired("AccessKeyId")) + } + if s.AccessKeyId != nil && len(*s.AccessKeyId) < 16 { + invalidParams.Add(request.NewErrParamMinLen("AccessKeyId", 16)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetAccessKeyId sets the AccessKeyId field's value. +func (s *GetAccessKeyInfoInput) SetAccessKeyId(v string) *GetAccessKeyInfoInput { + s.AccessKeyId = &v + return s +} + +type GetAccessKeyInfoOutput struct { + _ struct{} `type:"structure"` + + // The number used to identify the AWS account. + Account *string `type:"string"` +} + +// String returns the string representation +func (s GetAccessKeyInfoOutput) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s GetAccessKeyInfoOutput) GoString() string { + return s.String() +} + +// SetAccount sets the Account field's value. +func (s *GetAccessKeyInfoOutput) SetAccount(v string) *GetAccessKeyInfoOutput { + s.Account = &v + return s +} + type GetCallerIdentityInput struct { _ struct{} `type:"structure"` } @@ -2090,8 +2597,8 @@ type GetCallerIdentityOutput struct { Arn *string `min:"20" type:"string"` // The unique identifier of the calling entity. The exact value depends on the - // type of entity making the call. The values returned are those listed in the - // aws:userid column in the Principal table (http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html#principaltable) + // type of entity that is making the call. The values returned are those listed + // in the aws:userid column in the Principal table (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_variables.html#principaltable) // found on the Policy Variables reference page in the IAM User Guide. UserId *string `type:"string"` } @@ -2128,12 +2635,11 @@ type GetFederationTokenInput struct { _ struct{} `type:"structure"` // The duration, in seconds, that the session should last. Acceptable durations - // for federation sessions range from 900 seconds (15 minutes) to 129600 seconds - // (36 hours), with 43200 seconds (12 hours) as the default. Sessions obtained - // using AWS account (root) credentials are restricted to a maximum of 3600 + // for federation sessions range from 900 seconds (15 minutes) to 129,600 seconds + // (36 hours), with 43,200 seconds (12 hours) as the default. Sessions obtained + // using AWS account root user credentials are restricted to a maximum of 3,600 // seconds (one hour). If the specified duration is longer than one hour, the - // session obtained by using AWS account (root) credentials defaults to one - // hour. + // session obtained by using root user credentials defaults to one hour. DurationSeconds *int64 `min:"900" type:"integer"` // The name of the federated user. The name is used as an identifier for the @@ -2148,36 +2654,107 @@ type GetFederationTokenInput struct { // Name is a required field Name *string `min:"2" type:"string" required:"true"` - // An IAM policy in JSON format that is passed with the GetFederationToken call - // and evaluated along with the policy or policies that are attached to the - // IAM user whose credentials are used to call GetFederationToken. The passed - // policy is used to scope down the permissions that are available to the IAM - // user, by allowing only a subset of the permissions that are granted to the - // IAM user. The passed policy cannot grant more permissions than those granted - // to the IAM user. The final permissions for the federated user are the most - // restrictive set based on the intersection of the passed policy and the IAM - // user policy. - // - // If you do not pass a policy, the resulting temporary security credentials - // have no effective permissions. The only exception is when the temporary security - // credentials are used to access a resource that has a resource-based policy - // that specifically allows the federated user to access the resource. - // - // The format for this parameter, as described by its regex pattern, is a string - // of characters up to 2048 characters in length. The characters can be any - // ASCII character from the space character to the end of the valid character - // list (\u0020-\u00FF). It can also include the tab (\u0009), linefeed (\u000A), - // and carriage return (\u000D) characters. + // An IAM policy in JSON format that you want to use as an inline session policy. + // + // You must pass an inline or managed session policy (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) + // to this operation. You can pass a single JSON policy document to use as an + // inline session policy. You can also specify up to 10 managed policies to + // use as managed session policies. + // + // This parameter is optional. However, if you do not pass any session policies, + // then the resulting federated user session has no permissions. + // + // When you pass session policies, the session permissions are the intersection + // of the IAM user policies and the session policies that you pass. This gives + // you a way to further restrict the permissions for a federated user. You cannot + // use session policies to grant more permissions than those that are defined + // in the permissions policy of the IAM user. For more information, see Session + // Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) + // in the IAM User Guide. + // + // The resulting credentials can be used to access a resource that has a resource-based + // policy. If that policy specifically references the federated user session + // in the Principal element of the policy, the session has the permissions allowed + // by the policy. These permissions are granted in addition to the permissions + // that are granted by the session policies. // - // The policy plain text must be 2048 bytes or shorter. However, an internal - // conversion compresses it into a packed binary format with a separate limit. - // The PackedPolicySize response element indicates by percentage how close to - // the upper size limit the policy is, with 100% equaling the maximum allowed - // size. + // The plain text that you use for both inline and managed session policies + // can't exceed 2,048 characters. The JSON policy characters can be any ASCII + // character from the space character to the end of the valid character list + // (\u0020 through \u00FF). It can also include the tab (\u0009), linefeed (\u000A), + // and carriage return (\u000D) characters. // - // For more information about how permissions work, see Permissions for GetFederationToken - // (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_control-access_getfederationtoken.html). + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. Policy *string `min:"1" type:"string"` + + // The Amazon Resource Names (ARNs) of the IAM managed policies that you want + // to use as a managed session policy. The policies must exist in the same account + // as the IAM user that is requesting federated access. + // + // You must pass an inline or managed session policy (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) + // to this operation. You can pass a single JSON policy document to use as an + // inline session policy. You can also specify up to 10 managed policies to + // use as managed session policies. The plain text that you use for both inline + // and managed session policies can't exceed 2,048 characters. You can provide + // up to 10 managed policy ARNs. For more information about ARNs, see Amazon + // Resource Names (ARNs) and AWS Service Namespaces (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) + // in the AWS General Reference. + // + // This parameter is optional. However, if you do not pass any session policies, + // then the resulting federated user session has no permissions. + // + // When you pass session policies, the session permissions are the intersection + // of the IAM user policies and the session policies that you pass. This gives + // you a way to further restrict the permissions for a federated user. You cannot + // use session policies to grant more permissions than those that are defined + // in the permissions policy of the IAM user. For more information, see Session + // Policies (https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies.html#policies_session) + // in the IAM User Guide. + // + // The resulting credentials can be used to access a resource that has a resource-based + // policy. If that policy specifically references the federated user session + // in the Principal element of the policy, the session has the permissions allowed + // by the policy. These permissions are granted in addition to the permissions + // that are granted by the session policies. + // + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. + PolicyArns []*PolicyDescriptorType `type:"list"` + + // A list of session tags. Each session tag consists of a key name and an associated + // value. For more information about session tags, see Passing Session Tags + // in STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) + // in the IAM User Guide. + // + // This parameter is optional. You can pass up to 50 session tags. The plain + // text session tag keys can’t exceed 128 characters and the values can’t + // exceed 256 characters. For these and additional limits, see IAM and STS Character + // Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-limits.html#reference_iam-limits-entity-length) + // in the IAM User Guide. + // + // An AWS conversion compresses the passed session policies and session tags + // into a packed binary format that has a separate limit. Your request can fail + // for this limit even if your plain text meets the other requirements. The + // PackedPolicySize response element indicates by percentage how close the policies + // and tags for your request are to the upper size limit. + // + // You can pass a session tag with the same key as a tag that is already attached + // to the user you are federating. When you do, session tags override a user + // tag with the same key. + // + // Tag key–value pairs are not case sensitive, but case is preserved. This + // means that you cannot have separate Department and department tag keys. Assume + // that the role has the Department=Marketing tag and you pass the department=engineering + // session tag. Department and department are not saved as separate tags, and + // the session tag passed in the request takes precedence over the role tag. + Tags []*Tag `type:"list"` } // String returns the string representation @@ -2205,6 +2782,26 @@ func (s *GetFederationTokenInput) Validate() error { if s.Policy != nil && len(*s.Policy) < 1 { invalidParams.Add(request.NewErrParamMinLen("Policy", 1)) } + if s.PolicyArns != nil { + for i, v := range s.PolicyArns { + if v == nil { + continue + } + if err := v.Validate(); err != nil { + invalidParams.AddNested(fmt.Sprintf("%s[%v]", "PolicyArns", i), err.(request.ErrInvalidParams)) + } + } + } + if s.Tags != nil { + for i, v := range s.Tags { + if v == nil { + continue + } + if err := v.Validate(); err != nil { + invalidParams.AddNested(fmt.Sprintf("%s[%v]", "Tags", i), err.(request.ErrInvalidParams)) + } + } + } if invalidParams.Len() > 0 { return invalidParams @@ -2230,6 +2827,18 @@ func (s *GetFederationTokenInput) SetPolicy(v string) *GetFederationTokenInput { return s } +// SetPolicyArns sets the PolicyArns field's value. +func (s *GetFederationTokenInput) SetPolicyArns(v []*PolicyDescriptorType) *GetFederationTokenInput { + s.PolicyArns = v + return s +} + +// SetTags sets the Tags field's value. +func (s *GetFederationTokenInput) SetTags(v []*Tag) *GetFederationTokenInput { + s.Tags = v + return s +} + // Contains the response to a successful GetFederationToken request, including // temporary AWS credentials that can be used to make AWS requests. type GetFederationTokenOutput struct { @@ -2238,10 +2847,8 @@ type GetFederationTokenOutput struct { // The temporary security credentials, which include an access key ID, a secret // access key, and a security (or session) token. // - // Note: The size of the security token that STS APIs return is not fixed. We - // strongly recommend that you make no assumptions about the maximum size. As - // of this writing, the typical size is less than 4096 bytes, but that can vary. - // Also, future updates to AWS might require larger sizes. + // The size of the security token that STS API operations return is not fixed. + // We strongly recommend that you make no assumptions about the maximum size. Credentials *Credentials `type:"structure"` // Identifiers for the federated user associated with the credentials (such @@ -2250,9 +2857,10 @@ type GetFederationTokenOutput struct { // an Amazon S3 bucket policy. FederatedUser *FederatedUser `type:"structure"` - // A percentage value indicating the size of the policy in packed form. The - // service rejects policies for which the packed size is greater than 100 percent - // of the allowed value. + // A percentage value that indicates the packed size of the session policies + // and session tags combined passed in the request. The request fails if the + // packed size is greater than 100 percent, which means the policies and tags + // exceeded the allowed space. PackedPolicySize *int64 `type:"integer"` } @@ -2288,11 +2896,11 @@ type GetSessionTokenInput struct { _ struct{} `type:"structure"` // The duration, in seconds, that the credentials should remain valid. Acceptable - // durations for IAM user sessions range from 900 seconds (15 minutes) to 129600 - // seconds (36 hours), with 43200 seconds (12 hours) as the default. Sessions - // for AWS account owners are restricted to a maximum of 3600 seconds (one hour). - // If the duration is longer than one hour, the session for AWS account owners - // defaults to one hour. + // durations for IAM user sessions range from 900 seconds (15 minutes) to 129,600 + // seconds (36 hours), with 43,200 seconds (12 hours) as the default. Sessions + // for AWS account owners are restricted to a maximum of 3,600 seconds (one + // hour). If the duration is longer than one hour, the session for AWS account + // owners defaults to one hour. DurationSeconds *int64 `min:"900" type:"integer"` // The identification number of the MFA device that is associated with the IAM @@ -2303,16 +2911,16 @@ type GetSessionTokenInput struct { // You can find the device for an IAM user by going to the AWS Management Console // and viewing the user's security credentials. // - // The regex used to validated this parameter is a string of characters consisting + // The regex used to validate this parameter is a string of characters consisting // of upper- and lower-case alphanumeric characters with no spaces. You can // also include underscores or any of the following characters: =,.@:/- SerialNumber *string `min:"9" type:"string"` // The value provided by the MFA device, if MFA is required. If any policy requires // the IAM user to submit an MFA code, specify this value. If MFA authentication - // is required, and the user does not provide a code when requesting a set of - // temporary security credentials, the user will receive an "access denied" - // response when requesting resources that require MFA authentication. + // is required, the user must provide a code when requesting a set of temporary + // security credentials. A user who fails to provide the code receives an "access + // denied" response when requesting resources that require MFA authentication. // // The format for this parameter, as described by its regex pattern, is a sequence // of six numeric digits. @@ -2374,10 +2982,8 @@ type GetSessionTokenOutput struct { // The temporary security credentials, which include an access key ID, a secret // access key, and a security (or session) token. // - // Note: The size of the security token that STS APIs return is not fixed. We - // strongly recommend that you make no assumptions about the maximum size. As - // of this writing, the typical size is less than 4096 bytes, but that can vary. - // Also, future updates to AWS might require larger sizes. + // The size of the security token that STS API operations return is not fixed. + // We strongly recommend that you make no assumptions about the maximum size. Credentials *Credentials `type:"structure"` } @@ -2396,3 +3002,114 @@ func (s *GetSessionTokenOutput) SetCredentials(v *Credentials) *GetSessionTokenO s.Credentials = v return s } + +// A reference to the IAM managed policy that is passed as a session policy +// for a role session or a federated user session. +type PolicyDescriptorType struct { + _ struct{} `type:"structure"` + + // The Amazon Resource Name (ARN) of the IAM managed policy to use as a session + // policy for the role. For more information about ARNs, see Amazon Resource + // Names (ARNs) and AWS Service Namespaces (https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) + // in the AWS General Reference. + Arn *string `locationName:"arn" min:"20" type:"string"` +} + +// String returns the string representation +func (s PolicyDescriptorType) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s PolicyDescriptorType) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *PolicyDescriptorType) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "PolicyDescriptorType"} + if s.Arn != nil && len(*s.Arn) < 20 { + invalidParams.Add(request.NewErrParamMinLen("Arn", 20)) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetArn sets the Arn field's value. +func (s *PolicyDescriptorType) SetArn(v string) *PolicyDescriptorType { + s.Arn = &v + return s +} + +// You can pass custom key-value pair attributes when you assume a role or federate +// a user. These are called session tags. You can then use the session tags +// to control access to resources. For more information, see Tagging AWS STS +// Sessions (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) +// in the IAM User Guide. +type Tag struct { + _ struct{} `type:"structure"` + + // The key for a session tag. + // + // You can pass up to 50 session tags. The plain text session tag keys can’t + // exceed 128 characters. For these and additional limits, see IAM and STS Character + // Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-limits.html#reference_iam-limits-entity-length) + // in the IAM User Guide. + // + // Key is a required field + Key *string `min:"1" type:"string" required:"true"` + + // The value for a session tag. + // + // You can pass up to 50 session tags. The plain text session tag values can’t + // exceed 256 characters. For these and additional limits, see IAM and STS Character + // Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_iam-limits.html#reference_iam-limits-entity-length) + // in the IAM User Guide. + // + // Value is a required field + Value *string `type:"string" required:"true"` +} + +// String returns the string representation +func (s Tag) String() string { + return awsutil.Prettify(s) +} + +// GoString returns the string representation +func (s Tag) GoString() string { + return s.String() +} + +// Validate inspects the fields of the type to determine if they are valid. +func (s *Tag) Validate() error { + invalidParams := request.ErrInvalidParams{Context: "Tag"} + if s.Key == nil { + invalidParams.Add(request.NewErrParamRequired("Key")) + } + if s.Key != nil && len(*s.Key) < 1 { + invalidParams.Add(request.NewErrParamMinLen("Key", 1)) + } + if s.Value == nil { + invalidParams.Add(request.NewErrParamRequired("Value")) + } + + if invalidParams.Len() > 0 { + return invalidParams + } + return nil +} + +// SetKey sets the Key field's value. +func (s *Tag) SetKey(v string) *Tag { + s.Key = &v + return s +} + +// SetValue sets the Value field's value. +func (s *Tag) SetValue(v string) *Tag { + s.Value = &v + return s +} diff --git a/github.com/aws/aws-sdk-go/service/sts/customizations.go b/github.com/aws/aws-sdk-go/service/sts/customizations.go index 4010cc7fa1..d5307fcaa0 100644 --- a/github.com/aws/aws-sdk-go/service/sts/customizations.go +++ b/github.com/aws/aws-sdk-go/service/sts/customizations.go @@ -3,10 +3,9 @@ package sts import "github.com/aws/aws-sdk-go/aws/request" func init() { - initRequest = func(r *request.Request) { - switch r.Operation.Name { - case opAssumeRoleWithSAML, opAssumeRoleWithWebIdentity: - r.Handlers.Sign.Clear() // these operations are unsigned - } - } + initRequest = customizeRequest +} + +func customizeRequest(r *request.Request) { + r.RetryErrorCodes = append(r.RetryErrorCodes, ErrCodeIDPCommunicationErrorException) } diff --git a/github.com/aws/aws-sdk-go/service/sts/doc.go b/github.com/aws/aws-sdk-go/service/sts/doc.go index ef681ab0c6..fcb720dcac 100644 --- a/github.com/aws/aws-sdk-go/service/sts/doc.go +++ b/github.com/aws/aws-sdk-go/service/sts/doc.go @@ -7,22 +7,14 @@ // request temporary, limited-privilege credentials for AWS Identity and Access // Management (IAM) users or for users that you authenticate (federated users). // This guide provides descriptions of the STS API. For more detailed information -// about using this service, go to Temporary Security Credentials (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html). -// -// As an alternative to using the API, you can use one of the AWS SDKs, which -// consist of libraries and sample code for various programming languages and -// platforms (Java, Ruby, .NET, iOS, Android, etc.). The SDKs provide a convenient -// way to create programmatic access to STS. For example, the SDKs take care -// of cryptographically signing requests, managing errors, and retrying requests -// automatically. For information about the AWS SDKs, including how to download -// and install them, see the Tools for Amazon Web Services page (http://aws.amazon.com/tools/). +// about using this service, go to Temporary Security Credentials (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html). // // For information about setting up signatures and authorization through the -// API, go to Signing AWS API Requests (http://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html) +// API, go to Signing AWS API Requests (https://docs.aws.amazon.com/general/latest/gr/signing_aws_api_requests.html) // in the AWS General Reference. For general information about the Query API, -// go to Making Query Requests (http://docs.aws.amazon.com/IAM/latest/UserGuide/IAM_UsingQueryAPI.html) +// go to Making Query Requests (https://docs.aws.amazon.com/IAM/latest/UserGuide/IAM_UsingQueryAPI.html) // in Using IAM. For information about using security tokens with other AWS -// products, go to AWS Services That Work with IAM (http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html) +// products, go to AWS Services That Work with IAM (https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_aws-services-that-work-with-iam.html) // in the IAM User Guide. // // If you're new to AWS and need additional technical information about a specific @@ -31,14 +23,38 @@ // // Endpoints // -// The AWS Security Token Service (STS) has a default endpoint of https://sts.amazonaws.com -// that maps to the US East (N. Virginia) region. Additional regions are available -// and are activated by default. For more information, see Activating and Deactivating -// AWS STS in an AWS Region (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// By default, AWS Security Token Service (STS) is available as a global service, +// and all AWS STS requests go to a single endpoint at https://sts.amazonaws.com. +// Global requests map to the US East (N. Virginia) region. AWS recommends using +// Regional AWS STS endpoints instead of the global endpoint to reduce latency, +// build in redundancy, and increase session token validity. For more information, +// see Managing AWS STS in an AWS Region (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) +// in the IAM User Guide. +// +// Most AWS Regions are enabled for operations in all AWS services by default. +// Those Regions are automatically activated for use with AWS STS. Some Regions, +// such as Asia Pacific (Hong Kong), must be manually enabled. To learn more +// about enabling and disabling AWS Regions, see Managing AWS Regions (https://docs.aws.amazon.com/general/latest/gr/rande-manage.html) +// in the AWS General Reference. When you enable these AWS Regions, they are +// automatically activated for use with AWS STS. You cannot activate the STS +// endpoint for a Region that is disabled. Tokens that are valid in all AWS +// Regions are longer than tokens that are valid in Regions that are enabled +// by default. Changing this setting might affect existing systems where you +// temporarily store tokens. For more information, see Managing Global Endpoint +// Session Tokens (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html#sts-regions-manage-tokens) // in the IAM User Guide. // -// For information about STS endpoints, see Regions and Endpoints (http://docs.aws.amazon.com/general/latest/gr/rande.html#sts_region) -// in the AWS General Reference. +// After you activate a Region for use with AWS STS, you can direct AWS STS +// API calls to that Region. AWS STS recommends that you provide both the Region +// and endpoint when you make calls to a Regional endpoint. You can provide +// the Region alone for manually enabled Regions, such as Asia Pacific (Hong +// Kong). In this case, the calls are directed to the STS Regional endpoint. +// However, if you provide the Region alone for Regions enabled by default, +// the calls are directed to the global endpoint of https://sts.amazonaws.com. +// +// To view the list of AWS STS endpoints and whether they are active by default, +// see Writing Code to Use AWS STS Regions (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html#id_credentials_temp_enable-regions_writing_code) +// in the IAM User Guide. // // Recording API requests // @@ -46,8 +62,28 @@ // your AWS account and delivers log files to an Amazon S3 bucket. By using // information collected by CloudTrail, you can determine what requests were // successfully made to STS, who made the request, when it was made, and so -// on. To learn more about CloudTrail, including how to turn it on and find -// your log files, see the AWS CloudTrail User Guide (http://docs.aws.amazon.com/awscloudtrail/latest/userguide/what_is_cloud_trail_top_level.html). +// on. +// +// If you activate AWS STS endpoints in Regions other than the default global +// endpoint, then you must also turn on CloudTrail logging in those Regions. +// This is necessary to record any AWS STS API calls that are made in those +// Regions. For more information, see Turning On CloudTrail in Additional Regions +// (https://docs.aws.amazon.com/awscloudtrail/latest/userguide/aggregating_logs_regions_turn_on_ct.html) +// in the AWS CloudTrail User Guide. +// +// AWS Security Token Service (STS) is a global service with a single endpoint +// at https://sts.amazonaws.com. Calls to this endpoint are logged as calls +// to a global service. However, because this endpoint is physically located +// in the US East (N. Virginia) Region, your logs list us-east-1 as the event +// Region. CloudTrail does not write these logs to the US East (Ohio) Region +// unless you choose to include global service logs in that Region. CloudTrail +// writes calls to all Regional endpoints to their respective Regions. For example, +// calls to sts.us-east-2.amazonaws.com are published to the US East (Ohio) +// Region and calls to sts.eu-central-1.amazonaws.com are published to the EU +// (Frankfurt) Region. +// +// To learn more about CloudTrail, including how to turn it on and find your +// log files, see the AWS CloudTrail User Guide (https://docs.aws.amazon.com/awscloudtrail/latest/userguide/what_is_cloud_trail_top_level.html). // // See https://docs.aws.amazon.com/goto/WebAPI/sts-2011-06-15 for more information on this service. // diff --git a/github.com/aws/aws-sdk-go/service/sts/errors.go b/github.com/aws/aws-sdk-go/service/sts/errors.go index e24884ef37..a233f542ef 100644 --- a/github.com/aws/aws-sdk-go/service/sts/errors.go +++ b/github.com/aws/aws-sdk-go/service/sts/errors.go @@ -14,11 +14,11 @@ const ( // ErrCodeIDPCommunicationErrorException for service response error code // "IDPCommunicationError". // - // The request could not be fulfilled because the non-AWS identity provider - // (IDP) that was asked to verify the incoming identity token could not be reached. - // This is often a transient error caused by network conditions. Retry the request + // The request could not be fulfilled because the identity provider (IDP) that + // was asked to verify the incoming identity token could not be reached. This + // is often a transient error caused by network conditions. Retry the request // a limited number of times so that you don't exceed the request rate. If the - // error persists, the non-AWS identity provider might be down or not responding. + // error persists, the identity provider might be down or not responding. ErrCodeIDPCommunicationErrorException = "IDPCommunicationError" // ErrCodeIDPRejectedClaimException for service response error code @@ -56,9 +56,18 @@ const ( // ErrCodePackedPolicyTooLargeException for service response error code // "PackedPolicyTooLarge". // - // The request was rejected because the policy document was too large. The error - // message describes how big the policy document is, in packed form, as a percentage - // of what the API allows. + // The request was rejected because the total packed size of the session policies + // and session tags combined was too large. An AWS conversion compresses the + // session policy document, session policy ARNs, and session tags into a packed + // binary format that has a separate limit. The error message indicates by percentage + // how close the policies and tags are to the upper size limit. For more information, + // see Passing Session Tags in STS (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_session-tags.html) + // in the IAM User Guide. + // + // You could receive this error even though you meet other defined session policy + // and session tag limits. For more information, see IAM and STS Entity Character + // Limits (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) + // in the IAM User Guide. ErrCodePackedPolicyTooLargeException = "PackedPolicyTooLarge" // ErrCodeRegionDisabledException for service response error code @@ -67,7 +76,7 @@ const ( // STS is not activated in the requested region for the account that is being // asked to generate credentials. The account administrator must use the IAM // console to activate STS in that region. For more information, see Activating - // and Deactivating AWS STS in an AWS Region (http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) + // and Deactivating AWS STS in an AWS Region (https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_enable-regions.html) // in the IAM User Guide. ErrCodeRegionDisabledException = "RegionDisabledException" ) diff --git a/github.com/aws/aws-sdk-go/service/sts/service.go b/github.com/aws/aws-sdk-go/service/sts/service.go index 185c914d1b..d34a685533 100644 --- a/github.com/aws/aws-sdk-go/service/sts/service.go +++ b/github.com/aws/aws-sdk-go/service/sts/service.go @@ -31,7 +31,7 @@ var initRequest func(*request.Request) const ( ServiceName = "sts" // Name of service. EndpointsID = ServiceName // ID to lookup a service endpoint with. - ServiceID = "STS" // ServiceID is a unique identifer of a specific service. + ServiceID = "STS" // ServiceID is a unique identifier of a specific service. ) // New creates a new instance of the STS client with a session. @@ -39,6 +39,8 @@ const ( // aws.Config parameter to add your extra config. // // Example: +// mySession := session.Must(session.NewSession()) +// // // Create a STS client from just a session. // svc := sts.New(mySession) // @@ -46,11 +48,11 @@ const ( // svc := sts.New(mySession, aws.NewConfig().WithRegion("us-west-2")) func New(p client.ConfigProvider, cfgs ...*aws.Config) *STS { c := p.ClientConfig(EndpointsID, cfgs...) - return newClient(*c.Config, c.Handlers, c.Endpoint, c.SigningRegion, c.SigningName) + return newClient(*c.Config, c.Handlers, c.PartitionID, c.Endpoint, c.SigningRegion, c.SigningName) } // newClient creates, initializes and returns a new service client instance. -func newClient(cfg aws.Config, handlers request.Handlers, endpoint, signingRegion, signingName string) *STS { +func newClient(cfg aws.Config, handlers request.Handlers, partitionID, endpoint, signingRegion, signingName string) *STS { svc := &STS{ Client: client.New( cfg, @@ -59,6 +61,7 @@ func newClient(cfg aws.Config, handlers request.Handlers, endpoint, signingRegio ServiceID: ServiceID, SigningName: signingName, SigningRegion: signingRegion, + PartitionID: partitionID, Endpoint: endpoint, APIVersion: "2011-06-15", }, diff --git a/github.com/aws/aws-sdk-go/service/sts/stsiface/interface.go b/github.com/aws/aws-sdk-go/service/sts/stsiface/interface.go new file mode 100644 index 0000000000..e2e1d6efe5 --- /dev/null +++ b/github.com/aws/aws-sdk-go/service/sts/stsiface/interface.go @@ -0,0 +1,96 @@ +// Code generated by private/model/cli/gen-api/main.go. DO NOT EDIT. + +// Package stsiface provides an interface to enable mocking the AWS Security Token Service service client +// for testing your code. +// +// It is important to note that this interface will have breaking changes +// when the service model is updated and adds new API operations, paginators, +// and waiters. +package stsiface + +import ( + "github.com/aws/aws-sdk-go/aws" + "github.com/aws/aws-sdk-go/aws/request" + "github.com/aws/aws-sdk-go/service/sts" +) + +// STSAPI provides an interface to enable mocking the +// sts.STS service client's API operation, +// paginators, and waiters. This make unit testing your code that calls out +// to the SDK's service client's calls easier. +// +// The best way to use this interface is so the SDK's service client's calls +// can be stubbed out for unit testing your code with the SDK without needing +// to inject custom request handlers into the SDK's request pipeline. +// +// // myFunc uses an SDK service client to make a request to +// // AWS Security Token Service. +// func myFunc(svc stsiface.STSAPI) bool { +// // Make svc.AssumeRole request +// } +// +// func main() { +// sess := session.New() +// svc := sts.New(sess) +// +// myFunc(svc) +// } +// +// In your _test.go file: +// +// // Define a mock struct to be used in your unit tests of myFunc. +// type mockSTSClient struct { +// stsiface.STSAPI +// } +// func (m *mockSTSClient) AssumeRole(input *sts.AssumeRoleInput) (*sts.AssumeRoleOutput, error) { +// // mock response/functionality +// } +// +// func TestMyFunc(t *testing.T) { +// // Setup Test +// mockSvc := &mockSTSClient{} +// +// myfunc(mockSvc) +// +// // Verify myFunc's functionality +// } +// +// It is important to note that this interface will have breaking changes +// when the service model is updated and adds new API operations, paginators, +// and waiters. Its suggested to use the pattern above for testing, or using +// tooling to generate mocks to satisfy the interfaces. +type STSAPI interface { + AssumeRole(*sts.AssumeRoleInput) (*sts.AssumeRoleOutput, error) + AssumeRoleWithContext(aws.Context, *sts.AssumeRoleInput, ...request.Option) (*sts.AssumeRoleOutput, error) + AssumeRoleRequest(*sts.AssumeRoleInput) (*request.Request, *sts.AssumeRoleOutput) + + AssumeRoleWithSAML(*sts.AssumeRoleWithSAMLInput) (*sts.AssumeRoleWithSAMLOutput, error) + AssumeRoleWithSAMLWithContext(aws.Context, *sts.AssumeRoleWithSAMLInput, ...request.Option) (*sts.AssumeRoleWithSAMLOutput, error) + AssumeRoleWithSAMLRequest(*sts.AssumeRoleWithSAMLInput) (*request.Request, *sts.AssumeRoleWithSAMLOutput) + + AssumeRoleWithWebIdentity(*sts.AssumeRoleWithWebIdentityInput) (*sts.AssumeRoleWithWebIdentityOutput, error) + AssumeRoleWithWebIdentityWithContext(aws.Context, *sts.AssumeRoleWithWebIdentityInput, ...request.Option) (*sts.AssumeRoleWithWebIdentityOutput, error) + AssumeRoleWithWebIdentityRequest(*sts.AssumeRoleWithWebIdentityInput) (*request.Request, *sts.AssumeRoleWithWebIdentityOutput) + + DecodeAuthorizationMessage(*sts.DecodeAuthorizationMessageInput) (*sts.DecodeAuthorizationMessageOutput, error) + DecodeAuthorizationMessageWithContext(aws.Context, *sts.DecodeAuthorizationMessageInput, ...request.Option) (*sts.DecodeAuthorizationMessageOutput, error) + DecodeAuthorizationMessageRequest(*sts.DecodeAuthorizationMessageInput) (*request.Request, *sts.DecodeAuthorizationMessageOutput) + + GetAccessKeyInfo(*sts.GetAccessKeyInfoInput) (*sts.GetAccessKeyInfoOutput, error) + GetAccessKeyInfoWithContext(aws.Context, *sts.GetAccessKeyInfoInput, ...request.Option) (*sts.GetAccessKeyInfoOutput, error) + GetAccessKeyInfoRequest(*sts.GetAccessKeyInfoInput) (*request.Request, *sts.GetAccessKeyInfoOutput) + + GetCallerIdentity(*sts.GetCallerIdentityInput) (*sts.GetCallerIdentityOutput, error) + GetCallerIdentityWithContext(aws.Context, *sts.GetCallerIdentityInput, ...request.Option) (*sts.GetCallerIdentityOutput, error) + GetCallerIdentityRequest(*sts.GetCallerIdentityInput) (*request.Request, *sts.GetCallerIdentityOutput) + + GetFederationToken(*sts.GetFederationTokenInput) (*sts.GetFederationTokenOutput, error) + GetFederationTokenWithContext(aws.Context, *sts.GetFederationTokenInput, ...request.Option) (*sts.GetFederationTokenOutput, error) + GetFederationTokenRequest(*sts.GetFederationTokenInput) (*request.Request, *sts.GetFederationTokenOutput) + + GetSessionToken(*sts.GetSessionTokenInput) (*sts.GetSessionTokenOutput, error) + GetSessionTokenWithContext(aws.Context, *sts.GetSessionTokenInput, ...request.Option) (*sts.GetSessionTokenOutput, error) + GetSessionTokenRequest(*sts.GetSessionTokenInput) (*request.Request, *sts.GetSessionTokenOutput) +} + +var _ STSAPI = (*sts.STS)(nil) diff --git a/github.com/cockroachdb/datadriven/datadriven.go b/github.com/cockroachdb/datadriven/datadriven.go index ab1bb63dd8..9849b845a8 100644 --- a/github.com/cockroachdb/datadriven/datadriven.go +++ b/github.com/cockroachdb/datadriven/datadriven.go @@ -25,6 +25,9 @@ import ( "strconv" "strings" "testing" + + "github.com/cockroachdb/errors" + "github.com/pmezard/go-difflib/difflib" ) var ( @@ -34,8 +37,33 @@ var ( "run. Used to update tests when a change affects many cases; please verify the testfile "+ "diffs carefully!", ) + + quietLog = flag.Bool( + "datadriven-quiet", false, + "avoid echoing the directives and responses from test files.", + ) ) +// Verbose returns true iff -datadriven-quiet was not passed. +func Verbose() bool { + return testing.Verbose() && !*quietLog +} + +// In CockroachDB we want to quiesce all the logs across all packages. +// If we had only a flag to work with, we'd get command line parsing +// errors on all packages that do not use datadriven. So +// we make do by also making a command line parameter available. +func init() { + const quietEnvVar = "DATADRIVEN_QUIET_LOG" + if str, ok := os.LookupEnv(quietEnvVar); ok { + v, err := strconv.ParseBool(str) + if err != nil { + panic(fmt.Sprintf("error parsing %s: %s", quietEnvVar, err)) + } + *quietLog = v + } +} + // RunTest invokes a data-driven test. The test cases are contained in a // separate test file and are dynamically loaded, parsed, and executed by this // testing framework. By convention, test files are typically located in a @@ -102,17 +130,28 @@ func RunTest(t *testing.T, path string, f func(t *testing.T, d *TestData) string if err != nil { t.Fatal(err) } else if finfo.IsDir() { - t.Fatalf("%s is a directly, not a file; consider using datadriven.Walk", path) + t.Fatalf("%s is a directory, not a file; consider using datadriven.Walk", path) } - runTestInternal(t, path, file, f, *rewriteTestFiles) + rewriteData := runTestInternal(t, path, file, f, *rewriteTestFiles) + if *rewriteTestFiles { + if _, err := file.WriteAt(rewriteData, 0); err != nil { + t.Fatal(err) + } + if err := file.Truncate(int64(len(rewriteData))); err != nil { + t.Fatal(err) + } + if err := file.Sync(); err != nil { + t.Fatal(err) + } + } } // RunTestFromString is a version of RunTest which takes the contents of a test // directly. func RunTestFromString(t *testing.T, input string, f func(t *testing.T, d *TestData) string) { t.Helper() - runTestInternal(t, "" /* optionalPath */, strings.NewReader(input), f, *rewriteTestFiles) + runTestInternal(t, "" /* sourceName */, strings.NewReader(input), f, *rewriteTestFiles) } func runTestInternal( @@ -121,7 +160,7 @@ func runTestInternal( reader io.Reader, f func(t *testing.T, d *TestData) string, rewrite bool, -) { +) (rewriteOutput []byte) { t.Helper() r := newTestDataReader(t, sourceName, reader, rewrite) @@ -131,23 +170,13 @@ func runTestInternal( if r.rewrite != nil { data := r.rewrite.Bytes() + // Remove any trailing blank line. if l := len(data); l > 2 && data[l-1] == '\n' && data[l-2] == '\n' { data = data[:l-1] } - if dest, ok := reader.(*os.File); ok { - if _, err := dest.WriteAt(data, 0); err != nil { - t.Fatal(err) - } - if err := dest.Truncate(int64(len(data))); err != nil { - t.Fatal(err) - } - if err := dest.Sync(); err != nil { - t.Fatal(err) - } - } else { - t.Logf("input is not a file; rewritten output is:\n%s", data) - } + return data } + return nil } // runDirectiveOrSubTest runs either a "subtest" directive or an @@ -159,6 +188,7 @@ func runDirectiveOrSubTest( mandatorySubTestPrefix string, f func(*testing.T, *TestData) string, ) { + t.Helper() if subTestName, ok := isSubTestStart(t, r, mandatorySubTestPrefix); ok { runSubTest(subTestName, t, r, f) } else { @@ -284,7 +314,7 @@ func runDirective(t *testing.T, r *testDataReader, f func(*testing.T, *TestData) actual := func() string { defer func() { if r := recover(); r != nil { - fmt.Printf("\npanic during %s:\n%s\n", d.Pos, d.Input) + t.Logf("\npanic during %s:\n%s\n", d.Pos, d.Input) panic(r) } }() @@ -315,18 +345,35 @@ func runDirective(t *testing.T, r *testDataReader, f func(*testing.T, *TestData) r.rewrite.WriteString(actual) r.emit("----") r.emit("----") + r.emit("") } else { + // Here actual already ends in \n so emit adds a blank line. r.emit(actual) } } else if d.Expected != actual { + expectedLines := difflib.SplitLines(d.Expected) + actualLines := difflib.SplitLines(actual) + if len(expectedLines) > 5 { + // Print a unified diff if there is a lot of output to compare. + diff, err := difflib.GetUnifiedDiffString(difflib.UnifiedDiff{ + Context: 5, + A: expectedLines, + B: actualLines, + }) + if err == nil { + t.Fatalf("output didn't match expected:\n%s", diff) + return + } + t.Logf("Failed to produce diff %v", err) + } t.Fatalf("\n%s: %s\nexpected:\n%s\nfound:\n%s", d.Pos, d.Input, d.Expected, actual) - } else if testing.Verbose() { + } else if Verbose() { input := d.Input if input == "" { input = "" } // TODO(tbg): it's awkward to reproduce the args, but it would be helpful. - fmt.Printf("\n%s:\n%s [%d args]\n%s\n----\n%s", d.Pos, d.Cmd, len(d.CmdArgs), input, actual) + t.Logf("\n%s:\n%s [%d args]\n%s\n----\n%s", d.Pos, d.Cmd, len(d.CmdArgs), input, actual) } return } @@ -381,6 +428,32 @@ func Walk(t *testing.T, path string, f func(t *testing.T, path string)) { } } +func ClearResults(path string) error { + file, err := os.OpenFile(path, os.O_RDWR, 0644 /* irrelevant */) + if err != nil { + return err + } + defer func() { + _ = file.Close() + }() + finfo, err := file.Stat() + if err != nil { + return err + } + + if finfo.IsDir() { + return errors.Newf("%s is a directory, not a file", path) + } + + runTestInternal( + &testing.T{}, path, file, + func(t *testing.T, d *TestData) string { return "" }, + true, /* rewrite */ + ) + + return nil +} + // Ignore files named .XXXX, XXX~ or #XXX#. var tempFileRe = regexp.MustCompile(`(^\..*)|(.*~$)|(^#.*#$)`) @@ -444,14 +517,16 @@ func (td *TestData) ScanArgs(t *testing.T, key string, dests ...interface{}) { } } if arg.Key == "" { - t.Fatalf("missing argument: %s", key) + td.Fatalf(t, "missing argument: %s", key) } if len(dests) != len(arg.Vals) { - t.Fatalf("%s: got %d destinations, but %d values", arg.Key, len(dests), len(arg.Vals)) + td.Fatalf(t, "%s: got %d destinations, but %d values", arg.Key, len(dests), len(arg.Vals)) } for i := range dests { - arg.Scan(t, i, dests[i]) + if err := arg.scanErr(i, dests[i]); err != nil { + td.Fatalf(t, "%s: failed to scan argument %d: %v", arg.Key, i, err) + } } } @@ -480,8 +555,15 @@ func (arg CmdArg) String() string { // Scan attempts to parse the value at index i into the dest. func (arg CmdArg) Scan(t *testing.T, i int, dest interface{}) { + if err := arg.scanErr(i, dest); err != nil { + t.Fatal(err) + } +} + +// scanErr is like Scan but returns an error rather than taking a testing.T to fatal. +func (arg CmdArg) scanErr(i int, dest interface{}) error { if i < 0 || i >= len(arg.Vals) { - t.Fatalf("cannot scan index %d of key %s", i, arg.Key) + return errors.Errorf("cannot scan index %d of key %s", i, arg.Key) } val := arg.Vals[i] switch dest := dest.(type) { @@ -490,24 +572,25 @@ func (arg CmdArg) Scan(t *testing.T, i int, dest interface{}) { case *int: n, err := strconv.ParseInt(val, 10, 64) if err != nil { - t.Fatal(err) + return err } *dest = int(n) // assume 64bit ints case *uint64: n, err := strconv.ParseUint(val, 10, 64) if err != nil { - t.Fatal(err) + return err } *dest = n case *bool: b, err := strconv.ParseBool(val) if err != nil { - t.Fatal(err) + return err } *dest = b default: - t.Fatalf("unsupported type %T for destination #%d (might be easy to add it)", dest, i+1) + return errors.Errorf("unsupported type %T for destination #%d (might be easy to add it)", dest, i+1) } + return nil } // Fatalf wraps a fatal testing error with test file position information, so diff --git a/github.com/cockroachdb/datadriven/go.mod b/github.com/cockroachdb/datadriven/go.mod index 38b4e87a19..4460349a23 100644 --- a/github.com/cockroachdb/datadriven/go.mod +++ b/github.com/cockroachdb/datadriven/go.mod @@ -3,11 +3,7 @@ module github.com/cockroachdb/datadriven go 1.13 require ( - github.com/certifi/gocertifi v0.0.0-20191021191039-0944d244cd40 // indirect - github.com/cockroachdb/errors v1.2.4 - github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f // indirect - github.com/getsentry/raven-go v0.2.0 // indirect - github.com/gogo/protobuf v1.3.1 // indirect - github.com/kr/pretty v0.1.0 // indirect - github.com/pkg/errors v0.8.1 // indirect + github.com/cockroachdb/errors v1.6.1 + github.com/pkg/errors v0.9.1 // indirect + github.com/pmezard/go-difflib v1.0.0 ) diff --git a/github.com/cockroachdb/datadriven/go.sum b/github.com/cockroachdb/datadriven/go.sum index cfda93a49b..70b208972f 100644 --- a/github.com/cockroachdb/datadriven/go.sum +++ b/github.com/cockroachdb/datadriven/go.sum @@ -1,20 +1,252 @@ -github.com/certifi/gocertifi v0.0.0-20191021191039-0944d244cd40 h1:xvUo53O5MRZhVMJAxWCJcS5HHrqAiAG9SJ1LpMu6aAI= -github.com/certifi/gocertifi v0.0.0-20191021191039-0944d244cd40/go.mod h1:sGbDF6GwGcLpkNXPUTkMRoywsNa/ol15pxFe6ERfguA= -github.com/cockroachdb/errors v1.2.4 h1:Lap807SXTH5tri2TivECb/4abUkMZC9zRoLarvcKDqs= -github.com/cockroachdb/errors v1.2.4/go.mod h1:rQD95gz6FARkaKkQXUksEje/d9a6wBJoCr5oaCLELYA= +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +github.com/AndreasBriese/bbloom v0.0.0-20190306092124-e2d15f34fcf9/go.mod h1:bOvUY6CB00SOBii9/FifXqc0awNKxLFCL/+pkDPuyl8= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= +github.com/CloudyKit/fastprinter v0.0.0-20170127035650-74b38d55f37a/go.mod h1:EFZQ978U7x8IRnstaskI3IysnWY5Ao3QgZUKOXlsAdw= +github.com/CloudyKit/jet v2.1.3-0.20180809161101-62edd43e4f88+incompatible/go.mod h1:HPYO+50pSWkPoj9Q/eq0aRGByCL6ScRlUmiEX5Zgm+w= +github.com/Joker/hpp v1.0.0/go.mod h1:8x5n+M1Hp5hC0g8okX3sR3vFQwynaX/UgSOM9MeBKzY= +github.com/Joker/jade v1.0.1-0.20190614124447-d475f43051e7/go.mod h1:6E6s8o2AE4KhCrqr6GRJjdC/gNfTdxkIXvuGZZda2VM= +github.com/Shopify/goreferrer v0.0.0-20181106222321-ec9c9a553398/go.mod h1:a1uqRtAwp2Xwc6WNPJEufxJ7fx3npB4UV/JOLmbu5I0= +github.com/ajg/form v1.5.1/go.mod h1:uL1WgH+h2mgNtvBq0339dVnzXdBETtL2LeUXaIv25UY= +github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8= +github.com/aymerick/raymond v2.0.3-0.20180322193309-b565731e1464+incompatible/go.mod h1:osfaiScAUVup+UC9Nfq76eWqDhXlp+4UYaA8uhTBO6g= +github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= +github.com/cockroachdb/errors v1.6.1 h1:GqOx5BYPdco9HPaoUuy8W53kZsSEbPu5P/d4WLhanmM= +github.com/cockroachdb/errors v1.6.1/go.mod h1:tm6FTP5G81vwJ5lC0SizQo374JNCOPrHyXGitRJoDqM= github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f h1:o/kfcElHqOiXqcou5a3rIlMc7oJbMQkeLk0VQJ7zgqY= github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f/go.mod h1:i/u985jwjWRlyHXQbwatDASoW0RMlZ/3i9yJHE2xLkI= -github.com/getsentry/raven-go v0.2.0 h1:no+xWJRb5ZI7eE8TWgIq1jLulQiIoLG0IfYxv5JYMGs= -github.com/getsentry/raven-go v0.2.0/go.mod h1:KungGk8q33+aIAZUIVWZDr2OfAEBsO49PX4NzFV5kcQ= +github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2 h1:IKgmqgMQlVJIZj19CdocBeSfSaiCbEBZGKODaixqtHM= +github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2/go.mod h1:8BT+cPK6xvFOcRlk0R8eg+OTkcqI6baNH4xAkpiYVvQ= +github.com/codegangsta/inject v0.0.0-20150114235600-33e0aa1cb7c0/go.mod h1:4Zcjuz89kmFXt9morQgcfYZAYZ5n8WHjt81YYWIwtTM= +github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE= +github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk= +github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk= +github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/dgraph-io/badger v1.6.0/go.mod h1:zwt7syl517jmP8s94KqSxTlM6IMsdhYy6psNgSztDR4= +github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ= +github.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw= +github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= +github.com/eknkc/amber v0.0.0-20171010120322-cdade1c07385/go.mod h1:0vRUJqYpeSZifjYj7uP3BG/gKcuzL9xWVV/Y+cK33KM= +github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= +github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= +github.com/etcd-io/bbolt v1.3.3/go.mod h1:ZF2nL25h33cCyBtcyWeZ2/I3HQOfTP+0PIEvHjkjCrw= +github.com/fasthttp-contrib/websocket v0.0.0-20160511215533-1f3b11f56072/go.mod h1:duJ4Jxv5lDcvg4QuQr0oowTf7dz4/CR8NtyCooz9HL8= +github.com/fatih/structs v1.1.0/go.mod h1:9NiDSp5zOcgEDl+j00MP/WkGVPOlPRLejGD8Ga6PJ7M= +github.com/flosch/pongo2 v0.0.0-20190707114632-bbf5a6c351f4/go.mod h1:T9YF2M40nIgbVgp3rreNmTged+9HrbNTIQf1PsaIiTA= +github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= +github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= +github.com/gavv/httpexpect v2.0.0+incompatible/go.mod h1:x+9tiU1YnrOvnB725RkpoLv1M62hOWzwo5OXotisrKc= +github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s= +github.com/gin-gonic/gin v1.4.0/go.mod h1:OW2EZn3DO8Ln9oIKOvM++LBO+5UPHJJDH72/q/3rZdM= +github.com/go-check/check v0.0.0-20180628173108-788fd7840127/go.mod h1:9ES+weclKsC9YodN5RgxqK/VD9HM9JsCSh7rNhMZE98= +github.com/go-errors/errors v1.0.1 h1:LUHzmkK3GUKUrL/1gfBUxAHzcev3apQlezX/+O7ma6w= +github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q= +github.com/go-martini/martini v0.0.0-20170121215854-22fa46961aab/go.mod h1:/P9AEU963A2AYjv4d1V5eVL1CQbEJq6aCNHDDjibzu8= +github.com/gobwas/httphead v0.0.0-20180130184737-2c6c146eadee/go.mod h1:L0fX3K22YWvt/FAX9NnzrNzcI4wNYi9Yku4O0LKYflo= +github.com/gobwas/pool v0.2.0/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw= +github.com/gobwas/ws v1.0.2/go.mod h1:szmBTxLgaFppYjEmNtny/v3w89xOydFnnZMcgRRu/EM= +github.com/gogo/googleapis v0.0.0-20180223154316-0cd9801be74a/go.mod h1:gf4bu3Q80BeJ6H1S1vYPm8/ELATdvryBaNFGgqEef3s= +github.com/gogo/protobuf v1.2.0/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ= github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= +github.com/gogo/status v1.1.0/go.mod h1:BFv9nrluPLmrS0EmGVvLaPNmRosr9KapBYd5/hpY1WM= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= +github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= +github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= +github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= +github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= +github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= +github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= +github.com/gomodule/redigo v1.7.1-0.20190724094224-574c33c3df38/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4= +github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4= +github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck= +github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY= +github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ= +github.com/hashicorp/go-version v1.2.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA= +github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= +github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= +github.com/hydrogen18/memlistener v0.0.0-20141126152155-54553eb933fb/go.mod h1:qEIFzExnS6016fRpRfxrExeVn2gbClQA99gQhnIcdhE= +github.com/imkira/go-interpol v1.1.0/go.mod h1:z0h2/2T3XF8kyEPpRgJ3kmNv+C43p+I/CoI+jC3w2iA= +github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8= +github.com/iris-contrib/blackfriday v2.0.0+incompatible/go.mod h1:UzZ2bDEoaSGPbkg6SAB4att1aAwTmVIx/5gCVqeyUdI= +github.com/iris-contrib/go.uuid v2.0.0+incompatible/go.mod h1:iz2lgM/1UnEf1kP0L/+fafWORmlnuysV2EMP8MW+qe0= +github.com/iris-contrib/i18n v0.0.0-20171121225848-987a633949d0/go.mod h1:pMCz62A0xJL6I+umB2YTlFRwWXaDFA0jy+5HzGiJjqI= +github.com/iris-contrib/schema v0.0.1/go.mod h1:urYA3uvUNG1TIIjOSCzHr9/LmbQo8LrOcOqfqxa4hXw= +github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU= +github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU= +github.com/juju/errors v0.0.0-20181118221551-089d3ea4e4d5/go.mod h1:W54LbzXuIE0boCoNJfwqpmkKJ1O4TCTZMetAt6jGk7Q= +github.com/juju/loggo v0.0.0-20180524022052-584905176618/go.mod h1:vgyd7OREkbtVEN/8IXZe5Ooef3LQePvuBm9UWj6ZL8U= +github.com/juju/testing v0.0.0-20180920084828-472a3e8b2073/go.mod h1:63prj8cnj0tU0S9OHjGJn+b1h0ZghCndfnbQolrYTwA= +github.com/k0kubun/colorstring v0.0.0-20150214042306-9440f1994b88/go.mod h1:3w7q1U84EfirKl04SVQ/s7nPm1ZPhiXd34z40TNz36k= +github.com/kataras/golog v0.0.9/go.mod h1:12HJgwBIZFNGL0EJnMRhmvGA0PQGx8VFwrZtM4CqbAk= +github.com/kataras/iris/v12 v12.0.1/go.mod h1:udK4vLQKkdDqMGJJVd/msuMtN6hpYJhg/lSzuxjhO+U= +github.com/kataras/neffos v0.0.10/go.mod h1:ZYmJC07hQPW67eKuzlfY7SO3bC0mw83A3j6im82hfqw= +github.com/kataras/pio v0.0.0-20190103105442-ea782b38602d/go.mod h1:NV88laa9UiiDuX9AhMbDPkGYSPugBOV6yTZB1l2K9Z0= github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00= github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck= +github.com/klauspost/compress v1.8.2/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A= +github.com/klauspost/compress v1.9.0/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A= +github.com/klauspost/cpuid v1.2.1/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek= github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= +github.com/labstack/echo/v4 v4.1.11/go.mod h1:i541M3Fj6f76NZtHSj7TXnyM8n2gaodfvfxNnFqi74g= +github.com/labstack/gommon v0.3.0/go.mod h1:MULnywXg0yavhxWKc+lOruYdAhDwPK9wf0OL7NoOu+k= +github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ= +github.com/mattn/go-colorable v0.1.2/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE= +github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= +github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s= +github.com/mattn/go-isatty v0.0.9/go.mod h1:YNRxwqDuOph6SZLI9vUUz6OYw3QyUt7WiY2yME+cCiQ= +github.com/mattn/goveralls v0.0.2/go.mod h1:8d1ZMHsd7fW6IRPKQh46F2WRpyib5/X4FOpevwGNQEw= +github.com/mediocregopher/mediocre-go-lib v0.0.0-20181029021733-cb65787f37ed/go.mod h1:dSsfyI2zABAdhcbvkXqgxOxrCsbYeHCPgrZkku60dSg= +github.com/mediocregopher/radix/v3 v3.3.0/go.mod h1:EmfVyvspXz1uZEyPBMyGK+kjWiKQGvsUt6O3Pj+LDCQ= +github.com/microcosm-cc/bluemonday v1.0.2/go.mod h1:iVP4YcDBq+n/5fb23BhYFvIMq/leAFZyRl6bYmGDlGc= +github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0= +github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= +github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q= +github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0= +github.com/moul/http2curl v1.0.0/go.mod h1:8UbvGypXm98wA/IqH45anm5Y2Z6ep6O31QGOAZ3H0fQ= +github.com/nats-io/nats.go v1.8.1/go.mod h1:BrFz9vVn0fU3AcH9Vn4Kd7W0NpJ651tD5omQ3M8LwxM= +github.com/nats-io/nkeys v0.0.2/go.mod h1:dab7URMsZm6Z/jp9Z5UGa87Uutgc2mVpXLC4B7TDb/4= +github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c= +github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= +github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= +github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= +github.com/onsi/ginkgo v1.13.0/go.mod h1:+REjRxOmWfHCjfv9TTWB1jD1Frx4XydAD3zm1lskyM0= +github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= +github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= +github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= +github.com/pingcap/errors v0.11.4 h1:lFuQV/oaUMGcD2tqt+01ROSmJs75VG1ToEOkZIZ4nE4= +github.com/pingcap/errors v0.11.4/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= +github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= +github.com/ryanuber/columnize v2.1.0+incompatible/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts= +github.com/sclevine/agouti v3.0.0+incompatible/go.mod h1:b4WX9W9L1sfQKXeJf1mUTLZKJ48R1S7H23Ji7oFO5Bw= +github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM= +github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc= +github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc= +github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA= +github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ= +github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= +github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU= +github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo= +github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= +github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= +github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0= +github.com/urfave/negroni v1.0.0/go.mod h1:Meg73S6kFm/4PpbYdq35yYWoCZ9mS/YSx+lKnmiohz4= +github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc= +github.com/valyala/fasthttp v1.6.0/go.mod h1:FstJa9V+Pj9vQ7OJie2qMHdwemEDaDiSdBnvPM1Su9w= +github.com/valyala/fasttemplate v1.0.1/go.mod h1:UQGH1tvbgY+Nz5t2n7tXsz52dQxojPUpymEIMZ47gx8= +github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a/go.mod h1:v3UYOV9WzVtRmSR+PDvWpU/qWl4Wa5LApYYX4ZtKbio= +github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU= +github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ= +github.com/xeipuuv/gojsonschema v1.2.0/go.mod h1:anYRn/JVcOK2ZgGU+IjEV4nwlhoK5sQluxsYJ78Id3Y= +github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= +github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0/go.mod h1:/LWChgwKmvncFJFHJ7Gvn9wZArjbV5/FppcK2fKk/tI= +github.com/yudai/gojsondiff v1.0.0/go.mod h1:AY32+k2cwILAkW1fbgxQ5mUmMiZFgLIV+FBNExI05xg= +github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82/go.mod h1:lgjkn3NuSvDfVJdfcVVdX+jpBxNmX4rDAzaS45IcYoM= +github.com/yudai/pp v2.0.1+incompatible/go.mod h1:PuxR/8QJ7cyCkFp/aUDS+JY727OFEZkTdatxwunjIkc= +golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20190701094942-4def268fd1a4/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190327091125-710a502c58a2/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20190827160401-ba9fcec4b297/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190626221950-04f50cda93cb/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20181221001348-537d06c36207/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190327201419-c70d86f8b7cf/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/genproto v0.0.0-20180518175338-11a468237815/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= +google.golang.org/grpc v1.12.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= +google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= +google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk= +google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= +google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= +google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= +google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= +google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= +google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= +gopkg.in/go-playground/assert.v1 v1.2.1/go.mod h1:9RXL0bg/zibRAgZUYszZSwO/z8Y/a8bDuhia5mkpMnE= +gopkg.in/go-playground/validator.v8 v8.18.2/go.mod h1:RX2a/7Ha8BgOhfk7j780h4/u/RRjR0eouCJSH80/M2Y= +gopkg.in/mgo.v2 v2.0.0-20180705113604-9856a29383ce/go.mod h1:yeKp02qBN3iKW1OzL3MGk2IdtZzaj7SFntXj72NppTA= +gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= diff --git a/github.com/cockroachdb/datadriven/test_data_reader.go b/github.com/cockroachdb/datadriven/test_data_reader.go index bfe5c64009..2f5397b1c4 100644 --- a/github.com/cockroachdb/datadriven/test_data_reader.go +++ b/github.com/cockroachdb/datadriven/test_data_reader.go @@ -112,14 +112,14 @@ func (r *testDataReader) Next(t *testing.T) bool { r.data.Input = strings.TrimSpace(buf.String()) if separator { - r.readExpected() + r.readExpected(t) } return true } return false } -func (r *testDataReader) readExpected() { +func (r *testDataReader) readExpected(t *testing.T) { var buf bytes.Buffer var line string var allowBlankLines bool @@ -140,6 +140,11 @@ func (r *testDataReader) readExpected() { if r.scanner.Scan() { line2 := r.scanner.Text() if line2 == "----" { + // Read the following blank line (if we don't do this, we will emit + // an extra blank line when rewriting). + if r.scanner.Scan() && r.scanner.Text() != "" { + t.Fatal("non-blank line after end of double ---- separator section") + } break } diff --git a/github.com/cockroachdb/errors/.travis.yml b/github.com/cockroachdb/errors/.travis.yml index 00f925c140..a91ba657b2 100644 --- a/github.com/cockroachdb/errors/.travis.yml +++ b/github.com/cockroachdb/errors/.travis.yml @@ -1,6 +1,5 @@ language: go go: -- 1.12.x - 1.13.x - 1.14.x diff --git a/github.com/cockroachdb/errors/README.md b/github.com/cockroachdb/errors/README.md index 55979da394..13b80ef468 100644 --- a/github.com/cockroachdb/errors/README.md +++ b/github.com/cockroachdb/errors/README.md @@ -45,7 +45,8 @@ Table of contents: | wrappers for user-facing hints and details | | | | ✔ | | wrappers to attach secondary causes | | | | ✔ | | wrappers to attach [`logtags`](https://github.com/cockroachdb/logtags) details from `context.Context` | | | | ✔ | -| `errors.FormatError()`, `Formatter`, `Printer` | | | (under construction) | (under construction) | +| `errors.FormatError()`, `Formatter`, `Printer` | | | (under construction) | ✔ | +| `errors.SafeFormatError()`, `SafeFormatter` | | | | ✔ | "Forward compatibility" above refers to the ability of this library to recognize and properly handle network communication of error types it @@ -75,8 +76,8 @@ older version of the package. | Error detail | `Error()` and format `%s`/`%q`/`%v` | format `%+v` | `GetSafeDetails()` | Sentry report via `ReportError()` | |-----------------------------------------------------------------|-------------------------------------|--------------|-------------------------------|-----------------------------------| -| main message, eg `New()` | visible | visible | redacted | redacted | -| wrap prefix, eg `WithMessage()` | visible (as prefix) | visible | redacted | redacted | +| main message, eg `New()` | visible | visible | yes (CHANGED IN v1.6) | full (CHANGED IN v1.6) | +| wrap prefix, eg `WithMessage()` | visible (as prefix) | visible | yes (CHANGED IN v1.6) | full (CHANGED IN v1.6) | | stack trace, eg `WithStack()` | not visible | simplified | yes | full | | hint , eg `WithHint()` | not visible | visible | no | type only | | detail, eg `WithDetail()` | not visible | visible | no | type only | @@ -98,7 +99,7 @@ method. - `New(string) error`, `Newf(string, ...interface{}) error`, `Errorf(string, ...interface{}) error`: leaf errors with message - **when to use: common error cases.** - what it does: also captures the stack trace at point of call and redacts the provided message for safe reporting. - - how to access the detail: `Error()`, regular Go formatting. Details redacted in Sentry report. + - how to access the detail: `Error()`, regular Go formatting. **Details in Sentry report.** - see also: Section [Error composition](#Error-composition-summary) below. `errors.NewWithDepth()` variants to customize at which call depth the stack trace is captured. - `AssertionFailedf(string, ...interface{}) error`, `NewAssertionFailureWithWrappedErrf(error, string, ...interface{}) error`: signals an assertion failure / programming error. @@ -141,7 +142,7 @@ return errors.Wrap(foo(), "foo") - `Wrap(error, string) error`, `Wrapf(error, string, ...interface{}) error`: - **when to use: on error return paths.** - what it does: combines `WithMessage()`, `WithStack()`, `WithSafeDetails()`. - - how to access the details: `Error()`, regular Go formatting. Details redacted in Sentry report. + - how to access the details: `Error()`, regular Go formatting. **Details in Sentry report.** - see also: Section [Error composition](#Error-composition-summary) below. `WrapWithDepth()` variants to customize at which depth the stack trace is captured. - `WithSecondaryError(error, error) error`: annotate an error with a secondary error. @@ -201,17 +202,17 @@ return errors.Wrap(foo(), "foo") - `WithMessage(error, string) error`, `WithMessagef(error, string, ...interface{}) error`: message prefix. - when to use: probably never. Use `errors.Wrap()`/`errors.Wrapf()` instead. - what it does: adds a message prefix. - - how to access the detail: `Error()`, regular Go formatting. Not included in Sentry reports. + - how to access the detail: `Error()`, regular Go formatting, Sentry Report. - `WithDetail(error, string) error`, `WithDetailf(error, string, ...interface{}) error`, user-facing detail with contextual information. - **when to use: need to embark a message string to output when the error is presented to a human.** - what it does: captures detail strings. - - how to access the detail: `errors.GetAllDetails()`, `errors.FlattenDetails()` (all details are preserved), format with `%+v`. + - how to access the detail: `errors.GetAllDetails()`, `errors.FlattenDetails()` (all details are preserved), format with `%+v`. Not included in Sentry reports. - `WithHint(error, string) error`, `WithHintf(error, string, ...interface{}) error`: user-facing detail with suggestion for action to take. - **when to use: need to embark a message string to output when the error is presented to a human.** - what it does: captures hint strings. - - how to access the detail: `errors.GetAllHints()`, `errors.FlattenHints()` (hints are de-duplicated), format with `%+v`. + - how to access the detail: `errors.GetAllHints()`, `errors.FlattenHints()` (hints are de-duplicated), format with `%+v`. Not included in Sentry reports. - `WithIssueLink(error, IssueLink) error`: annotate an error with an URL and arbitrary string. - **when to use: to refer (human) users to some external resources.** @@ -243,13 +244,14 @@ return errors.Wrap(foo(), "foo") The library support PII-free strings essentially as follows: -- by default, strings included in an error object are considered to be - PII-unsafe, and are stripped out when building a Sentry report. -- some fields in the library are whitelisted by default. +- by default, many strings included in an error object are considered + to be PII-unsafe, and are stripped out when building a Sentry + report. +- some fields in the library are assumed to be PII-safe by default. - you can opt additional strings in to Sentry reports. -The following strings from this library are "whitelisted" upfront, -considered to be PII-free, and thus included in Sentry reports automatically: +The following strings from this library are considered to be PII-free, +and thus included in Sentry reports automatically: - the *type* of error objects, - stack traces (containing only file paths, line numbers, function names - arguments are not included), @@ -260,7 +262,7 @@ considered to be PII-free, and thus included in Sentry reports automatically: - the `format string` argument of `Newf`, `AssertionFailedf`, etc (the constructors ending with `...f()`), - the *type* of the additional arguments passed to the `...f()` constructors, - the *value of specific argument types* passed to the `...f()` constructors, when known to be PII-safe. - For details of which arguments are considered PII-free, see the [`Redact()` function](safedetails/redact.go). + For details of which arguments are considered PII-free, see the [`redact` package](https://github.com/cockroachdb/redact). It is possible to opt additional in to Sentry reporting, using either of the following methods: @@ -293,7 +295,7 @@ If your error type is a wrapper, you should implement a `Format()` method that redirects to `errors.FormatError()`, otherwise `%+v` will not work. Additionally, if your type has a payload not otherwise visible via `Error()`, you may want to implement -`errors.Formatter`. See [making `%+v` work with your +`errors.SafeFormatter`. See [making `%+v` work with your type](#Making-v-work-with-your-type) below for details. Finally, you may want your new error type to be portable across @@ -445,8 +447,8 @@ In short: type, this will disable the recursive application of the `%+v` flag to the causes chained from your wrapper.) -- You may optionally implement the `errors.Formatter` interface: - `FormatError(p errors.Printer) (next error)`. This is optional, but +- You may optionally implement the `errors.SafeFormatter` interface: + `SafeFormatError(p errors.Printer) (next error)`. This is optional, but should be done when some details are not included by `Error()` and should be emitted upon `%+v`. @@ -458,11 +460,11 @@ achieves this as follows: func (w *withHTTPCode) Format(s fmt.State, verb rune) { errors.FormatError(w, s, verb) } // FormatError() formats the error. -func (w *withHTTPCode) FormatError(p errors.Printer) (next error) { +func (w *withHTTPCode) SafeFormatError(p errors.Printer) (next error) { // Note: no need to print out the cause here! // FormatError() knows how to do this automatically. if p.Detail() { - p.Printf("http code: %d", w.code) + p.Printf("http code: %d", errors.Safe(w.code)) } return w.cause } @@ -492,22 +494,22 @@ proposal](https://go.googlesource.com/proposal/+/master/design/29934-error-value ## Error composition (summary) -| Constructor | Composes | -|------------------------------------|--------------------------------------------------------------------------------------------------| -| `New` | `NewWithDepth` (see below) | -| `Errorf` | = `Newf` | -| `Newf` | `NewWithDepthf` (see below) | -| `WithMessage` | = `pkgErr.WithMessage` | -| `Wrap` | `WrapWithDepth` (see below) | -| `Wrapf` | `WrapWithDepthf` (see below) | -| `AssertionFailed` | `AssertionFailedWithDepthf` (see below) | -| `NewWithDepth` | `goErr.New` + `WithStackDepth` (see below) | -| `NewWithDepthf` | `fmt.Errorf` + `WithSafeDetails` + `WithStackDepth` | -| `WithMessagef` | `pkgErr.WithMessagef` + `WithSafeDetails` | -| `WrapWithDepth` | `WithMessage` + `WithStackDepth` | -| `WrapWithDepthf` | `WithMessage` + `WithStackDepth` + `WithSafeDetails` | -| `AssertionFailedWithDepthf` | `fmt.Errorf` + `WithStackDepth` + `WithSafeDetails` + `WithAssertionFailure` | -| `NewAssertionErrorWithWrappedErrf` | `HandledWithMessagef` (barrier) + `WithStackDepth` + `WithSafeDetails` + `WithAssertionFailure` | +| Constructor | Composes | +|------------------------------------|-----------------------------------------------------------------------------------| +| `New` | `NewWithDepth` (see below) | +| `Errorf` | = `Newf` | +| `Newf` | `NewWithDepthf` (see below) | +| `WithMessage` | custom wrapper with message prefix and knowledge of safe strings | +| `Wrap` | `WrapWithDepth` (see below) | +| `Wrapf` | `WrapWithDepthf` (see below) | +| `AssertionFailed` | `AssertionFailedWithDepthf` (see below) | +| `NewWithDepth` | custom leaf with knowledge of safe strings + `WithStackDepth` (see below) | +| `NewWithDepthf` | custom leaf with knowledge of safe strings + `WithSafeDetails` + `WithStackDepth` | +| `WithMessagef` | custom wrapper with message prefix and knowledge of safe strings | +| `WrapWithDepth` | `WithMessage` + `WithStackDepth` | +| `WrapWithDepthf` | `WithMessagef` + `WithStackDepth` | +| `AssertionFailedWithDepthf` | `NewWithDepthf` + `WithAssertionFailure` | +| `NewAssertionErrorWithWrappedErrf` | `HandledWithMessagef` (barrier) + `WrapWithDepthf` + `WithAssertionFailure` | ## API (not constructing error objects) @@ -519,6 +521,13 @@ func Cause(err error) error // compatibility func Unwrap(err error) error // compatibility type Wrapper interface { ... } // compatibility +// Error formatting. +type Formatter interface { ... } // compatibility, not recommended +type SafeFormatter interface { ... } +type Printer interface { ... } +func FormatError(err error, s fmt.State, verb rune) +func Formattable(err error) fmt.Formatter + // Identify errors. func Is(err, reference error) bool func IsAny(err error, references ...error) bool @@ -553,10 +562,14 @@ func GetReportableStackTrace(err error) *ReportableStackTrace type SafeDetailPayload struct { ... } func GetAllSafeDetails(err error) []SafeDetailPayload func GetSafeDetails(err error) (payload SafeDetailPayload) + +// Obsolete APIs. type SafeMessager interface { ... } -func Safe(v interface{}) SafeMessager func Redact(r interface{}) string +// Aliases redact.Safe. +func Safe(v interface{}) SafeMessager + // Assertion failures. func HasAssertionFailure(err error) bool func IsAssertionFailure(err error) bool diff --git a/github.com/cockroachdb/errors/assert/assert.go b/github.com/cockroachdb/errors/assert/assert.go index 9c7ddd9cde..5c641a5f10 100644 --- a/github.com/cockroachdb/errors/assert/assert.go +++ b/github.com/cockroachdb/errors/assert/assert.go @@ -62,7 +62,7 @@ type withAssertionFailure struct { var _ error = (*withAssertionFailure)(nil) var _ fmt.Formatter = (*withAssertionFailure)(nil) -var _ errbase.Formatter = (*withAssertionFailure)(nil) +var _ errbase.SafeFormatter = (*withAssertionFailure)(nil) // ErrorHint implements the hintdetail.ErrorHinter interface. func (w *withAssertionFailure) ErrorHint() string { @@ -77,9 +77,9 @@ func (w *withAssertionFailure) Cause() error { return w.cause } func (w *withAssertionFailure) Unwrap() error { return w.cause } func (w *withAssertionFailure) Format(s fmt.State, verb rune) { errbase.FormatError(w, s, verb) } -func (w *withAssertionFailure) FormatError(p errbase.Printer) error { +func (w *withAssertionFailure) SafeFormatError(p errbase.Printer) error { if p.Detail() { - p.Print("assertion failure") + p.Printf("assertion failure") } return w.cause } diff --git a/github.com/cockroachdb/errors/assert_api.go b/github.com/cockroachdb/errors/assert_api.go index 419402f2fc..d01e0bf63f 100644 --- a/github.com/cockroachdb/errors/assert_api.go +++ b/github.com/cockroachdb/errors/assert_api.go @@ -16,11 +16,20 @@ package errors import "github.com/cockroachdb/errors/assert" -// WithAssertionFailure forwards a definition. +// WithAssertionFailure decorates the error with an assertion failure marker. +// This is not intended to be used directly (see AssertionFailed() for +// further decoration). +// +// Detail is shown: +// - when formatting with `%+v`. +// - in Sentry reports. func WithAssertionFailure(err error) error { return assert.WithAssertionFailure(err) } -// HasAssertionFailure forwards a definition. +// HasAssertionFailure returns true if the error or any of its causes +// is an assertion failure annotation. func HasAssertionFailure(err error) bool { return assert.HasAssertionFailure(err) } -// IsAssertionFailure forwards a definition. +// IsAssertionFailure returns true if the error (not its causes) is an +// assertion failure annotation. Consider using markers.If or +// HasAssertionFailure to test both the error and its causes. func IsAssertionFailure(err error) bool { return assert.IsAssertionFailure(err) } diff --git a/github.com/cockroachdb/errors/barriers/barriers.go b/github.com/cockroachdb/errors/barriers/barriers.go index 24b03d73b0..fa4f431180 100644 --- a/github.com/cockroachdb/errors/barriers/barriers.go +++ b/github.com/cockroachdb/errors/barriers/barriers.go @@ -75,7 +75,7 @@ type barrierError struct { var _ error = (*barrierError)(nil) var _ errbase.SafeDetailer = (*barrierError)(nil) -var _ errbase.Formatter = (*barrierError)(nil) +var _ errbase.SafeFormatter = (*barrierError)(nil) var _ fmt.Formatter = (*barrierError)(nil) // barrierError is an error. @@ -94,8 +94,8 @@ func (e *barrierError) SafeDetails() []string { // Printing a barrier reveals the details. func (e *barrierError) Format(s fmt.State, verb rune) { errbase.FormatError(e, s, verb) } -func (e *barrierError) FormatError(p errbase.Printer) (next error) { - p.Printf(e.msg) +func (e *barrierError) SafeFormatError(p errbase.Printer) (next error) { + p.Print(e.msg) if p.Detail() { p.Printf("-- cause hidden behind barrier\n%+v", e.maskedErr) } diff --git a/github.com/cockroachdb/errors/barriers_api.go b/github.com/cockroachdb/errors/barriers_api.go index 61757112e2..feeb618e5c 100644 --- a/github.com/cockroachdb/errors/barriers_api.go +++ b/github.com/cockroachdb/errors/barriers_api.go @@ -16,8 +16,19 @@ package errors import "github.com/cockroachdb/errors/barriers" -// Handled forwards a definition. +// Handled swallows the provided error and hides it from the +// Cause()/Unwrap() interface, and thus the Is() facility that +// identifies causes. However, it retains it for the purpose of +// printing the error out (e.g. for troubleshooting). The error +// message is preserved in full. +// +// Detail is shown: +// - via `errors.GetSafeDetails()`, shows details from hidden error. +// - when formatting with `%+v`. +// - in Sentry reports. func Handled(err error) error { return barriers.Handled(err) } -// HandledWithMessage forwards a definition. +// HandledWithMessage is like Handled except the message is overridden. +// This can be used e.g. to hide message details or to prevent +// downstream code to make assertions on the message's contents. func HandledWithMessage(err error, msg string) error { return barriers.HandledWithMessage(err, msg) } diff --git a/github.com/cockroachdb/errors/contexttags/contexttags.go b/github.com/cockroachdb/errors/contexttags/contexttags.go index 6b520c7cc4..8b8277f290 100644 --- a/github.com/cockroachdb/errors/contexttags/contexttags.go +++ b/github.com/cockroachdb/errors/contexttags/contexttags.go @@ -18,8 +18,8 @@ import ( "context" "github.com/cockroachdb/errors/errbase" - "github.com/cockroachdb/errors/safedetails" "github.com/cockroachdb/logtags" + "github.com/cockroachdb/redact" ) // WithContextTags captures the k/v pairs stored in the context via the @@ -88,15 +88,26 @@ func convertToStringsOnly(b *logtags.Buffer) (res *logtags.Buffer) { func redactTags(b *logtags.Buffer) []string { res := make([]string, len(b.Get())) + redactableTagsIterate(b, func(i int, r redact.RedactableString) { + res[i] = r.Redact().StripMarkers() + }) + return res +} + +func redactableTagsIterate(b *logtags.Buffer, fn func(i int, s redact.RedactableString)) { + var empty redact.SafeString for i, t := range b.Get() { - res[i] = t.Key() + k := t.Key() v := t.Value() + eq := empty + var val interface{} = empty if v != nil { - if len(res[i]) > 1 { - res[i] += "=" + if len(k) > 1 { + eq = "=" } - res[i] += safedetails.Redact(v) + val = v } + res := redact.Sprintf("%s%s%v", redact.Safe(k), eq, val) + fn(i, res) } - return res } diff --git a/github.com/cockroachdb/errors/contexttags/with_context.go b/github.com/cockroachdb/errors/contexttags/with_context.go index 407edb2e61..f1e63c7f03 100644 --- a/github.com/cockroachdb/errors/contexttags/with_context.go +++ b/github.com/cockroachdb/errors/contexttags/with_context.go @@ -21,6 +21,7 @@ import ( "github.com/cockroachdb/errors/errbase" "github.com/cockroachdb/errors/errorspb" "github.com/cockroachdb/logtags" + "github.com/cockroachdb/redact" "github.com/gogo/protobuf/proto" ) @@ -39,8 +40,8 @@ type withContext struct { var _ error = (*withContext)(nil) var _ errbase.SafeDetailer = (*withContext)(nil) +var _ errbase.SafeFormatter = (*withContext)(nil) var _ fmt.Formatter = (*withContext)(nil) -var _ errbase.Formatter = (*withContext)(nil) // withContext is an error. The original error message is preserved. func (w *withContext) Error() string { return w.cause.Error() } @@ -52,9 +53,16 @@ func (w *withContext) Unwrap() error { return w.cause } // Printing a withContext reveals the tags. func (w *withContext) Format(s fmt.State, verb rune) { errbase.FormatError(w, s, verb) } -func (w *withContext) FormatError(p errbase.Printer) error { +func (w *withContext) SafeFormatError(p errbase.Printer) error { if p.Detail() && w.tags != nil { - p.Printf("tags: [%s]", w.tags.String()) + p.Printf("tags: [") + redactableTagsIterate(w.tags, func(i int, r redact.RedactableString) { + if i > 0 { + p.Printf(",") + } + p.Print(r) + }) + p.Printf("]") } return w.cause } diff --git a/github.com/cockroachdb/errors/contexttags_api.go b/github.com/cockroachdb/errors/contexttags_api.go index 51ac6ed32b..3f77960e84 100644 --- a/github.com/cockroachdb/errors/contexttags_api.go +++ b/github.com/cockroachdb/errors/contexttags_api.go @@ -21,10 +21,27 @@ import ( "github.com/cockroachdb/logtags" ) -// WithContextTags forwards a definition. +// WithContextTags captures the k/v pairs stored in the context via the +// `logtags` package and annotates them on the error. +// +// Only the stromg representation of values remains available. This is +// because the library cannot guarantee that the underlying value is +// preserved across the network. To avoid creating a stateful interface +// (where the user code needs to know whether an error has traveled +// through the network or not), the library restricts access to the +// value part as strings. See GetContextTags() below. +// +// Detail is shown: +// - via `errors.GetSafeDetails()`. +// - via `GetContextTags()` below. +// - when formatting with `%+v`. +// - in Sentry reports. func WithContextTags(err error, ctx context.Context) error { return contexttags.WithContextTags(err, ctx) } -// GetContextTags forwards a definition. +// GetContextTags retrieves the k/v pairs stored in the error. +// The sets are returned from outermost to innermost level of cause. +// The returned logtags.Buffer only know about the string +// representation of the values originally captured by the error. func GetContextTags(err error) []*logtags.Buffer { return contexttags.GetContextTags(err) } diff --git a/github.com/cockroachdb/errors/domains/domains.go b/github.com/cockroachdb/errors/domains/domains.go index dbc5be8425..a7b0dde5cd 100644 --- a/github.com/cockroachdb/errors/domains/domains.go +++ b/github.com/cockroachdb/errors/domains/domains.go @@ -93,7 +93,7 @@ func HandledInDomainWithMessage(err error, domain Domain, msg string) error { // Handled creates a handled error in the implicit domain (see // PackageDomain() below) of its caller. // -// See the documentation of `errors.Handled()` for details. +// See the documentation of `barriers.Handled()` for details. func Handled(err error) error { return HandledInDomain(err, PackageDomainAtDepth(1)) } @@ -144,7 +144,7 @@ func PackageDomain() Domain { } // PackageDomainAtDepth returns an error domain that describes the -// package at the given depth. +// package at the given call depth. func PackageDomainAtDepth(depth int) Domain { _, f, _, _ := runtime.Caller(1 + depth) return Domain("error domain: pkg " + filepath.Dir(f)) diff --git a/github.com/cockroachdb/errors/domains/with_domain.go b/github.com/cockroachdb/errors/domains/with_domain.go index d881244384..fbf24e3826 100644 --- a/github.com/cockroachdb/errors/domains/with_domain.go +++ b/github.com/cockroachdb/errors/domains/with_domain.go @@ -19,6 +19,7 @@ import ( "fmt" "github.com/cockroachdb/errors/errbase" + "github.com/cockroachdb/redact" "github.com/gogo/protobuf/proto" ) @@ -36,7 +37,7 @@ var _ error = (*withDomain)(nil) var _ errbase.SafeDetailer = (*withDomain)(nil) var _ errbase.TypeKeyMarker = (*withDomain)(nil) var _ fmt.Formatter = (*withDomain)(nil) -var _ errbase.Formatter = (*withDomain)(nil) +var _ errbase.SafeFormatter = (*withDomain)(nil) // withDomain is an error. The original error message is preserved. func (e *withDomain) Error() string { return e.cause.Error() } @@ -58,9 +59,9 @@ func (e *withDomain) SafeDetails() []string { func (e *withDomain) Format(s fmt.State, verb rune) { errbase.FormatError(e, s, verb) } -func (e *withDomain) FormatError(p errbase.Printer) error { +func (e *withDomain) SafeFormatError(p errbase.Printer) error { if p.Detail() { - p.Print(e.domain) + p.Print(redact.Safe(e.domain)) } return e.cause } diff --git a/github.com/cockroachdb/errors/domains_api.go b/github.com/cockroachdb/errors/domains_api.go index fea98f31e5..5e965e5494 100644 --- a/github.com/cockroachdb/errors/domains_api.go +++ b/github.com/cockroachdb/errors/domains_api.go @@ -16,42 +16,63 @@ package errors import "github.com/cockroachdb/errors/domains" -// Domain forwards a definition. +// Domain is the type of a domain annotation. type Domain = domains.Domain -// NoDomain forwards a definition. +// NoDomain is the domain of errors that don't originate +// from a barrier. const NoDomain Domain = domains.NoDomain -// NamedDomain forwards a definition. +// NamedDomain returns an error domain identified by the given string. func NamedDomain(domainName string) Domain { return domains.NamedDomain(domainName) } -// PackageDomain forwards a definition. +// PackageDomain returns an error domain that represents the +// package of its caller. func PackageDomain() Domain { return domains.PackageDomainAtDepth(1) } -// PackageDomainAtDepth forwards a definition. +// PackageDomainAtDepth returns an error domain that describes the +// package at the given call depth. func PackageDomainAtDepth(depth int) Domain { return domains.PackageDomainAtDepth(depth) } -// WithDomain forwards a definition. +// WithDomain wraps an error so that it appears to come from the given domain. +// +// Domain is shown: +// - via `errors.GetSafeDetails()`. +// - when formatting with `%+v`. +// - in Sentry reports. func WithDomain(err error, domain Domain) error { return domains.WithDomain(err, domain) } -// NotInDomain forwards a definition. +// NotInDomain returns true if and only if the error's +// domain is not one of the specified domains. func NotInDomain(err error, doms ...Domain) bool { return domains.NotInDomain(err, doms...) } -// EnsureNotInDomain forwards a definition. +// EnsureNotInDomain checks whether the error is in the given domain(s). +// If it is, the given constructor if provided is called to construct +// an alternate error. If no error constructor is provided, +// a new barrier is constructed automatically using the first +// provided domain as new domain. The original error message +// is preserved. func EnsureNotInDomain(err error, constructor DomainOverrideFn, forbiddenDomains ...Domain) error { return domains.EnsureNotInDomain(err, constructor, forbiddenDomains...) } -// DomainOverrideFn forwards a definition. +// DomainOverrideFn is the type of the callback function passed to EnsureNotInDomain(). type DomainOverrideFn = func(originalDomain Domain, err error) error -// HandledInDomain forwards a definition. +// HandledInDomain creates an error in the given domain and retains +// the details of the given original error as context for +// debugging. The original error is hidden and does not become a +// "cause" for the new error. The original's error _message_ +// is preserved. +// +// See the documentation of `WithDomain()` and `errors.Handled()` for details. func HandledInDomain(err error, domain Domain) error { return domains.HandledInDomain(err, domain) } -// HandledInDomainWithMessage forwards a definition. +// HandledInDomainWithMessage is like HandledWithMessage but with a domain. func HandledInDomainWithMessage(err error, domain Domain, msg string) error { return domains.HandledInDomainWithMessage(err, domain, msg) } -// GetDomain forwards a definition. +// GetDomain extracts the domain of the given error, or NoDomain if +// the error's cause does not have a domain annotation. func GetDomain(err error) Domain { return domains.GetDomain(err) } diff --git a/github.com/cockroachdb/errors/errbase/encode.go b/github.com/cockroachdb/errors/errbase/encode.go index ca37de7568..a4857632e5 100644 --- a/github.com/cockroachdb/errors/errbase/encode.go +++ b/github.com/cockroachdb/errors/errbase/encode.go @@ -211,6 +211,9 @@ func getTypeDetails( // TypeKeyMarker can be implemented by errors that wish to extend // their type name as seen by GetTypeKey(). +// +// Note: the key marker is considered safe for reporting and +// is included in sentry reports. type TypeKeyMarker interface { ErrorKeyMarker() string } diff --git a/github.com/cockroachdb/errors/errbase/format_error.go b/github.com/cockroachdb/errors/errbase/format_error.go index 94ccb05099..534db0809e 100644 --- a/github.com/cockroachdb/errors/errbase/format_error.go +++ b/github.com/cockroachdb/errors/errbase/format_error.go @@ -28,12 +28,16 @@ import ( "fmt" "io" "reflect" - "strconv" + "strings" + "github.com/cockroachdb/redact" + "github.com/kr/pretty" pkgErr "github.com/pkg/errors" ) // FormatError formats an error according to s and verb. +// This is a helper meant for use when implementing the fmt.Formatter +// interface on custom error objects. // // If the error implements errors.Formatter, FormatError calls its // FormatError method of f with an errors.Printer configured according @@ -44,16 +48,53 @@ import ( // // Otherwise, its Error() text is printed. func FormatError(err error, s fmt.State, verb rune) { + formatErrorInternal(err, s, verb, false /* redactableOutput */) +} + +// FormatRedactableError formats an error as a safe object. +// +// Note that certain verb/flags combinations are currently not +// supported, and result in a rendering that considers the entire +// object as unsafe. For example, %q, %#v are not yet supported. +func FormatRedactableError(err error, s redact.SafePrinter, verb rune) { + formatErrorInternal(err, s, verb, true /* redactable */) +} + +func init() { + // Also inform the redact package of how to print an error + // safely. This is used when an error is passed as argument + // to one of the redact print functions. + redact.RegisterRedactErrorFn(FormatRedactableError) +} + +// Formattable wraps an error into a fmt.Formatter which +// will provide "smart" formatting even if the outer layer +// of the error does not implement the Formatter interface. +func Formattable(err error) fmt.Formatter { + return &errorFormatter{err} +} + +// formatErrorInternal is the shared logic between FormatError +// and FormatErrorRedactable. +// +// When the redactableOutput argument is true, the fmt.State argument +// is really a redact.SafePrinter and casted down as necessary. +// +// If verb and flags are not one of the supported error formatting +// combinations (in particular, %q, %#v etc), then the redactableOutput +// argument is ignored. This limitation may be lifted in a later +// version. +func formatErrorInternal(err error, s fmt.State, verb rune, redactableOutput bool) { // Assuming this function is only called from the Format method, and given // that FormatError takes precedence over Format, it cannot be called from // any package that supports errors.Formatter. It is therefore safe to // disregard that State may be a specific printer implementation and use one // of our choice instead. - p := state{State: s} + p := state{State: s, redactableOutput: redactableOutput} switch { - case verb == 'v' && s.Flag('+'): + case verb == 'v' && s.Flag('+') && !s.Flag('#'): // Here we are going to format as per %+v, into p.buf. // // We need to start with the innermost (root cause) error first, @@ -69,17 +110,30 @@ func FormatError(err error, s fmt.State, verb rune) { // We're done formatting. Apply width/precision parameters. p.finishDisplay(verb) - case verb == 'v' && s.Flag('#'): + case !redactableOutput && verb == 'v' && s.Flag('#'): + // We only know how to process %#v if redactable output is not + // requested. This is because the structured output may emit + // arbitrary unsafe strings without redaction markers, + // or improperly balanced/escaped redaction markers. if stringer, ok := err.(fmt.GoStringer); ok { - io.WriteString(&p.buf, stringer.GoString()) - p.finishDisplay(verb) - return + io.WriteString(&p.finalBuf, stringer.GoString()) + } else { + // Not a GoStringer: delegate to the pretty library. + fmt.Fprintf(&p.finalBuf, "%# v", pretty.Formatter(err)) } - // Not a stringer. Proceed as if it were %v. - fallthrough + p.finishDisplay(verb) - case verb == 's' || verb == 'q' || - verb == 'x' || verb == 'X' || verb == 'v': + case verb == 's' || + // We only handle %v/%+v or other combinations here; %#v is unsupported. + (verb == 'v' && !s.Flag('#')) || + // If redactable output is not requested, then we also + // know how to format %x/%X (print bytes of error message in hex) + // and %q (quote the result). + // If redactable output is requested, then we don't know + // how to perform these exotic verbs, because they + // may muck with the redaction markers. In this case, + // we simply refuse the format as per the default clause below. + (!redactableOutput && (verb == 'x' || verb == 'X' || verb == 'q')): // Only the error message. // // Use an intermediate buffer because there may be alignment @@ -99,61 +153,121 @@ func FormatError(err error, s fmt.State, verb rune) { default: // Unknown verb. Do like fmt.Printf and tell the user we're // confused. - p.buf.WriteString("%!") - p.buf.WriteRune(verb) - p.buf.WriteByte('(') + // + // Note that the following logic is correct regardless of the + // value of 'redactableOutput', because the display of the verb and type + // are always safe for redaction. If/when this code is changed to + // print more details, care is to be taken to add redaction + // markers if s.redactableOutput is set. + p.finalBuf.WriteString("%!") + p.finalBuf.WriteRune(verb) + p.finalBuf.WriteByte('(') switch { case err != nil: - p.buf.WriteString(reflect.TypeOf(err).String()) + p.finalBuf.WriteString(reflect.TypeOf(err).String()) default: - p.buf.WriteString("") + p.finalBuf.WriteString("") } - p.buf.WriteByte(')') - io.Copy(s, &p.buf) + p.finalBuf.WriteByte(')') + io.Copy(s, &p.finalBuf) } } +// formatEntries reads the entries from s.entries and produces a +// detailed rendering in s.finalBuf. +// +// Note that if s.redactableOutput is true, s.finalBuf is to contain a +// RedactableBytes. However, we are not using the helper facilities +// from redact.SafePrinter to do this, so care should be taken below +// to properly escape markers, etc. func (s *state) formatEntries(err error) { // The first entry at the top is special. We format it as follows: // // // (1)
s.formatSingleLineOutput() - s.buf.WriteString("\n(1)") - printEntry(&s.buf, s.entries[len(s.entries)-1]) + s.finalBuf.WriteString("\n(1)") + + s.printEntry(s.entries[len(s.entries)-1]) // All the entries that follow are printed as follows: // // Wraps: (N)
// for i, j := len(s.entries)-2, 2; i >= 0; i, j = i-1, j+1 { - fmt.Fprintf(&s.buf, "\nWraps: (%d)", j) + fmt.Fprintf(&s.finalBuf, "\nWraps: (%d)", j) entry := s.entries[i] - printEntry(&s.buf, entry) + s.printEntry(entry) } // At the end, we link all the (N) references to the Go type of the // error. - s.buf.WriteString("\nError types:") + s.finalBuf.WriteString("\nError types:") for i, j := len(s.entries)-1, 1; i >= 0; i, j = i-1, j+1 { - fmt.Fprintf(&s.buf, " (%d) %T", j, s.entries[i].err) + fmt.Fprintf(&s.finalBuf, " (%d) %T", j, s.entries[i].err) } } -func printEntry(buf *bytes.Buffer, entry formatEntry) { +// printEntry renders the entry given as argument +// into s.finalBuf. +// +// If s.redactableOutput is set, then s.finalBuf is to contain +// a RedactableBytes, with redaction markers. In that +// case, we must be careful to escape (or not) the entry +// depending on entry.redactable. +// +// If s.redactableOutput is unset, then we are not caring about +// redactability. In that case entry.redactable is not set +// anyway and we can pass contents through. +func (s *state) printEntry(entry formatEntry) { if len(entry.head) > 0 { if entry.head[0] != '\n' { - buf.WriteByte(' ') + s.finalBuf.WriteByte(' ') + } + if len(entry.head) > 0 { + if !s.redactableOutput || entry.redactable { + // If we don't care about redaction, then we can pass the string + // through. + // + // If we do care about redaction, and entry.redactable is true, + // then entry.head is already a RedactableBytes. Then we can + // also pass it through. + s.finalBuf.Write(entry.head) + } else { + // We care about redaction, and the head is unsafe. Escape it + // and enclose the result within redaction markers. + s.finalBuf.Write([]byte(redact.EscapeBytes(entry.head))) + } } - buf.Write(entry.head) } if len(entry.details) > 0 { if len(entry.head) == 0 { if entry.details[0] != '\n' { - buf.WriteByte(' ') + s.finalBuf.WriteByte(' ') } } - buf.Write(entry.details) + if !s.redactableOutput || entry.redactable { + // If we don't care about redaction, then we can pass the string + // through. + // + // If we do care about redaction, and entry.redactable is true, + // then entry.details is already a RedactableBytes. Then we can + // also pass it through. + s.finalBuf.Write(entry.details) + } else { + // We care about redaction, and the details are unsafe. Escape + // them and enclose the result within redaction markers. + s.finalBuf.Write([]byte(redact.EscapeBytes(entry.details))) + } + } + if entry.stackTrace != nil { + s.finalBuf.WriteString("\n -- stack trace:") + s.finalBuf.WriteString(strings.ReplaceAll( + fmt.Sprintf("%+v", entry.stackTrace), + "\n", string(detailSep))) + if entry.elidedStackTrace { + fmt.Fprintf(&s.finalBuf, "%s[...repeated from below...]", detailSep) + } } } @@ -168,23 +282,53 @@ func printEntry(buf *bytes.Buffer, entry formatEntry) { // (e *myType) Format(s fmt.State, verb rune) { errors.FormatError(s, verb, e) } // // and also to print the first line in the output of a %+v format. +// +// It reads from s.entries and writes to s.finalBuf. +// s.buf is left untouched. +// +// Note that if s.redactableOutput is true, s.finalBuf is to contain a +// RedactableBytes. However, we are not using the helper facilities +// from redact.SafePrinter to do this, so care should be taken below +// to properly escape markers, etc. func (s *state) formatSingleLineOutput() { for i := len(s.entries) - 1; i >= 0; i-- { entry := &s.entries[i] if entry.elideShort { continue } - if s.buf.Len() > 0 && len(entry.head) > 0 { - s.buf.WriteString(": ") + if s.finalBuf.Len() > 0 && len(entry.head) > 0 { + s.finalBuf.WriteString(": ") + } + if len(entry.head) == 0 { + // shortcut, to avoid the copy below. + continue + } + if !s.redactableOutput || entry.redactable { + // If we don't care about redaction, then we can pass the string + // through. + // + // If we do care about redaction, and entry.redactable is true, + // then entry.head is already a RedactableBytes. Then we can + // also pass it through. + s.finalBuf.Write(entry.head) + } else { + // We do care about redaction, but entry.redactable is unset. + // This means entry.head is unsafe. We need to escape it. + s.finalBuf.Write([]byte(redact.EscapeBytes(entry.head))) } - s.buf.Write(entry.head) } } // formatRecursive performs a post-order traversal on the chain of // errors to collect error details from innermost to outermost. // +// It uses s.buf as an intermediate buffer to collect strings. // It populates s.entries as a result. +// Between each layer of error, s.buf is reset. +// +// s.finalBuf is untouched. The conversion of s.entries +// to s.finalBuf is done by formatSingleLineOutput() and/or +// formatEntries(). func (s *state) formatRecursive(err error, isOutermost, withDetail bool) { cause := UnwrapOnce(err) if cause != nil { @@ -203,68 +347,93 @@ func (s *state) formatRecursive(err error, isOutermost, withDetail bool) { seenTrace := false - switch v := err.(type) { - case Formatter: - desiredShortening := v.FormatError((*printer)(s)) - if desiredShortening == nil { - // The error wants to elide the short messages. Do it. - for i := range s.entries { - s.entries[i].elideShort = true + bufIsRedactable := false + + printDone := false + for _, fn := range specialCases { + if handled, desiredShortening := fn(err, (*safePrinter)(s), cause == nil /* leaf */); handled { + printDone = true + bufIsRedactable = true + if desiredShortening == nil { + // The error wants to elide the short messages from inner + // causes. Do it. + for i := range s.entries { + s.entries[i].elideShort = true + } } + break } - case fmt.Formatter: - // We can only use a fmt.Formatter when both the following - // conditions are true: - // - when it is the leaf error, because a fmt.Formatter - // on a wrapper also recurses. - // - when it is not the outermost wrapper, because - // the Format() method is likely to be calling FormatError() - // to do its job and we want to avoid an infinite recursion. - if !isOutermost && cause == nil { - v.Format(s, 'v') - if st, ok := err.(StackTraceProvider); ok { - // This is likely a leaf error from github/pkg/errors. - // The thing probably printed its stack trace on its own. - seenTrace = true - // We'll subsequently simplify stack traces in wrappers. - s.lastStack = st.StackTrace() + } + if !printDone { + switch v := err.(type) { + case SafeFormatter: + bufIsRedactable = true + desiredShortening := v.SafeFormatError((*safePrinter)(s)) + if desiredShortening == nil { + // The error wants to elide the short messages from inner + // causes. Do it. + for i := range s.entries { + s.entries[i].elideShort = true + } } - } else { + + case Formatter: + desiredShortening := v.FormatError((*printer)(s)) + if desiredShortening == nil { + // The error wants to elide the short messages from inner + // causes. Do it. + for i := range s.entries { + s.entries[i].elideShort = true + } + } + + case fmt.Formatter: + // We can only use a fmt.Formatter when both the following + // conditions are true: + // - when it is the leaf error, because a fmt.Formatter + // on a wrapper also recurses. + // - when it is not the outermost wrapper, because + // the Format() method is likely to be calling FormatError() + // to do its job and we want to avoid an infinite recursion. + if !isOutermost && cause == nil { + v.Format(s, 'v') + if st, ok := err.(StackTraceProvider); ok { + // This is likely a leaf error from github/pkg/errors. + // The thing probably printed its stack trace on its own. + seenTrace = true + // We'll subsequently simplify stack traces in wrappers. + s.lastStack = st.StackTrace() + } + } else { + s.formatSimple(err, cause) + } + + default: + // If the error did not implement errors.Formatter nor + // fmt.Formatter, but it is a wrapper, still attempt best effort: + // print what we can at this level. s.formatSimple(err, cause) } - default: - // If the error did not implement errors.Formatter nor - // fmt.Formatter, but it is a wrapper, still attempt best effort: - // print what we can at this level. - s.formatSimple(err, cause) } - // If there's an embedded stack trace, print it. + // Collect the result. + entry := s.collectEntry(err, bufIsRedactable) + + // If there's an embedded stack trace, also collect it. // This will get either a stack from pkg/errors, or ours. if !seenTrace { if st, ok := err.(StackTraceProvider); ok { - newStack, elided := ElideSharedStackTraceSuffix(s.lastStack, st.StackTrace()) - s.lastStack = newStack - if s.wantDetail { - s.switchOver() - if s.multiLine { - s.Write([]byte("\n-- stack trace:")) - } - newStack.Format(s, 'v') - if elided { - s.Write([]byte("\n[...repeated from below...]")) - } - } + entry.stackTrace, entry.elidedStackTrace = ElideSharedStackTraceSuffix(s.lastStack, st.StackTrace()) + s.lastStack = entry.stackTrace } } - // Collect the result. - entry := s.collectEntry(err) + // Remember the entry for later rendering. s.entries = append(s.entries, entry) s.buf = bytes.Buffer{} } -func (s *state) collectEntry(err error) formatEntry { +func (s *state) collectEntry(err error, bufIsRedactable bool) formatEntry { entry := formatEntry{err: err} if s.wantDetail { // The buffer has been populated as a result of formatting with @@ -284,9 +453,40 @@ func (s *state) collectEntry(err error) formatEntry { } entry.head = append(entry.head, s.buf.Bytes()...) } + + if bufIsRedactable { + // In this case, we've produced entry.head/entry.details using a + // SafeFormatError() invocation. The strings in + // entry.head/entry.detail contain redaction markers at this + // point. + if s.redactableOutput { + // Redaction markers desired in the final output. Keep the + // redaction markers. + entry.redactable = true + } else { + // Markers not desired in the final output: strip the markers. + entry.head = redact.RedactableBytes(entry.head).StripMarkers() + entry.details = redact.RedactableBytes(entry.details).StripMarkers() + } + } + return entry } +// safeErrorPrinterFn is the type of a function that can take +// over the safe printing of an error. This is used to inject special +// cases into the formatting in errutil. We need this machinery to +// prevent import cycles. +type safeErrorPrinterFn = func(err error, p Printer, isLeaf bool) (handled bool, next error) + +// specialCases is a list of functions to apply for special cases. +var specialCases []safeErrorPrinterFn + +// RegisterSpecialCasePrinter registers a handler. +func RegisterSpecialCasePrinter(fn safeErrorPrinterFn) { + specialCases = append(specialCases, fn) +} + // formatSimple performs a best effort at extracting the details at a // given level of wrapping when the error object does not implement // the Formatter interface. @@ -302,14 +502,23 @@ func (s *state) formatSimple(err, cause error) { } } -// finishDisplay renders the buffer in state into the fmt.State. +// finishDisplay renders s.finalBuf into s.State. func (p *state) finishDisplay(verb rune) { + if p.redactableOutput { + // If we're rendering in redactable form, then s.finalBuf contains + // a RedactableBytes. We can emit that directly. + sp := p.State.(redact.SafePrinter) + sp.Print(redact.RedactableBytes(p.finalBuf.Bytes())) + return + } + // Not redactable: render depending on flags and verb. + width, okW := p.Width() - prec, okP := p.Precision() + _, okP := p.Precision() // If `direct` is set to false, then the buffer is always // passed through fmt.Printf regardless of the width and alignment - // settings. This is important or e.g. %q where quotes must be added + // settings. This is important for e.g. %q where quotes must be added // in any case. // If `direct` is set to true, then the detour via // fmt.Printf only occurs if there is a width or alignment @@ -317,28 +526,10 @@ func (p *state) finishDisplay(verb rune) { direct := verb == 'v' || verb == 's' if !direct || (okW && width > 0) || okP { - // Construct format string from State s. - format := []byte{'%'} - if p.Flag('-') { - format = append(format, '-') - } - if p.Flag('+') { - format = append(format, '+') - } - if p.Flag(' ') { - format = append(format, ' ') - } - if okW { - format = strconv.AppendInt(format, int64(width), 10) - } - if okP { - format = append(format, '.') - format = strconv.AppendInt(format, int64(prec), 10) - } - format = append(format, string(verb)...) - fmt.Fprintf(p.State, string(format), p.buf.String()) + _, format := redact.MakeFormat(p, verb) + fmt.Fprintf(p.State, format, p.finalBuf.String()) } else { - io.Copy(p.State, &p.buf) + io.Copy(p.State, &p.finalBuf) } } @@ -346,12 +537,35 @@ var detailSep = []byte("\n | ") // state tracks error printing state. It implements fmt.State. type state struct { + // state inherits fmt.State. + // + // If we are rendering with redactableOutput=true, then fmt.State + // can be downcasted to redact.SafePrinter. fmt.State - // buf collects the details of the current error object - // at a given stage of recursion in formatRecursive(). - // At each stage of recursion (level of wrapping), buf - // contains successively: + // redactableOutput indicates whether we want the output + // to use redaction markers. When set to true, + // the fmt.State above is actually a redact.SafePrinter. + redactableOutput bool + + // finalBuf contains the final rendered string, prior to being + // copied to the fmt.State above. + // + // If redactableOutput is true, then finalBuf contains a RedactableBytes + // and safe redaction markers. Otherwise, it can be considered + // an unsafe string. + finalBuf bytes.Buffer + + // entries collect the result of formatRecursive(). They are + // consumed by formatSingleLineOutput() and formatEntries() to + // procude the contents of finalBuf. + entries []formatEntry + + // buf collects the details of the current error object at a given + // stage of recursion in formatRecursive(). + // + // At each stage of recursion (level of wrapping), buf contains + // successively: // // - at the beginning, the "simple" part of the error message -- // either the pre-Detail() string if the error implements Formatter, @@ -360,15 +574,53 @@ type state struct { // - after the first call to Detail(), buf is copied to headBuf, // then reset, then starts collecting the "advanced" part of the // error message. + // + // At the end of an error layer, the contents of buf and headBuf + // are collected into a formatEntry by collectEntry(). + // This collection does not touch finalBuf above. + // + // The entries are later consumed by formatSingleLineOutput() or + // formatEntries() to produce the contents of finalBuf. + // + // + // Notes regarding redaction markers and string safety. Throughout a + // single "level" of error, there are three cases to consider: + // + // - the error level implements SafeErrorFormatter and + // s.redactableOutput is set. In that case, the error's + // SafeErrorFormat() is used to produce a RedactableBytes in + // buf/headBuf via safePrinter{}, and an entry is collected at the + // end of that with the redactable bit set on the entry. + // + // - the error level implements SafeErrorFormatter + // and s.redactableOutput is *not* set. In this case, + // for convenience we implement non-redactable output by using + // SafeErrorFormat() to generate a RedactableBytes into + // buf/headBuf via safePrinter{}, and then stripping the redaction + // markers to produce the entry. The entry is not marked as + // redactable. + // + // - in the remaining case (s.redactableOutput is not set or the + // error only implements Formatter), then we use FormatError() + // to produce a non-redactable string into buf/headBuf, + // and mark the resulting entry as non-redactable. buf bytes.Buffer // When an error's FormatError() calls Detail(), the current // value of buf above is copied to headBuf, and a new // buf is initialized. headBuf []byte - // entries collects the result of formatRecursive(). - entries []formatEntry - // hasDetail becomes true at each level of th formatRecursive() + // lastStack tracks the last stack trace observed when looking at + // the errors from innermost to outermost. This is used to elide + // redundant stack trace entries. + lastStack StackTrace + + // --------------------------------------------------------------- + // The following attributes organize the synchronization of writes + // to buf and headBuf, during the rendering of a single error + // layer. They get reset between layers. + + // hasDetail becomes true at each level of the formatRecursive() // recursion after the first call to .Detail(). It is used to // determine how to translate buf/headBuf into a formatEntry. hasDetail bool @@ -381,10 +633,10 @@ type state struct { // errors.FormatError().) wantDetail bool - // lastStack tracks the last stack trace observed when looking at - // the errors from innermost to outermost. This is used to elide - // redundant stack trace entries. - lastStack StackTrace + // collectingRedactableString is true iff the data being accumulated + // into buf comes from a redact string. It ensures that newline + // characters are not included inside redaction markers. + collectingRedactableString bool // notEmpty tracks, at each level of recursion of formatRecursive(), // whether there were any details printed by an error's @@ -407,6 +659,10 @@ type state struct { // wrapping or the leaf error in an error chain. type formatEntry struct { err error + // redactable is true iff the data in head and details + // are RedactableBytes. See the explanatory comments + // on (state).buf for when this is set. + redactable bool // head is the part of the text that is suitable for printing in the // one-liner summary, or when producing the output of .Error(). head []byte @@ -416,6 +672,14 @@ type formatEntry struct { // elideShort, if true, elides the value of 'head' from concatenated // "short" messages produced by formatSingleLineOutput(). elideShort bool + + // stackTrace, if non-nil, reports the stack trace embedded at this + // level of error. + stackTrace StackTrace + // elidedStackTrace, if true, indicates that the stack trace was + // truncated to avoid duplication of entries. This is used to + // display a truncation indicator during verbose rendering. + elidedStackTrace bool } // String is used for debugging only. @@ -453,8 +717,6 @@ func (s *state) Write(b []byte) (n int, err error) { } else { if s.needNewline > 0 && s.notEmpty { // If newline chars were pending, display them now. - // The s.notEmpty condition ensures that we don't - // start a detail string with excess newline characters. for i := 0; i < s.needNewline-1; i++ { s.buf.Write(detailSep[:len(sep)-1]) } @@ -525,11 +787,60 @@ func (s *printer) enhanceArgs(args []interface{}) { s.lastStack = lastSeen } +// safePrinter is a variant to printer used when the current error +// level implements SafeFormatter. +// +// In any case, it uses the error's SafeFormatError() method to +// prepare a RedactableBytes into s.buf / s.headBuf. +// The the explanation for `buf` in the state struct. +type safePrinter state + +func (s *safePrinter) Detail() bool { + return ((*state)(s)).detail() +} + +func (s *safePrinter) Print(args ...interface{}) { + s.enhanceArgs(args) + redact.Fprint((*state)(s), args...) +} + +func (s *safePrinter) Printf(format string, args ...interface{}) { + s.enhanceArgs(args) + redact.Fprintf((*state)(s), format, args...) +} + +func (s *safePrinter) enhanceArgs(args []interface{}) { + prevStack := s.lastStack + lastSeen := prevStack + for i := range args { + if st, ok := args[i].(pkgErr.StackTrace); ok { + thisStack, _ := ElideSharedStackTraceSuffix(prevStack, st) + // Stack traces are safe strings. + args[i] = redact.Safe(thisStack) + lastSeen = st + } + // In contrast with (*printer).enhanceArgs(), we dont use a + // special case for `error` here, because the redact package + // already helps us recursing into a safe print for + // error objects. + } + s.lastStack = lastSeen +} + type errorFormatter struct{ err error } // Format implements the fmt.Formatter interface. func (ef *errorFormatter) Format(s fmt.State, verb rune) { FormatError(ef.err, s, verb) } +// Error implements error, so that `redact` knows what to do with it. +func (ef *errorFormatter) Error() string { return ef.err.Error() } + +// Unwrap makes it a wrapper. +func (ef *errorFormatter) Unwrap() error { return ef.err } + +// Cause makes it a wrapper. +func (ef *errorFormatter) Cause() error { return ef.err } + // ElideSharedStackTraceSuffix removes the suffix of newStack that's already // present in prevStack. The function returns true if some entries // were elided. diff --git a/github.com/cockroachdb/errors/errbase/formatter.go b/github.com/cockroachdb/errors/errbase/formatter.go index e8b8273e54..68262428a3 100644 --- a/github.com/cockroachdb/errors/errbase/formatter.go +++ b/github.com/cockroachdb/errors/errbase/formatter.go @@ -22,13 +22,16 @@ package errbase // A Formatter formats error messages. +// +// NB: Consider implementing SafeFormatter instead. This will ensure +// that error displays can distinguish bits that are PII-safe. type Formatter interface { error // FormatError prints the receiver's first error. // The return value decides what happens in the case // FormatError() is used to produce a "short" message, - // eg. when it is used to implementError(): + // eg. when it is used to implement Error(): // // - if it returns nil, then the short message // contains no more than that produced for this error, @@ -44,6 +47,37 @@ type Formatter interface { FormatError(p Printer) (next error) } +// SafeFormatter is implemented by error leaf or wrapper types that want +// to separate safe and non-safe information when printed out. +// +// When multiple errors are chained (e.g. via errors.Wrap), intermediate +// layers in the error that do not implement SafeError are considered +// “unsafe†+type SafeFormatter interface { + // SafeFormatError prints the receiver's first error. + // + // The provided Printer behaves like a redact.SafePrinter its + // Print() and Printf() methods conditionally add redaction markers + // around unsafe bits. + // + // The return value of SafeFormatError() decides what happens in the + // case the method is used to produce a "short" message, eg. when it + // is used to implement Error(): + // + // - if it returns nil, then the short message + // contains no more than that produced for this error, + // even if the error has a further causal chain. + // + // - if it returns non-nil, then the short message + // contains the value printed by this error, + // followed by that of its causal chain. + // (e.g. thiserror: itscause: furthercause) + // + // Note that all the causal chain is reported in verbose reports in + // any case. + SafeFormatError(p Printer) (next error) +} + // A Printer formats error messages. // // The most common implementation of Printer is the one provided by package fmt diff --git a/github.com/cockroachdb/errors/errbase/opaque.go b/github.com/cockroachdb/errors/errbase/opaque.go index 56ddfaf4a7..81ca032954 100644 --- a/github.com/cockroachdb/errors/errbase/opaque.go +++ b/github.com/cockroachdb/errors/errbase/opaque.go @@ -18,6 +18,7 @@ import ( "fmt" "github.com/cockroachdb/errors/errorspb" + "github.com/cockroachdb/redact" ) // opaqueLeaf is used when receiving an unknown leaf type. @@ -32,7 +33,7 @@ type opaqueLeaf struct { var _ error = (*opaqueLeaf)(nil) var _ SafeDetailer = (*opaqueLeaf)(nil) var _ fmt.Formatter = (*opaqueLeaf)(nil) -var _ Formatter = (*opaqueLeaf)(nil) +var _ SafeFormatter = (*opaqueLeaf)(nil) // opaqueWrapper is used when receiving an unknown wrapper type. // Its important property is that if it is communicated @@ -47,7 +48,7 @@ type opaqueWrapper struct { var _ error = (*opaqueWrapper)(nil) var _ SafeDetailer = (*opaqueWrapper)(nil) var _ fmt.Formatter = (*opaqueWrapper)(nil) -var _ Formatter = (*opaqueWrapper)(nil) +var _ SafeFormatter = (*opaqueWrapper)(nil) func (e *opaqueLeaf) Error() string { return e.msg } @@ -68,31 +69,38 @@ func (e *opaqueWrapper) SafeDetails() []string { return e.details.ReportablePayl func (e *opaqueLeaf) Format(s fmt.State, verb rune) { FormatError(e, s, verb) } func (e *opaqueWrapper) Format(s fmt.State, verb rune) { FormatError(e, s, verb) } -func (e *opaqueLeaf) FormatError(p Printer) (next error) { +func (e *opaqueLeaf) SafeFormatError(p Printer) (next error) { p.Print(e.msg) if p.Detail() { - p.Print("\n(opaque error leaf)") - p.Printf("\ntype name: %s", e.details.OriginalTypeName) + p.Printf("\n(opaque error leaf)") + p.Printf("\ntype name: %s", redact.Safe(e.details.OriginalTypeName)) for i, d := range e.details.ReportablePayload { - p.Printf("\nreportable %d:\n%s", i, d) + p.Printf("\nreportable %d:\n%s", redact.Safe(i), redact.Safe(d)) } if e.details.FullDetails != nil { - p.Printf("\npayload type: %s", e.details.FullDetails.TypeUrl) + p.Printf("\npayload type: %s", redact.Safe(e.details.FullDetails.TypeUrl)) } } return nil } -func (e *opaqueWrapper) FormatError(p Printer) (next error) { - p.Print(e.prefix) +func (e *opaqueWrapper) SafeFormatError(p Printer) (next error) { + if len(e.prefix) > 0 { + // We use the condition if len(msg) > 0 because + // otherwise an empty string would cause a "redactable + // empty string" to be emitted (something that looks like "<>") + // and the error formatting code only cleanly elides + // the prefix properly if the output string is completely empty. + p.Print(e.prefix) + } if p.Detail() { - p.Print("\n(opaque error wrapper)") - p.Printf("\ntype name: %s", e.details.OriginalTypeName) + p.Printf("\n(opaque error wrapper)") + p.Printf("\ntype name: %s", redact.Safe(e.details.OriginalTypeName)) for i, d := range e.details.ReportablePayload { - p.Printf("\nreportable %d:\n%s", i, d) + p.Printf("\nreportable %d:\n%s", redact.Safe(i), redact.Safe(d)) } if e.details.FullDetails != nil { - p.Printf("\npayload type: %s", e.details.FullDetails.TypeUrl) + p.Printf("\npayload type: %s", redact.Safe(e.details.FullDetails.TypeUrl)) } } return e.cause diff --git a/github.com/cockroachdb/errors/errbase/safe_details.go b/github.com/cockroachdb/errors/errbase/safe_details.go index 8eebdb30f7..0f692c4e5f 100644 --- a/github.com/cockroachdb/errors/errbase/safe_details.go +++ b/github.com/cockroachdb/errors/errbase/safe_details.go @@ -81,8 +81,13 @@ type SafeDetailPayload struct { // Fill can be used to concatenate multiple SafeDetailPayloads. func (s *SafeDetailPayload) Fill(slice []string) []string { - slice = append(slice, fmt.Sprintf("-- details for %s::%s:", + if len(s.SafeDetails) == 0 { + return slice + } + slice = append(slice, fmt.Sprintf("details for %s::%s:", s.ErrorTypeMark.FamilyName, s.ErrorTypeMark.Extension)) - slice = append(slice, s.SafeDetails...) + for _, sd := range s.SafeDetails { + slice = append(slice, " "+sd) + } return slice } diff --git a/github.com/cockroachdb/errors/errbase_api.go b/github.com/cockroachdb/errors/errbase_api.go index f1569dc21b..7e5bfbff40 100644 --- a/github.com/cockroachdb/errors/errbase_api.go +++ b/github.com/cockroachdb/errors/errbase_api.go @@ -16,74 +16,164 @@ package errors import ( "context" + "fmt" "github.com/cockroachdb/errors/errbase" ) -// UnwrapOnce forwards a definition. +// UnwrapOnce accesses the direct cause of the error if any, otherwise +// returns nil. +// +// It supports both errors implementing causer (`Cause()` method, from +// github.com/pkg/errors) and `Wrapper` (`Unwrap()` method, from the +// Go 2 error proposal). func UnwrapOnce(err error) error { return errbase.UnwrapOnce(err) } -// UnwrapAll forwards a definition. +// UnwrapAll accesses the root cause object of the error. +// If the error has no cause (leaf error), it is returned directly. func UnwrapAll(err error) error { return errbase.UnwrapAll(err) } -// EncodedError forwards a definition. +// EncodedError is the type of an encoded (and protobuf-encodable) error. type EncodedError = errbase.EncodedError -// EncodeError forwards a definition. +// EncodeError encodes an error. func EncodeError(ctx context.Context, err error) EncodedError { return errbase.EncodeError(ctx, err) } -// DecodeError forwards a definition. +// DecodeError decodes an error. func DecodeError(ctx context.Context, enc EncodedError) error { return errbase.DecodeError(ctx, enc) } -// SafeDetailer forwards a definition. +// SafeDetailer is an interface that can be implemented by errors that +// can provide PII-free additional strings suitable for reporting or +// telemetry. type SafeDetailer = errbase.SafeDetailer -// GetAllSafeDetails forwards a definition. +// GetAllSafeDetails collects the safe details from the given error object +// and all its causes. +// The details are collected from outermost to innermost level of cause. func GetAllSafeDetails(err error) []SafeDetailPayload { return errbase.GetAllSafeDetails(err) } -// GetSafeDetails forwards a definition. +// GetSafeDetails collects the safe details from the given error +// object. If it is a wrapper, only the details from the wrapper are +// returned. func GetSafeDetails(err error) (payload SafeDetailPayload) { return errbase.GetSafeDetails(err) } -// SafeDetailPayload forwards a definition. +// SafeDetailPayload captures the safe strings for one +// level of wrapping. type SafeDetailPayload = errbase.SafeDetailPayload -// RegisterLeafDecoder forwards a definition. +// RegisterLeafDecoder can be used to register new leaf error types to +// the library. Registered types will be decoded using their own +// Go type when an error is decoded. Wrappers that have not been +// registered will be decoded using the opaqueLeaf type. +// +// Note: if the error type has been migrated from a previous location +// or a different type, ensure that RegisterTypeMigration() was called +// prior to RegisterLeafDecoder(). func RegisterLeafDecoder(typeName TypeKey, decoder LeafDecoder) { errbase.RegisterLeafDecoder(typeName, decoder) } -// TypeKey forwards a definition. +// TypeKey identifies an error for the purpose of looking up decoders. +// It is equivalent to the "family name" in ErrorTypeMarker. type TypeKey = errbase.TypeKey -// GetTypeKey forwards a definition. +// GetTypeKey retrieve the type key for a given error object. This +// is meant for use in combination with the Register functions. func GetTypeKey(err error) TypeKey { return errbase.GetTypeKey(err) } -// LeafDecoder forwards a definition. +// LeafDecoder is to be provided (via RegisterLeafDecoder above) +// by additional wrapper types not yet known to this library. +// A nil return indicates that decoding was not successful. type LeafDecoder = errbase.LeafDecoder -// RegisterWrapperDecoder forwards a definition. +// RegisterWrapperDecoder can be used to register new wrapper types to +// the library. Registered wrappers will be decoded using their own +// Go type when an error is decoded. Wrappers that have not been +// registered will be decoded using the opaqueWrapper type. +// +// Note: if the error type has been migrated from a previous location +// or a different type, ensure that RegisterTypeMigration() was called +// prior to RegisterWrapperDecoder(). func RegisterWrapperDecoder(typeName TypeKey, decoder WrapperDecoder) { errbase.RegisterWrapperDecoder(typeName, decoder) } -// WrapperDecoder forwards a definition. +// WrapperDecoder is to be provided (via RegisterWrapperDecoder above) +// by additional wrapper types not yet known to this library. +// A nil return indicates that decoding was not successful. type WrapperDecoder = errbase.WrapperDecoder -// RegisterLeafEncoder forwards a definition. +// RegisterLeafEncoder can be used to register new leaf error types to +// the library. Registered types will be encoded using their own +// Go type when an error is encoded. Wrappers that have not been +// registered will be encoded using the opaqueLeaf type. +// +// Note: if the error type has been migrated from a previous location +// or a different type, ensure that RegisterTypeMigration() was called +// prior to RegisterLeafEncoder(). func RegisterLeafEncoder(typeName TypeKey, encoder LeafEncoder) { errbase.RegisterLeafEncoder(typeName, encoder) } -// LeafEncoder forwards a definition. +// LeafEncoder is to be provided (via RegisterLeafEncoder above) +// by additional wrapper types not yet known to this library. type LeafEncoder = errbase.LeafEncoder -// RegisterWrapperEncoder forwards a definition. +// RegisterWrapperEncoder can be used to register new wrapper types to +// the library. Registered wrappers will be encoded using their own +// Go type when an error is encoded. Wrappers that have not been +// registered will be encoded using the opaqueWrapper type. +// +// Note: if the error type has been migrated from a previous location +// or a different type, ensure that RegisterTypeMigration() was called +// prior to RegisterWrapperEncoder(). func RegisterWrapperEncoder(typeName TypeKey, encoder WrapperEncoder) { errbase.RegisterWrapperEncoder(typeName, encoder) } -// WrapperEncoder forwards a definition. +// WrapperEncoder is to be provided (via RegisterWrapperEncoder above) +// by additional wrapper types not yet known to this library. type WrapperEncoder = errbase.WrapperEncoder -// SetWarningFn forwards a definition. +// SetWarningFn enables configuration of the warning function. func SetWarningFn(fn func(context.Context, string, ...interface{})) { errbase.SetWarningFn(fn) } + +// A Formatter formats error messages. +// +// NB: Consider implementing SafeFormatter instead. This will ensure +// that error displays can distinguish bits that are PII-safe. +type Formatter = errbase.Formatter + +// SafeFormatter is implemented by error leaf or wrapper types that want +// to separate safe and non-safe information when printed out. +// +// When multiple errors are chained (e.g. via errors.Wrap), intermediate +// layers in the error that do not implement SafeError are considered +// “unsafe†+type SafeFormatter = errbase.SafeFormatter + +// A Printer formats error messages. +// +// The most common implementation of Printer is the one provided by package fmt +// during Printf (as of Go 1.13). Localization packages such as golang.org/x/text/message +// typically provide their own implementations. +type Printer = errbase.Printer + +// FormatError formats an error according to s and verb. +// This is a helper meant for use when implementing the fmt.Formatter +// interface on custom error objects. +// +// If the error implements errors.Formatter, FormatError calls its +// FormatError method of f with an errors.Printer configured according +// to s and verb, and writes the result to s. +// +// Otherwise, if it is a wrapper, FormatError prints out its error prefix, +// then recurses on its cause. +// +// Otherwise, its Error() text is printed. +func FormatError(err error, s fmt.State, verb rune) { errbase.FormatError(err, s, verb) } + +// Formattable wraps an error into a fmt.Formatter which +// will provide "smart" formatting even if the outer layer +// of the error does not implement the Formatter interface. +func Formattable(err error) fmt.Formatter { return errbase.Formattable(err) } diff --git a/github.com/cockroachdb/errors/errutil/format_error_special.go b/github.com/cockroachdb/errors/errutil/format_error_special.go new file mode 100644 index 0000000000..fa7548859c --- /dev/null +++ b/github.com/cockroachdb/errors/errutil/format_error_special.go @@ -0,0 +1,88 @@ +// Copyright 2020 The Cockroach Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +// implied. See the License for the specific language governing +// permissions and limitations under the License. + +package errutil + +import ( + "context" + "net" + "os" + "runtime" + "syscall" + + "github.com/cockroachdb/errors/errbase" + "github.com/cockroachdb/errors/markers" + "github.com/cockroachdb/redact" +) + +func init() { + errbase.RegisterSpecialCasePrinter(specialCaseFormat) +} + +func specialCaseFormat(err error, p errbase.Printer, isLeaf bool) (handled bool, next error) { + if isLeaf && markers.IsAny(err, + context.DeadlineExceeded, + context.Canceled, + os.ErrInvalid, + os.ErrPermission, + os.ErrExist, + os.ErrNotExist, + os.ErrClosed, + os.ErrNoDeadline) { + p.Print(redact.Safe(err.Error())) + return true, nil + } + + switch v := err.(type) { + // The following two types are safe too. + case runtime.Error: + p.Print(redact.Safe(v.Error())) + return true, err + case syscall.Errno: + p.Print(redact.Safe(v.Error())) + return true, err + case *os.SyscallError: + p.Print(redact.Safe(v.Syscall)) + return true, err + case *os.PathError: + p.Printf("%s %s", redact.Safe(v.Op), v.Path) + return true, err + case *os.LinkError: + p.Printf("%s %s %s", redact.Safe(v.Op), v.Old, v.New) + return true, err + case *net.OpError: + p.Print(redact.Safe(v.Op)) + if v.Net != "" { + p.Printf(" %s", redact.Safe(v.Net)) + } + if v.Source != nil { + p.Printf(" %s", v.Source) + } + if v.Addr != nil { + if v.Source != nil { + p.Printf(" ->") + } + p.Printf(" %s", v.Addr) + } + return true, err + case redact.SafeMessager: + // Backward-compatibility with previous versions + // of the errors library: if an error type implements + // SafeMessage(), use that instead of its error message. + p.Print(redact.Safe(v.SafeMessage())) + // It also short-cuts any further causes. + return true, nil + } + return false, nil +} diff --git a/github.com/cockroachdb/errors/errutil/message.go b/github.com/cockroachdb/errors/errutil/message.go index 4d2599b9b7..2ee60b842a 100644 --- a/github.com/cockroachdb/errors/errutil/message.go +++ b/github.com/cockroachdb/errors/errutil/message.go @@ -14,61 +14,32 @@ package errutil -import ( - "context" - "fmt" - - "github.com/cockroachdb/errors/errbase" - "github.com/gogo/protobuf/proto" -) +import "github.com/cockroachdb/redact" // WithMessage annotates err with a new message. // If err is nil, WithMessage returns nil. +// The message is considered safe for reporting +// and is included in Sentry reports. func WithMessage(err error, message string) error { if err == nil { return nil } - return &withMessage{ - cause: err, - msg: message, + return &withPrefix{ + cause: err, + prefix: redact.Sprint(redact.Safe(message)), } } // WithMessagef annotates err with the format specifier. // If err is nil, WithMessagef returns nil. +// The message is formatted as per redact.Sprintf, +// to separate safe and unsafe strings for Sentry reporting. func WithMessagef(err error, format string, args ...interface{}) error { if err == nil { return nil } - return &withMessage{ - cause: err, - msg: fmt.Sprintf(format, args...), + return &withPrefix{ + cause: err, + prefix: redact.Sprintf(format, args...), } } - -type withMessage struct { - cause error - msg string -} - -func (w *withMessage) Error() string { return fmt.Sprintf("%s: %v", w.msg, w.cause) } -func (w *withMessage) Cause() error { return w.cause } -func (w *withMessage) Unwrap() error { return w.cause } - -func (w *withMessage) Format(s fmt.State, verb rune) { errbase.FormatError(w, s, verb) } -func (w *withMessage) FormatError(p errbase.Printer) (next error) { - p.Print(w.msg) - return w.cause -} - -// A message wrapper is decoded exactly. -func decodeMessage( - ctx context.Context, cause error, msg string, _ []string, _ proto.Message, -) error { - return &withMessage{cause: cause, msg: msg} -} - -func init() { - tn := errbase.GetTypeKey((*withMessage)(nil)) - errbase.RegisterWrapperDecoder(tn, decodeMessage) -} diff --git a/github.com/cockroachdb/errors/errutil/redactable.go b/github.com/cockroachdb/errors/errutil/redactable.go new file mode 100644 index 0000000000..6d34f250a5 --- /dev/null +++ b/github.com/cockroachdb/errors/errutil/redactable.go @@ -0,0 +1,163 @@ +package errutil + +import ( + "context" + "fmt" + + "github.com/cockroachdb/errors/errbase" + "github.com/cockroachdb/errors/errorspb" + "github.com/cockroachdb/redact" + "github.com/gogo/protobuf/proto" +) + +// leafError is like the basic error string in the stdlib except the +// message can contain redactable and non-redactable parts. +type leafError struct { + msg redact.RedactableString +} + +var _ error = (*leafError)(nil) +var _ fmt.Formatter = (*leafError)(nil) +var _ errbase.SafeFormatter = (*leafError)(nil) +var _ errbase.SafeDetailer = (*leafError)(nil) + +func (l *leafError) Error() string { return l.msg.StripMarkers() } +func (l *leafError) Format(s fmt.State, verb rune) { errbase.FormatError(l, s, verb) } +func (l *leafError) SafeFormatError(p errbase.Printer) (next error) { + p.Print(l.msg) + return nil +} +func (l *leafError) SafeDetails() []string { + return []string{l.msg.Redact().StripMarkers()} +} + +func encodeLeaf(_ context.Context, err error) (string, []string, proto.Message) { + l := err.(*leafError) + return l.Error(), l.SafeDetails(), &errorspb.StringPayload{Msg: string(l.msg)} +} + +func decodeLeaf(_ context.Context, _ string, _ []string, payload proto.Message) error { + m, ok := payload.(*errorspb.StringPayload) + if !ok { + // If this ever happens, this means some version of the library + // (presumably future) changed the payload type, and we're + // receiving this here. In this case, give up and let + // DecodeError use the opaque type. + return nil + } + return &leafError{msg: redact.RedactableString(m.Msg)} +} + +func init() { + errbase.RegisterLeafEncoder(errbase.GetTypeKey((*leafError)(nil)), encodeLeaf) + errbase.RegisterLeafDecoder(errbase.GetTypeKey((*leafError)(nil)), decodeLeaf) +} + +// withPrefix is like withMessage but the +// message can contain redactable and non-redactable parts. +type withPrefix struct { + cause error + prefix redact.RedactableString +} + +var _ error = (*withPrefix)(nil) +var _ fmt.Formatter = (*withPrefix)(nil) +var _ errbase.SafeFormatter = (*withPrefix)(nil) +var _ errbase.SafeDetailer = (*withPrefix)(nil) + +func (l *withPrefix) Error() string { + if l.prefix == "" { + return l.cause.Error() + } + return fmt.Sprintf("%s: %v", l.prefix.StripMarkers(), l.cause) +} + +func (l *withPrefix) Cause() error { return l.cause } +func (l *withPrefix) Unwrap() error { return l.cause } + +func (l *withPrefix) Format(s fmt.State, verb rune) { errbase.FormatError(l, s, verb) } +func (l *withPrefix) SafeFormatError(p errbase.Printer) (next error) { + p.Print(l.prefix) + return l.cause +} + +func (l *withPrefix) SafeDetails() []string { + return []string{l.prefix.Redact().StripMarkers()} +} + +func encodeWithPrefix(_ context.Context, err error) (string, []string, proto.Message) { + l := err.(*withPrefix) + return l.Error(), l.SafeDetails(), &errorspb.StringPayload{Msg: string(l.prefix)} +} + +func decodeWithPrefix( + _ context.Context, cause error, _ string, _ []string, payload proto.Message, +) error { + m, ok := payload.(*errorspb.StringPayload) + if !ok { + // If this ever happens, this means some version of the library + // (presumably future) changed the payload type, and we're + // receiving this here. In this case, give up and let + // DecodeError use the opaque type. + return nil + } + return &withPrefix{cause: cause, prefix: redact.RedactableString(m.Msg)} +} + +func init() { + errbase.RegisterWrapperEncoder(errbase.GetTypeKey((*withPrefix)(nil)), encodeWithPrefix) + errbase.RegisterWrapperDecoder(errbase.GetTypeKey((*withPrefix)(nil)), decodeWithPrefix) +} + +// withNewMessage is like withPrefix but the message completely +// overrides that of the underlying error. +type withNewMessage struct { + cause error + message redact.RedactableString +} + +var _ error = (*withNewMessage)(nil) +var _ fmt.Formatter = (*withNewMessage)(nil) +var _ errbase.SafeFormatter = (*withNewMessage)(nil) +var _ errbase.SafeDetailer = (*withNewMessage)(nil) + +func (l *withNewMessage) Error() string { + return l.message.StripMarkers() +} + +func (l *withNewMessage) Cause() error { return l.cause } +func (l *withNewMessage) Unwrap() error { return l.cause } + +func (l *withNewMessage) Format(s fmt.State, verb rune) { errbase.FormatError(l, s, verb) } +func (l *withNewMessage) SafeFormatError(p errbase.Printer) (next error) { + p.Print(l.message) + return nil /* nil here overrides the cause's message */ +} + +func (l *withNewMessage) SafeDetails() []string { + return []string{l.message.Redact().StripMarkers()} +} + +func encodeWithNewMessage(_ context.Context, err error) (string, []string, proto.Message) { + l := err.(*withNewMessage) + return l.Error(), l.SafeDetails(), &errorspb.StringPayload{Msg: string(l.message)} +} + +func decodeWithNewMessage( + _ context.Context, cause error, _ string, _ []string, payload proto.Message, +) error { + m, ok := payload.(*errorspb.StringPayload) + if !ok { + // If this ever happens, this means some version of the library + // (presumably future) changed the payload type, and we're + // receiving this here. In this case, give up and let + // DecodeError use the opaque type. + return nil + } + return &withNewMessage{cause: cause, message: redact.RedactableString(m.Msg)} +} + +func init() { + errbase.RegisterWrapperEncoder(errbase.GetTypeKey((*withNewMessage)(nil)), encodeWithNewMessage) + errbase.RegisterWrapperDecoder(errbase.GetTypeKey((*withNewMessage)(nil)), decodeWithNewMessage) +} diff --git a/github.com/cockroachdb/errors/errutil/utilities.go b/github.com/cockroachdb/errors/errutil/utilities.go index 28e1ed29ae..2fd126d74b 100644 --- a/github.com/cockroachdb/errors/errutil/utilities.go +++ b/github.com/cockroachdb/errors/errutil/utilities.go @@ -15,21 +15,24 @@ package errutil import ( - goErr "errors" - "fmt" - - "github.com/cockroachdb/errors/safedetails" + "github.com/cockroachdb/errors/secondary" "github.com/cockroachdb/errors/withstack" + "github.com/cockroachdb/redact" ) // New creates an error with a simple error message. // A stack trace is retained. // +// Note: the message string is assumed to not contain +// PII and is included in Sentry reports. +// Use errors.Newf("%s", ) for errors +// strings that may contain PII information. +// // Detail output: // - message via `Error()` and formatting using `%v`/`%s`/`%q`. // - everything when formatting with `%+v`. -// - stack trace (not message) via `errors.GetSafeDetails()`. -// - stack trace (not message) in Sentry reports. +// - stack trace and message via `errors.GetSafeDetails()`. +// - stack trace and message in Sentry reports. func New(msg string) error { return NewWithDepth(1, msg) } @@ -38,13 +41,19 @@ func New(msg string) error { // trace is configurable. // See the doc of `New()` for more details. func NewWithDepth(depth int, msg string) error { - err := goErr.New(msg) + err := error(&leafError{redact.Sprint(redact.Safe(msg))}) err = withstack.WithStackDepth(err, 1+depth) return err } // Newf creates an error with a formatted error message. // A stack trace is retained. +// +// Note: the format string is assumed to not contain +// PII and is included in Sentry reports. +// Use errors.Newf("%s", ) for errors +// strings that may contain PII information. +// // See the doc of `New()` for more details. func Newf(format string, args ...interface{}) error { return NewWithDepthf(1, format, args...) @@ -54,9 +63,25 @@ func Newf(format string, args ...interface{}) error { // trace is configurable. // See the doc of `New()` for more details. func NewWithDepthf(depth int, format string, args ...interface{}) error { - err := fmt.Errorf(format, args...) - if format != "" || len(args) > 0 { - err = safedetails.WithSafeDetails(err, format, args...) + // If there's the verb %w in here, shortcut to fmt.Errorf() + // and store the safe details as extra payload. That's + // because we don't want to re-implement the error wrapping + // logic from 'fmt' in there. + var err error + var errRefs []error + for _, a := range args { + if e, ok := a.(error); ok { + errRefs = append(errRefs, e) + } + } + redactable, wrappedErr := redact.HelperForErrorf(format, args...) + if wrappedErr != nil { + err = &withNewMessage{cause: wrappedErr, message: redactable} + } else { + err = &leafError{redactable} + } + for _, e := range errRefs { + err = secondary.WithSecondaryError(err, e) } err = withstack.WithStackDepth(err, 1+depth) return err @@ -65,11 +90,16 @@ func NewWithDepthf(depth int, format string, args ...interface{}) error { // Wrap wraps an error with a message prefix. // A stack trace is retained. // +// Note: the prefix string is assumed to not contain +// PII and is included in Sentry reports. +// Use errors.Wrapf(err, "%s", ) for errors +// strings that may contain PII information. +// // Detail output: // - original error message + prefix via `Error()` and formatting using `%v`/`%s`/`%q`. // - everything when formatting with `%+v`. -// - stack trace (not message) via `errors.GetSafeDetails()`. -// - stack trace (not message) in Sentry reports. +// - stack trace and message via `errors.GetSafeDetails()`. +// - stack trace and message in Sentry reports. func Wrap(err error, msg string) error { return WrapWithDepth(1, err, msg) } @@ -78,6 +108,9 @@ func Wrap(err error, msg string) error { // trace is configurable. // The the doc of `Wrap()` for more details. func WrapWithDepth(depth int, err error, msg string) error { + if err == nil { + return nil + } if msg != "" { err = WithMessage(err, msg) } @@ -89,11 +122,16 @@ func WrapWithDepth(depth int, err error, msg string) error { // trace is also retained. If the format is empty, no prefix is added, // but the extra arguments are still processed for reportable strings. // +// Note: the format string is assumed to not contain +// PII and is included in Sentry reports. +// Use errors.Wrapf(err, "%s", ) for errors +// strings that may contain PII information. +// // Detail output: // - original error message + prefix via `Error()` and formatting using `%v`/`%s`/`%q`. // - everything when formatting with `%+v`. -// - stack trace (not message) and redacted details via `errors.GetSafeDetails()`. -// - stack trace (not message) and redacted details in Sentry reports. +// - stack trace, format, and redacted details via `errors.GetSafeDetails()`. +// - stack trace, format, and redacted details in Sentry reports. func Wrapf(err error, format string, args ...interface{}) error { return WrapWithDepthf(1, err, format, args...) } @@ -102,9 +140,20 @@ func Wrapf(err error, format string, args ...interface{}) error { // trace is configurable. // The the doc of `Wrapf()` for more details. func WrapWithDepthf(depth int, err error, format string, args ...interface{}) error { + if err == nil { + return nil + } + var errRefs []error + for _, a := range args { + if e, ok := a.(error); ok { + errRefs = append(errRefs, e) + } + } if format != "" || len(args) > 0 { err = WithMessagef(err, format, args...) - err = safedetails.WithSafeDetails(err, format, args...) + } + for _, e := range errRefs { + err = secondary.WithSecondaryError(err, e) } err = withstack.WithStackDepth(err, depth+1) return err diff --git a/github.com/cockroachdb/errors/errutil_api.go b/github.com/cockroachdb/errors/errutil_api.go index 9deae51416..512bdedb9e 100644 --- a/github.com/cockroachdb/errors/errutil_api.go +++ b/github.com/cockroachdb/errors/errutil_api.go @@ -15,105 +15,180 @@ package errors import ( - "fmt" - "github.com/cockroachdb/errors/barriers" "github.com/cockroachdb/errors/errbase" "github.com/cockroachdb/errors/errutil" ) -// New forwards a definition. +// New creates an error with a simple error message. +// A stack trace is retained. +// +// Note: the message string is assumed to not contain +// PII and is included in Sentry reports. +// Use errors.Newf("%s", ) for errors +// strings that may contain PII information. +// +// Detail output: +// - message via `Error()` and formatting using `%v`/`%s`/`%q`. +// - everything when formatting with `%+v`. +// - stack trace and message via `errors.GetSafeDetails()`. +// - stack trace and message in Sentry reports. func New(msg string) error { return errutil.NewWithDepth(1, msg) } -// NewWithDepth forwards a definition. +// NewWithDepth is like New() except the depth to capture the stack +// trace is configurable. +// See the doc of `New()` for more details. func NewWithDepth(depth int, msg string) error { return errutil.NewWithDepth(depth+1, msg) } -// Newf forwards a definition. +// Newf creates an error with a formatted error message. +// A stack trace is retained. +// +// Note: the format string is assumed to not contain +// PII and is included in Sentry reports. +// Use errors.Newf("%s", ) for errors +// strings that may contain PII information. +// +// See the doc of `New()` for more details. func Newf(format string, args ...interface{}) error { return errutil.NewWithDepthf(1, format, args...) } -// NewWithDepthf forwards a definition. +// NewWithDepthf is like Newf() except the depth to capture the stack +// trace is configurable. +// See the doc of `New()` for more details. func NewWithDepthf(depth int, format string, args ...interface{}) error { return errutil.NewWithDepthf(depth+1, format, args...) } -// Errorf forwards a definition. +// Errorf aliases Newf(). func Errorf(format string, args ...interface{}) error { return errutil.NewWithDepthf(1, format, args...) } -// Cause is provided for compatibility with github.com/pkg/errors. +// Cause aliases UnwrapAll() for compatibility with github.com/pkg/errors. func Cause(err error) error { return errbase.UnwrapAll(err) } -// Unwrap is provided for compatibility with xerrors. +// Unwrap aliases UnwrapOnce() for compatibility with xerrors. func Unwrap(err error) error { return errbase.UnwrapOnce(err) } -// Wrapper is provided for compatibility with xerrors. +// Wrapper is the type of an error wrapper. type Wrapper interface { Unwrap() error } -// Formatter is provided for compatibility with xerrors. -type Formatter = errbase.Formatter - -// Printer is provided for compatibility with xerrors. -type Printer = errbase.Printer - -// FormatError can be used to implement the fmt.Formatter interface. -func FormatError(err error, s fmt.State, verb rune) { errbase.FormatError(err, s, verb) } - -// Opaque is provided for compatibility with xerrors. +// Opaque aliases barrier.Handled(), for compatibility with xerrors. func Opaque(err error) error { return barriers.Handled(err) } -// WithMessage forwards a definition. +// WithMessage annotates err with a new message. +// If err is nil, WithMessage returns nil. +// The message is considered safe for reporting +// and is included in Sentry reports. func WithMessage(err error, msg string) error { return errutil.WithMessage(err, msg) } -// WithMessagef forwards a definition. +// WithMessagef annotates err with the format specifier. +// If err is nil, WithMessagef returns nil. +// The message is formatted as per redact.Sprintf, +// to separate safe and unsafe strings for Sentry reporting. func WithMessagef(err error, format string, args ...interface{}) error { return errutil.WithMessagef(err, format, args...) } -// Wrap forwards a definition. +// Wrap wraps an error with a message prefix. +// A stack trace is retained. +// +// Note: the prefix string is assumed to not contain +// PII and is included in Sentry reports. +// Use errors.Wrapf(err, "%s", ) for errors +// strings that may contain PII information. +// +// Detail output: +// - original error message + prefix via `Error()` and formatting using `%v`/`%s`/`%q`. +// - everything when formatting with `%+v`. +// - stack trace and message via `errors.GetSafeDetails()`. +// - stack trace and message in Sentry reports. func Wrap(err error, msg string) error { return errutil.WrapWithDepth(1, err, msg) } -// WrapWithDepth forwards a definition. +// WrapWithDepth is like Wrap except the depth to capture the stack +// trace is configurable. +// The the doc of `Wrap()` for more details. func WrapWithDepth(depth int, err error, msg string) error { return errutil.WrapWithDepth(depth+1, err, msg) } -// Wrapf forwards a definition. +// Wrapf wraps an error with a formatted message prefix. A stack +// trace is also retained. If the format is empty, no prefix is added, +// but the extra arguments are still processed for reportable strings. +// +// Note: the format string is assumed to not contain +// PII and is included in Sentry reports. +// Use errors.Wrapf(err, "%s", ) for errors +// strings that may contain PII information. +// +// Detail output: +// - original error message + prefix via `Error()` and formatting using `%v`/`%s`/`%q`. +// - everything when formatting with `%+v`. +// - stack trace, format, and redacted details via `errors.GetSafeDetails()`. +// - stack trace, format, and redacted details in Sentry reports. func Wrapf(err error, format string, args ...interface{}) error { return errutil.WrapWithDepthf(1, err, format, args...) } -// WrapWithDepthf forwards a definition. +// WrapWithDepthf is like Wrapf except the depth to capture the stack +// trace is configurable. +// The the doc of `Wrapf()` for more details. func WrapWithDepthf(depth int, err error, format string, args ...interface{}) error { return errutil.WrapWithDepthf(depth+1, err, format, args...) } -// AssertionFailedf forwards a definition. +// AssertionFailedf creates an internal error. +// +// Detail is shown: +// - via `errors.GetSafeDetails()`, shows redacted strings. +// - when formatting with `%+v`. +// - in Sentry reports. func AssertionFailedf(format string, args ...interface{}) error { return errutil.AssertionFailedWithDepthf(1, format, args...) } -// AssertionFailedWithDepthf forwards a definition. +// AssertionFailedWithDepthf creates an internal error +// with a stack trace collected at the specified depth. +// See the doc of `AssertionFailedf()` for more details. func AssertionFailedWithDepthf(depth int, format string, args ...interface{}) error { return errutil.AssertionFailedWithDepthf(depth+1, format, args...) } -// NewAssertionErrorWithWrappedErrf forwards a definition. +// NewAssertionErrorWithWrappedErrf wraps an error and turns it into +// an assertion error. Both details from the original error and the +// context of the caller are preserved. The original error is not +// visible as cause any more. The original error message is preserved. +// See the doc of `AssertionFailedf()` for more details. func NewAssertionErrorWithWrappedErrf(origErr error, format string, args ...interface{}) error { return errutil.NewAssertionErrorWithWrappedErrDepthf(1, origErr, format, args...) } -// HandleAsAssertionFailure forwards a definition. +// HandleAsAssertionFailure hides an error and turns it into +// an assertion failure. Both details from the original error and the +// context of the caller are preserved. The original error is not +// visible as cause any more. The original error message is preserved. +// See the doc of `AssertionFailedf()` for more details. func HandleAsAssertionFailure(origErr error) error { return errutil.HandleAsAssertionFailureDepth(1, origErr) } -// HandleAsAssertionFailureDepth forwards a definition. +// HandleAsAssertionFailureDepth is like HandleAsAssertionFailure but +// the depth at which the call stack is captured can be specified. func HandleAsAssertionFailureDepth(depth int, origErr error) error { return errutil.HandleAsAssertionFailureDepth(1+depth, origErr) } -// As forwards a definition +// As finds the first error in err's chain that matches the type to which target +// points, and if so, sets the target to its value and returns true. An error +// matches a type if it is assignable to the target type, or if it has a method +// As(interface{}) bool such that As(target) returns true. As will panic if target +// is not a non-nil pointer to a type which implements error or is of interface type. +// +// The As method should set the target to its value and return true if err +// matches the type to which target points. +// +// Note: this implementation differs from that of xerrors as follows: +// - it also supports recursing through causes with Cause(). +// - if it detects an API use error, its panic object is a valid error. func As(err error, target interface{}) bool { return errutil.As(err, target) } diff --git a/github.com/cockroachdb/errors/go.mod b/github.com/cockroachdb/errors/go.mod index 1eedc27ecd..3e5cade00f 100644 --- a/github.com/cockroachdb/errors/go.mod +++ b/github.com/cockroachdb/errors/go.mod @@ -3,13 +3,15 @@ module github.com/cockroachdb/errors go 1.13 require ( + github.com/cockroachdb/datadriven v1.0.0 github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f + github.com/cockroachdb/redact v1.0.6 github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2 github.com/gogo/protobuf v1.3.1 github.com/gogo/status v1.1.0 github.com/golang/protobuf v1.4.2 github.com/hydrogen18/memlistener v0.0.0-20141126152155-54553eb933fb github.com/kr/pretty v0.1.0 - github.com/pkg/errors v0.8.1 + github.com/pkg/errors v0.9.1 google.golang.org/grpc v1.29.1 ) diff --git a/github.com/cockroachdb/errors/go.sum b/github.com/cockroachdb/errors/go.sum index 556824e7b6..fd9be7e02e 100644 --- a/github.com/cockroachdb/errors/go.sum +++ b/github.com/cockroachdb/errors/go.sum @@ -12,8 +12,13 @@ github.com/aymerick/raymond v2.0.3-0.20180322193309-b565731e1464+incompatible/go github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= +github.com/cockroachdb/datadriven v1.0.0 h1:uhZrAfEayBecH2w2tZmhe20HJ7hDvrrA4x2Bg9YdZKM= +github.com/cockroachdb/datadriven v1.0.0/go.mod h1:5Ib8Meh+jk1RlHIXej6Pzevx/NLlNvQB9pmSBZErGA4= +github.com/cockroachdb/errors v1.6.1/go.mod h1:tm6FTP5G81vwJ5lC0SizQo374JNCOPrHyXGitRJoDqM= github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f h1:o/kfcElHqOiXqcou5a3rIlMc7oJbMQkeLk0VQJ7zgqY= github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f/go.mod h1:i/u985jwjWRlyHXQbwatDASoW0RMlZ/3i9yJHE2xLkI= +github.com/cockroachdb/redact v1.0.6 h1:W34uRRyNR4dlZFd0MibhNELsZSgMkl52uRV/tA1xToY= +github.com/cockroachdb/redact v1.0.6/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg= github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2 h1:IKgmqgMQlVJIZj19CdocBeSfSaiCbEBZGKODaixqtHM= github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2/go.mod h1:8BT+cPK6xvFOcRlk0R8eg+OTkcqI6baNH4xAkpiYVvQ= github.com/codegangsta/inject v0.0.0-20150114235600-33e0aa1cb7c0/go.mod h1:4Zcjuz89kmFXt9morQgcfYZAYZ5n8WHjt81YYWIwtTM= @@ -141,6 +146,9 @@ github.com/pingcap/errors v0.11.4 h1:lFuQV/oaUMGcD2tqt+01ROSmJs75VG1ToEOkZIZ4nE4 github.com/pingcap/errors v0.11.4/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8= github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g= diff --git a/github.com/cockroachdb/errors/hintdetail_api.go b/github.com/cockroachdb/errors/hintdetail_api.go index 8df54e4cb7..623bf64af2 100644 --- a/github.com/cockroachdb/errors/hintdetail_api.go +++ b/github.com/cockroachdb/errors/hintdetail_api.go @@ -16,36 +16,64 @@ package errors import "github.com/cockroachdb/errors/hintdetail" -// ErrorHinter forwards a definition. +// ErrorHinter is implemented by types that can provide +// user-informing detail strings. This is implemented by withHint +// here, withIssueLink, assertionFailure and pgerror.Error. type ErrorHinter = hintdetail.ErrorHinter -// ErrorDetailer forwards a definition. +// ErrorDetailer is implemented by types that can provide +// user-informing detail strings. type ErrorDetailer = hintdetail.ErrorDetailer -// WithHint forwards a definition. +// WithHint decorates an error with a textual hint. +// The hint may contain PII and thus will not reportable. +// +// Hint is shown: +// - when formatting with `%+v`. +// - with `GetAllHints()` / `FlattenHints()` below. +// +// Note: the hint does not appear in the main error message returned +// with Error(). Use GetAllHints() or FlattenHints() to retrieve it. func WithHint(err error, msg string) error { return hintdetail.WithHint(err, msg) } -// WithHintf forwards a definition. +// WithHintf is a helper that formats the hint. +// See the documentation of WithHint() for details. func WithHintf(err error, format string, args ...interface{}) error { return hintdetail.WithHintf(err, format, args...) } -// WithDetail forwards a definition. +// WithDetail decorates an error with a textual detail. +// The detail may contain PII and thus will not reportable. +// +// Detail is shown: +// - when formatting with `%+v`. +// - with `GetAllHints()` / `FlattenHints()` below. +// +// Note: the detail does not appear in the main error message returned +// with Error(). Use GetAllDetails() or FlattenDetails() to retrieve +// it. func WithDetail(err error, msg string) error { return hintdetail.WithDetail(err, msg) } -// WithDetailf forwards a definition. +// WithDetailf is a helper that formats the detail string. +// See the documentation of WithDetail() for details. func WithDetailf(err error, format string, args ...interface{}) error { return hintdetail.WithDetailf(err, format, args...) } -// GetAllHints forwards a definition. +// GetAllHints retrieves the hints from the error using in post-order +// traversal. The hints are de-duplicated. Assertion failures, issue +// links and unimplemented errors are detected and receive standard +// hints. func GetAllHints(err error) []string { return hintdetail.GetAllHints(err) } -// FlattenHints forwards a definition. +// FlattenHints retrieves the hints as per GetAllHints() and +// concatenates them into a single string. func FlattenHints(err error) string { return hintdetail.FlattenHints(err) } -// GetAllDetails forwards a definition. +// GetAllDetails retrieves the details from the error using in post-order +// traversal. func GetAllDetails(err error) []string { return hintdetail.GetAllDetails(err) } -// FlattenDetails forwards a definition. +// FlattenDetails retrieves the details as per GetAllDetails() and +// concatenates them into a single string. func FlattenDetails(err error) string { return hintdetail.FlattenDetails(err) } diff --git a/github.com/cockroachdb/errors/issuelink/unimplemented_error.go b/github.com/cockroachdb/errors/issuelink/unimplemented_error.go index 2611b87978..af64df12a4 100644 --- a/github.com/cockroachdb/errors/issuelink/unimplemented_error.go +++ b/github.com/cockroachdb/errors/issuelink/unimplemented_error.go @@ -20,17 +20,19 @@ import ( "fmt" "github.com/cockroachdb/errors/errbase" + "github.com/cockroachdb/redact" "github.com/gogo/protobuf/proto" ) type unimplementedError struct { + // For now, msg is non-reportable. msg string IssueLink } var _ error = (*unimplementedError)(nil) var _ fmt.Formatter = (*unimplementedError)(nil) -var _ errbase.Formatter = (*unimplementedError)(nil) +var _ errbase.SafeFormatter = (*unimplementedError)(nil) var _ errbase.SafeDetailer = (*unimplementedError)(nil) func (w *unimplementedError) Error() string { return w.msg } @@ -51,15 +53,17 @@ const UnimplementedErrorHint = `You have attempted to use a feature that is not func (w *unimplementedError) Format(s fmt.State, verb rune) { errbase.FormatError(w, s, verb) } -func (w *unimplementedError) FormatError(p errbase.Printer) error { +func (w *unimplementedError) SafeFormatError(p errbase.Printer) error { + // For now, msg is non-reportable. p.Print(w.msg) if p.Detail() { - p.Print("unimplemented") + // But the details are. + p.Printf("unimplemented") if w.IssueURL != "" { - p.Printf("\nissue: %s", w.IssueURL) + p.Printf("\nissue: %s", redact.Safe(w.IssueURL)) } if w.Detail != "" { - p.Printf("\ndetail: %s", w.Detail) + p.Printf("\ndetail: %s", redact.Safe(w.Detail)) } } return nil diff --git a/github.com/cockroachdb/errors/issuelink/with_issuelink.go b/github.com/cockroachdb/errors/issuelink/with_issuelink.go index 83c5d56543..422333820b 100644 --- a/github.com/cockroachdb/errors/issuelink/with_issuelink.go +++ b/github.com/cockroachdb/errors/issuelink/with_issuelink.go @@ -21,6 +21,7 @@ import ( "github.com/cockroachdb/errors/errbase" "github.com/cockroachdb/errors/stdstrings" + "github.com/cockroachdb/redact" "github.com/gogo/protobuf/proto" ) @@ -32,7 +33,7 @@ type withIssueLink struct { var _ error = (*withIssueLink)(nil) var _ errbase.SafeDetailer = (*withIssueLink)(nil) var _ fmt.Formatter = (*withIssueLink)(nil) -var _ errbase.Formatter = (*withIssueLink)(nil) +var _ errbase.SafeFormatter = (*withIssueLink)(nil) func (w *withIssueLink) Error() string { return w.cause.Error() } func (w *withIssueLink) Cause() error { return w.cause } @@ -64,15 +65,15 @@ func maybeAppendReferral(buf *bytes.Buffer, link IssueLink) { func (w *withIssueLink) Format(s fmt.State, verb rune) { errbase.FormatError(w, s, verb) } -func (w *withIssueLink) FormatError(p errbase.Printer) error { +func (w *withIssueLink) SafeFormatError(p errbase.Printer) error { if p.Detail() { - sep := "" + sep := redact.SafeString("") if w.IssueURL != "" { - p.Printf("issue: %s", w.IssueURL) + p.Printf("issue: %s", redact.Safe(w.IssueURL)) sep = "\n" } if w.Detail != "" { - p.Printf("%sdetail: %s", sep, w.Detail) + p.Printf("%sdetail: %s", sep, redact.Safe(w.Detail)) } } return w.cause diff --git a/github.com/cockroachdb/errors/issuelink_api.go b/github.com/cockroachdb/errors/issuelink_api.go index 22f6db13da..431915864b 100644 --- a/github.com/cockroachdb/errors/issuelink_api.go +++ b/github.com/cockroachdb/errors/issuelink_api.go @@ -16,36 +16,58 @@ package errors import "github.com/cockroachdb/errors/issuelink" -// WithIssueLink forwards a definition. +// WithIssueLink adds an annotation to a know issue +// on a web issue tracker. +// +// The url and detail strings may contain PII and will +// be considered reportable. +// +// Detail is shown: +// - via `errors.GetSafeDetails()` +// - when formatting with `%+v`. +// - in Sentry reports. +// - via `errors.GetAllHints()` / `errors.FlattenHints()` func WithIssueLink(err error, issue IssueLink) error { return issuelink.WithIssueLink(err, issue) } -// IssueLink forwards a definition. +// IssueLink is the payload for a linked issue annotation. type IssueLink = issuelink.IssueLink -// UnimplementedError forwards a definition. +// UnimplementedError creates a new leaf error that indicates that +// some feature was not (yet) implemented. +// +// Detail is shown: +// - via `errors.GetSafeDetails()` +// - when formatting with `%+v`. +// - in Sentry reports. +// - via `errors.GetAllHints()` / `errors.FlattenHints()` func UnimplementedError(issueLink IssueLink, msg string) error { return issuelink.UnimplementedError(issueLink, msg) } -// UnimplementedErrorf forwards a definition. +// UnimplementedErrorf creates a new leaf error that indicates that +// some feature was not (yet) implemented. The message is formatted. func UnimplementedErrorf(issueLink IssueLink, format string, args ...interface{}) error { return issuelink.UnimplementedErrorf(issueLink, format, args...) } -// GetAllIssueLinks forwards a definition. +// GetAllIssueLinks retrieves the linked issue carried +// by the error or its direct causes. func GetAllIssueLinks(err error) (issues []IssueLink) { return issuelink.GetAllIssueLinks(err) } -// HasIssueLink forwards a definition. +// HasIssueLink returns true iff the error or one of its +// causes has a linked issue payload. func HasIssueLink(err error) bool { return issuelink.HasIssueLink(err) } -// IsIssueLink forwards a definition. +// IsIssueLink returns true iff the error (not its +// causes) has a linked issue payload. func IsIssueLink(err error) bool { return issuelink.IsIssueLink(err) } -// HasUnimplementedError forwards a definition. +// HasUnimplementedError returns iff if err or its cause is an +// unimplemented error. func HasUnimplementedError(err error) bool { return issuelink.HasUnimplementedError(err) } -// IsUnimplementedError forwards a definition. +// IsUnimplementedError returns iff if err is an unimplemented error. func IsUnimplementedError(err error) bool { return issuelink.IsUnimplementedError(err) } -// UnimplementedErrorHint forwards a definition. +// UnimplementedErrorHint is the hint emitted upon unimplemented errors. const UnimplementedErrorHint = issuelink.UnimplementedErrorHint diff --git a/github.com/cockroachdb/errors/markers/markers.go b/github.com/cockroachdb/errors/markers/markers.go index 75139b4b13..8a2c2e62ec 100644 --- a/github.com/cockroachdb/errors/markers/markers.go +++ b/github.com/cockroachdb/errors/markers/markers.go @@ -21,6 +21,7 @@ import ( "github.com/cockroachdb/errors/errbase" "github.com/cockroachdb/errors/errorspb" + "github.com/cockroachdb/redact" "github.com/gogo/protobuf/proto" ) @@ -97,7 +98,8 @@ func getInterfaceType(context string, referenceInterface interface{}) reflect.Ty return typ.Elem() } -// If returns a predicate's return value the first time the predicate returns true. +// If iterates on the error's causal chain and returns a predicate's +// return value the first time the predicate returns true. // // Note: if any of the error types has been migrated from a previous // package location or a different type, ensure that @@ -225,7 +227,7 @@ type withMark struct { var _ error = (*withMark)(nil) var _ fmt.Formatter = (*withMark)(nil) -var _ errbase.Formatter = (*withMark)(nil) +var _ errbase.SafeFormatter = (*withMark)(nil) func (m *withMark) Error() string { return m.cause.Error() } func (m *withMark) Cause() error { return m.cause } @@ -233,13 +235,13 @@ func (m *withMark) Unwrap() error { return m.cause } func (m *withMark) Format(s fmt.State, verb rune) { errbase.FormatError(m, s, verb) } -func (m *withMark) FormatError(p errbase.Printer) error { +func (m *withMark) SafeFormatError(p errbase.Printer) error { if p.Detail() { - p.Print("forced error mark\n") + p.Printf("forced error mark\n") p.Printf("%q\n%s::%s", m.mark.msg, - m.mark.types[0].FamilyName, - m.mark.types[0].Extension, + redact.Safe(m.mark.types[0].FamilyName), + redact.Safe(m.mark.types[0].Extension), ) } return m.cause diff --git a/github.com/cockroachdb/errors/markers_api.go b/github.com/cockroachdb/errors/markers_api.go index aced76fbbb..64c8f29415 100644 --- a/github.com/cockroachdb/errors/markers_api.go +++ b/github.com/cockroachdb/errors/markers_api.go @@ -16,24 +16,47 @@ package errors import "github.com/cockroachdb/errors/markers" -// Is forwards a definition. +// Is determines whether one of the causes of the given error or any +// of its causes is equivalent to some reference error. +// +// Note: if any of the error types has been migrated from a previous +// package location or a different type, ensure that +// RegisterTypeMigration() was called prior to Is(). func Is(err, reference error) bool { return markers.Is(err, reference) } -// HasType forwards a definition. +// HasType returns true iff err contains an error whose concrete type +// matches that of referenceType. func HasType(err, referenceType error) bool { return markers.HasType(err, referenceType) } -// HasInterface forwards a definition. +// HasInterface returns true if err contains an error which implements the +// interface pointed to by referenceInterface. The type of referenceInterface +// must be a pointer to an interface type. If referenceInterface is not a +// pointer to an interface, this function will panic. func HasInterface(err error, referenceInterface interface{}) bool { return markers.HasInterface(err, referenceInterface) } -// If forwards a definition. +// If iterates on the error's causal chain and returns a predicate's +// return value the first time the predicate returns true. +// +// Note: if any of the error types has been migrated from a previous +// package location or a different type, ensure that +// RegisterTypeMigration() was called prior to If(). func If(err error, pred func(err error) (interface{}, bool)) (interface{}, bool) { return markers.If(err, pred) } -// IsAny forwards a definition. +// IsAny is like Is except that multiple references are compared. +// +// Note: if any of the error types has been migrated from a previous +// package location or a different type, ensure that +// RegisterTypeMigration() was called prior to IsAny(). func IsAny(err error, references ...error) bool { return markers.IsAny(err, references...) } -// Mark forwards a definition. +// Mark creates an explicit mark for the given error, using +// the same mark as some reference error. +// +// Note: if any of the error types has been migrated from a previous +// package location or a different type, ensure that +// RegisterTypeMigration() was called prior to Mark(). func Mark(err error, reference error) error { return markers.Mark(err, reference) } diff --git a/github.com/cockroachdb/errors/report/report.go b/github.com/cockroachdb/errors/report/report.go index c793b234f4..33d507ae7c 100644 --- a/github.com/cockroachdb/errors/report/report.go +++ b/github.com/cockroachdb/errors/report/report.go @@ -22,6 +22,7 @@ import ( "github.com/cockroachdb/errors/domains" "github.com/cockroachdb/errors/errbase" "github.com/cockroachdb/errors/withstack" + "github.com/cockroachdb/redact" "github.com/cockroachdb/sentry-go" ) @@ -111,13 +112,24 @@ func BuildSentryReport(err error) (event *sentry.Event, extraDetails map[string] } module := string(domains.GetDomain(err)) - // For the summary, we collect the innermost source context. - // file, line, fn, hasOneLineSource := withstack.GetOneLineSource(err) + // firstDetailLine is the first detail string encountered. + // This is added as decoration to the first Exception + // payload (either from the error object or synthetic) + // so as to populate the Sentry report title. + var firstDetailLine string // longMsgBuf will become the Message field, which contains the full // structure of the error with cross-references to "Exception" and // "Additional data" fields. var longMsgBuf strings.Builder + redactedErrStr := redact.Sprint(err).Redact() + if redactedErrStr != redactedMarker { + if f, l, _, ok := withstack.GetOneLineSource(err); ok { + fmt.Fprintf(&longMsgBuf, "%s:%d: ", f, l) + } + firstDetailLine = redactedErrStr.StripMarkers() + fmt.Fprintf(&longMsgBuf, "%v\n--\n", firstDetailLine) + } // sep is used to separate the entries in the longMsgBuf / Message // payload. @@ -143,12 +155,6 @@ func BuildSentryReport(err error) (event *sentry.Event, extraDetails map[string] // This is used as fallback when no Exception payload is generated. var leafErrorType string - // firstDetailLine is the first detail string encountered. - // This is added as decoration to the first Exception - // payload (either from the error object or synthetic) - // so as to populate the Sentry report title. - var firstDetailLine string - // Iterate from the last (innermost) to first (outermost) error // layer. We iterate in this order because we want to describe the // error from innermost to outermost layer in longMsgBuf and @@ -269,7 +275,7 @@ func BuildSentryReport(err error) (event *sentry.Event, extraDetails map[string] stKey := fmt.Sprintf("%d: details", extraNum) var extraStr bytes.Buffer for _, d := range details[i].SafeDetails { - fmt.Fprintln(&extraStr, d) + fmt.Fprintln(&extraStr, strings.ReplaceAll(d, "\n", "\n ")) } extras[stKey] = extraStr.String() fmt.Fprintf(&longMsgBuf, " (%d)", extraNum) @@ -348,8 +354,13 @@ func BuildSentryReport(err error) (event *sentry.Event, extraDetails map[string] return event, extras } +var redactedMarker = redact.RedactableString(redact.RedactedMarker()) + // ReportError reports the given error to Sentry. The caller is responsible for -// checking whether telemetry is enabled. +// checking whether telemetry is enabled, and calling the sentry.Flush() +// function to wait for the report to be uploaded. (By default, +// Sentry submits reports asynchronously.) +// // Note: an empty 'eventID' can be returned which signifies that the error was // not reported. This can occur when Sentry client hasn't been properly // configured or Sentry client decided to not report the error (due to diff --git a/github.com/cockroachdb/errors/report_api.go b/github.com/cockroachdb/errors/report_api.go index 2e4d60ffc6..c346990ff4 100644 --- a/github.com/cockroachdb/errors/report_api.go +++ b/github.com/cockroachdb/errors/report_api.go @@ -19,10 +19,85 @@ import ( "github.com/cockroachdb/sentry-go" ) -// BuildSentryReport forwards a definition. +// BuildSentryReport builds the components of a sentry report. This +// can be used instead of ReportError() below to use additional custom +// conditions in the reporting or add additional reporting tags. +// +// The Sentry Event is populated for maximal utility when exploited in +// the Sentry.io web interface and database. +// +// A Sentry report is displayed visually in the Sentry UI as follows: +// +//////////////// +// Title: (1) some prefix in bold (2) one line for a stack trace +// (3) a single-line subtitle +// +// (4) the tags, as a tag soup (concatenated in a single paragrah, +// unsorted) +// +// (5) a "message" +// +// (6) zero or more "exceptions", each composed of: +// (7) a bold title +// (8) some freeform text +// (9) a stack trace +// +// (10) metadata fields: environment, arch, etc +// +// (11) "Additional data" fields +// +// (12) SDK version +///////////////// +// +// These visual items map to the Sentry Event object as follows: +// +// (1) the Type field of the 1st Exception object, if any +// otherwise the Message field +// (2) the topmost entry from the Stacktrace field of the 1st Exception object, if any +// (3) the Value field of the 1st Exception object, if any, unwrapped as a single line +// (4) the Tags field +// (5) the Message field +// (7) the Type field (same as (1) for 1st execption) +// (8) the Value field (same as (3) for 1st exception) +// (9) the Stacktrace field (input to (2) on 1st exception) +// (10) the other fields on the Event object +// (11) the Extra field +// +// (Note how the top-level title fields (1) (3) are unrelated to the +// Message field in the event, which is surprising!) +// +// Given this mapping, an error object is decomposed as follows: +// +// (1)/(7): : () +// (3)/(8): : +// (4): not populated in this function, caller is to manage this +// (5): detailed structure of the entire error object, with references to "additional data" +// and additional "exception" objects +// (9): generated from innermost stack trace +// (6): every exception object after the 1st reports additional stack trace contexts +// (11): "additional data" populated from safe detail payloads +// +// If there is no stack trace in the error, a synthetic Exception +// object is still produced to provide visual detail in the Sentry UI. +// +// Note that if a layer in the error has both a stack trace (ie +// provides the `StackTrace()` interface) and also safe details +// (`SafeDetails()`) other than the stack trace, only the stack trace +// is included in the Sentry report. This does not affect error types +// provided by the library, but could impact error types defined by +// 3rd parties. This limitation may be lifted in a later version. +// func BuildSentryReport(err error) (*sentry.Event, map[string]interface{}) { return report.BuildSentryReport(err) } -// ReportError forwards a definition. +// ReportError reports the given error to Sentry. The caller is responsible for +// checking whether telemetry is enabled, and calling the sentry.Flush() +// function to wait for the report to be uploaded. (By default, +// Sentry submits reports asynchronously.) +// +// Note: an empty 'eventID' can be returned which signifies that the error was +// not reported. This can occur when Sentry client hasn't been properly +// configured or Sentry client decided to not report the error (due to +// configured sampling rate, callbacks, Sentry's event processors, etc). func ReportError(err error) string { return report.ReportError(err) } diff --git a/github.com/cockroachdb/errors/safedetails/redact.go b/github.com/cockroachdb/errors/safedetails/redact.go index d4d4f5d127..d9ce8b6b7d 100644 --- a/github.com/cockroachdb/errors/safedetails/redact.go +++ b/github.com/cockroachdb/errors/safedetails/redact.go @@ -14,124 +14,12 @@ package safedetails -import ( - "context" - "fmt" - "net" - "os" - "runtime" - "strings" - "syscall" - - "github.com/cockroachdb/errors/errbase" - "github.com/cockroachdb/errors/markers" - "github.com/cockroachdb/errors/withstack" -) +import "github.com/cockroachdb/redact" // Redact returns a redacted version of the supplied item that is safe to use in // anonymized reporting. +// +// NB: this interface is obsolete. Use redact.Sprint() directly. func Redact(r interface{}) string { - var buf strings.Builder - - switch t := r.(type) { - case SafeMessager: - buf.WriteString(t.SafeMessage()) - case error: - if file, line, _, ok := withstack.GetOneLineSource(t); ok { - fmt.Fprintf(&buf, "%s:%d: ", file, line) - } - redactErr(&buf, t) - default: - typAnd(&buf, r, "") - } - - return buf.String() -} - -func redactErr(buf *strings.Builder, err error) { - if c := errbase.UnwrapOnce(err); c == nil { - // This is a leaf error. Decode the leaf and return. - redactLeafErr(buf, err) - } else /* c != nil */ { - // Print the inner error before the outer error. - redactErr(buf, c) - redactWrapper(buf, err) - } - - // Add any additional safe strings from the wrapper, if present. - if payload := errbase.GetSafeDetails(err); len(payload.SafeDetails) > 0 { - buf.WriteString("\n(more details:)") - for _, sd := range payload.SafeDetails { - buf.WriteByte('\n') - buf.WriteString(strings.TrimSpace(sd)) - } - } -} - -func redactWrapper(buf *strings.Builder, err error) { - buf.WriteString("\nwrapper: ") - switch t := err.(type) { - case *os.SyscallError: - typAnd(buf, t, t.Syscall) - case *os.PathError: - typAnd(buf, t, t.Op) - case *os.LinkError: - fmt.Fprintf(buf, "%T: %s ", t, t.Op) - case *net.OpError: - typAnd(buf, t, t.Op) - if t.Net != "" { - fmt.Fprintf(buf, " %s", t.Net) - } - if t.Source != nil { - buf.WriteString("") - } - if t.Addr != nil { - if t.Source != nil { - buf.WriteString("->") - } - buf.WriteString("") - } - default: - typAnd(buf, err, "") - } -} - -func redactLeafErr(buf *strings.Builder, err error) { - // Is it a sentinel error? These are safe. - if markers.IsAny(err, - context.DeadlineExceeded, - context.Canceled, - os.ErrInvalid, - os.ErrPermission, - os.ErrExist, - os.ErrNotExist, - os.ErrClosed, - os.ErrNoDeadline, - ) { - typAnd(buf, err, err.Error()) - return - } - - if redactPre113Wrappers(buf, err) { - return - } - - // The following two types are safe too. - switch t := err.(type) { - case runtime.Error: - typAnd(buf, t, t.Error()) - case syscall.Errno: - typAnd(buf, t, t.Error()) - default: - // No further information about this error, simply report its type. - typAnd(buf, err, "") - } -} - -func typAnd(buf *strings.Builder, r interface{}, msg string) { - if msg == "" { - fmt.Fprintf(buf, "<%T>", r) - } else { - fmt.Fprintf(buf, "%T: %s", r, msg) - } + return redact.Sprint(r).Redact().StripMarkers() } diff --git a/github.com/cockroachdb/errors/safedetails/redact_go112.go b/github.com/cockroachdb/errors/safedetails/redact_go112.go deleted file mode 100644 index fca1e24cb5..0000000000 --- a/github.com/cockroachdb/errors/safedetails/redact_go112.go +++ /dev/null @@ -1,48 +0,0 @@ -// Copyright 2019 The Cockroach Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or -// implied. See the License for the specific language governing -// permissions and limitations under the License. - -// +build !go1.13 - -package safedetails - -import ( - "net" - "os" - "strings" -) - -func redactPre113Wrappers(buf *strings.Builder, err error) bool { - // The following cases are needed for go 1.12 and previous - // versions. Go 1.13 instances of these errors will be recognized - // by the unwrapping performed in redactErr(). - switch t := err.(type) { - case *os.SyscallError: - redactErr(buf, t.Err) - redactWrapper(buf, err) - return true - case *os.PathError: - redactErr(buf, t.Err) - redactWrapper(buf, err) - return true - case *net.OpError: - redactErr(buf, t.Err) - redactWrapper(buf, err) - return true - case *os.LinkError: - redactErr(buf, t.Err) - redactWrapper(buf, err) - return true - } - return false -} diff --git a/github.com/cockroachdb/errors/safedetails/redact_go113.go b/github.com/cockroachdb/errors/safedetails/redact_go113.go deleted file mode 100644 index 5c4cd5ef10..0000000000 --- a/github.com/cockroachdb/errors/safedetails/redact_go113.go +++ /dev/null @@ -1,23 +0,0 @@ -// Copyright 2019 The Cockroach Authors. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or -// implied. See the License for the specific language governing -// permissions and limitations under the License. - -// +build go1.13 - -package safedetails - -import "strings" - -func redactPre113Wrappers(buf *strings.Builder, err error) bool { - return false -} diff --git a/github.com/cockroachdb/errors/safedetails/safedetails.go b/github.com/cockroachdb/errors/safedetails/safedetails.go index b3cd35495d..bf60350df8 100644 --- a/github.com/cockroachdb/errors/safedetails/safedetails.go +++ b/github.com/cockroachdb/errors/safedetails/safedetails.go @@ -14,7 +14,11 @@ package safedetails -import "fmt" +import ( + "reflect" + + "github.com/cockroachdb/redact" +) // WithSafeDetails annotates an error with the given reportable details. // The format is made available as a PII-free string, alongside @@ -22,6 +26,9 @@ import "fmt" // Arguments can be reported as-is (without redaction) by wrapping // them using the Safe() function. // +// If the format is empty and there are no arguments, the +// error argument is returned unchanged. +// // Detail is shown: // - via `errors.GetSafeDetails()` // - when formatting with `%+v`. @@ -30,53 +37,28 @@ func WithSafeDetails(err error, format string, args ...interface{}) error { if err == nil { return nil } - - details := make([]string, 1, 1+len(args)) - details[0] = format - for i, a := range args { - details = append(details, fmt.Sprintf("-- arg %d: %s", i+1, Redact(a))) + if len(format) == 0 && len(args) == 0 { + return err + } + details := []string{ + redact.Sprintf(format, args...).Redact().StripMarkers(), } return &withSafeDetails{cause: err, safeDetails: details} } -// SafeMessager is implemented by objects which have a way of representing -// themselves suitably redacted for anonymized reporting. -type SafeMessager interface { - SafeMessage() string -} +var refSafeType = reflect.TypeOf(redact.Safe("")) + +// SafeMessager is implemented by objects which have a way of +// representing themselves suitably redacted for anonymized reporting. +// +// NB: this interface is obsolete. Use redact.SafeFormatter instead. +type SafeMessager = redact.SafeMessager // Safe wraps the given object into an opaque struct that implements // SafeMessager: its contents can be included as-is in PII-free // strings in error objects and reports. -func Safe(v interface{}) SafeMessager { - return safeType{V: v} -} - -// A safeType panic can be reported verbatim, i.e. does not leak -// information. A nil `*safeType` is not valid for use and may cause -// panics. -type safeType struct { - V interface{} -} - -var _ SafeMessager = safeType{} - -// SafeMessage implements SafeMessager. -func (st safeType) SafeMessage() string { - return fmt.Sprintf("%+v", st.V) -} - -// safeType implements fmt.Stringer as a convenience. -func (st safeType) String() string { - return st.SafeMessage() -} - -// Format implements fmt.Formatter. -func (st safeType) Format(s fmt.State, verb rune) { - flags := "" - if s.Flag('+') { - flags += "+" - } - fmtString := fmt.Sprintf("%%%s%c", flags, verb) - fmt.Fprintf(s, fmtString, st.V) +// +// NB: this is obsolete. Use redact.Safe instead. +func Safe(v interface{}) redact.SafeValue { + return redact.Safe(v) } diff --git a/github.com/cockroachdb/errors/safedetails/with_safedetails.go b/github.com/cockroachdb/errors/safedetails/with_safedetails.go index f710925f29..03ba7e5151 100644 --- a/github.com/cockroachdb/errors/safedetails/with_safedetails.go +++ b/github.com/cockroachdb/errors/safedetails/with_safedetails.go @@ -19,6 +19,7 @@ import ( "fmt" "github.com/cockroachdb/errors/errbase" + "github.com/cockroachdb/redact" "github.com/gogo/protobuf/proto" ) @@ -33,17 +34,29 @@ func (e *withSafeDetails) SafeDetails() []string { } var _ fmt.Formatter = (*withSafeDetails)(nil) +var _ errbase.SafeFormatter = (*withSafeDetails)(nil) // Printing a withSecondary reveals the details. func (e *withSafeDetails) Format(s fmt.State, verb rune) { errbase.FormatError(e, s, verb) } -func (e *withSafeDetails) FormatError(p errbase.Printer) error { +// SafeFormatError implements errbase.SafeFormatter. +func (e *withSafeDetails) SafeFormatError(p errbase.Printer) error { if p.Detail() { - plural := "s" - if len(e.safeDetails) == 1 { - plural = "" + comma := redact.SafeString("") + if len(e.safeDetails) != 1 { + plural := redact.SafeString("s") + if len(e.safeDetails) == 1 { + plural = "" + } + p.Printf("%d safe detail%s enclosed", redact.Safe(len(e.safeDetails)), plural) + comma = "\n" + } + // We hide the details from %+v; they are included + // during Sentry reporting. + for _, s := range e.safeDetails { + p.Printf("%s%s", comma, redact.Safe(s)) + comma = "\n" } - p.Printf("%d safe detail%s enclosed", len(e.safeDetails), plural) } return e.cause } diff --git a/github.com/cockroachdb/errors/safedetails_api.go b/github.com/cockroachdb/errors/safedetails_api.go index e4c0ee1584..4caac9d4f2 100644 --- a/github.com/cockroachdb/errors/safedetails_api.go +++ b/github.com/cockroachdb/errors/safedetails_api.go @@ -14,19 +14,43 @@ package errors -import "github.com/cockroachdb/errors/safedetails" +import ( + "github.com/cockroachdb/errors/safedetails" + "github.com/cockroachdb/redact" +) -// WithSafeDetails forwards a definition. +// WithSafeDetails annotates an error with the given reportable details. +// The format is made available as a PII-free string, alongside +// with a PII-free representation of every additional argument. +// Arguments can be reported as-is (without redaction) by wrapping +// them using the Safe() function. +// +// If the format is empty and there are no arguments, the +// error argument is returned unchanged. +// +// Detail is shown: +// - via `errors.GetSafeDetails()` +// - when formatting with `%+v`. +// - in Sentry reports. func WithSafeDetails(err error, format string, args ...interface{}) error { return safedetails.WithSafeDetails(err, format, args...) } -// SafeMessager forwards a definition. -type SafeMessager = safedetails.SafeMessager +// SafeMessager aliases redact.SafeMessager. +// +// NB: this is obsolete. Use redact.SafeFormatter or +// errors.SafeFormatter instead. +type SafeMessager = redact.SafeMessager -// Safe forwards a definition. -func Safe(v interface{}) SafeMessager { return safedetails.Safe(v) } +// Safe wraps the given object into an opaque struct that implements +// SafeMessager: its contents can be included as-is in PII-free +// strings in error objects and reports. +// +// NB: this is obsolete. Use redact.Safe instead. +func Safe(v interface{}) redact.SafeValue { return safedetails.Safe(v) } -// Redact returns a redacted version of the supplied item that is safe -// to use in anonymized reporting. +// Redact returns a redacted version of the supplied item that is safe to use in +// anonymized reporting. +// +// NB: this interface is obsolete. Use redact.Sprint() directly. func Redact(r interface{}) string { return safedetails.Redact(r) } diff --git a/github.com/cockroachdb/errors/secondary/with_secondary.go b/github.com/cockroachdb/errors/secondary/with_secondary.go index e2563cd8b8..5822b2c1cb 100644 --- a/github.com/cockroachdb/errors/secondary/with_secondary.go +++ b/github.com/cockroachdb/errors/secondary/with_secondary.go @@ -33,7 +33,7 @@ type withSecondaryError struct { var _ error = (*withSecondaryError)(nil) var _ errbase.SafeDetailer = (*withSecondaryError)(nil) var _ fmt.Formatter = (*withSecondaryError)(nil) -var _ errbase.Formatter = (*withSecondaryError)(nil) +var _ errbase.SafeFormatter = (*withSecondaryError)(nil) // SafeDetails reports the PII-free details from the secondary error. func (e *withSecondaryError) SafeDetails() []string { @@ -48,10 +48,9 @@ func (e *withSecondaryError) SafeDetails() []string { // Printing a withSecondary reveals the details. func (e *withSecondaryError) Format(s fmt.State, verb rune) { errbase.FormatError(e, s, verb) } -func (e *withSecondaryError) FormatError(p errbase.Printer) (next error) { +func (e *withSecondaryError) SafeFormatError(p errbase.Printer) (next error) { if p.Detail() { - p.Print("secondary error attachment\n") - p.Printf("%+v", e.secondaryError) + p.Printf("secondary error attachment\n%+v", e.secondaryError) } return e.cause } diff --git a/github.com/cockroachdb/errors/secondary_api.go b/github.com/cockroachdb/errors/secondary_api.go index 3b74d9a832..2f5e34f39b 100644 --- a/github.com/cockroachdb/errors/secondary_api.go +++ b/github.com/cockroachdb/errors/secondary_api.go @@ -16,12 +16,27 @@ package errors import "github.com/cockroachdb/errors/secondary" -// WithSecondaryError forwards a definition. +// WithSecondaryError enhances the error given as first argument with +// an annotation that carries the error given as second argument. The +// second error does not participate in cause analysis (Is, etc) and +// is only revealed when printing out the error or collecting safe +// (PII-free) details for reporting. +// +// If additionalErr is nil, the first error is returned as-is. +// +// Tip: consider using CombineErrors() below in the general case. +// +// Detail is shown: +// - via `errors.GetSafeDetails()`, shows details from secondary error. +// - when formatting with `%+v`. +// - in Sentry reports. func WithSecondaryError(err error, additionalErr error) error { return secondary.WithSecondaryError(err, additionalErr) } -// CombineErrors forwards a definition. +// CombineErrors returns err, or, if err is nil, otherErr. +// if err is non-nil, otherErr is attached as secondary error. +// See the documentation of `WithSecondaryError()` for details. func CombineErrors(err, otherErr error) error { return secondary.CombineErrors(err, otherErr) } diff --git a/github.com/cockroachdb/errors/telemetrykeys/with_telemetry.go b/github.com/cockroachdb/errors/telemetrykeys/with_telemetry.go index 20f799fb68..d495329c4a 100644 --- a/github.com/cockroachdb/errors/telemetrykeys/with_telemetry.go +++ b/github.com/cockroachdb/errors/telemetrykeys/with_telemetry.go @@ -17,8 +17,10 @@ package telemetrykeys import ( "context" "fmt" + "strings" "github.com/cockroachdb/errors/errbase" + "github.com/cockroachdb/redact" "github.com/gogo/protobuf/proto" ) @@ -30,7 +32,7 @@ type withTelemetry struct { var _ error = (*withTelemetry)(nil) var _ errbase.SafeDetailer = (*withTelemetry)(nil) var _ fmt.Formatter = (*withTelemetry)(nil) -var _ errbase.Formatter = (*withTelemetry)(nil) +var _ errbase.SafeFormatter = (*withTelemetry)(nil) func (w *withTelemetry) Error() string { return w.cause.Error() } func (w *withTelemetry) Cause() error { return w.cause } @@ -40,9 +42,9 @@ func (w *withTelemetry) SafeDetails() []string { return w.keys } func (w *withTelemetry) Format(s fmt.State, verb rune) { errbase.FormatError(w, s, verb) } -func (w *withTelemetry) FormatError(p errbase.Printer) (next error) { +func (w *withTelemetry) SafeFormatError(p errbase.Printer) (next error) { if p.Detail() { - p.Printf("keys: %+v", w.keys) + p.Printf("keys: [%s]", redact.Safe(strings.Join(w.keys, " "))) } return w.cause } diff --git a/github.com/cockroachdb/errors/telemetrykeys_api.go b/github.com/cockroachdb/errors/telemetrykeys_api.go index 0ccb615a69..145d9b9974 100644 --- a/github.com/cockroachdb/errors/telemetrykeys_api.go +++ b/github.com/cockroachdb/errors/telemetrykeys_api.go @@ -16,8 +16,17 @@ package errors import "github.com/cockroachdb/errors/telemetrykeys" -// WithTelemetry forwards a definition. +// WithTelemetry annotates err with the given telemetry key(s). +// The telemetry keys must be PII-free. +// +// Detail is shown: +// - via `errors.GetSafeDetails()`. +// - via `GetTelemetryKeys()` below. +// - when formatting with `%+v`. +// - in Sentry reports. func WithTelemetry(err error, keys ...string) error { return telemetrykeys.WithTelemetry(err, keys...) } -// GetTelemetryKeys forwards a definition. +// GetTelemetryKeys retrieves the (de-duplicated) set of +// all telemetry keys present in the direct causal chain +// of the error. The keys may not be sorted. func GetTelemetryKeys(err error) []string { return telemetrykeys.GetTelemetryKeys(err) } diff --git a/github.com/cockroachdb/errors/withstack/withstack.go b/github.com/cockroachdb/errors/withstack/withstack.go index 402608999d..56f5d68486 100644 --- a/github.com/cockroachdb/errors/withstack/withstack.go +++ b/github.com/cockroachdb/errors/withstack/withstack.go @@ -58,7 +58,7 @@ type withStack struct { var _ error = (*withStack)(nil) var _ fmt.Formatter = (*withStack)(nil) -var _ errbase.Formatter = (*withStack)(nil) +var _ errbase.SafeFormatter = (*withStack)(nil) var _ errbase.SafeDetailer = (*withStack)(nil) func (w *withStack) Error() string { return w.cause.Error() } @@ -68,15 +68,17 @@ func (w *withStack) Unwrap() error { return w.cause } // Format implements the fmt.Formatter interface. func (w *withStack) Format(s fmt.State, verb rune) { errbase.FormatError(w, s, verb) } -func (w *withStack) FormatError(p errbase.Printer) error { +// SafeFormatError implements the errbase.SafeFormatter interface. +func (w *withStack) SafeFormatError(p errbase.Printer) error { if p.Detail() { - p.Print("attached stack trace") + p.Printf("attached stack trace") } // We do not print the stack trace ourselves - errbase.FormatError() // does this for us. return w.cause } +// SafeDetails implements the errbase.SafeDetailer interface. func (w *withStack) SafeDetails() []string { return []string{fmt.Sprintf("%+v", w.StackTrace())} } diff --git a/github.com/cockroachdb/errors/withstack_api.go b/github.com/cockroachdb/errors/withstack_api.go index 1da56f576c..db6eb25c03 100644 --- a/github.com/cockroachdb/errors/withstack_api.go +++ b/github.com/cockroachdb/errors/withstack_api.go @@ -16,21 +16,53 @@ package errors import "github.com/cockroachdb/errors/withstack" -// WithStack forwards a definition. +// This file mirrors the WithStack functionality from +// github.com/pkg/errors. We would prefer to reuse the withStack +// struct from that package directly (the library recognizes it well) +// unfortunately github.com/pkg/errors does not enable client code to +// customize the depth at which the stack trace is captured. + +// WithStack annotates err with a stack trace at the point WithStack was called. +// +// Detail is shown: +// - via `errors.GetSafeDetails()` +// - when formatting with `%+v`. +// - in Sentry reports. +// - when innermost stack capture, with `errors.GetOneLineSource()`. func WithStack(err error) error { return withstack.WithStackDepth(err, 1) } -// WithStackDepth forwards a definition. +// WithStackDepth annotates err with a stack trace starting from the +// given call depth. The value zero identifies the caller +// of WithStackDepth itself. +// See the documentation of WithStack() for more details. func WithStackDepth(err error, depth int) error { return withstack.WithStackDepth(err, depth+1) } -// ReportableStackTrace forwards a definition. +// ReportableStackTrace aliases the type of the same name in the sentry +// package. This is used by SendReport(). type ReportableStackTrace = withstack.ReportableStackTrace -// GetOneLineSource forwards a definition. +// GetOneLineSource extracts the file/line/function information +// of the topmost caller in the innermost recorded stack trace. +// The filename is simplified to remove the path prefix. +// +// This is used e.g. to populate the "source" field in +// PostgreSQL errors in CockroachDB. func GetOneLineSource(err error) (file string, line int, fn string, ok bool) { return withstack.GetOneLineSource(err) } -// GetReportableStackTrace forwards a definition. +// GetReportableStackTrace extracts a stack trace embedded in the +// given error in the format suitable for Sentry reporting. +// +// This supports: +// - errors generated by github.com/pkg/errors (either generated +// locally or after transfer through the network), +// - errors generated with WithStack() in this package, +// - any other error that implements a StackTrace() method +// returning a StackTrace from github.com/pkg/errors. +// +// Note: Sentry wants the oldest call frame first, so +// the entries are reversed in the result. func GetReportableStackTrace(err error) *ReportableStackTrace { return withstack.GetReportableStackTrace(err) } diff --git a/github.com/cockroachdb/pebble/.travis.yml b/github.com/cockroachdb/pebble/.travis.yml index 4fae68c91b..b5ff32883b 100644 --- a/github.com/cockroachdb/pebble/.travis.yml +++ b/github.com/cockroachdb/pebble/.travis.yml @@ -5,7 +5,6 @@ branches: - master install: - - go get -d -t -v ./... - go get -v golang.org/x/lint/golint matrix: @@ -13,32 +12,32 @@ matrix: - name: "go1.13.x-linux" go: 1.13.x os: linux + script: make test + - name: "go1.14.x-linux" + go: 1.14.x + os: linux script: make test generate - - name: "go1.13.x-linux-race" - go: 1.13.x + - name: "go1.14.x-linux-race" + go: 1.14.x os: linux script: make testrace TAGS= - - name: "go1.13.x-linux-no-invariants" - go: 1.13.x + - name: "go1.14.x-linux-no-invariants" + go: 1.14.x os: linux script: make test TAGS= - - name: "go1.13.x-darwin" - go: 1.13.x + - name: "go1.14.x-darwin" + go: 1.14.x os: osx script: make test - - name: "go1.13.x-windows" - go: 1.13.x + - name: "go1.14.x-windows" + go: 1.14.x os: windows script: go test ./... - - name: "go1.13.x-freebsd" - go: 1.13.x + - name: "go1.14.x-freebsd" + go: 1.14.x os: linux # NB: "env: GOOS=freebsd" does not have the desired effect. script: GOOS=freebsd go build -v ./... - - name: "go1.14.x-linux" - go: 1.14.x - os: linux - script: make test notifications: email: diff --git a/github.com/cockroachdb/pebble/Makefile b/github.com/cockroachdb/pebble/Makefile index 88192e7ee4..ede486fdd9 100644 --- a/github.com/cockroachdb/pebble/Makefile +++ b/github.com/cockroachdb/pebble/Makefile @@ -1,6 +1,6 @@ GO := go -GOFLAGS := PKG := ./... +GOFLAGS := STRESSFLAGS := TAGS := invariants TESTS := . @@ -16,29 +16,29 @@ all: @echo " make mod-update" @echo " make clean" +override testflags := .PHONY: test test: - GO111MODULE=off ${GO} test -tags '$(TAGS)' ${GOFLAGS} -run ${TESTS} ${PKG} + ${GO} test -mod=vendor -tags '$(TAGS)' ${testflags} -run ${TESTS} ${PKG} .PHONY: testrace -testrace: GOFLAGS += -race +testrace: testflags += -race testrace: test .PHONY: stress stressrace -stressrace: GOFLAGS += -race -stress stressrace: - GO111MODULE=off ${GO} test -v -tags '$(TAGS)' ${GOFLAGS} -exec 'stress ${STRESSFLAGS}' -run '${TESTS}' -timeout 0 ${PKG} +stressrace: testflags += -race +stress stressrace: testflags += -exec 'stress ${STRESSFLAGS}' -timeout 0 -test.v +stress stressrace: test .PHONY: stressmeta -stressmeta: PKG = ./internal/metamorphic -stressmeta: STRESSFLAGS += -p 1 -stressmeta: TESTS = TestMeta$$ -stressmeta: - GO111MODULE=off ${GO} test -v -tags '$(TAGS)' ${GOFLAGS} -exec 'stress ${STRESSFLAGS}' -run '${TESTS}' -timeout 0 ${PKG} +stressmeta: override PKG = ./internal/metamorphic +stressmeta: override STRESSFLAGS += -p 1 +stressmeta: override TESTS = TestMeta$$ +stressmeta: stress .PHONY: generate generate: - GO111MODULE=off ${GO} generate ${PKG} + ${GO} generate -mod=vendor ${PKG} # The cmd/pebble/{badger,boltdb,rocksdb}.go files causes various # cockroach dependencies to be pulled in which is undesirable. Hack @@ -46,8 +46,9 @@ generate: mod-update: mkdir -p cmd/pebble/_bak mv cmd/pebble/{badger,boltdb,rocksdb}.go cmd/pebble/_bak - GO111MODULE=on ${GO} get -u - GO111MODULE=on ${GO} mod vendor + ${GO} get -u + ${GO} mod tidy + ${GO} mod vendor mv cmd/pebble/_bak/* cmd/pebble && rmdir cmd/pebble/_bak .PHONY: clean diff --git a/github.com/cockroachdb/pebble/README.md b/github.com/cockroachdb/pebble/README.md index 6b095e00ee..3af5200648 100644 --- a/github.com/cockroachdb/pebble/README.md +++ b/github.com/cockroachdb/pebble/README.md @@ -1,4 +1,6 @@ -# Pebble [![Build Status](https://travis-ci.org/cockroachdb/pebble.svg?branch=master)](https://travis-ci.org/cockroachdb/pebble) +# Pebble [![Build Status](https://travis-ci.org/cockroachdb/pebble.svg?branch=master)](https://travis-ci.org/cockroachdb/pebble) [![GoDoc](https://godoc.org/github.com/cockroachdb/pebble?status.svg)](https://godoc.org/github.com/cockroachdb/pebble) + +#### [Nightly benchmarks](https://cockroachdb.github.io/pebble/) Pebble is a LevelDB/RocksDB inspired key-value store focused on performance and internal usage by CockroachDB. Pebble inherits the @@ -107,3 +109,40 @@ https://github.com/google/leveldb Optimizations and inspiration were drawn from RocksDB: https://github.com/facebook/rocksdb + +## Getting Started + +### Example Code + +```go +package main + +import ( + "fmt" + "log" + + "github.com/cockroachdb/pebble" +) + +func main() { + db, err := pebble.Open("demo", &pebble.Options{}) + if err != nil { + log.Fatal(err) + } + key := []byte("hello") + if err := db.Set(key, []byte("world"), pebble.Sync); err != nil { + log.Fatal(err) + } + value, closer, err := db.Get(key) + if err != nil { + log.Fatal(err) + } + fmt.Printf("%s %s\n", key, value) + if err := closer.Close(); err != nil { + log.Fatal(err) + } + if err := db.Close(); err != nil { + log.Fatal(err) + } +} +``` diff --git a/github.com/cockroachdb/pebble/batch.go b/github.com/cockroachdb/pebble/batch.go index bf7f9f6cdd..e4f89126c3 100644 --- a/github.com/cockroachdb/pebble/batch.go +++ b/github.com/cockroachdb/pebble/batch.go @@ -184,6 +184,7 @@ type Batch struct { // memtables. data []byte cmp Compare + formatKey base.FormatKey abbreviatedKey AbbreviatedKey memTableSize uint32 @@ -196,6 +197,10 @@ type Batch struct { // data whenever Repr() is called. count uint64 + // The count of range deletions in the batch. Updated every time a range + // deletion is added. + countRangeDels uint64 + // A deferredOp struct, stored in the Batch so that a pointer can be returned // from the *Deferred() methods rather than a value. deferredOp DeferredBatchOp @@ -247,6 +252,7 @@ func newBatch(db *DB) *Batch { func newIndexedBatch(db *DB, comparer *Comparer) *Batch { i := indexedBatchPool.Get().(*indexedBatch) i.batch.cmp = comparer.Compare + i.batch.formatKey = comparer.FormatKey i.batch.abbreviatedKey = comparer.AbbreviatedKey i.batch.db = db i.batch.index = &i.index @@ -271,20 +277,12 @@ func (b *Batch) release() { // complains. b.Reset() b.cmp = nil + b.formatKey = nil b.abbreviatedKey = nil - b.memTableSize = 0 - - b.deferredOp = DeferredBatchOp{} - b.tombstones = nil - b.flushable = nil - b.commit = sync.WaitGroup{} - b.commitErr = nil - atomic.StoreUint32(&b.applied, 0) if b.index == nil { batchPool.Put(b) } else { - b.index.Reset() b.index, b.rangeDelIndex = nil, nil indexedBatchPool.Put((*indexedBatch)(unsafe.Pointer(b))) } @@ -296,12 +294,16 @@ func (b *Batch) refreshMemTableSize() { return } + b.countRangeDels = 0 for r := b.Reader(); ; { - _, key, value, ok := r.Next() + kind, key, value, ok := r.Next() if !ok { break } b.memTableSize += memTableEntrySize(len(key), len(value)) + if kind == InternalKeyKindRangeDelete { + b.countRangeDels++ + } } } @@ -313,7 +315,7 @@ func (b *Batch) Apply(batch *Batch, _ *WriteOptions) error { return nil } if len(batch.data) < batchHeaderLen { - return errors.New("pebble: invalid batch") + return base.CorruptionErrorf("pebble: invalid batch") } offset := len(b.data) @@ -334,6 +336,9 @@ func (b *Batch) Apply(batch *Batch, _ *WriteOptions) error { if !ok { break } + if kind == InternalKeyKindRangeDelete { + b.countRangeDels++ + } if b.index != nil { var err error if kind == InternalKeyKindRangeDelete { @@ -594,6 +599,7 @@ func (b *Batch) DeleteRange(start, end []byte, _ *WriteOptions) error { // with the end key. func (b *Batch) DeleteRangeDeferred(startLen, endLen int) *DeferredBatchOp { b.prepareDeferredKeyValueRecord(startLen, endLen, InternalKeyKindRangeDelete) + b.countRangeDels++ if b.index != nil { b.tombstones = nil // Range deletions are rare, so we lazily allocate the index for them. @@ -642,7 +648,7 @@ func (b *Batch) Repr() []byte { // Batch is no longer in use. func (b *Batch) SetRepr(data []byte) error { if len(data) < batchHeaderLen { - return errors.New("invalid batch") + return base.CorruptionErrorf("invalid batch") } b.data = data b.count = uint64(binary.LittleEndian.Uint32(b.countData())) @@ -690,7 +696,8 @@ func (b *Batch) newRangeDelIter(o *IterOptions) internalIterator { // tombstone is added to the batch. if b.tombstones == nil { frag := &rangedel.Fragmenter{ - Cmp: b.cmp, + Cmp: b.cmp, + Format: b.formatKey, Emit: func(fragmented []rangedel.Tombstone) { b.tombstones = append(b.tombstones, fragmented...) }, @@ -743,12 +750,21 @@ func (b *Batch) init(cap int) { b.data = b.data[:batchHeaderLen] } -// Reset clears the underlying byte slice and effectively empties the batch for -// reuse. Used in cases where Batch is only being used to build a batch, and -// where the end result is a Repr() call, not a Commit call or a Close call. -// Commits and Closes take care of releasing resources when appropriate. +// Reset resets the batch for reuse. The underlying byte slice (that is +// returned by Repr()) is not modified. It is only necessary to call this +// method if a batch is explicitly being reused. Close automatically takes are +// of releasing resources when appropriate for batches that are internally +// being reused. func (b *Batch) Reset() { b.count = 0 + b.countRangeDels = 0 + b.memTableSize = 0 + b.deferredOp = DeferredBatchOp{} + b.tombstones = nil + b.flushable = nil + b.commit = sync.WaitGroup{} + b.commitErr = nil + atomic.StoreUint32(&b.applied, 0) if b.data != nil { if cap(b.data) > batchMaxRetainedSize { // If the capacity of the buffer is larger than our maximum @@ -762,6 +778,10 @@ func (b *Batch) Reset() { b.setSeqNum(0) } } + if b.index != nil { + b.index.Init(&b.data, b.cmp, b.abbreviatedKey) + b.rangeDelIndex = nil + } } // seqNumData returns the 8 byte little-endian sequence number. Zero means that @@ -778,7 +798,7 @@ func (b *Batch) countData() []byte { func (b *Batch) grow(n int) { newSize := len(b.data) + n - if newSize >= maxBatchSize { + if uint64(newSize) >= maxBatchSize { panic(ErrBatchTooLarge) } if newSize > cap(b.data) { @@ -799,8 +819,11 @@ func (b *Batch) setSeqNum(seqNum uint64) { // SeqNum returns the batch sequence number which is applied to the first // record in the batch. The sequence number is incremented for each subsequent -// record. +// record. It returns zero if the batch is empty. func (b *Batch) SeqNum() uint64 { + if len(b.data) == 0 { + b.init(batchHeaderLen) + } return binary.LittleEndian.Uint64(b.seqNumData()) } @@ -820,27 +843,30 @@ func (b *Batch) Count() uint32 { // Reader returns a BatchReader for the current batch contents. If the batch is // mutated, the new entries will not be visible to the reader. func (b *Batch) Reader() BatchReader { + if len(b.data) == 0 { + b.init(batchHeaderLen) + } return b.data[batchHeaderLen:] } func batchDecodeStr(data []byte) (odata []byte, s []byte, ok bool) { var v uint32 var n int - src := (*[5]uint8)(unsafe.Pointer(&data[0])) - if a := (*src)[0]; a < 128 { + ptr := unsafe.Pointer(&data[0]) + if a := *((*uint8)(ptr)); a < 128 { v = uint32(a) n = 1 - } else if a, b := a&0x7f, (*src)[1]; b < 128 { + } else if a, b := a&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 1))); b < 128 { v = uint32(b)<<7 | uint32(a) n = 2 - } else if b, c := b&0x7f, (*src)[2]; c < 128 { + } else if b, c := b&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 2))); c < 128 { v = uint32(c)<<14 | uint32(b)<<7 | uint32(a) n = 3 - } else if c, d := c&0x7f, (*src)[3]; d < 128 { + } else if c, d := c&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 3))); d < 128 { v = uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) n = 4 } else { - d, e := d&0x7f, (*src)[4] + d, e := d&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 4))) v = uint32(e)<<28 | uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) n = 5 } @@ -858,6 +884,9 @@ type BatchReader []byte // MakeBatchReader constructs a BatchReader from a batch representation. The // header (containing the batch count and seqnum) is ignored. func MakeBatchReader(repr []byte) BatchReader { + if len(repr) <= batchHeaderLen { + return nil + } return repr[batchHeaderLen:] } @@ -966,7 +995,7 @@ func (i *batchIter) Value() []byte { offset, _, keyEnd := i.iter.KeyInfo() data := i.batch.data if len(data[offset:]) == 0 { - i.err = errors.New("corrupted batch") + i.err = base.CorruptionErrorf("corrupted batch") return nil } @@ -1015,8 +1044,9 @@ type flushableBatchEntry struct { // flushableBatch wraps an existing batch and provides the interfaces needed // for making the batch flushable (i.e. able to mimic a memtable). type flushableBatch struct { - cmp Compare - data []byte + cmp Compare + formatKey base.FormatKey + data []byte // The base sequence number for the entries in the batch. This is the same // value as Batch.seqNum() and is cached here for performance. @@ -1045,9 +1075,10 @@ var _ flushable = (*flushableBatch)(nil) // of the batch data. func newFlushableBatch(batch *Batch, comparer *Comparer) *flushableBatch { b := &flushableBatch{ - data: batch.data, - cmp: comparer.Compare, - offsets: make([]flushableBatchEntry, 0, batch.Count()), + data: batch.data, + cmp: comparer.Compare, + formatKey: comparer.FormatKey, + offsets: make([]flushableBatchEntry, 0, batch.Count()), } if b.data != nil { // Note that this sequence number is not correct when this batch has not @@ -1096,7 +1127,8 @@ func newFlushableBatch(batch *Batch, comparer *Comparer) *flushableBatch { if len(rangeDelOffsets) > 0 { frag := &rangedel.Fragmenter{ - Cmp: b.cmp, + Cmp: b.cmp, + Format: b.formatKey, Emit: func(fragmented []rangedel.Tombstone) { b.tombstones = append(b.tombstones, fragmented...) }, @@ -1349,12 +1381,12 @@ func (i *flushableBatchIter) Key() *InternalKey { func (i *flushableBatchIter) Value() []byte { p := i.data[i.offsets[i.index].offset:] if len(p) == 0 { - i.err = errors.New("corrupted batch") + i.err = base.CorruptionErrorf("corrupted batch") return nil } kind := InternalKeyKind(p[0]) if kind > InternalKeyKindMax { - i.err = errors.New("corrupted batch") + i.err = base.CorruptionErrorf("corrupted batch") return nil } var value []byte @@ -1364,7 +1396,7 @@ func (i *flushableBatchIter) Value() []byte { keyEnd := i.offsets[i.index].keyEnd _, value, ok = batchDecodeStr(i.data[keyEnd:]) if !ok { - i.err = errors.New("corrupted batch") + i.err = base.CorruptionErrorf("corrupted batch") return nil } } @@ -1448,12 +1480,12 @@ func (i flushFlushableBatchIter) Prev() (*InternalKey, []byte) { func (i flushFlushableBatchIter) valueSize() uint64 { p := i.data[i.offsets[i.index].offset:] if len(p) == 0 { - i.err = errors.New("corrupted batch") + i.err = base.CorruptionErrorf("corrupted batch") return 0 } kind := InternalKeyKind(p[0]) if kind > InternalKeyKindMax { - i.err = errors.New("corrupted batch") + i.err = base.CorruptionErrorf("corrupted batch") return 0 } var length uint64 @@ -1462,7 +1494,7 @@ func (i flushFlushableBatchIter) valueSize() uint64 { keyEnd := i.offsets[i.index].keyEnd v, n := binary.Uvarint(i.data[keyEnd:]) if n <= 0 { - i.err = errors.New("corrupted batch") + i.err = base.CorruptionErrorf("corrupted batch") return 0 } length = v + uint64(n) diff --git a/github.com/cockroachdb/pebble/bloom/bloom.go b/github.com/cockroachdb/pebble/bloom/bloom.go index 9f3ce5c057..726b879ab8 100644 --- a/github.com/cockroachdb/pebble/bloom/bloom.go +++ b/github.com/cockroachdb/pebble/bloom/bloom.go @@ -111,7 +111,7 @@ func hash(b []byte) uint32 { seed = 0xbc9f1d34 m = 0xc6a4a793 ) - h := uint32(seed) ^ uint32(len(b)*m) + h := uint32(seed) ^ uint32(uint64(uint32(len(b))*m)) for ; len(b) >= 4; b = b[4:] { h += uint32(b[0]) | uint32(b[1])<<8 | uint32(b[2])<<16 | uint32(b[3])<<24 h *= m diff --git a/github.com/cockroachdb/pebble/checkpoint.go b/github.com/cockroachdb/pebble/checkpoint.go index 605de15c86..b276194f4e 100644 --- a/github.com/cockroachdb/pebble/checkpoint.go +++ b/github.com/cockroachdb/pebble/checkpoint.go @@ -119,9 +119,9 @@ func (d *DB) Checkpoint(destDir string) (err error) { // Link or copy the sstables. for l := range current.Levels { - level := current.Levels[l] - for i := range level { - srcPath := base.MakeFilename(fs, d.dirname, fileTypeTable, level[i].FileNum) + iter := current.Levels[l].Iter() + for f := iter.First(); f != nil; f = iter.Next() { + srcPath := base.MakeFilename(fs, d.dirname, fileTypeTable, f.FileNum) destPath := fs.PathJoin(destDir, fs.PathBase(srcPath)) if err := vfs.LinkOrCopy(fs, srcPath, destPath); err != nil { return err diff --git a/github.com/cockroachdb/pebble/commit.go b/github.com/cockroachdb/pebble/commit.go index e55cf27299..7132daebd5 100644 --- a/github.com/cockroachdb/pebble/commit.go +++ b/github.com/cockroachdb/pebble/commit.go @@ -207,13 +207,20 @@ type commitEnv struct { // that committing of that unapplied batch will eventually find our (applied) // batch in the queue. See commitPipeline.publish for additional commentary. type commitPipeline struct { - env commitEnv - sem chan struct{} + // WARNING: The following struct `commitQueue` contains fields which will + // be accessed atomically. + // + // Go allocations are guaranteed to be 64-bit aligned which we take advantage + // of by placing the 64-bit fields which we access atomically at the beginning + // of the commitPipeline struct. + // For more information, see https://golang.org/pkg/sync/atomic/#pkg-note-BUG. + // Queue of pending batches to commit. + pending commitQueue + env commitEnv + sem chan struct{} // The mutex to use for synchronizing access to logSeqNum and serializing // calls to commitEnv.write(). mu sync.Mutex - // Queue of pending batches to commit. - pending commitQueue } func newCommitPipeline(env commitEnv) *commitPipeline { diff --git a/github.com/cockroachdb/pebble/compaction.go b/github.com/cockroachdb/pebble/compaction.go index 6be91ff938..cc8cd4df7c 100644 --- a/github.com/cockroachdb/pebble/compaction.go +++ b/github.com/cockroachdb/pebble/compaction.go @@ -16,7 +16,6 @@ import ( "strings" "sync/atomic" "time" - "unsafe" "github.com/cockroachdb/errors" "github.com/cockroachdb/pebble/internal/base" @@ -41,20 +40,12 @@ func expandedCompactionByteSizeLimit(opts *Options, level int) uint64 { return uint64(25 * opts.Level(level).TargetFileSize) } -// maxGrandparentOverlapBytes is the maximum bytes of overlap with level+2 -// before we stop building a single file in a level to level+1 compaction. +// maxGrandparentOverlapBytes is the maximum bytes of overlap with level+1 +// before we stop building a single file in a level-1 to level compaction. func maxGrandparentOverlapBytes(opts *Options, level int) uint64 { return uint64(10 * opts.Level(level).TargetFileSize) } -// totalSize returns the total size of all the files in f. -func totalSize(f []*fileMetadata) (size uint64) { - for _, x := range f { - size += x.Size - } - return size -} - // noCloseIter wraps around an internal iterator, intercepting and eliding // calls to Close. It is used during compaction to ensure that rangeDelIters // are not closed prematurely. @@ -70,9 +61,372 @@ type userKeyRange struct { start, end []byte } +type compactionLevel struct { + level int + files manifest.LevelSlice +} + +// Return output from compactionOutputSplitters. See comment on +// compactionOutputSplitter.shouldSplitBefore() on how this value is used. +type compactionSplitSuggestion int + +const ( + noSplit compactionSplitSuggestion = iota + splitSoon + splitNow +) + +// String implements the Stringer interface. +func (c compactionSplitSuggestion) String() string { + switch c { + case noSplit: + return "no-split" + case splitSoon: + return "split-soon" + } + return "split-now" +} + +// compactionOutputSplitter is an interface for encapsulating logic around +// switching the output of a compaction to a new output file. Additional +// constraints around switching compaction outputs that are specific to that +// compaction type (eg. flush splits) are implemented in +// compactionOutputSplitters that compose other child compactionOutputSplitters. +type compactionOutputSplitter interface { + // shouldSplitBefore returns whether we should split outputs before the + // specified "current key". The return value is one of splitNow, splitSoon, + // or noSlit. splitNow means a split is advised before the specified key, + // splitSoon means no split is advised yet but the limit returned in + // onNewOutput can be considered invalidated and a splitNow suggestion will + // be made on an upcoming key shortly, and noSplit means no split is + // advised. + shouldSplitBefore(key *InternalKey, tw *sstable.Writer) compactionSplitSuggestion + // onNewOutput updates internal splitter state when the compaction switches + // to a new sstable, and returns the next limit for the new output which + // would get used to truncate range tombstones if the compaction iterator + // runs out of keys. The limit returned MUST be > key according to the + // compaction's comparator. The specified key is the first key in the new + // output, or nil if this sstable will only contain range tombstones already + // in the fragmenter. + onNewOutput(key *InternalKey) []byte +} + +// fileSizeSplitter is a compactionOutputSplitter that makes a determination +// to split outputs based on the estimated file size of the current output. +// Note that, unlike most other splitters, this splitter does not guarantee +// that it will advise splits only at user key change boundaries. +type fileSizeSplitter struct { + maxFileSize uint64 +} + +func (f *fileSizeSplitter) shouldSplitBefore( + key *InternalKey, tw *sstable.Writer, +) compactionSplitSuggestion { + // The Kind != RangeDelete part exists because EstimatedSize doesn't grow + // rightaway when a range tombstone is added to the fragmenter. It's always + // better to make a sequence of range tombstones visible to the fragmenter. + if key.Kind() != InternalKeyKindRangeDelete && tw != nil && + tw.EstimatedSize() >= f.maxFileSize { + return splitNow + } + return noSplit +} + +func (f *fileSizeSplitter) onNewOutput(key *InternalKey) []byte { + return nil +} + +type grandparentLimitSplitter struct { + c *compaction + ve *versionEdit + limit []byte +} + +func (g *grandparentLimitSplitter) shouldSplitBefore( + key *InternalKey, tw *sstable.Writer, +) compactionSplitSuggestion { + if g.limit != nil && g.c.cmp(key.UserKey, g.limit) > 0 { + return splitNow + } + return noSplit +} + +func (g *grandparentLimitSplitter) onNewOutput(key *InternalKey) []byte { + g.limit = nil + if g.c.rangeDelFrag.Empty() || len(g.ve.NewFiles) == 0 { + // In this case, `limit` will be a larger user key than `key.UserKey`, or + // nil. In either case, the inner loop will execute at least once to + // process `key`, and the input iterator will be advanced. + if key != nil { + g.limit = g.c.findGrandparentLimit(key.UserKey) + } else { // !c.rangeDelFrag.Empty() + g.limit = g.c.findGrandparentLimit(g.c.rangeDelFrag.Start()) + } + } else { + // There is a range tombstone spanning from the last file into the + // current one. Therefore this file's smallest boundary will overlap the + // last file's largest boundary. + // + // In this case, `limit` will be a larger user key than the previous + // file's largest key and correspond to a grandparent file's largest user + // key, or nil. Then, it is possible the inner loop executes zero times, + // and the output file contains only range tombstones. That is fine as + // long as the number of times we execute this case is bounded. Since + // `findGrandparentLimit()` returns a strictly larger user key each time + // and it corresponds to a grandparent file largest key, the number of + // times this case can execute is bounded by the number of grandparent + // files (plus one for the final time it returns nil). + // + // n > 0 since we cannot have seen range tombstones at the + // beginning of the first file. + n := len(g.ve.NewFiles) + g.limit = g.c.findGrandparentLimit(g.ve.NewFiles[n-1].Meta.Largest.UserKey) + // This conditional is necessary to maintain the invariant that + // successive calls to finishOutput() have an increasing + // limit, and that the fragmenter's *FlushTo() and Add() calls are + // always made with successively increasing user keys. It's possible + // for the last file's Meta.Largest.UserKey + // to be substantially less than the last key returned by + // the compactionIter, such as in a sequence of consecutive + // rangedel tombstones, of which some but not all get elided. + // Consider this example: + // + // Compaction input (all range tombstones in this snippet): + // a-b + // c-d + // e-f (elided) + // g-h (elided) <-- grandparent limit before this (at ff) + // i-j (elided) + // k-q <-- grandparent limit at k + // m-q + // + // Note that elided tombstones are added to the fragmenter, but + // removed before they make their way from the fragmenter onto + // iter.tombstones. They still affect the fragmenter's internal + // tracking of the key up to which all tombstones have been flushed. + // After the first output is cut with limit = g, the fragmenter + // is empty, so the next grandparent calculation happens with + // key = g, and returns k. We continue adding range tombstones to + // the fragmenter between [g,k], which includes k-q as we only + // switch outputs after the limit is exceeded. So key = m and + // limit = k when finishOutput is called. Since the start key in + // the fragmenter (k) matches the limit, and no point keys were + // added, we actually don't produce a new output (see the + // conditional in finishOutput() on why). + // + // When we try to calculate the next grandparent limit, since + // c.rangeDelFrag.Empty() == false (as k-q is sitting in it), + // we fall into this case, and use the end key of the last written + // sstable (which was [a-d]) to calculate the grandparent limit. + // That gets us g again, which gets us to call + // c.rangeDelFrag.FlushTo(g), which violates the + // invariant that the current flush-to key (g) be greater than the + // last one (k). + // + // To solve this, if the grandparent limit falls "in between" + // ve.NewFiles[n-1].Meta.Largest.UserKey and c.rangeDelFrag.Start(), + // re-calculate a higher limit ahead of that key. + if g.c.rangeDelFrag.Start() != nil && g.c.cmp(g.limit, g.c.rangeDelFrag.Start()) <= 0 { + g.limit = g.c.findGrandparentLimit(g.c.rangeDelFrag.Start()) + } + } + return g.limit +} + +type l0LimitSplitter struct { + c *compaction + ve *versionEdit + limit []byte +} + +func (l *l0LimitSplitter) shouldSplitBefore( + key *InternalKey, tw *sstable.Writer, +) compactionSplitSuggestion { + if l.limit != nil && l.c.cmp(key.UserKey, l.limit) > 0 { + return splitNow + } + return noSplit +} + +func (l *l0LimitSplitter) onNewOutput(key *InternalKey) []byte { + l.limit = nil + // For flushes being split across multiple sstables, call + // findL0Limit to find the next L0 limit. + if key != nil { + l.limit = l.c.findL0Limit(key.UserKey) + } else { + // Use the start key of the first pending tombstone to find the + // next limit. All pending tombstones have the same start key. + // We use this as opposed to the end key of the + // last written sstable to effectively handle cases like these: + // + // a.SET.3 + // (L0 limit at b) + // d.RANGEDEL.4:f + // + // In this case, the partition after b has only range deletions, + // so if we were to find the L0 limit after the last written + // key at the split point (key a), we'd get the limit b again, + // and finishOutput() would not advance any further because + // the next range tombstone to write does not start until after + // the L0 split point. + startKey := l.c.rangeDelFrag.Start() + if startKey != nil { + l.limit = l.c.findL0Limit(startKey) + } + } + return l.limit +} + +// splitterGroup is a compactionOutputSplitter that splits whenever one of its +// child splitters advises a compaction split. +type splitterGroup struct { + cmp Compare + splitters []compactionOutputSplitter +} + +func (a *splitterGroup) shouldSplitBefore( + key *InternalKey, tw *sstable.Writer, +) (suggestion compactionSplitSuggestion) { + suggestion = noSplit + for _, splitter := range a.splitters { + switch splitter.shouldSplitBefore(key, tw) { + case splitNow: + return splitNow + case splitSoon: + suggestion = splitSoon + } + } + return suggestion +} + +func (a *splitterGroup) onNewOutput(key *InternalKey) []byte { + var earliestLimit []byte + for _, splitter := range a.splitters { + limit := splitter.onNewOutput(key) + if limit == nil { + continue + } + if earliestLimit == nil || a.cmp(limit, earliestLimit) < 0 { + earliestLimit = limit + } + } + return earliestLimit +} + +// userKeyChangeSplitter is a compactionOutputSplitter that takes in a child +// splitter, and splits when 1) that child splitter has advised a split, and 2) +// the compaction output is at the boundary between two user keys (also +// the boundary between atomic compaction units). Use this splitter to wrap +// any splitters that don't guarantee user key splits (i.e. splitters that make +// their determinatino in ways other than comparing the current key against a +// limit key. +type userKeyChangeSplitter struct { + cmp Compare + splitOnNextUserKey bool + savedKey []byte + splitter compactionOutputSplitter +} + +func (u *userKeyChangeSplitter) shouldSplitBefore( + key *InternalKey, tw *sstable.Writer, +) compactionSplitSuggestion { + if u.splitOnNextUserKey && u.cmp(u.savedKey, key.UserKey) != 0 { + u.splitOnNextUserKey = false + u.savedKey = u.savedKey[:0] + return splitNow + } + if split := u.splitter.shouldSplitBefore(key, tw); split == splitNow { + u.splitOnNextUserKey = true + u.savedKey = append(u.savedKey[:0], key.UserKey...) + return noSplit + } + return noSplit +} + +func (u *userKeyChangeSplitter) onNewOutput(key *InternalKey) []byte { + return u.splitter.onNewOutput(key) +} + +// nonZeroSeqNumSplitter is a compactionOutputSplitter that takes in a child +// splitter, and advises a split when 1) that child splitter advises a split, +// and 2) the compaction output is at a point where the previous point sequence +// number is nonzero. +type nonZeroSeqNumSplitter struct { + c *compaction + splitter compactionOutputSplitter + prevPointSeqNum uint64 + splitOnNonZeroSeqNum bool +} + +func (n *nonZeroSeqNumSplitter) shouldSplitBefore( + key *InternalKey, tw *sstable.Writer, +) compactionSplitSuggestion { + curSeqNum := key.SeqNum() + keyKind := key.Kind() + prevPointSeqNum := n.prevPointSeqNum + if keyKind != InternalKeyKindRangeDelete { + n.prevPointSeqNum = curSeqNum + } + + if n.splitOnNonZeroSeqNum { + if prevPointSeqNum > 0 || n.c.rangeDelFrag.Empty() { + n.splitOnNonZeroSeqNum = false + return splitNow + } + } else if split := n.splitter.shouldSplitBefore(key, tw); split == splitNow { + userKeyChange := curSeqNum > prevPointSeqNum + if prevPointSeqNum > 0 || n.c.rangeDelFrag.Empty() || userKeyChange { + return splitNow + } + n.splitOnNonZeroSeqNum = true + return splitSoon + } + return noSplit +} + +func (n *nonZeroSeqNumSplitter) onNewOutput(key *InternalKey) []byte { + n.prevPointSeqNum = InternalKeySeqNumMax + n.splitOnNonZeroSeqNum = false + return n.splitter.onNewOutput(key) +} + +// compactionFile is a vfs.File wrapper that, on every write, updates a metric +// in `versions` on bytes written by in-progress compactions so far. It also +// increments a per-compaction `written` int. +type compactionFile struct { + vfs.File + + versions *versionSet + written *int64 +} + +// Write implements the io.Writer interface. +func (c *compactionFile) Write(p []byte) (n int, err error) { + n, err = c.File.Write(p) + if err != nil { + return n, err + } + + *c.written += int64(n) + c.versions.incrementCompactionBytes(int64(n)) + return n, err +} + +type compactionKind string + +const ( + compactionKindDefault compactionKind = "default" + compactionKindFlush compactionKind = "flush" + compactionKindMove compactionKind = "move" + compactionKindDeleteOnly compactionKind = "delete-only" + compactionKindElisionOnly compactionKind = "elision-only" +) + // compaction is a table compaction from one level to the next, starting from a // given version. type compaction struct { + kind compactionKind cmp Compare formatKey base.FormatKey logger Logger @@ -82,11 +436,13 @@ type compaction struct { // startLevel is the level that is being compacted. Inputs from startLevel // and outputLevel will be merged to produce a set of outputLevel files. - startLevel int + startLevel *compactionLevel // outputLevel is the level that files are being produced in. outputLevel is // equal to startLevel+1 except when startLevel is 0 in which case it is // equal to compactionPicker.baseLevel(). - outputLevel int + outputLevel *compactionLevel + + inputs []compactionLevel // maxOutputFileSize is the maximum size of an individual table created // during compaction. @@ -94,10 +450,6 @@ type compaction struct { // maxOverlapBytes is the maximum number of bytes of overlap allowed for a // single output table with the tables in the grandparent level. maxOverlapBytes uint64 - // maxExpandedBytes is the maximum size of an expanded compaction. If growing - // a compaction results in a larger size, the original compaction is used - // instead. - maxExpandedBytes uint64 // disableRangeTombstoneElision disables elision of range tombstones. Used by // tests to allow range tombstones to be added to tables where they would // otherwise be elided. @@ -107,14 +459,15 @@ type compaction struct { flushing flushableList // bytesIterated contains the number of bytes that have been flushed/compacted. bytesIterated uint64 + // bytesWritten contains the number of bytes that have been written to outputs. + bytesWritten int64 // atomicBytesIterated points to the variable to increment during iteration. // atomicBytesIterated must be read/written atomically. Flushing will increment // the shared variable which compaction will read. This allows for the // compaction routine to know how many bytes have been flushed before the flush // is applied. atomicBytesIterated *uint64 - // inputs are the tables to be compacted. - inputs [2][]*fileMetadata + // The boundaries of the input data. smallest InternalKey largest InternalKey @@ -132,7 +485,7 @@ type compaction struct { // grandparents are the tables in level+2 that overlap with the files being // compacted. Used to determine output table boundaries. Do not assume that the actual files // in the grandparent when this compaction finishes will be the same. - grandparents []*fileMetadata + grandparents manifest.LevelSlice // Boundaries at which flushes to L0 should be split. Determined by // L0Sublevels. If nil, flushes aren't split. @@ -145,64 +498,122 @@ type compaction struct { inuseKeyRanges []userKeyRange elideTombstoneIndex int - // L0-specific compaction info. Set to a non-nil value for all compactions - // where startLevel == 0 that were generated by L0Sublevels. - lcf *manifest.L0CompactionFiles + // allowedZeroSeqNum is true if seqnums can be zeroed if there are no + // snapshots requiring them to be kept. This determination is made by + // looking for an sstable which overlaps the bounds of the compaction at a + // lower level in the LSM during runCompaction. + allowedZeroSeqNum bool metrics map[int]*LevelMetrics } -func newCompaction( - opts *Options, cur *version, startLevel, - baseLevel int, bytesCompacted *uint64, -) *compaction { - if startLevel > 0 && startLevel < baseLevel { - panic(fmt.Sprintf("invalid compaction: start level %d should be empty (base level %d)", - startLevel, baseLevel)) +func (c *compaction) makeInfo(jobID int) CompactionInfo { + info := CompactionInfo{ + JobID: jobID, + Input: make([]LevelInfo, 0, len(c.inputs)), } - - outputLevel := startLevel + 1 - if startLevel == 0 { - outputLevel = baseLevel + for _, cl := range c.inputs { + inputInfo := LevelInfo{Level: cl.level, Tables: nil} + iter := cl.files.Iter() + for m := iter.First(); m != nil; m = iter.Next() { + inputInfo.Tables = append(inputInfo.Tables, m.TableInfo()) + } + info.Input = append(info.Input, inputInfo) } - if outputLevel >= numLevels-1 { - outputLevel = numLevels - 1 + if c.outputLevel != nil { + info.Output.Level = c.outputLevel.level + // If there are no inputs from the output level (eg, a move + // compaction), add an empty LevelInfo to info.Input. + if len(c.inputs) > 0 && c.inputs[len(c.inputs)-1].level != c.outputLevel.level { + info.Input = append(info.Input, LevelInfo{Level: c.outputLevel.level}) + } + } else { + // For a delete-only compaction, set the output level to L6. The + // output level is not meaningful here, but complicating the + // info.Output interface with a pointer doesn't seem worth the + // semantic distinction. + info.Output.Level = numLevels - 1 } - // Output level is in the range [baseLevel,numLevels]. For the purpose of - // determining the target output file size, overlap bytes, and expanded - // bytes, we want to adjust the range to [1,numLevels]. - adjustedOutputLevel := 1 + outputLevel - baseLevel + return info +} - return &compaction{ - cmp: opts.Comparer.Compare, +func newCompaction(pc *pickedCompaction, opts *Options, bytesCompacted *uint64) *compaction { + c := &compaction{ + kind: compactionKindDefault, + cmp: pc.cmp, formatKey: opts.Comparer.FormatKey, + score: pc.score, + inputs: pc.inputs, + smallest: pc.smallest, + largest: pc.largest, logger: opts.Logger, - version: cur, - startLevel: startLevel, - outputLevel: outputLevel, - maxOutputFileSize: uint64(opts.Level(adjustedOutputLevel).TargetFileSize), - maxOverlapBytes: maxGrandparentOverlapBytes(opts, adjustedOutputLevel), - maxExpandedBytes: expandedCompactionByteSizeLimit(opts, adjustedOutputLevel), + version: pc.version, + maxOutputFileSize: pc.maxOutputFileSize, + maxOverlapBytes: pc.maxOverlapBytes, atomicBytesIterated: bytesCompacted, } + c.startLevel = &c.inputs[0] + c.outputLevel = &c.inputs[1] + + // Compute the set of outputLevel+1 files that overlap this compaction (these + // are the grandparent sstables). + if c.outputLevel.level+1 < numLevels { + c.grandparents = c.version.Overlaps(c.outputLevel.level+1, c.cmp, + c.smallest.UserKey, c.largest.UserKey) + } + c.setupInuseKeyRanges() + + if c.startLevel.level == numLevels-1 { + // This compaction is an L6->L6 elision-only compaction to rewrite + // a sstable without unnecessary tombstones. + c.kind = compactionKindElisionOnly + } else if c.outputLevel.files.Empty() && c.startLevel.files.Len() == 1 && + c.grandparents.SizeSum() <= c.maxOverlapBytes { + // This compaction can be converted into a trivial move from one level + // to the next. We avoid such a move if there is lots of overlapping + // grandparent data. Otherwise, the move could create a parent file + // that will require a very expensive merge later on. + c.kind = compactionKindMove + } + return c +} + +func newDeleteOnlyCompaction(opts *Options, cur *version, inputs []compactionLevel) *compaction { + c := &compaction{ + kind: compactionKindDeleteOnly, + cmp: opts.Comparer.Compare, + formatKey: opts.Comparer.FormatKey, + logger: opts.Logger, + version: cur, + inputs: inputs, + } + + // Set c.smallest, c.largest. + files := make([]manifest.LevelIterator, 0, len(inputs)) + for _, in := range inputs { + files = append(files, in.files.Iter()) + } + c.smallest, c.largest = manifest.KeyRange(opts.Comparer.Compare, files...) + return c } func newFlush( opts *Options, cur *version, baseLevel int, flushing flushableList, bytesFlushed *uint64, ) *compaction { c := &compaction{ + kind: compactionKindFlush, cmp: opts.Comparer.Compare, formatKey: opts.Comparer.FormatKey, logger: opts.Logger, version: cur, - startLevel: -1, - outputLevel: 0, + inputs: []compactionLevel{{level: -1}, {level: 0}}, maxOutputFileSize: math.MaxUint64, maxOverlapBytes: math.MaxUint64, - maxExpandedBytes: math.MaxUint64, flushing: flushing, atomicBytesIterated: bytesFlushed, } + c.startLevel = &c.inputs[0] + c.outputLevel = &c.inputs[1] if cur.L0Sublevels != nil { c.l0Limits = cur.L0Sublevels.FlushSplitKeys() } @@ -257,177 +668,17 @@ func newFlush( if opts.Experimental.FlushSplitBytes > 0 { c.maxOutputFileSize = uint64(opts.Level(0).TargetFileSize) c.maxOverlapBytes = maxGrandparentOverlapBytes(opts, 0) - c.maxExpandedBytes = expandedCompactionByteSizeLimit(opts, 0) - c.grandparents = c.version.Overlaps(baseLevel, c.cmp, c.smallest.UserKey, c.largest.UserKey) + c.grandparents = c.version.Overlaps(baseLevel, c.cmp, + c.smallest.UserKey, c.largest.UserKey) } c.setupInuseKeyRanges() return c } -// setupInputs fills in the rest of the compaction inputs, regardless of -// whether the compaction was automatically scheduled or user initiated. -func (c *compaction) setupInputs() { - // Expand the initial inputs to a clean cut. - c.inputs[0] = c.expandInputs(c.startLevel, c.inputs[0]) - c.smallest, c.largest = manifest.KeyRange(c.cmp, c.inputs[0], nil) - - // Determine the sstables in the output level which overlap with the input - // sstables, and then expand those tables to a clean cut. - c.inputs[1] = c.version.Overlaps(c.outputLevel, c.cmp, c.smallest.UserKey, c.largest.UserKey) - c.inputs[1] = c.expandInputs(c.outputLevel, c.inputs[1]) - c.smallest, c.largest = manifest.KeyRange(c.cmp, c.inputs[0], c.inputs[1]) - - // Grow the sstables in c.startLevel as long as it doesn't affect the number - // of sstables included from c.outputLevel. - if c.lcf != nil && c.startLevel == 0 && c.outputLevel != 0 { - // Call the L0-specific compaction extension method. Similar logic as - // c.grow. Additional L0 files are optionally added to the compaction at - // this step. - if c.version.L0Sublevels.ExtendL0ForBaseCompactionTo(c.smallest.UserKey, c.largest.UserKey, c.lcf) { - c.inputs[0] = c.inputs[0][:0] - for j := range c.lcf.FilesIncluded { - if c.lcf.FilesIncluded[j] { - c.inputs[0] = append(c.inputs[0], c.version.Levels[0][j]) - } - } - c.smallest, c.largest = manifest.KeyRange(c.cmp, c.inputs[0], c.inputs[1]) - } - } else if c.grow(c.smallest, c.largest) { - c.smallest, c.largest = manifest.KeyRange(c.cmp, c.inputs[0], c.inputs[1]) - } - - // Compute the set of outputLevel+1 files that overlap this compaction (these - // are the grandparent sstables). - if c.outputLevel+1 < numLevels { - c.grandparents = c.version.Overlaps(c.outputLevel+1, c.cmp, c.smallest.UserKey, c.largest.UserKey) - } - - c.setupInuseKeyRanges() -} - -// expandInputs expands the files in inputs in order to maintain the invariant -// that the versions of keys at level+1 are older than the versions of keys at -// level. This is achieved by adding tables to the right of the current input -// tables such that the rightmost table has a "clean cut". A clean cut is -// either a change in user keys, or when the largest key in the left sstable is -// a range tombstone sentinel key (InternalKeyRangeDeleteSentinel). -// -// In addition to maintaining the seqnum invariant, expandInputs is used to -// provide clean boundaries for range tombstone truncation during -// compaction. In order to achieve these clean boundaries, expandInputs needs -// to find a "clean cut" on the left edge of the compaction as well. This is -// necessary in order for "atomic compaction units" to always be compacted as a -// unit. Failure to do this leads to a subtle bug with truncation of range -// tombstones to atomic compaction unit boundaries. Consider the scenario: -// -// L3: -// 12:[a#2,15-b#1,1] -// 13:[b#0,15-d#72057594037927935,15] -// -// These sstables contain a range tombstone [a-d)#2 which spans the two -// sstables. The two sstables need to always be kept together. Compacting -// sstable 13 independently of sstable 12 would result in: -// -// L3: -// 12:[a#2,15-b#1,1] -// L4: -// 14:[b#0,15-d#72057594037927935,15] -// -// This state is still ok, but when sstable 12 is next compacted, its range -// tombstones will be truncated at "b" (the largest key in its atomic -// compaction unit). In the scenario here, that could result in b#1 becoming -// visible when it should be deleted. -func (c *compaction) expandInputs(level int, inputs []*fileMetadata) []*fileMetadata { - if level == 0 { - // We already called version.Overlaps for L0 and that call guarantees that - // we get a "clean cut". - return inputs - } - if len(inputs) == 0 { - // Nothing to expand. - return inputs - } - files := c.version.Levels[level] - // Pointer arithmetic to figure out the index if inputs[0] with - // files[0]. This requires that the inputs slice is a sub-slice of - // files. This is true for non-L0 files returned from version.overlaps. - if uintptr(unsafe.Pointer(&inputs[0])) < uintptr(unsafe.Pointer(&files[0])) { - panic("pebble: invalid input slice") - } - start := int((uintptr(unsafe.Pointer(&inputs[0])) - - uintptr(unsafe.Pointer(&files[0]))) / unsafe.Sizeof(inputs[0])) - if start >= len(files) { - panic("pebble: invalid input slice") - } - end := start + len(inputs) - - for ; start > 0; start-- { - cur := files[start] - prev := files[start-1] - if c.cmp(prev.Largest.UserKey, cur.Smallest.UserKey) < 0 { - break - } - if prev.Largest.Trailer == InternalKeyRangeDeleteSentinel { - // The range deletion sentinel key is set for the largest key in a - // table when a range deletion tombstone straddles a table. It - // isn't necessary to include the prev table in the atomic - // compaction unit as prev.largest.UserKey does not actually exist - // in the prev table. - break - } - // prev.Largest.UserKey == cur.Smallest.UserKey, so we need to include prev - // in the compaction. - } - - for ; end < len(files); end++ { - cur := files[end-1] - next := files[end] - if c.cmp(cur.Largest.UserKey, next.Smallest.UserKey) < 0 { - break - } - if cur.Largest.Trailer == InternalKeyRangeDeleteSentinel { - // The range deletion sentinel key is set for the largest key in a table - // when a range deletion tombstone straddles a table. It isn't necessary - // to include the next table in the compaction as cur.largest.UserKey - // does not actually exist in the table. - break - } - // cur.Largest.UserKey == next.Smallest.UserKey, so we need to include next - // in the compaction. - } - return files[start:end] -} - -// grow grows the number of inputs at c.level without changing the number of -// c.level+1 files in the compaction, and returns whether the inputs grew. sm -// and la are the smallest and largest InternalKeys in all of the inputs. -func (c *compaction) grow(sm, la InternalKey) bool { - if len(c.inputs[1]) == 0 { - return false - } - grow0 := c.version.Overlaps(c.startLevel, c.cmp, sm.UserKey, la.UserKey) - grow0 = c.expandInputs(c.startLevel, grow0) - if len(grow0) <= len(c.inputs[0]) { - return false - } - if totalSize(grow0)+totalSize(c.inputs[1]) >= c.maxExpandedBytes { - return false - } - sm1, la1 := manifest.KeyRange(c.cmp, grow0, nil) - grow1 := c.version.Overlaps(c.outputLevel, c.cmp, sm1.UserKey, la1.UserKey) - grow1 = c.expandInputs(c.outputLevel, grow1) - if len(grow1) != len(c.inputs[1]) { - return false - } - c.inputs[0] = grow0 - c.inputs[1] = grow1 - return true -} - func (c *compaction) setupInuseKeyRanges() { - level := c.outputLevel + 1 - if c.outputLevel == 0 { + level := c.outputLevel.level + 1 + if c.outputLevel.level == 0 { // Level 0 can contain overlapping sstables so we need to check it for // overlaps. level = 0 @@ -437,9 +688,8 @@ func (c *compaction) setupInuseKeyRanges() { // levels. var input []userKeyRange for ; level < numLevels; level++ { - overlaps := c.version.Overlaps(level, c.cmp, c.smallest.UserKey, c.largest.UserKey) - for i := range overlaps { - m := overlaps[i] + iter := c.version.Overlaps(level, c.cmp, c.smallest.UserKey, c.largest.UserKey).Iter() + for m := iter.First(); m != nil; m = iter.Next() { input = append(input, userKeyRange{m.Smallest.UserKey, m.Largest.UserKey}) } } @@ -470,21 +720,6 @@ func (c *compaction) setupInuseKeyRanges() { } } -func (c *compaction) trivialMove() bool { - if len(c.flushing) != 0 { - return false - } - // Check for a trivial move of one table from one level to the next. We avoid - // such a move if there is lots of overlapping grandparent data. Otherwise, - // the move could create a parent file that will require a very expensive - // merge later on. - if len(c.inputs[0]) == 1 && len(c.inputs[1]) == 0 && - totalSize(c.grandparents) <= c.maxOverlapBytes { - return true - } - return false -} - // findGrandparentLimit takes the start user key for a table and returns the // user key to which that table can extend without excessively overlapping // the grandparent level. If no limit is needed considering the grandparent @@ -503,38 +738,53 @@ func (c *compaction) trivialMove() bool { // user-key. Perhaps this isn't a problem if the compaction picking heuristics // always pick the right (older) sibling for compaction first. func (c *compaction) findGrandparentLimit(start []byte) []byte { - lower := sort.Search(len(c.grandparents), func(i int) bool { - return c.cmp(start, c.grandparents[i].Largest.UserKey) <= 0 - }) + iter := c.grandparents.Iter() var overlappedBytes uint64 - for upper := lower; upper < len(c.grandparents); upper++ { - overlappedBytes += c.grandparents[upper].Size + for f := iter.SeekGE(c.cmp, start); f != nil; f = iter.Next() { + overlappedBytes += f.Size // To ensure forward progress we always return a larger user // key than where we started. See comments above clients of // this function for how this is used. - if overlappedBytes > c.maxOverlapBytes && c.cmp(start, c.grandparents[upper].Largest.UserKey) < 0 { - return c.grandparents[upper].Largest.UserKey + if overlappedBytes > c.maxOverlapBytes && c.cmp(start, f.Largest.UserKey) < 0 { + return f.Largest.UserKey } } return nil } // findL0Limit takes the start key for a table and returns the user key to which -// that table can be extended without excessively overlapping the grandparent -// level, or hitting the next l0Limit, whichever happens sooner. For non-flushes -// this function passes through to findGrandparentLimit. +// that table can be extended without hitting the next l0Limit. Having flushed +// sstables "bridging across" an l0Limit could lead to increased L0 -> LBase +// compaction sizes as well as elevated read amplification. func (c *compaction) findL0Limit(start []byte) []byte { - grandparentLimit := c.findGrandparentLimit(start) - if c.startLevel > -1 || c.outputLevel != 0 || len(c.l0Limits) == 0 { - return grandparentLimit + if c.startLevel.level > -1 || c.outputLevel.level != 0 || len(c.l0Limits) == 0 { + return nil } index := sort.Search(len(c.l0Limits), func(i int) bool { return c.cmp(c.l0Limits[i], start) > 0 }) - if index < len(c.l0Limits) && (grandparentLimit == nil || c.cmp(c.l0Limits[index], grandparentLimit) < 0) { + if index < len(c.l0Limits) { return c.l0Limits[index] } - return grandparentLimit + return nil +} + +// errorOnUserKeyOverlap returns an error if the last two written sstables in +// this compaction have revisions of the same user key present in both sstables, +// when it shouldn't (eg. when splitting flushes). +func (c *compaction) errorOnUserKeyOverlap(ve *versionEdit) error { + if n := len(ve.NewFiles); n > 1 { + meta := ve.NewFiles[n-1].Meta + prevMeta := ve.NewFiles[n-2].Meta + if prevMeta.Largest.Trailer != InternalKeyRangeDeleteSentinel && + c.cmp(prevMeta.Largest.UserKey, meta.Smallest.UserKey) >= 0 { + return errors.Errorf("pebble: compaction split user key across two sstables: %s in %s and %s", + prevMeta.Largest.Pretty(c.formatKey), + prevMeta.FileNum, + meta.FileNum) + } + } + return nil } // allowZeroSeqNum returns true if seqnum's can be zeroed if there are no @@ -569,7 +819,7 @@ func (c *compaction) elideTombstone(key []byte) bool { // elideRangeTombstone returns true if it is ok to elide the specified range // tombstone. A return value of true guarantees that there are no key/value -// pairs at c.outputLevel+1 or higher that possibly overlap the specified +// pairs at c.outputLevel.level+1 or higher that possibly overlap the specified // tombstone. func (c *compaction) elideRangeTombstone(start, end []byte) bool { // Disable range tombstone elision if the testing knob for that is enabled, @@ -578,8 +828,8 @@ func (c *compaction) elideRangeTombstone(start, end []byte) bool { // being flushed in the same compaction. It's possible for a range tombstone // in one memtable to overlap keys in a preceding memtable in c.flushing. // - // This function is also used in allowZeroSeqNum, so disabling elision of - // range tombstones also disables zeroing of SeqNums. + // This function is also used in setting allowZeroSeqNum, so disabling + // elision of range tombstones also disables zeroing of SeqNums. // // TODO(peter): we disable zeroing of seqnums during flushing to match // RocksDB behavior and to avoid generating overlapping sstables during @@ -602,67 +852,6 @@ func (c *compaction) elideRangeTombstone(start, end []byte) bool { return lower >= upper } -// atomicUnitBounds returns the bounds of the atomic compaction unit containing -// the specified sstable (identified by a pointer to its fileMetadata). -func (c *compaction) atomicUnitBounds(f *fileMetadata) (lower, upper []byte) { - for i := range c.inputs { - files := c.inputs[i] - for j := range files { - if f == files[j] { - // Note that if this file is in a multi-file atomic compaction unit, this file - // may not be the first file in that unit. An example in Pebble would be a - // preceding file with Largest c#12,1 and this file with Smallest c#9,1 and - // containing a range tombstone [c, g)#11,15. The start of the range tombstone - // is already truncated to this file's Smallest.UserKey, due to the code in - // rangedel.Fragmenter.FlushTo(), so this walking back should not be necessary - // (also see range_deletions.md for more details). - // - // We do this walking back to be extra cautious, in case it helps with RocksDB - // compatibility. - lowerBound := f.Smallest.UserKey - for k := j; k > 0; k-- { - cur := files[k] - prev := files[k-1] - if c.cmp(prev.Largest.UserKey, cur.Smallest.UserKey) < 0 { - break - } - if prev.Largest.Trailer == InternalKeyRangeDeleteSentinel { - // The range deletion sentinel key is set for the largest key in a - // table when a range deletion tombstone straddles a table. It - // isn't necessary to include the prev table in the atomic - // compaction unit as prev.largest.UserKey does not actually exist - // in the prev table. - break - } - lowerBound = prev.Smallest.UserKey - } - - upperBound := f.Largest.UserKey - for k := j + 1; k < len(files); k++ { - cur := files[k-1] - next := files[k] - if c.cmp(cur.Largest.UserKey, next.Smallest.UserKey) < 0 { - break - } - if cur.Largest.Trailer == InternalKeyRangeDeleteSentinel { - // The range deletion sentinel key is set for the largest key in a - // table when a range deletion tombstone straddles a table. It - // isn't necessary to include the next table in the atomic - // compaction unit as cur.largest.UserKey does not actually exist - // in the current table. - break - } - // cur.largest.UserKey == next.largest.UserKey, so next is part of - // the atomic compaction unit. - upperBound = next.Largest.UserKey - } - return lowerBound, upperBound - } - } - } - return nil, nil -} - // newInputIter returns an iterator over all the input tables in a compaction. func (c *compaction) newInputIter(newIters tableNewIters) (_ internalIterator, retErr error) { if len(c.flushing) != 0 { @@ -688,16 +877,20 @@ func (c *compaction) newInputIter(newIters tableNewIters) (_ internalIterator, r // Check that the LSM ordering invariants are ok in order to prevent // generating corrupted sstables due to a violation of those invariants. - if c.startLevel >= 0 { - if err := manifest.CheckOrdering(c.cmp, c.formatKey, manifest.Level(c.startLevel), c.inputs[0]); err != nil { - c.logger.Fatalf("%s", err) + if c.startLevel.level >= 0 { + err := manifest.CheckOrdering(c.cmp, c.formatKey, + manifest.Level(c.startLevel.level), c.startLevel.files.Iter()) + if err != nil { + return nil, err } } - if err := manifest.CheckOrdering(c.cmp, c.formatKey, manifest.Level(c.outputLevel), c.inputs[1]); err != nil { - c.logger.Fatalf("%s", err) + err := manifest.CheckOrdering(c.cmp, c.formatKey, + manifest.Level(c.outputLevel.level), c.outputLevel.files.Iter()) + if err != nil { + return nil, err } - iters := make([]internalIterator, 0, 2*len(c.inputs[0])+1) + iters := make([]internalIterator, 0, 2*c.startLevel.files.Len()+1) defer func() { if retErr != nil { for _, iter := range iters { @@ -716,7 +909,7 @@ func (c *compaction) newInputIter(newIters tableNewIters) (_ internalIterator, r // one which iterates over the range deletions. These two iterators are // combined with a mergingIter. newRangeDelIter := func( - f *fileMetadata, _ *IterOptions, bytesIterated *uint64, + f manifest.LevelFile, _ *IterOptions, bytesIterated *uint64, ) (internalIterator, internalIterator, error) { iter, rangeDelIter, err := newIters(f, nil /* iter options */, &c.bytesIterated) if err == nil { @@ -745,10 +938,16 @@ func (c *compaction) newInputIter(newIters tableNewIters) (_ internalIterator, r // boundaries at write time. Because we're doing the truncation at read // time, we follow RocksDB's lead and do not truncate tombstones to // atomic unit boundaries at compaction time. - lowerBound, upperBound := c.atomicUnitBounds(f) - if lowerBound != nil || upperBound != nil { - rangeDelIter = rangedel.Truncate(c.cmp, rangeDelIter, lowerBound, upperBound) - } + atomicUnit, _ := expandToAtomicUnit(c.cmp, f.Slice()) + lowerBound, upperBound := manifest.KeyRange(c.cmp, atomicUnit.Iter()) + // Range deletion tombstones are often written to sstables + // untruncated on the end key side. However, they are still only + // valid within a given file's bounds. The logic for writing range + // tombstones to an output file sometimes has an incomplete view + // of range tombstones outside the file's internal key bounds. Skip + // any range tombstones completely outside file bounds. + rangeDelIter = rangedel.Truncate( + c.cmp, rangeDelIter, lowerBound.UserKey, upperBound.UserKey, &f.Smallest, &f.Largest) } if rangeDelIter == nil { rangeDelIter = emptyIter @@ -757,15 +956,61 @@ func (c *compaction) newInputIter(newIters tableNewIters) (_ internalIterator, r } iterOpts := IterOptions{logger: c.logger} - if c.startLevel != 0 { - iters = append(iters, newLevelIter(iterOpts, c.cmp, newIters, c.inputs[0], - manifest.Level(c.startLevel), &c.bytesIterated)) - iters = append(iters, newLevelIter(iterOpts, c.cmp, newRangeDelIter, c.inputs[0], - manifest.Level(c.startLevel), &c.bytesIterated)) + addItersForLevel := func(iters []internalIterator, level *compactionLevel) ([]internalIterator, error) { + iters = append(iters, newLevelIter(iterOpts, c.cmp, newIters, level.files.Iter(), + manifest.Level(level.level), &c.bytesIterated)) + // Add the range deletion iterator for each file as an independent level + // in mergingIter, as opposed to making a levelIter out of those. This + // is safer as levelIter expects all keys coming from underlying + // iterators to be in order. Due to compaction / tombstone writing + // logic in finishOutput(), it is possible for range tombstones to not + // be strictly ordered across all files in one level. + // + // Consider this example from the metamorphic tests (also repeated in + // finishOutput()), consisting of three L3 files with their bounds + // specified in square brackets next to the file name: + // + // ./000240.sst [tmgc#391,MERGE-tmgc#391,MERGE] + // tmgc#391,MERGE [786e627a] + // tmgc-udkatvs#331,RANGEDEL + // + // ./000241.sst [tmgc#384,MERGE-tmgc#384,MERGE] + // tmgc#384,MERGE [666c7070] + // tmgc-tvsalezade#383,RANGEDEL + // tmgc-tvsalezade#331,RANGEDEL + // + // ./000242.sst [tmgc#383,RANGEDEL-tvsalezade#72057594037927935,RANGEDEL] + // tmgc-tvsalezade#383,RANGEDEL + // tmgc#375,SET [72646c78766965616c72776865676e79] + // tmgc-tvsalezade#356,RANGEDEL + // + // Here, the range tombstone in 000240.sst falls "after" one in + // 000241.sst, despite 000240.sst being ordered "before" 000241.sst for + // levelIter's purposes. While each file is still consistent before its + // bounds, it's safer to have all rangedel iterators be visible to + // mergingIter. + iter := level.files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { + rangeDelIter, _, err := newRangeDelIter(iter.Take(), nil, &c.bytesIterated) + if err != nil { + return nil, errors.Wrapf(err, "pebble: could not open table %s", errors.Safe(f.FileNum)) + } + if rangeDelIter != emptyIter { + iters = append(iters, rangeDelIter) + } + } + return iters, nil + } + + if c.startLevel.level != 0 { + iters, err = addItersForLevel(iters, c.startLevel) + if err != nil { + return nil, err + } } else { - for i := range c.inputs[0] { - f := c.inputs[0][i] - iter, rangeDelIter, err := newIters(f, nil /* iter options */, &c.bytesIterated) + iter := c.startLevel.files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { + iter, rangeDelIter, err := newIters(iter.Take(), nil /* iter options */, &c.bytesIterated) if err != nil { return nil, errors.Wrapf(err, "pebble: could not open table %s", errors.Safe(f.FileNum)) } @@ -776,10 +1021,10 @@ func (c *compaction) newInputIter(newIters tableNewIters) (_ internalIterator, r } } - iters = append(iters, newLevelIter(iterOpts, c.cmp, newIters, c.inputs[1], - manifest.Level(c.outputLevel), &c.bytesIterated)) - iters = append(iters, newLevelIter(iterOpts, c.cmp, newRangeDelIter, c.inputs[1], - manifest.Level(c.outputLevel), &c.bytesIterated)) + iters, err = addItersForLevel(iters, c.outputLevel) + if err != nil { + return nil, err + } return newMergingIter(c.logger, c.cmp, iters...), nil } @@ -790,12 +1035,13 @@ func (c *compaction) String() string { var buf bytes.Buffer for i := range c.inputs { - level := c.startLevel + level := c.startLevel.level if i == 1 { - level = c.outputLevel + level = c.outputLevel.level } fmt.Fprintf(&buf, "%d:", level) - for _, f := range c.inputs[i] { + iter := c.inputs[i].files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { fmt.Fprintf(&buf, " %s:%s-%s", f.FileNum, f.Smallest, f.Largest) } fmt.Fprintf(&buf, "\n") @@ -817,14 +1063,15 @@ type manualCompaction struct { func (d *DB) addInProgressCompaction(c *compaction) { d.mu.compact.inProgress[c] = struct{}{} var isBase, isIntraL0 bool - for _, files := range c.inputs { - for _, f := range files { + for _, cl := range c.inputs { + iter := cl.files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { if f.Compacting { - d.opts.Logger.Fatalf("L%d->L%d: %s already being compacted", c.startLevel, c.outputLevel, f.FileNum) + d.opts.Logger.Fatalf("L%d->L%d: %s already being compacted", c.startLevel.level, c.outputLevel.level, f.FileNum) } f.Compacting = true - if c.startLevel == 0 { - if c.outputLevel == 0 { + if c.startLevel != nil && c.outputLevel != nil && c.startLevel.level == 0 { + if c.outputLevel.level == 0 { f.IsIntraL0Compacting = true isIntraL0 = true } else { @@ -835,9 +1082,9 @@ func (d *DB) addInProgressCompaction(c *compaction) { } if (isIntraL0 || isBase) && c.version.L0Sublevels != nil { - l0Inputs := [][]*fileMetadata{c.inputs[0]} + l0Inputs := []manifest.LevelSlice{c.startLevel.files} if isIntraL0 { - l0Inputs = c.inputs[:] + l0Inputs = append(l0Inputs, c.outputLevel.files) } if err := c.version.L0Sublevels.UpdateStateForStartedCompaction(l0Inputs, isBase); err != nil { d.opts.Logger.Fatalf("could not update state for compaction: %s", err) @@ -852,10 +1099,10 @@ func (d *DB) addInProgressCompaction(c *compaction) { strs := make([]string, 0, len(d.mu.compact.inProgress)) for c := range d.mu.compact.inProgress { var s string - if c.startLevel == -1 { - s = fmt.Sprintf("mem->L%d", c.outputLevel) + if c.startLevel.level == -1 { + s = fmt.Sprintf("mem->L%d", c.outputLevel.level) } else { - s = fmt.Sprintf("L%d->L%d:%.1f", c.startLevel, c.outputLevel, c.score) + s = fmt.Sprintf("L%d->L%d:%.1f", c.startLevel.level, c.outputLevel.level, c.score) } strs = append(strs, s) } @@ -875,21 +1122,24 @@ func (d *DB) addInProgressCompaction(c *compaction) { // DB.mu must be held when calling this method. All writes to the manifest // for this compaction should have completed by this point. func (d *DB) removeInProgressCompaction(c *compaction) { - for _, files := range c.inputs { - for _, f := range files { + for _, cl := range c.inputs { + iter := cl.files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { if !f.Compacting { - d.opts.Logger.Fatalf("L%d->L%d: %s already being compacted", c.startLevel, c.outputLevel, f.FileNum) + d.opts.Logger.Fatalf("L%d->L%d: %s not being compacted", c.startLevel.level, c.outputLevel.level, f.FileNum) } f.Compacting = false f.IsIntraL0Compacting = false } } delete(d.mu.compact.inProgress, c) - d.mu.versions.currentVersion().L0Sublevels.InitCompactingFileInfo() + + l0InProgress := inProgressL0Compactions(d.getInProgressCompactionInfoLocked(c)) + d.mu.versions.currentVersion().L0Sublevels.InitCompactingFileInfo(l0InProgress) } func (d *DB) getCompactionPacerInfo() compactionPacerInfo { - bytesFlushed := atomic.LoadUint64(&d.bytesFlushed) + bytesFlushed := atomic.LoadUint64(&d.atomic.bytesFlushed) d.mu.Lock() estimatedMaxWAmp := d.mu.versions.picker.getEstimatedMaxWAmp() @@ -919,7 +1169,7 @@ func (d *DB) getFlushPacerInfo() flushPacerInfo { // // d.mu must be held when calling this. func (d *DB) maybeScheduleFlush() { - if d.mu.compact.flushing || atomic.LoadInt32(&d.closed) != 0 || d.opts.ReadOnly { + if d.mu.compact.flushing || d.closed.Load() != nil || d.opts.ReadOnly { return } if len(d.mu.mem.queue) <= 1 { @@ -958,6 +1208,53 @@ func (d *DB) maybeScheduleFlush() { go d.flush() } +func (d *DB) maybeScheduleDelayedFlush(tbl *memTable) { + var mem *flushableEntry + for _, m := range d.mu.mem.queue { + if m.flushable == tbl { + mem = m + break + } + } + if mem == nil || mem.flushForced || mem.delayedFlushForced { + return + } + mem.delayedFlushForced = true + go func() { + timer := time.NewTimer(d.opts.Experimental.DeleteRangeFlushDelay) + defer timer.Stop() + + select { + case <-d.closedCh: + return + case <-mem.flushed: + return + case <-timer.C: + d.commit.mu.Lock() + defer d.commit.mu.Unlock() + d.mu.Lock() + defer d.mu.Unlock() + + // NB: The timer may fire concurrently with a call to Close. If a + // Close call beat us to acquiring d.mu, d.closed is 1, and it's + // too late to flush anything. Otherwise, the Close call will + // block on locking d.mu until we've finished scheduling the flush + // and set `d.mu.compact.flushing` to true. Close will wait for + // the current flush to complete. + if d.closed.Load() != nil { + return + } + + if d.mu.mem.mutable == tbl { + d.makeRoomForWrite(nil) + } else { + mem.flushForced = true + d.maybeScheduleFlush() + } + } + }() +} + func (d *DB) flush() { pprof.Do(context.Background(), flushLabels, func(context.Context) { d.mu.Lock() @@ -1007,14 +1304,16 @@ func (d *DB) flush1() error { } c := newFlush(d.opts, d.mu.versions.currentVersion(), - d.mu.versions.picker.getBaseLevel(), d.mu.mem.queue[:n], &d.bytesFlushed) + d.mu.versions.picker.getBaseLevel(), d.mu.mem.queue[:n], &d.atomic.bytesFlushed) d.addInProgressCompaction(c) jobID := d.mu.nextJobID d.mu.nextJobID++ d.opts.EventListener.FlushBegin(FlushInfo{ JobID: jobID, + Input: n, }) + startTime := d.timeNow() flushPacer := (pacer)(nilPacer) if d.opts.private.enablePacing { @@ -1029,9 +1328,11 @@ func (d *DB) flush1() error { ve, pendingOutputs, err := d.runCompaction(jobID, c, flushPacer) info := FlushInfo{ - JobID: jobID, - Done: true, - Err: err, + JobID: jobID, + Input: n, + Duration: d.timeNow().Sub(startTime), + Done: true, + Err: err, } if err == nil { for i := range ve.NewFiles { @@ -1060,12 +1361,14 @@ func (d *DB) flush1() error { } } + d.maybeUpdateDeleteCompactionHints(c) d.removeInProgressCompaction(c) + d.mu.versions.incrementCompactionBytes(-c.bytesWritten) d.mu.versions.incrementFlushes() d.opts.EventListener.FlushEnd(info) // Refresh bytes flushed count. - atomic.StoreUint64(&d.bytesFlushed, 0) + atomic.StoreUint64(&d.atomic.bytesFlushed, 0) var flushed flushableList if err == nil { @@ -1097,7 +1400,25 @@ func (d *DB) flush1() error { // // d.mu must be held when calling this. func (d *DB) maybeScheduleCompaction() { - if atomic.LoadInt32(&d.closed) != 0 || d.opts.ReadOnly { + d.maybeScheduleCompactionPicker(pickAuto) +} + +func pickAuto(picker compactionPicker, env compactionEnv) *pickedCompaction { + return picker.pickAuto(env) +} + +func pickElisionOnly(picker compactionPicker, env compactionEnv) *pickedCompaction { + return picker.pickElisionOnlyCompaction(env) +} + +// maybeScheduleCompactionPicker schedules a compaction if necessary, +// calling `pickFunc` to pick automatic compactions. +// +// d.mu must be held when calling this. +func (d *DB) maybeScheduleCompactionPicker( + pickFunc func(compactionPicker, compactionEnv) *pickedCompaction, +) { + if d.closed.Load() != nil || d.opts.ReadOnly { return } if d.mu.compact.compactingCount >= d.opts.MaxConcurrentCompactions { @@ -1117,19 +1438,40 @@ func (d *DB) maybeScheduleCompaction() { // Check for the closed flag again, in case the DB was closed while we were // waiting for logLock(). - if atomic.LoadInt32(&d.closed) != 0 { + if d.closed.Load() != nil { return } env := compactionEnv{ - bytesCompacted: &d.bytesCompacted, + bytesCompacted: &d.atomic.bytesCompacted, + earliestSnapshotSeqNum: d.mu.snapshots.earliest(), earliestUnflushedSeqNum: d.getEarliestUnflushedSeqNumLocked(), } + + // Check for delete-only compactions first, because they're expected to be + // cheap and reduce future compaction work. + if len(d.mu.compact.deletionHints) > 0 && + d.mu.compact.compactingCount < d.opts.MaxConcurrentCompactions && + !d.opts.private.disableAutomaticCompactions { + v := d.mu.versions.currentVersion() + snapshots := d.mu.snapshots.toSlice() + inputs, unresolvedHints := checkDeleteCompactionHints(d.cmp, v, d.mu.compact.deletionHints, snapshots) + d.mu.compact.deletionHints = unresolvedHints + + if len(inputs) > 0 { + c := newDeleteOnlyCompaction(d.opts, v, inputs) + d.mu.compact.compactingCount++ + d.addInProgressCompaction(c) + go d.compact(c, nil) + } + } + for len(d.mu.compact.manual) > 0 && d.mu.compact.compactingCount < d.opts.MaxConcurrentCompactions { manual := d.mu.compact.manual[0] env.inProgressCompactions = d.getInProgressCompactionInfoLocked(nil) - c, retryLater := d.mu.versions.picker.pickManual(env, manual) - if c != nil { + pc, retryLater := d.mu.versions.picker.pickManual(env, manual) + if pc != nil { + c := newCompaction(pc, d.opts, env.bytesCompacted) d.mu.compact.manual = d.mu.compact.manual[1:] d.mu.compact.compactingCount++ d.addInProgressCompaction(c) @@ -1147,16 +1489,214 @@ func (d *DB) maybeScheduleCompaction() { for !d.opts.private.disableAutomaticCompactions && d.mu.compact.compactingCount < d.opts.MaxConcurrentCompactions { env.inProgressCompactions = d.getInProgressCompactionInfoLocked(nil) - c := d.mu.versions.picker.pickAuto(env) - if c == nil { + pc := pickFunc(d.mu.versions.picker, env) + if pc == nil { break } + c := newCompaction(pc, d.opts, env.bytesCompacted) d.mu.compact.compactingCount++ d.addInProgressCompaction(c) go d.compact(c, nil) } } +// A deleteCompactionHint records a user key and sequence number span that has been +// deleted by a range tombstone. A hint is recorded if at least one sstable +// falls completely within both the user key and sequence number spans. +// Once the tombstones and the observed completely-contained sstables fall +// into the same snapshot stripe, a delete-only compaction may delete any +// sstables within the range. +type deleteCompactionHint struct { + start []byte + end []byte + // The level of the file containing the range tombstone(s) when the hint + // was created. Only lower levels need to be searched for files that may + // be deleted. + tombstoneLevel int + // The file containing the range tombstone(s) that created the hint. + tombstoneFile *fileMetadata + // The smallest and largest sequence numbers of the abutting tombstones + // merged to form this hint. All of a tables' keys must be less than the + // tombstone smallest sequence number to be deleted. All of a tables' + // sequence numbers must fall into the same snapshot stripe as the + // tombstone largest sequence number to be deleted. + tombstoneLargestSeqNum uint64 + tombstoneSmallestSeqNum uint64 + // The smallest sequence number of a sstable that was found to be covered + // by this hint. The hint cannot be resolved until this sequence number is + // in the same snapshot stripe as the largest tombstone sequence number. + // This is set when a hint is created, so the LSM may look different and + // notably no longer contian the sstable that contained the key at this + // sequence number. + fileSmallestSeqNum uint64 +} + +func (h deleteCompactionHint) String() string { + return fmt.Sprintf("L%d.%s %s-%s seqnums(tombstone=%d-%d, file-smallest=%d)", + h.tombstoneLevel, h.tombstoneFile.FileNum, h.start, h.end, + h.tombstoneSmallestSeqNum, h.tombstoneLargestSeqNum, h.fileSmallestSeqNum) +} + +func (h *deleteCompactionHint) canDelete(cmp Compare, m *fileMetadata, snapshots []uint64) bool { + // The file can only be deleted if all of its keys are older than the + // earliest tombstone aggregated into the hint. + if m.LargestSeqNum >= h.tombstoneSmallestSeqNum || m.SmallestSeqNum < h.fileSmallestSeqNum { + return false + } + + // The file's oldest key must be in the same snapshot stripe as the + // newest tombstone. NB: We already checked the hint's sequence numbers, + // but this file's oldest sequence number might be lower than the hint's + // smallest sequence number despite the file falling within the key range + // if this file was constructed after the hint by a compaction. + ti, _ := snapshotIndex(h.tombstoneLargestSeqNum, snapshots) + fi, _ := snapshotIndex(m.SmallestSeqNum, snapshots) + if ti != fi { + return false + } + + // The file's keys must be completely contianed within the hint range. + return cmp(h.start, m.Smallest.UserKey) <= 0 && cmp(m.Largest.UserKey, h.end) < 0 +} + +func (d *DB) maybeUpdateDeleteCompactionHints(c *compaction) { + // Compactions that zero sequence numbers can interfere with compaction + // deletion hints. Deletion hints apply to tables containing keys older + // than a threshold. If a key more recent than the threshold is zeroed in + // a compaction, a delete-only compaction may mistake it as meeting the + // threshold and drop a table containing live data. + // + // To avoid this scenario, compactions that zero sequence numbers remove + // any conflicting deletion hints. A deletion hint is conflicting if both + // of the following conditions apply: + // * its key space overlaps with the compaction + // * at least one of its inputs contains a key as recent as one of the + // hint's tombstones. + // + if !c.allowedZeroSeqNum { + return + } + + updatedHints := d.mu.compact.deletionHints[:0] + for _, h := range d.mu.compact.deletionHints { + // If the compaction's key space is disjoint from the hint's key + // space, the zeroing of sequence numbers won't affect the hint. Keep + // the hint. + keysDisjoint := d.cmp(h.end, c.smallest.UserKey) < 0 || d.cmp(h.start, c.largest.UserKey) > 0 + if keysDisjoint { + updatedHints = append(updatedHints, h) + continue + } + + // All of the compaction's inputs must be older than the hint's + // tombstones. + inputsOlder := true + for _, in := range c.inputs { + iter := in.files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { + inputsOlder = inputsOlder && f.LargestSeqNum < h.tombstoneSmallestSeqNum + } + } + if inputsOlder { + updatedHints = append(updatedHints, h) + continue + } + + // Drop h, because the compaction c may have zeroed sequence numbers + // of keys more recent than some of h's tombstones. + } + d.mu.compact.deletionHints = updatedHints +} + +func checkDeleteCompactionHints( + cmp Compare, v *version, hints []deleteCompactionHint, snapshots []uint64, +) ([]compactionLevel, []deleteCompactionHint) { + var files map[*fileMetadata]bool + var byLevel [numLevels][]*fileMetadata + + unresolvedHints := hints[:0] + for _, h := range hints { + // Check each compaction hint to see if it's resolvable. Resolvable + // hints are removed and trigger a delete-only compaction if any files + // in the current LSM still meet their criteria. Unresolvable hints + // are saved and don't trigger a delete-only compaction. + // + // When a compaction hint is created, the sequence numbers of the + // range tombstones and the covered file with the oldest key are + // recorded. The largest tombstone sequence number and the smallest + // file sequence number must be in the same snapshot stripe for the + // hint to be resolved. The below graphic models a compaction hint + // covering the keyspace [b, r). The hint completely contains two + // files, 000002 and 000003. The file 000003 contains the lowest + // covered sequence number at #90. The tombstone b.RANGEDEL.230:h has + // the highest tombstone sequence number incorporated into the hint. + // The hint may be resolved only once the snapshots at #100, #180 and + // #210 are all closed. File 000001 is not included within the hint + // because it extends beyond the range tombstones in user key space. + // + // 250 + // + // |-b...230:h-| + // _____________________________________________________ snapshot #210 + // 200 |--h.RANGEDEL.200:r--| + // + // _____________________________________________________ snapshot #180 + // + // 150 +--------+ + // +---------+ | 000003 | + // | 000002 | | | + // +_________+ | | + // 100_____________________|________|___________________ snapshot #100 + // +--------+ + // _____________________________________________________ snapshot #70 + // +---------------+ + // 50 | 000001 | + // | | + // +---------------+ + // ______________________________________________________________ + // a b c d e f g h i j k l m n o p q r s t u v w x y z + + ti, _ := snapshotIndex(h.tombstoneLargestSeqNum, snapshots) + fi, _ := snapshotIndex(h.fileSmallestSeqNum, snapshots) + if ti != fi { + // Cannot resolve yet. + unresolvedHints = append(unresolvedHints, h) + continue + } + + // The hint h will be resolved and dropped, regardless of whether + // there are any tables that can be deleted. + for l := h.tombstoneLevel + 1; l < numLevels; l++ { + overlaps := v.Overlaps(l, cmp, h.start, h.end).Iter() + for m := overlaps.First(); m != nil; m = overlaps.Next() { + if m.Compacting || !h.canDelete(cmp, m, snapshots) || files[m] { + continue + } + + if files == nil { + // Construct files lazily, assuming most calls will not + // produce delete-only compactions. + files = make(map[*fileMetadata]bool) + } + files[m] = true + byLevel[l] = append(byLevel[l], m) + } + } + } + + var compactLevels []compactionLevel + for l, files := range byLevel { + if len(files) == 0 { + continue + } + compactLevels = append(compactLevels, compactionLevel{ + level: l, + files: manifest.NewLevelSlice(files), + }) + } + return compactLevels, unresolvedHints +} + // compact runs one compaction and maybe schedules another call to compact. func (d *DB) compact(c *compaction, errChannel chan error) { pprof.Do(context.Background(), compactLabels, func(context.Context) { @@ -1187,17 +1727,7 @@ func (d *DB) compact1(c *compaction, errChannel chan error) (err error) { jobID := d.mu.nextJobID d.mu.nextJobID++ - info := CompactionInfo{ - JobID: jobID, - } - info.Input.Level = c.startLevel - info.Output.Level = c.outputLevel - for i := range c.inputs { - for j := range c.inputs[i] { - m := c.inputs[i][j] - info.Input.Tables[i] = append(info.Input.Tables[i], m.TableInfo()) - } - } + info := c.makeInfo(jobID) d.opts.EventListener.CompactionBegin(info) startTime := d.timeNow() @@ -1234,8 +1764,10 @@ func (d *DB) compact1(c *compaction, errChannel chan error) (err error) { } } + d.maybeUpdateDeleteCompactionHints(c) d.removeInProgressCompaction(c) d.mu.versions.incrementCompactions() + d.mu.versions.incrementCompactionBytes(-c.bytesWritten) d.opts.EventListener.CompactionEnd(info) // Update the read state before deleting obsolete files because the @@ -1259,24 +1791,54 @@ func (d *DB) compact1(c *compaction, errChannel chan error) (err error) { func (d *DB) runCompaction( jobID int, c *compaction, pacer pacer, ) (ve *versionEdit, pendingOutputs []FileNum, retErr error) { + // Check for a delete-only compaction. This can occur when wide range + // tombstones completely contain sstables. + if c.kind == compactionKindDeleteOnly { + c.metrics = make(map[int]*LevelMetrics, len(c.inputs)) + ve := &versionEdit{ + DeletedFiles: map[deletedFileEntry]bool{}, + } + for _, cl := range c.inputs { + levelMetrics := &LevelMetrics{} + iter := cl.files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { + levelMetrics.NumFiles-- + levelMetrics.Size -= int64(f.Size) + ve.DeletedFiles[deletedFileEntry{ + Level: cl.level, + FileNum: f.FileNum, + }] = true + } + c.metrics[cl.level] = levelMetrics + } + return ve, nil, nil + } + // Check for a trivial move of one table from one level to the next. We avoid // such a move if there is lots of overlapping grandparent data. Otherwise, // the move could create a parent file that will require a very expensive // merge later on. - if c.trivialMove() { - meta := c.inputs[0][0] + if c.kind == compactionKindMove { + iter := c.startLevel.files.Iter() + meta := iter.First() c.metrics = map[int]*LevelMetrics{ - c.outputLevel: &LevelMetrics{ + c.startLevel.level: { + NumFiles: -1, + Size: -int64(meta.Size), + }, + c.outputLevel.level: { + NumFiles: 1, + Size: int64(meta.Size), BytesMoved: meta.Size, TablesMoved: 1, }, } ve := &versionEdit{ DeletedFiles: map[deletedFileEntry]bool{ - deletedFileEntry{Level: c.startLevel, FileNum: meta.FileNum}: true, + {Level: c.startLevel.level, FileNum: meta.FileNum}: true, }, NewFiles: []newFileEntry{ - {Level: c.outputLevel, Meta: meta}, + {Level: c.outputLevel.level, Meta: meta}, }, } return ve, nil, nil @@ -1299,9 +1861,9 @@ func (d *DB) runCompaction( if err != nil { return nil, pendingOutputs, err } - allowZeroSeqNum := c.allowZeroSeqNum(iiter) - iter := newCompactionIter(c.cmp, d.merge, iiter, snapshots, &c.rangeDelFrag, - allowZeroSeqNum, c.elideTombstone, c.elideRangeTombstone) + c.allowedZeroSeqNum = c.allowZeroSeqNum(iiter) + iter := newCompactionIter(c.cmp, c.formatKey, d.merge, iiter, snapshots, + &c.rangeDelFrag, c.allowedZeroSeqNum, c.elideTombstone, c.elideRangeTombstone) var ( filenames []string @@ -1328,16 +1890,19 @@ func (d *DB) runCompaction( DeletedFiles: map[deletedFileEntry]bool{}, } - metrics := &LevelMetrics{ - BytesIn: totalSize(c.inputs[0]), - BytesRead: totalSize(c.inputs[1]), + outputMetrics := &LevelMetrics{ + BytesIn: c.startLevel.files.SizeSum(), + BytesRead: c.outputLevel.files.SizeSum(), } - metrics.BytesRead += metrics.BytesIn + outputMetrics.BytesRead += outputMetrics.BytesIn c.metrics = map[int]*LevelMetrics{ - c.outputLevel: metrics, + c.outputLevel.level: outputMetrics, + } + if len(c.flushing) == 0 && c.metrics[c.startLevel.level] == nil { + c.metrics[c.startLevel.level] = &LevelMetrics{} } - writerOpts := d.opts.MakeWriterOptions(c.outputLevel) + writerOpts := d.opts.MakeWriterOptions(c.outputLevel.level) newOutput := func() error { d.mu.Lock() @@ -1363,13 +1928,18 @@ func (d *DB) runCompaction( file = vfs.NewSyncingFile(file, vfs.SyncingFileOptions{ BytesPerSync: d.opts.BytesPerSync, }) + file = &compactionFile{ + File: file, + versions: d.mu.versions, + written: &c.bytesWritten, + } filenames = append(filenames, filename) cacheOpts := private.SSTableCacheOpts(d.cacheID, fileNum).(sstable.WriterOption) internalTableOpt := private.SSTableInternalTableOpt.(sstable.WriterOption) tw = sstable.NewWriter(file, writerOpts, cacheOpts, internalTableOpt) ve.NewFiles = append(ve.NewFiles, newFileEntry{ - Level: c.outputLevel, + Level: c.outputLevel.level, Meta: &fileMetadata{ FileNum: fileNum, CreationTime: time.Now().Unix(), @@ -1378,20 +1948,64 @@ func (d *DB) runCompaction( return nil } - splittingFlush := c.startLevel < 0 && c.outputLevel == 0 && d.opts.Experimental.FlushSplitBytes > 0 + splitL0Outputs := c.outputLevel.level == 0 && d.opts.Experimental.FlushSplitBytes > 0 // finishOutput is called for an sstable with the first key of the next sstable, and for the // last sstable with an empty key. finishOutput := func(key []byte) error { + // If we haven't output any point records to the sstable (tw == nil) + // then the sstable will only contain range tombstones. The smallest + // key in the sstable will be the start key of the first range + // tombstone added. We need to ensure that this start key is distinct + // from the limit (key) passed to finishOutput, otherwise we would + // generate an sstable where the largest key is smaller than the + // smallest key due to how the largest key boundary is set below. + // TODO: It is unfortunate that we have to do this check here rather + // than when we decide to finish the sstable in the runCompaction + // loop. A better structure currently eludes us. + if tw == nil { + startKey := c.rangeDelFrag.Start() + if len(iter.tombstones) > 0 { + startKey = iter.tombstones[0].Start.UserKey + } + if d.cmp(startKey, key) == 0 { + return nil + } + } + // NB: clone the key because the data can be held on to by the call to // compactionIter.Tombstones via rangedel.Fragmenter.FlushTo. key = append([]byte(nil), key...) - for _, v := range iter.Tombstones(key, splittingFlush) { + for _, v := range iter.Tombstones(key, splitL0Outputs) { if tw == nil { if err := newOutput(); err != nil { return err } } + // The tombstone being added could be completely outside the + // eventual bounds of the sstable. Consider this example (bounds + // in square brackets next to table filename): + // + // ./000240.sst [tmgc#391,MERGE-tmgc#391,MERGE] + // tmgc#391,MERGE [786e627a] + // tmgc-udkatvs#331,RANGEDEL + // + // ./000241.sst [tmgc#384,MERGE-tmgc#384,MERGE] + // tmgc#384,MERGE [666c7070] + // tmgc-tvsalezade#383,RANGEDEL + // tmgc-tvsalezade#331,RANGEDEL + // + // ./000242.sst [tmgc#383,RANGEDEL-tvsalezade#72057594037927935,RANGEDEL] + // tmgc-tvsalezade#383,RANGEDEL + // tmgc#375,SET [72646c78766965616c72776865676e79] + // tmgc-tvsalezade#356,RANGEDEL + // + // Note that both of the top two SSTables have range tombstones + // that start after the file's end keys. Since the file bound + // computation happens well after all range tombstones have been + // added to the writer, eliding out-of-file range tombstones based + // on sequence number at this stage is difficult, and necessitates + // read-time logic to ignore range tombstones outside file bounds. if err := tw.Add(v.Start, v.End); err != nil { return err } @@ -1418,20 +2032,17 @@ func (d *DB) runCompaction( meta.MarkedForCompaction = writerMeta.MarkedForCompaction // If the file didn't contain any range deletions, we can fill its // table stats now, avoiding unnecessarily loading the table later. - if writerMeta.Properties.NumRangeDeletions == 0 { - meta.Stats = manifest.TableStats{ - Valid: true, - RangeDeletionsBytesEstimate: 0, - } - } + maybeSetStatsFromProperties(meta, &writerMeta.Properties) if c.flushing == nil { - metrics.TablesCompacted++ - metrics.BytesCompacted += meta.Size + outputMetrics.TablesCompacted++ + outputMetrics.BytesCompacted += meta.Size } else { - metrics.TablesFlushed++ - metrics.BytesFlushed += meta.Size + outputMetrics.TablesFlushed++ + outputMetrics.BytesFlushed += meta.Size } + outputMetrics.Size += int64(meta.Size) + outputMetrics.NumFiles++ // The handling of range boundaries is a bit complicated. if n := len(ve.NewFiles); n > 1 { @@ -1524,7 +2135,7 @@ func (d *DB) runCompaction( if meta.Largest.Trailer >= c.largest.Trailer { break } - if allowZeroSeqNum && meta.Largest.SeqNum() == 0 { + if c.allowedZeroSeqNum && meta.Largest.SeqNum() == 0 { break } fallthrough @@ -1534,9 +2145,79 @@ func (d *DB) runCompaction( c.largest.Pretty(d.opts.Comparer.FormatKey)) } } + // Verify that when splitting an output to L0, we never split different + // revisions of the same user key across two different sstables. + if splitL0Outputs { + if err := c.errorOnUserKeyOverlap(ve); err != nil { + return err + } + } + if err := meta.Validate(d.cmp, d.opts.Comparer.FormatKey); err != nil { + return err + } return nil } + // compactionOutputSplitters contain all logic to determine whether the + // compaction loop should stop writing to one output sstable and switch to + // a new one. Some splitters can wrap other splitters, and + // the splitterGroup can be composed of multiple splitters. In this case, + // we start off with splitters for file sizes, grandparent limits, and (for + // L0 splits) L0 limits, before wrapping them in an splitterGroup. + // + // There is a complication here: we can't split outputs where the largest + // key on the left side has a seqnum of zero. This limitation + // exists because a range tombstone which extends into the next sstable + // will cause the smallest key for the next sstable to have the same user + // key, but we need the two tables to be disjoint in key space. Consider + // the scenario: + // + // a#RANGEDEL-c,3 b#SET,0 + // + // If b#SET,0 is the last key added to an sstable, the range tombstone + // [b-c)#3 will extend into the next sstable. The boundary generation + // code in finishOutput() will compute the smallest key for that sstable + // as b#RANGEDEL,3 which sorts before b#SET,0. Normally we just adjust + // the seqnum of this key, but that isn't possible for seqnum 0. To ensure + // we only split where the previous point key has a zero seqnum, we wrap + // our splitters with a nonZeroSeqNumSplitter. + // + // Another case where we may not be able to switch SSTables right away is + // when we are splitting an L0 output. We do not split the same user key + // across different sstables within one flush, so the userKeyChangeSplitter + // ensures we are at a user key change boundary when doing a split. + outputSplitters := []compactionOutputSplitter{ + &fileSizeSplitter{maxFileSize: c.maxOutputFileSize}, + &grandparentLimitSplitter{c: c, ve: ve}, + } + var splitter compactionOutputSplitter + if splitL0Outputs { + // outputSplitters[0] is the file size splitter, which doesn't guarantee + // that all advised splits will be at user key change boundaries. Wrap + // it in a userKeyChangeSplitter. + outputSplitters[0] = &userKeyChangeSplitter{ + cmp: c.cmp, + splitter: outputSplitters[0], + } + outputSplitters = append(outputSplitters, &l0LimitSplitter{c: c, ve: ve}) + } + splitter = &splitterGroup{ + cmp: c.cmp, + splitters: outputSplitters, + } + // Compactions to L0 don't need nonzero last-point-key seqnums at split + // boundaries because when writing to L0, we are able to guarantee that + // the end key of tombstones will also be truncated (through the + // TruncateAndFlushTo call), and no user keys will + // be split between sstables. So a nonZeroSeqNumSplitter is unnecessary + // in that case. + if !splitL0Outputs { + splitter = &nonZeroSeqNumSplitter{ + c: c, + splitter: splitter, + } + } + // Each outer loop iteration produces one output file. An iteration that // produces a file containing point keys (and optionally range tombstones) // guarantees that the input iterator advanced. An iteration that produces @@ -1546,89 +2227,15 @@ func (d *DB) runCompaction( // progress guarantees ensure that eventually the input iterator will be // exhausted and the range tombstone fragments will all be flushed. for key, val := iter.First(); key != nil || !c.rangeDelFrag.Empty(); { - var limit []byte - if splittingFlush { - // For flushes being split across multiple sstables, call - // findL0Limit to find the next L0 limit. - if key != nil { - limit = c.findL0Limit(key.UserKey) - } else { - // Use the start key of the first pending tombstone to find the - // next limit. All pending tombstones have the same start key. - // We use this as opposed to the end key of the - // last written sstable to effectively handle cases like these: - // - // a.SET.3 - // (L0 limit at b) - // d.RANGEDEL.4:f - // - // In this case, the partition after b has only range deletions, - // so if we were to find the L0 limit after the last written - // key at the split point (key a), we'd get the limit b again, - // and finishOutput() would not advance any further because - // the next range tombstone to write does not start until after - // the L0 split point. - startKey := c.rangeDelFrag.Start() - if startKey != nil { - limit = c.findL0Limit(startKey) - } - } - } else if c.rangeDelFrag.Empty() { - // In this case, `limit` will be a larger user key than `key.UserKey`, or - // nil. In either case, the inner loop will execute at least once to - // process `key`, and the input iterator will be advanced. - limit = c.findGrandparentLimit(key.UserKey) - } else { - // There is a range tombstone spanning from the last file into the - // current one. Therefore this file's smallest boundary will overlap the - // last file's largest boundary. - // - // In this case, `limit` will be a larger user key than the previous - // file's largest key and correspond to a grandparent file's largest user - // key, or nil. Then, it is possible the inner loop executes zero times, - // and the output file contains only range tombstones. That is fine as - // long as the number of times we execute this case is bounded. Since - // `findGrandparentLimit()` returns a strictly larger user key each time - // and it corresponds to a grandparent file largest key, the number of - // times this case can execute is bounded by the number of grandparent - // files (plus one for the final time it returns nil). - // - // n > 0 since we cannot have seen range tombstones at the - // beginning of the first file. - n := len(ve.NewFiles) - limit = c.findGrandparentLimit(ve.NewFiles[n-1].Meta.Largest.UserKey) - } + limit := splitter.onNewOutput(key) // Each inner loop iteration processes one key from the input iterator. - passedGrandparentLimit := false prevPointSeqNum := InternalKeySeqNumMax for ; key != nil; key, val = iter.Next() { - // Break out of this loop and switch to a new sstable if we've reached - // the grandparent limit. There is a complication here: we can't create - // an sstable where the largest key has a seqnum of zero. This limitation - // exists because a range tombstone which extends into the next sstable - // will cause the smallest key for the next sstable to have the same user - // key, but we need the two tables to be disjoint in key space. Consider - // the scenario: - // - // a#RANGEDEL-c,3 b#SET,0 - // - // If b#SET,0 is the last key added to an sstable, the range tombstone - // [b-c)#3 will extend into the next sstable. The boundary generation - // code in finishOutput() will compute the smallest key for that sstable - // as b#RANGEDEL,3 which sorts before b#SET,0. Normally we just adjust - // the seqnum of this key, but that isn't possible for seqnum 0. - if passedGrandparentLimit || (limit != nil && c.cmp(key.UserKey, limit) > 0) { - // The flush split exception exists because, when flushing - // to L0, we are able to guarantee that the end key of - // tombstones will also be truncated (through the - // TruncateAndFlushTo call), and no user keys will - // be split between sstables. - if prevPointSeqNum != 0 || c.rangeDelFrag.Empty() || splittingFlush { - if passedGrandparentLimit || splittingFlush { - limit = key.UserKey - } - if splittingFlush { + if split := splitter.shouldSplitBefore(key, tw); split != noSplit { + if split == splitNow { + limit = key.UserKey + if splitL0Outputs { // Flush all tombstones up until key.UserKey, and // truncate them at that key. // @@ -1639,13 +2246,16 @@ func (d *DB) runCompaction( } break } - // If we can't cut the sstable at limit, we need to ensure that the - // limit is at least the last key added to the table. We can't actually - // use key.UserKey here because it will be invalidated by the next call - // to iter.Next(). Instead, we set a flag indicating that we've passed - // the grandparent limit and clear the limit. If we can stop output at - // a subsequent key, we'll use that key as the new limit. - passedGrandparentLimit = true + // split == splitSoon + // + // Invalidate the limit here. It has probably been exceeded + // by the current key, but we can't split just yet, such as to + // maintain the nonzero sequence number invariant mentioned + // above. Setting limit to nil is okay as it's just a transient + // setting, as when split eventually equals splitNow, we will + // set the limit to the key after that. If the compaction were + // to run out of keys before we get to that point, limit would + // be nil as it should be for all end-of-compaction cases. limit = nil } @@ -1661,31 +2271,6 @@ func (d *DB) runCompaction( c.rangeDelFrag.Add(iter.cloneKey(*key), val) continue } - if tw != nil && tw.EstimatedSize() >= c.maxOutputFileSize { - // Use the next key as the sstable boundary. Note that we already - // checked this key against the grandparent limit above. - if !splittingFlush { - limit = key.UserKey - break - } - // We do not split user keys across different sstables when - // splitting flushes. This includes range tombstones; a range - // tombstone covering a key in a given flush is guaranteed to - // end up in the same sstable as that key. Wide range tombstones - // spanning split boundaries are split to keep this invariant - // true. This invariant exists because L0 compaction picking and - // sublevel construction does not account for atomic compaction - // units, unlike regular compactions. Atomic compaction units - // resulting from user keys being split across sstables - // are an unhelpful side-effects of regular compactions; and the - // only reason it's done for regular compactions is to maintain - // the same behaviour as RocksDB. - // - // Set the limit to a copy of the current key, as the current - // key is backed by compactionIter. The inner loop will break - // and switch outputs when it iterates past this user key. - limit = key.Clone().UserKey - } if tw == nil { if err := newOutput(); err != nil { return nil, pendingOutputs, err @@ -1702,9 +2287,11 @@ func (d *DB) runCompaction( // We ran out of keys and the last key added to the sstable has a zero // seqnum and there are buffered range tombstones, so we're unable to use // the grandparent/flush limit for the sstable boundary. See the example in the - // in the loop above with range tombstones straddling sstables. + // in the loop above with range tombstones straddling sstables. Setting + // limit to nil ensures that we flush the entirety of the rangedel + // fragmenter when writing the last output. limit = nil - case key == nil && splittingFlush && !c.rangeDelFrag.Empty(): + case key == nil && splitL0Outputs && !c.rangeDelFrag.Empty(): // We ran out of keys with flush splits enabled, and have remaining // buffered range tombstones. Set limit to nil so all range // tombstones get flushed in the current sstable. Consider this @@ -1742,14 +2329,13 @@ func (d *DB) runCompaction( } } - for i := range c.inputs { - level := c.startLevel - if i == 1 { - level = c.outputLevel - } - for _, f := range c.inputs[i] { + for _, cl := range c.inputs { + iter := cl.files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { + c.metrics[cl.level].NumFiles-- + c.metrics[cl.level].Size -= int64(f.Size) ve.DeletedFiles[deletedFileEntry{ - Level: level, + Level: cl.level, FileNum: f.FileNum, }] = true } diff --git a/github.com/cockroachdb/pebble/compaction_iter.go b/github.com/cockroachdb/pebble/compaction_iter.go index 5e1c6e36e8..e044d6f714 100644 --- a/github.com/cockroachdb/pebble/compaction_iter.go +++ b/github.com/cockroachdb/pebble/compaction_iter.go @@ -9,6 +9,7 @@ import ( "sort" "github.com/cockroachdb/errors" + "github.com/cockroachdb/pebble/internal/base" "github.com/cockroachdb/pebble/internal/bytealloc" "github.com/cockroachdb/pebble/internal/rangedel" ) @@ -181,6 +182,7 @@ type compactionIter struct { func newCompactionIter( cmp Compare, + formatKey base.FormatKey, merge Merge, iter internalIterator, snapshots []uint64, @@ -200,6 +202,7 @@ func newCompactionIter( elideRangeTombstone: elideRangeTombstone, } i.rangeDelFrag.Cmp = cmp + i.rangeDelFrag.Format = formatKey i.rangeDelFrag.Emit = i.emitRangeDelChunk return i } @@ -318,7 +321,10 @@ func (i *compactionIter) Next() (*InternalKey, []byte) { change = i.mergeNext(valueMerger) } if i.err == nil { - i.value, i.valueCloser, i.err = valueMerger.Finish() + // includesBase is true whenever we've transformed the MERGE record + // into a SET. + includesBase := i.key.Kind() == InternalKeyKindSet + i.value, i.valueCloser, i.err = valueMerger.Finish(includesBase) } if i.err == nil { // A non-skippable entry does not necessarily cover later merge @@ -337,10 +343,13 @@ func (i *compactionIter) Next() (*InternalKey, []byte) { } return &i.key, i.value } + if i.err != nil { + i.err = base.MarkCorruptionError(i.err) + } return nil, nil default: - i.err = errors.Errorf("invalid internal key kind: %d", errors.Safe(i.iterKey.Kind())) + i.err = base.CorruptionErrorf("invalid internal key kind: %d", errors.Safe(i.iterKey.Kind())) return nil, nil } } @@ -492,7 +501,7 @@ func (i *compactionIter) mergeNext(valueMerger ValueMerger) stripeChangeType { } default: - i.err = errors.Errorf("invalid internal key kind: %d", errors.Safe(i.iterKey.Kind())) + i.err = base.CorruptionErrorf("invalid internal key kind: %d", errors.Safe(i.iterKey.Kind())) return sameStripeSkippable } } @@ -528,7 +537,7 @@ func (i *compactionIter) singleDeleteNext() bool { continue default: - i.err = errors.Errorf("invalid internal key kind: %d", errors.Safe(i.iterKey.Kind())) + i.err = base.CorruptionErrorf("invalid internal key kind: %d", errors.Safe(i.iterKey.Kind())) return false } } diff --git a/github.com/cockroachdb/pebble/compaction_picker.go b/github.com/cockroachdb/pebble/compaction_picker.go index 186e16ca50..ba7e4b4992 100644 --- a/github.com/cockroachdb/pebble/compaction_picker.go +++ b/github.com/cockroachdb/pebble/compaction_picker.go @@ -5,10 +5,13 @@ package pebble import ( + "bytes" + "fmt" "math" "sort" "github.com/cockroachdb/pebble/internal/base" + "github.com/cockroachdb/pebble/internal/humanize" "github.com/cockroachdb/pebble/internal/manifest" ) @@ -19,6 +22,7 @@ const minIntraL0Count = 4 type compactionEnv struct { bytesCompacted *uint64 earliestUnflushedSeqNum uint64 + earliestSnapshotSeqNum uint64 inProgressCompactions []compactionInfo } @@ -27,8 +31,9 @@ type compactionPicker interface { getBaseLevel() int getEstimatedMaxWAmp() float64 estimatedCompactionDebt(l0ExtraSize uint64) uint64 - pickAuto(env compactionEnv) (c *compaction) - pickManual(env compactionEnv, manual *manualCompaction) (c *compaction, retryLater bool) + pickAuto(env compactionEnv) (pc *pickedCompaction) + pickManual(env compactionEnv, manual *manualCompaction) (c *pickedCompaction, retryLater bool) + pickElisionOnlyCompaction(env compactionEnv) (pc *pickedCompaction) forceBaseLevel1() } @@ -36,12 +41,34 @@ type compactionPicker interface { // Information about in-progress compactions provided to the compaction picker. These are used to // constrain the new compactions that will be picked. type compactionInfo struct { - startLevel int + inputs []compactionLevel outputLevel int - inputs [2][]*fileMetadata + smallest InternalKey + largest InternalKey } -type sortCompactionLevelsDecreasingScore []pickedCompactionInfo +func (info compactionInfo) String() string { + var buf bytes.Buffer + var largest int + for i, in := range info.inputs { + if i > 0 { + fmt.Fprintf(&buf, " -> ") + } + fmt.Fprintf(&buf, "L%d", in.level) + in.files.Each(func(m *fileMetadata) { + fmt.Fprintf(&buf, " %s", m.FileNum) + }) + if largest < in.level { + largest = in.level + } + } + if largest != info.outputLevel || len(info.inputs) == 1 { + fmt.Fprintf(&buf, " -> L%d", info.outputLevel) + } + return buf.String() +} + +type sortCompactionLevelsDecreasingScore []candidateLevelInfo func (s sortCompactionLevelsDecreasingScore) Len() int { return len(s) @@ -56,28 +83,353 @@ func (s sortCompactionLevelsDecreasingScore) Swap(i, j int) { s[i], s[j] = s[j], s[i] } +// pickedCompaction contains information about a compaction that has already +// been chosen, and is being constructed. Compaction construction info lives in +// this struct, and is copied over into the compaction struct when that's +// created. +type pickedCompaction struct { + cmp Compare + + // score of the chosen compaction. Taken from candidateLevelInfo. + score float64 + + // startLevel is the level that is being compacted. Inputs from startLevel + // and outputLevel will be merged to produce a set of outputLevel files. + startLevel *compactionLevel + // outputLevel is the level that files are being produced in. outputLevel is + // equal to startLevel+1 except when startLevel is 0 in which case it is + // equal to compactionPicker.baseLevel(). + outputLevel *compactionLevel + + inputs []compactionLevel + + // L0-specific compaction info. Set to a non-nil value for all compactions + // where startLevel == 0 that were generated by L0Sublevels. + lcf *manifest.L0CompactionFiles + + // maxOutputFileSize is the maximum size of an individual table created + // during compaction. + maxOutputFileSize uint64 + // maxOverlapBytes is the maximum number of bytes of overlap allowed for a + // single output table with the tables in the grandparent level. + maxOverlapBytes uint64 + // maxExpandedBytes is the maximum size of an expanded compaction. If growing + // a compaction results in a larger size, the original compaction is used + // instead. + maxExpandedBytes uint64 + + // The boundaries of the input data. + smallest InternalKey + largest InternalKey + + version *version +} + +func newPickedCompaction(opts *Options, cur *version, startLevel, baseLevel int) *pickedCompaction { + if startLevel > 0 && startLevel < baseLevel { + panic(fmt.Sprintf("invalid compaction: start level %d should not be empty (base level %d)", + startLevel, baseLevel)) + } + + outputLevel := startLevel + 1 + if startLevel == 0 { + outputLevel = baseLevel + } + if outputLevel >= numLevels-1 { + outputLevel = numLevels - 1 + } + // Output level is in the range [baseLevel,numLevels]. For the purpose of + // determining the target output file size, overlap bytes, and expanded + // bytes, we want to adjust the range to [1,numLevels]. + adjustedOutputLevel := 1 + outputLevel - baseLevel + + pc := &pickedCompaction{ + cmp: opts.Comparer.Compare, + version: cur, + inputs: []compactionLevel{{level: startLevel}, {level: outputLevel}}, + maxOutputFileSize: uint64(opts.Level(adjustedOutputLevel).TargetFileSize), + maxOverlapBytes: maxGrandparentOverlapBytes(opts, adjustedOutputLevel), + maxExpandedBytes: expandedCompactionByteSizeLimit(opts, adjustedOutputLevel), + } + pc.startLevel = &pc.inputs[0] + pc.outputLevel = &pc.inputs[1] + return pc +} + +func newPickedCompactionFromL0( + lcf *manifest.L0CompactionFiles, opts *Options, vers *version, baseLevel int, isBase bool, +) *pickedCompaction { + pc := newPickedCompaction(opts, vers, 0, baseLevel) + pc.lcf = lcf + if !isBase { + pc.outputLevel.level = 0 + } + + // Manually build the compaction as opposed to calling + // pickAutoHelper. This is because L0Sublevels has already added + // any overlapping L0 SSTables that need to be added, and + // because compactions built by L0SSTables do not necessarily + // pick contiguous sequences of files in pc.version.Levels[0]. + files := make([]*manifest.FileMetadata, 0, len(lcf.Files)) + iter := vers.Levels[0].Iter() + for j, f := 0, iter.First(); f != nil; j, f = j+1, iter.Next() { + if lcf.FilesIncluded[j] { + files = append(files, f) + } + } + pc.startLevel.files = manifest.NewLevelSlice(files) + return pc +} + +func (pc *pickedCompaction) setupInputs() bool { + // Expand the initial inputs to a clean cut. + var isCompacting bool + pc.startLevel.files, isCompacting = expandToAtomicUnit(pc.cmp, pc.startLevel.files) + if isCompacting { + return false + } + pc.smallest, pc.largest = manifest.KeyRange(pc.cmp, pc.startLevel.files.Iter()) + + // Determine the sstables in the output level which overlap with the input + // sstables, and then expand those tables to a clean cut. No need to do + // this for intra-L0 compactions; outputLevel.files is left empty for those. + if pc.startLevel.level != pc.outputLevel.level { + pc.outputLevel.files = pc.version.Overlaps(pc.outputLevel.level, pc.cmp, pc.smallest.UserKey, pc.largest.UserKey) + pc.outputLevel.files, isCompacting = expandToAtomicUnit(pc.cmp, pc.outputLevel.files) + if isCompacting { + return false + } + pc.smallest, pc.largest = manifest.KeyRange(pc.cmp, + pc.startLevel.files.Iter(), pc.outputLevel.files.Iter()) + } + + // Grow the sstables in pc.startLevel.level as long as it doesn't affect the number + // of sstables included from pc.outputLevel.level. + if pc.lcf != nil && pc.startLevel.level == 0 && pc.outputLevel.level != 0 { + // Call the L0-specific compaction extension method. Similar logic as + // pc.grow. Additional L0 files are optionally added to the compaction at + // this step. Note that the bounds passed in are not the bounds of the + // compaction, but rather the smallest and largest internal keys that + // the compaction cannot include from L0 without pulling in more Lbase + // files. Consider this example: + // + // L0: c-d e+f g-h + // Lbase: a-b e+f i-j + // a b c d e f g h i j + // + // The e-f files have already been chosen in the compaction. As pulling + // in more LBase files is undesirable, the logic below will pass in + // smallest = b and largest = i to ExtendL0ForBaseCompactionTo, which + // will expand the compaction to include c-d and g-h from L0. The + // bounds passed in are exclusive; the compaction cannot be expanded + // to include files that "touch" it. + smallestBaseKey := base.InvalidInternalKey + largestBaseKey := base.InvalidInternalKey + if pc.outputLevel.files.Empty() { + baseIter := pc.version.Levels[pc.outputLevel.level].Iter() + if sm := baseIter.SeekLT(pc.cmp, pc.smallest.UserKey); sm != nil { + smallestBaseKey = sm.Largest + } + if la := baseIter.SeekGE(pc.cmp, pc.largest.UserKey); la != nil { + largestBaseKey = la.Smallest + } + } else { + // NB: We use Reslice to access the underlying level's files, but + // we discard the returned slice. The pc.outputLevel.files slice + // is not modified. + _ = pc.outputLevel.files.Reslice(func(start, end *manifest.LevelIterator) { + if sm := start.Prev(); sm != nil { + smallestBaseKey = sm.Largest + } + if la := end.Next(); la != nil { + largestBaseKey = la.Smallest + } + }) + } + + oldLcf := *pc.lcf + if pc.version.L0Sublevels.ExtendL0ForBaseCompactionTo(smallestBaseKey, largestBaseKey, pc.lcf) { + var newStartLevelFiles []*fileMetadata + iter := pc.version.Levels[0].Iter() + var sizeSum uint64 + for j, f := 0, iter.First(); f != nil; j, f = j+1, iter.Next() { + if pc.lcf.FilesIncluded[j] { + newStartLevelFiles = append(newStartLevelFiles, f) + sizeSum += f.Size + } + } + if sizeSum+pc.outputLevel.files.SizeSum() < pc.maxExpandedBytes { + pc.startLevel.files = manifest.NewLevelSlice(newStartLevelFiles) + pc.smallest, pc.largest = manifest.KeyRange(pc.cmp, + pc.startLevel.files.Iter(), pc.outputLevel.files.Iter()) + } else { + *pc.lcf = oldLcf + } + } + } else if pc.grow(pc.smallest, pc.largest) { + pc.smallest, pc.largest = manifest.KeyRange(pc.cmp, + pc.startLevel.files.Iter(), pc.outputLevel.files.Iter()) + } + return true +} + +// grow grows the number of inputs at c.level without changing the number of +// c.level+1 files in the compaction, and returns whether the inputs grew. sm +// and la are the smallest and largest InternalKeys in all of the inputs. +func (pc *pickedCompaction) grow(sm, la InternalKey) bool { + if pc.outputLevel.files.Empty() { + return false + } + grow0 := pc.version.Overlaps(pc.startLevel.level, pc.cmp, sm.UserKey, la.UserKey) + grow0, isCompacting := expandToAtomicUnit(pc.cmp, grow0) + if isCompacting { + return false + } + if grow0.Len() <= pc.startLevel.files.Len() { + return false + } + if grow0.SizeSum()+pc.outputLevel.files.SizeSum() >= pc.maxExpandedBytes { + return false + } + sm1, la1 := manifest.KeyRange(pc.cmp, grow0.Iter()) + grow1 := pc.version.Overlaps(pc.outputLevel.level, pc.cmp, sm1.UserKey, la1.UserKey) + grow1, isCompacting = expandToAtomicUnit(pc.cmp, grow1) + if isCompacting { + return false + } + if grow1.Len() != pc.outputLevel.files.Len() { + return false + } + pc.startLevel.files = grow0 + pc.outputLevel.files = grow1 + return true +} + +// expandToAtomicUnit expands the provided level slice within its level both +// forwards and backwards to its "atomic compaction unit" boundaries, if +// necessary. +// +// While picking compaction inputs, this is required to maintain the invariant +// that the versions of keys at level+1 are older than the versions of keys at +// level. Tables are added to the right of the current slice tables such that +// the rightmost table has a "clean cut". A clean cut is either a change in +// user keys, or when the largest key in the left sstable is a range tombstone +// sentinel key (InternalKeyRangeDeleteSentinel). +// +// In addition to maintaining the seqnum invariant, expandToAtomicUnit is used +// to provide clean boundaries for range tombstone truncation during +// compaction. In order to achieve these clean boundaries, expandToAtomicUnit +// needs to find a "clean cut" on the left edge of the compaction as well. +// This is necessary in order for "atomic compaction units" to always be +// compacted as a unit. Failure to do this leads to a subtle bug with +// truncation of range tombstones to atomic compaction unit boundaries. +// Consider the scenario: +// +// L3: +// 12:[a#2,15-b#1,1] +// 13:[b#0,15-d#72057594037927935,15] +// +// These sstables contain a range tombstone [a-d)#2 which spans the two +// sstables. The two sstables need to always be kept together. Compacting +// sstable 13 independently of sstable 12 would result in: +// +// L3: +// 12:[a#2,15-b#1,1] +// L4: +// 14:[b#0,15-d#72057594037927935,15] +// +// This state is still ok, but when sstable 12 is next compacted, its range +// tombstones will be truncated at "b" (the largest key in its atomic +// compaction unit). In the scenario here, that could result in b#1 becoming +// visible when it should be deleted. +// +// isCompacting is returned true for any atomic units that contain files that +// have in-progress compactions, i.e. FileMetadata.Compacting == true. +func expandToAtomicUnit( + cmp Compare, inputs manifest.LevelSlice, +) (slice manifest.LevelSlice, isCompacting bool) { + // NB: Inputs for L0 can't be expanded and *version.Overlaps guarantees + // that we get a 'clean cut.' For L0, Overlaps will return a slice without + // access to the rest of the L0 files, so it's OK to try to reslice. + if inputs.Empty() { + // Nothing to expand. + return inputs, false + } + + inputs = inputs.Reslice(func(start, end *manifest.LevelIterator) { + iter := start.Clone() + iter.Prev() + for cur, prev := start.Current(), iter.Current(); prev != nil; cur, prev = start.Prev(), iter.Prev() { + if cur.Compacting { + isCompacting = true + } + if cmp(prev.Largest.UserKey, cur.Smallest.UserKey) < 0 { + break + } + if prev.Largest.Trailer == InternalKeyRangeDeleteSentinel { + // The range deletion sentinel key is set for the largest key in a + // table when a range deletion tombstone straddles a table. It + // isn't necessary to include the prev table in the atomic + // compaction unit as prev.largest.UserKey does not actually exist + // in the prev table. + break + } + // prev.Largest.UserKey == cur.Smallest.UserKey, so we need to + // include prev in the compaction. + } + + iter = end.Clone() + iter.Next() + for cur, next := end.Current(), iter.Current(); next != nil; cur, next = end.Next(), iter.Next() { + if cur.Compacting { + isCompacting = true + } + if cmp(cur.Largest.UserKey, next.Smallest.UserKey) < 0 { + break + } + if cur.Largest.Trailer == InternalKeyRangeDeleteSentinel { + // The range deletion sentinel key is set for the largest key + // in a table when a range deletion tombstone straddles a + // table. It isn't necessary to include the next table in the + // compaction as PeekPrev().Largest.UserKey does not actually + // exist in the table. + break + } + // cur.Largest.UserKey == next.Smallest.UserKey, so we need to + // include next in the compaction. + } + }) + inputIter := inputs.Iter() + return inputs, isCompacting || inputIter.First().Compacting || inputIter.Last().Compacting +} + func newCompactionPicker( - v *version, opts *Options, inProgressCompactions []compactionInfo, + v *version, + opts *Options, + inProgressCompactions []compactionInfo, + levelSizes [numLevels]int64, ) compactionPicker { p := &compactionPickerByScore{ opts: opts, vers: v, } - p.initLevelMaxBytes(inProgressCompactions) + p.initLevelMaxBytes(inProgressCompactions, levelSizes) return p } -// Information about a candidate compaction that has been identified by the -// compaction picker. -type pickedCompactionInfo struct { +// Information about a candidate compaction level that has been identified by +// the compaction picker. +type candidateLevelInfo struct { // The score of the level to be compacted. - score float64 - level int + score float64 + origScore float64 + level int // The level to compact to. outputLevel int - // The file in level that will be compacted. Additional files may be picked by the + // The file in level that will be compacted. Additional files may be + // picked by the compaction, and a pickedCompaction created for the // compaction. - file int + file manifest.LevelFile } // compensatedSize returns f's file size, inflated according to compaction @@ -90,9 +442,9 @@ func compensatedSize(f *fileMetadata) uint64 { return sz } -func totalCompensatedSize(files []*fileMetadata) uint64 { +func totalCompensatedSize(iter manifest.LevelIterator) uint64 { var sz uint64 - for _, f := range files { + for f := iter.First(); f != nil; f = iter.Next() { sz += compensatedSize(f) } return sz @@ -113,12 +465,17 @@ type compactionPickerByScore struct { // added to L0. estimatedMaxWAmp float64 - // smoothedLevelMultiplier is the size ratio between one level and the next. - smoothedLevelMultiplier float64 - // levelMaxBytes holds the dynamically adjusted max bytes setting for each // level. levelMaxBytes [numLevels]int64 + + // Information for facilitating L6->L6 elision-only compactions. If + // elisionThreshold is non-nil, it's the lowest LargestSeqNum of a file + // that meets the conditions for an elision-only compaction. It may be + // math.MaxUint64 if it's known that no file within L6 meets the + // conditions for an elision-only compaction. + elisionThreshold *uint64 + elisionCandidate *manifest.LevelFile } var _ compactionPicker = &compactionPickerByScore{} @@ -151,8 +508,8 @@ func (p *compactionPickerByScore) estimatedCompactionDebt(l0ExtraSize uint64) ui // We assume that all the bytes in L0 need to be compacted to Lbase. This is // unlike the RocksDB logic that figures out whether L0 needs compaction. - bytesAddedToNextLevel := l0ExtraSize + totalSize(p.vers.Levels[0]) - nextLevelSize := totalSize(p.vers.Levels[p.baseLevel]) + bytesAddedToNextLevel := l0ExtraSize + p.vers.Levels[0].Slice().SizeSum() + nextLevelSize := p.vers.Levels[p.baseLevel].Slice().SizeSum() var compactionDebt uint64 if bytesAddedToNextLevel > 0 && nextLevelSize > 0 { @@ -164,7 +521,7 @@ func (p *compactionPickerByScore) estimatedCompactionDebt(l0ExtraSize uint64) ui for level := p.baseLevel; level < numLevels-1; level++ { levelSize := nextLevelSize + bytesAddedToNextLevel - nextLevelSize = totalSize(p.vers.Levels[level+1]) + nextLevelSize = p.vers.Levels[level+1].Slice().SizeSum() if levelSize > uint64(p.levelMaxBytes[level]) { bytesAddedToNextLevel = levelSize - uint64(p.levelMaxBytes[level]) if nextLevelSize > 0 { @@ -181,14 +538,19 @@ func (p *compactionPickerByScore) estimatedCompactionDebt(l0ExtraSize uint64) ui return compactionDebt } -func (p *compactionPickerByScore) initLevelMaxBytes(inProgressCompactions []compactionInfo) { +func (p *compactionPickerByScore) initLevelMaxBytes( + inProgressCompactions []compactionInfo, levelSizes [numLevels]int64, +) { // The levelMaxBytes calculations here differ from RocksDB in two ways: // - // 1. The use of bottomLevelSize vs maxLevelSize. RocksDB uses the size of - // the maximum level in L1-L6, rather than the size of the bottommost - // non-empty level. In practice this seems to have little impact. + // 1. The use of dbSize vs maxLevelSize. RocksDB uses the size of the maximum + // level in L1-L6, rather than determining the size of the bottom level + // based on the total amount of data in the dB. The RocksDB calculation is + // problematic if L0 contains a significant fraction of data, or if the + // level sizes are roughly equal and thus there is a significant fraction + // of data outside of the largest level. // - // 2. Not adjusting the size of base level based on L0. RocksDB computes + // 2. Not adjusting the size of Lbase based on L0. RocksDB computes // baseBytesMax as the maximum of the configured LBaseMaxBytes and the // size of L0. This is problematic because baseBytesMax is used to compute // the max size of lower levels. A very large baseBytesMax will result in @@ -196,33 +558,23 @@ func (p *compactionPickerByScore) initLevelMaxBytes(inProgressCompactions []comp // those levels not to be compacted even when they should be // compacted. This often results in "inverted" LSM shapes where Ln is // larger than Ln+1. - // - // TODO(peter): An alternative to the current calculation of bottomLevelSize - // is to compute the total number of bytes in the LSM and then compute - // bottomLevelSize as 90% of that value (presuming a level multiplier of - // 10). This computation has the advantage of being stable: it changes at the - // rate that data is inserted into the DB, independently of - // compactions. Unfortunately, it performed worse experimentally and often - // resulted in "inverted" LSM shapes where L5 was significantly larger than - // L6. The reason for this inversion was not clear. - - // Determine the first non-empty level and the bottom level size. + + // Determine the first non-empty level and the total DB size. firstNonEmptyLevel := -1 - var bottomLevelSize int64 + var dbSize int64 for level := 1; level < numLevels; level++ { - levelSize := int64(totalSize(p.vers.Levels[level])) - if levelSize > 0 { + if levelSizes[level] > 0 { if firstNonEmptyLevel == -1 { firstNonEmptyLevel = level } - bottomLevelSize = levelSize + dbSize += levelSizes[level] } } for _, c := range inProgressCompactions { - if c.outputLevel == 0 { + if c.outputLevel == 0 || c.outputLevel == -1 { continue } - if c.startLevel == 0 && (firstNonEmptyLevel == -1 || c.outputLevel < firstNonEmptyLevel) { + if c.inputs[0].level == 0 && (firstNonEmptyLevel == -1 || c.outputLevel < firstNonEmptyLevel) { firstNonEmptyLevel = c.outputLevel } } @@ -234,7 +586,7 @@ func (p *compactionPickerByScore) initLevelMaxBytes(inProgressCompactions []comp p.levelMaxBytes[level] = math.MaxInt64 } - if bottomLevelSize == 0 { + if dbSize == 0 { // No levels for L1 and up contain any data. Target L0 compactions for the // last level or to the level to which there is an ongoing L0 compaction. p.baseLevel = numLevels - 1 @@ -244,44 +596,36 @@ func (p *compactionPickerByScore) initLevelMaxBytes(inProgressCompactions []comp return } - levelMultiplier := 10.0 - - baseBytesMax := p.opts.LBaseMaxBytes - baseBytesMin := int64(float64(baseBytesMax) / levelMultiplier) + const levelMultiplier = 10 + dbSize += levelSizes[0] + bottomLevelSize := dbSize - dbSize/levelMultiplier curLevelSize := bottomLevelSize for level := numLevels - 2; level >= firstNonEmptyLevel; level-- { curLevelSize = int64(float64(curLevelSize) / levelMultiplier) } - if curLevelSize <= baseBytesMin { - // If we make target size of last level to be bottomLevelSize, target size - // of the first non-empty level would be smaller than baseBytesMin. We set - // it be baseBytesMin. - p.baseLevel = firstNonEmptyLevel - } else { - // Compute base level (where L0 data is compacted to). - p.baseLevel = firstNonEmptyLevel - for p.baseLevel > 1 && curLevelSize > baseBytesMax { - p.baseLevel-- - curLevelSize = int64(float64(curLevelSize) / levelMultiplier) - } + // Compute base level (where L0 data is compacted to). + baseBytesMax := p.opts.LBaseMaxBytes + p.baseLevel = firstNonEmptyLevel + for p.baseLevel > 1 && curLevelSize > baseBytesMax { + p.baseLevel-- + curLevelSize = int64(float64(curLevelSize) / levelMultiplier) } + smoothedLevelMultiplier := 1.0 if p.baseLevel < numLevels-1 { - p.smoothedLevelMultiplier = math.Pow( + smoothedLevelMultiplier = math.Pow( float64(bottomLevelSize)/float64(baseBytesMax), 1.0/float64(numLevels-p.baseLevel-1)) - } else { - p.smoothedLevelMultiplier = 1.0 } - p.estimatedMaxWAmp = float64(numLevels-p.baseLevel) * (p.smoothedLevelMultiplier + 1) + p.estimatedMaxWAmp = float64(numLevels-p.baseLevel) * (smoothedLevelMultiplier + 1) levelSize := float64(baseBytesMax) for level := p.baseLevel; level < numLevels; level++ { if level > p.baseLevel && levelSize > 0 { - levelSize *= p.smoothedLevelMultiplier + levelSize *= smoothedLevelMultiplier } // Round the result since test cases use small target level sizes, which // can be impacted by floating-point imprecision + integer truncation. @@ -304,18 +648,26 @@ func calculateSizeAdjust(inProgressCompactions []compactionInfo) [numLevels]int6 var sizeAdjust [numLevels]int64 for i := range inProgressCompactions { c := &inProgressCompactions[i] - compensated := int64(totalCompensatedSize(c.inputs[0])) - real := int64(totalSize(c.inputs[0])) - sizeAdjust[c.startLevel] -= compensated - sizeAdjust[c.outputLevel] += real + + for _, input := range c.inputs { + real := int64(input.files.SizeSum()) + compensated := int64(totalCompensatedSize(input.files.Iter())) + + if input.level != c.outputLevel { + sizeAdjust[input.level] -= compensated + if c.outputLevel != -1 { + sizeAdjust[c.outputLevel] += real + } + } + } } return sizeAdjust } func (p *compactionPickerByScore) calculateScores( inProgressCompactions []compactionInfo, -) [numLevels]pickedCompactionInfo { - var scores [numLevels]pickedCompactionInfo +) [numLevels]candidateLevelInfo { + var scores [numLevels]candidateLevelInfo for i := range scores { scores[i].level = i scores[i].outputLevel = i + 1 @@ -323,31 +675,67 @@ func (p *compactionPickerByScore) calculateScores( scores[0] = p.calculateL0Score(inProgressCompactions) sizeAdjust := calculateSizeAdjust(inProgressCompactions) - for level := 1; level < numLevels-1; level++ { - // Use the "compensated" file size when scoring. The file size is - // compensated by artifically inflating it to account for other - // priorities like reclaiming disk space beneath range tombstones. - levelSize := int64(totalCompensatedSize(p.vers.Levels[level])) + sizeAdjust[level] + for level := 1; level < numLevels; level++ { + levelSize := int64(totalCompensatedSize(p.vers.Levels[level].Iter())) + sizeAdjust[level] scores[level].score = float64(levelSize) / float64(p.levelMaxBytes[level]) + scores[level].origScore = scores[level].score } + + // Adjust each level's score by the score of the next level. If the next + // level has a high score, and is thus a priority for compaction, this + // reduces the priority for compacting the current level. If the next level + // has a low score (i.e. it is below its target size), this increases the + // priority for compacting the current level. + // + // The effect of this adjustment is to help prioritize compactions in lower + // levels. The following shows the adjusted score and original score. In this + // scenario, L0 has 68 sublevels. L3 (a.k.a. Lbase) is significantly above + // its target size. The original score prioritizes compactions from those two + // levels, but doing so ends up causing a future problem: data piles up in + // the higher levels, starving L5->L6 compactions, and to a lesser degree + // starving L4->L5 compactions. + // + // adjusted original + // score score size max-size + // L0 3.2 68.0 2.2 G - + // L3 3.2 21.1 1.3 G 64 M + // L4 3.4 6.7 3.1 G 467 M + // L5 3.4 2.0 6.6 G 3.3 G + // L6 0.6 0.6 14 G 24 G + var prevLevel int + for level := p.baseLevel; level < numLevels; level++ { + if scores[prevLevel].score >= 1 { + // Avoid absurdly large scores by placing a floor on the score that we'll + // adjust a level by. The value of 0.01 was chosen somewhat arbitrarily + const minScore = 0.01 + if scores[level].score >= minScore { + scores[prevLevel].score /= scores[level].score + } else { + scores[prevLevel].score /= minScore + } + } + prevLevel = level + } + sort.Sort(sortCompactionLevelsDecreasingScore(scores[:])) return scores } func (p *compactionPickerByScore) calculateL0Score( inProgressCompactions []compactionInfo, -) pickedCompactionInfo { - var info pickedCompactionInfo +) candidateLevelInfo { + var info candidateLevelInfo info.outputLevel = p.baseLevel if p.opts.Experimental.L0SublevelCompactions { // If L0Sublevels are present, we use the sublevel count as opposed to // the L0 file count to score this level. The base vs intra-L0 // compaction determination happens in pickAuto, not here. - info.score = float64(p.vers.L0Sublevels.MaxDepthAfterOngoingCompactions()) / + info.score = float64(2*p.vers.L0Sublevels.MaxDepthAfterOngoingCompactions()) / float64(p.opts.L0CompactionThreshold) return info } + // TODO(peter): The current scoring logic precludes concurrent L0->Lbase // compactions in most cases because if there is an in-progress L0->Lbase // compaction we'll instead preferentially score an intra-L0 compaction. One @@ -368,11 +756,16 @@ func (p *compactionPickerByScore) calculateL0Score( // Score an L0->Lbase compaction by counting the number of idle // (non-compacting) files in L0. - var idleL0Count int - for _, f := range p.vers.Levels[0] { - if !f.Compacting { + var idleL0Count, totalL0Count, intraL0Count int + iter := p.vers.Levels[0].Iter() + for f := iter.First(); f != nil; f = iter.Next() { + if f.Compacting { + intraL0Count = 0 + } else { idleL0Count++ + intraL0Count++ } + totalL0Count++ } info.score = float64(idleL0Count) / float64(p.opts.L0CompactionThreshold) @@ -380,7 +773,7 @@ func (p *compactionPickerByScore) calculateL0Score( // compaction. var l0Compaction bool for i := range inProgressCompactions { - if inProgressCompactions[i].startLevel == 0 && + if inProgressCompactions[i].inputs[0].level == 0 && inProgressCompactions[i].outputLevel != 0 { l0Compaction = true break @@ -390,21 +783,12 @@ func (p *compactionPickerByScore) calculateL0Score( return info } - l0Files := p.vers.Levels[0] - if len(l0Files) < p.opts.L0CompactionThreshold+2 { + if totalL0Count < p.opts.L0CompactionThreshold+2 { // If L0 isn't accumulating many files beyond the regular L0 trigger, // don't resort to an intra-L0 compaction yet. This matches the RocksDB // heuristic. return info } - var end = len(l0Files) - for ; end >= 1; end-- { - if l0Files[end-1].Compacting { - break - } - } - - intraL0Count := len(l0Files) - end if intraL0Count < minIntraL0Count { // Not enough idle L0 files to perform an intra-L0 compaction. This // matches the RocksDB heuristic. Note that if another file is flushed @@ -420,7 +804,7 @@ func (p *compactionPickerByScore) calculateL0Score( return info } -func (p *compactionPickerByScore) pickFile(level, outputLevel int) int { +func (p *compactionPickerByScore) pickFile(level, outputLevel int) (manifest.LevelFile, bool) { // Select the file within the level to compact. We want to minimize write // amplification, but also ensure that deletes are propagated to the // bottom level in a timely fashion so as to reclaim disk space. A table's @@ -444,31 +828,34 @@ func (p *compactionPickerByScore) pickFile(level, outputLevel int) int { // an in-progress compaction. cmp := p.opts.Comparer.Compare - outputLevelFiles := p.vers.Levels[outputLevel] + startIter := p.vers.Levels[level].Iter() + outputIter := p.vers.Levels[outputLevel].Iter() - file := -1 + var file manifest.LevelFile smallestRatio := uint64(math.MaxUint64) - for i, f := range p.vers.Levels[level] { + outputFile := outputIter.First() + + for f := startIter.First(); f != nil; f = startIter.Next() { var overlappingBytes uint64 // Trim any output-level files smaller than f. - for len(outputLevelFiles) > 0 && base.InternalCompare(cmp, outputLevelFiles[0].Largest, f.Smallest) < 0 { - outputLevelFiles = outputLevelFiles[1:] + for outputFile != nil && base.InternalCompare(cmp, outputFile.Largest, f.Smallest) < 0 { + outputFile = outputIter.Next() } compacting := f.Compacting - for len(outputLevelFiles) > 0 && base.InternalCompare(cmp, outputLevelFiles[0].Smallest, f.Largest) < 0 { - overlappingBytes += outputLevelFiles[0].Size - compacting = compacting || outputLevelFiles[0].Compacting + for outputFile != nil && base.InternalCompare(cmp, outputFile.Smallest, f.Largest) < 0 { + overlappingBytes += outputFile.Size + compacting = compacting || outputFile.Compacting // If the file in the next level extends beyond f's largest key, - // break out and don't trim outputLevelFiles because f's - // successor might also overlap. - if base.InternalCompare(cmp, outputLevelFiles[0].Largest, f.Largest) > 0 { + // break out and don't advance outputIter because f's successor + // might also overlap. + if base.InternalCompare(cmp, outputFile.Largest, f.Largest) > 0 { break } - outputLevelFiles = outputLevelFiles[1:] + outputFile = outputIter.Next() } // If the input level file or one of the overlapping files is @@ -481,10 +868,10 @@ func (p *compactionPickerByScore) pickFile(level, outputLevel int) int { scaledRatio := overlappingBytes * 1024 / compensatedSize(f) if scaledRatio < smallestRatio && !f.Compacting { smallestRatio = scaledRatio - file = i + file = startIter.Take() } } - return file + return file, file.FileMetadata != nil } // pickAuto picks the best compaction, if any. @@ -496,77 +883,135 @@ func (p *compactionPickerByScore) pickFile(level, outputLevel int) int { // // If a score-based compaction cannot be found, pickAuto falls back to looking // for a forced compaction (identified by FileMetadata.MarkedForCompaction). -func (p *compactionPickerByScore) pickAuto(env compactionEnv) (c *compaction) { - // highPriorityThreshold controls compaction concurrency. If there is already - // a compaction in progress, highPriorityThreshold is set to the minimum - // score needed for a concurrent compaction to be initiated. Since all level - // scores are >= 0, a positive value will cause compactions to be - // disabled. We set highPriorityThreshold to a value only when a there is at - // least one in-progress compaction. Concurrent compactions are useful for - // ensuring that compaction doesn't fall far behind, but concurrent - // compactions can have an adverse affect on write throughput. - // - // There are a variety of possibilities for choosing highPriorityThreshold: - // - A fixed value: 1.5. - // - A value linear in the number of in-progress compactions: 2, 3, 4. - // - A value exponential in the number of in-progress compactions: 2, 4, 8. - // - // A fixed value tends to allow too much compaction concurrency. There was - // only a minor difference between the linear and exponential values, making - // the choice of exponential below somewhat arbitrary. - // - // For comparison, RocksDB alternates between only allowing a single - // compaction at a time to allowing the configured maximum number of - // concurrent compactions depending on whether compaction-debt has gotten too - // large or the number of L0 sstables has reached 2x the L0 compaction - // threshold. In testing, it is usually the latter condition that triggers - // concurrent compactions in RocksDB. - var highPriorityThreshold float64 - if len(env.inProgressCompactions) > 0 { - // Exponential high priority threshold: 2, 4, 8, ... - highPriorityThreshold = float64(int(1) << len(env.inProgressCompactions)) +func (p *compactionPickerByScore) pickAuto(env compactionEnv) (pc *pickedCompaction) { + // Compaction concurrency is controlled by L0 read-amp. We allow one + // additional compaction per L0CompactionConcurrency sublevels, as well as + // one additional compaction per CompactionDebtConcurrency bytes of + // compaction debt. Compaction concurrency is tied to L0 sublevels as that + // signal is independent of the database size. We tack on the compaction + // debt as a second signal to prevent compaction concurrency from dropping + // significantly right after a base compaction finishes, and before those + // bytes have been compacted further down the LSM. + if n := len(env.inProgressCompactions); n > 0 { + var l0ReadAmp int + if p.opts.Experimental.L0SublevelCompactions { + l0ReadAmp = p.vers.L0Sublevels.MaxDepthAfterOngoingCompactions() + } else { + l0ReadAmp = p.vers.Levels[0].Slice().Len() + } + compactionDebt := int(p.estimatedCompactionDebt(0)) + ccSignal1 := n * p.opts.Experimental.L0CompactionConcurrency + ccSignal2 := n * p.opts.Experimental.CompactionDebtConcurrency + if l0ReadAmp < ccSignal1 && compactionDebt < ccSignal2 { + return nil + } } scores := p.calculateScores(env.inProgressCompactions) + // TODO(peter): Either remove, or change this into an event sent to the + // EventListener. + logCompaction := func(pc *pickedCompaction) { + var buf bytes.Buffer + for i := 0; i < numLevels; i++ { + if i != 0 && i < p.baseLevel { + continue + } + + var info *candidateLevelInfo + for j := range scores { + if scores[j].level == i { + info = &scores[j] + break + } + } + + marker := " " + if pc.startLevel.level == info.level { + marker = "*" + } + fmt.Fprintf(&buf, " %sL%d: %5.1f %5.1f %8s %8s", + marker, info.level, info.score, info.origScore, + humanize.Int64(int64(totalCompensatedSize(p.vers.Levels[info.level].Iter()))), + humanize.Int64(p.levelMaxBytes[info.level]), + ) + + count := 0 + for i := range env.inProgressCompactions { + c := &env.inProgressCompactions[i] + if c.inputs[0].level != info.level { + continue + } + count++ + if count == 1 { + fmt.Fprintf(&buf, " [") + } else { + fmt.Fprintf(&buf, " ") + } + fmt.Fprintf(&buf, "L%d->L%d", c.inputs[0].level, c.outputLevel) + } + if count > 0 { + fmt.Fprintf(&buf, "]") + } + fmt.Fprintf(&buf, "\n") + } + p.opts.Logger.Infof("pickAuto: L%d->L%d\n%s", + pc.startLevel.level, pc.outputLevel.level, buf.String()) + } + // Check for a score-based compaction. "scores" has been sorted in order of // decreasing score. For each level with a score >= 1, we attempt to find a // compaction anchored at at that level. for i := range scores { info := &scores[i] - if info.score < highPriorityThreshold { - // Don't start a low priority compaction if there is already a compaction - // running. - return nil - } if info.score < 1 { break } + if info.level == numLevels-1 { + continue + } if info.level == 0 && p.opts.Experimental.L0SublevelCompactions { - c = pickL0(env, p.opts, p.vers, p.baseLevel) + pc = pickL0(env, p.opts, p.vers, p.baseLevel) // Fail-safe to protect against compacting the same sstable // concurrently. - if c != nil && !inputAlreadyCompacting(c) { - c.score = info.score - return c + if pc != nil && !inputRangeAlreadyCompacting(env, pc) { + pc.score = info.score + // TODO(peter): remove + if false { + logCompaction(pc) + } + return pc } continue } - info.file = p.pickFile(info.level, info.outputLevel) - if info.file == -1 { + var ok bool + info.file, ok = p.pickFile(info.level, info.outputLevel) + if !ok { continue } - c := pickAutoHelper(env, p.opts, p.vers, *info, p.baseLevel) + pc := pickAutoHelper(env, p.opts, p.vers, *info, p.baseLevel) // Fail-safe to protect against compacting the same sstable concurrently. - if c != nil && !inputAlreadyCompacting(c) { - c.score = info.score - return c + if pc != nil && !inputRangeAlreadyCompacting(env, pc) { + pc.score = info.score + // TODO(peter): remove + if false { + logCompaction(pc) + } + return pc } } + // Check for L6 files with tombstones that may be elided. These files may + // exist if a snapshot prevented the elision of a tombstone or because of + // a move compaction. These are low-priority compactions because they + // don't help us keep up with writes, just reclaim disk space. + if pc := p.pickElisionOnlyCompaction(env); pc != nil { + return pc + } + // Check for forced compactions. These are lower priority than score-based // compactions. Note that this loop only runs if we haven't already found a // score-based compaction. @@ -575,135 +1020,189 @@ func (p *compactionPickerByScore) pickAuto(env compactionEnv) (c *compaction) { // extremely wasteful in the common case. Could we maintain a // MarkedForCompaction map from fileNum to level? for level := 0; level < numLevels-1; level++ { - for file, f := range p.vers.Levels[level] { + iter := p.vers.Levels[level].Iter() + for f := iter.First(); f != nil; f = iter.Next() { if !f.MarkedForCompaction { continue } + var info *candidateLevelInfo for i := range scores { - if scores[i].level != level { - continue - } - info := &scores[i] - info.file = file - c := pickAutoHelper(env, p.opts, p.vers, *info, p.baseLevel) - // Fail-safe to protect against compacting the same sstable concurrently. - if c != nil && !inputAlreadyCompacting(c) { - c.score = info.score - return c + if scores[i].level == level { + info = &scores[i] + break } - break } - break + if info == nil { + panic(fmt.Sprintf("could not find candidate level info for level %d", level)) + } + info.file = iter.Take() + pc := pickAutoHelper(env, p.opts, p.vers, *info, p.baseLevel) + // Fail-safe to protect against compacting the same sstable concurrently. + if pc != nil && !inputRangeAlreadyCompacting(env, pc) { + pc.score = info.score + return pc + } } } + return nil +} - // TODO(peter): When a snapshot is released, we may need to compact tables at - // the bottom level in order to free up entries that were pinned by the - // snapshot. +// pickElisionOnlyCompaction looks for compactions of sstables in the +// bottommost level containing obsolete records that may now be dropped. +func (p *compactionPickerByScore) pickElisionOnlyCompaction( + env compactionEnv, +) (pc *pickedCompaction) { + if p.elisionThreshold == nil || (p.elisionCandidate != nil && p.elisionCandidate.Compacting) { + var lowestCandidateSeqNum uint64 = math.MaxUint64 + var lowestCandidate manifest.LevelFile + iter := p.vers.Levels[numLevels-1].Iter() + for f := iter.First(); f != nil; f = iter.Next() { + if f.Compacting { + continue + } + if f.LargestSeqNum >= lowestCandidateSeqNum { + continue + } + if !f.Stats.Valid { + continue + } + // Bottommost files are large and not worthwhile to compact just + // to remove a few tombstones. Consider a file ineligible if its + // own range deletions delete less than 10% of its data and its + // deletion tombstones make up less than 10% of its entries. + // + // TODO(jackson): This does not account for duplicate user keys + // which may be collapsed. Ideally, we would have 'obsolete keys' + // statistics that would include tombstones, the keys that are + // dropped by tombstones and duplicated user keys. See #847. + if f.Stats.RangeDeletionsBytesEstimate*10 < f.Size && + f.Stats.NumDeletions*10 < f.Stats.NumEntries { + continue + } + lowestCandidate = iter.Take() + lowestCandidateSeqNum = f.LargestSeqNum + } + p.elisionThreshold = &lowestCandidateSeqNum + if lowestCandidate.FileMetadata == nil { + p.elisionCandidate = nil + } else { + p.elisionCandidate = &lowestCandidate + } + } + + // We've already scanned L6 and know that we can perform an L6 + // elision-only compaction only if the sequence number + // `elisionThreshold` is in the last snapshot stripe. + if *p.elisionThreshold >= env.earliestSnapshotSeqNum { + return nil + } + + // Construct a picked compaction of the elision candidate's atomic + // compaction unit. + pc = newPickedCompaction(p.opts, p.vers, numLevels-1, numLevels-1) + var isCompacting bool + pc.startLevel.files, isCompacting = expandToAtomicUnit(p.opts.Comparer.Compare, p.elisionCandidate.Slice()) + if isCompacting { + return nil + } + + p.elisionThreshold = nil + p.elisionCandidate = nil + + // Fail-safe to protect against compacting the same sstable concurrently. + if !inputRangeAlreadyCompacting(env, pc) { + return pc + } return nil } func pickAutoHelper( - env compactionEnv, opts *Options, vers *version, cInfo pickedCompactionInfo, baseLevel int, -) (c *compaction) { + env compactionEnv, opts *Options, vers *version, cInfo candidateLevelInfo, baseLevel int, +) (pc *pickedCompaction) { if cInfo.outputLevel == 0 { return pickIntraL0(env, opts, vers) } - c = newCompaction(opts, vers, cInfo.level, baseLevel, env.bytesCompacted) - if c.outputLevel != cInfo.outputLevel { + pc = newPickedCompaction(opts, vers, cInfo.level, baseLevel) + if pc.outputLevel.level != cInfo.outputLevel { panic("pebble: compaction picked unexpected output level") } - c.inputs[0] = vers.Levels[c.startLevel][cInfo.file : cInfo.file+1] + pc.startLevel.files = cInfo.file.Slice() // Files in level 0 may overlap each other, so pick up all overlapping ones. - if c.startLevel == 0 { + if pc.startLevel.level == 0 { cmp := opts.Comparer.Compare - smallest, largest := manifest.KeyRange(cmp, c.inputs[0], nil) - c.inputs[0] = vers.Overlaps(0, cmp, smallest.UserKey, largest.UserKey) - if len(c.inputs[0]) == 0 { + smallest, largest := manifest.KeyRange(cmp, pc.startLevel.files.Iter()) + pc.startLevel.files = vers.Overlaps(0, cmp, smallest.UserKey, largest.UserKey) + if pc.startLevel.files.Empty() { panic("pebble: empty compaction") } } - c.setupInputs() - return c + if !pc.setupInputs() { + return nil + } + return pc } // Helper method to pick compactions originating from L0. Uses information about // sublevels to generate a compaction. -func pickL0(env compactionEnv, opts *Options, vers *version, baseLevel int) (c *compaction) { +func pickL0(env compactionEnv, opts *Options, vers *version, baseLevel int) (pc *pickedCompaction) { // It is important to pass information about Lbase files to L0Sublevels // so it can pick a compaction that does not conflict with an Lbase => Lbase+1 // compaction. Without this, we observed reduced concurrency of L0=>Lbase // compactions, and increasing read amplification in L0. - lcf, err := vers.L0Sublevels.PickBaseCompaction( - opts.L0CompactionThreshold, vers.Levels[baseLevel]) + // + // TODO(bilal) Remove the minCompactionDepth parameter once fixing it at 1 + // has been shown to not cause a performance regression. + lcf, err := vers.L0Sublevels.PickBaseCompaction(1, vers.Levels[baseLevel].Slice()) if err != nil { opts.Logger.Infof("error when picking base compaction: %s", err) return } if lcf != nil { - // Manually build the compaction as opposed to calling - // pickAutoHelper. This is because L0Sublevels has already added - // any overlapping L0 SSTables that need to be added, and - // because compactions built by L0SSTables do not necessarily - // pick contiguous sequences of files in p.vers.Levels[0]. - c = newCompaction(opts, vers, 0, baseLevel, env.bytesCompacted) - c.lcf = lcf - if c.outputLevel != baseLevel { - opts.Logger.Fatalf("compaction picked unexpected output level: %d != %d", c.outputLevel, baseLevel) - } - c.inputs[0] = make([]*manifest.FileMetadata, 0, len(lcf.Files)) - for j := range lcf.FilesIncluded { - if lcf.FilesIncluded[j] { - c.inputs[0] = append(c.inputs[0], vers.Levels[0][j]) - } - } - c.setupInputs() - if len(c.inputs[0]) == 0 { + pc = newPickedCompactionFromL0(lcf, opts, vers, baseLevel, true) + pc.setupInputs() + if pc.startLevel.files.Empty() { opts.Logger.Fatalf("empty compaction chosen") } - return c + return pc } // Couldn't choose a base compaction. Try choosing an intra-L0 - // compaction. - lcf, err = vers.L0Sublevels.PickIntraL0Compaction(env.earliestUnflushedSeqNum, opts.L0CompactionThreshold) + // compaction. Note that we pass in L0CompactionThreshold here as opposed to + // 1, since choosing a single sublevel intra-L0 compaction is + // counterproductive. + lcf, err = vers.L0Sublevels.PickIntraL0Compaction(env.earliestUnflushedSeqNum, minIntraL0Count) if err != nil { opts.Logger.Infof("error when picking intra-L0 compaction: %s", err) return } if lcf != nil { - c = newCompaction(opts, vers, 0, 0, env.bytesCompacted) - c.lcf = lcf - c.inputs[0] = make([]*manifest.FileMetadata, 0, len(lcf.Files)) - for j := range lcf.FilesIncluded { - if lcf.FilesIncluded[j] { - c.inputs[0] = append(c.inputs[0], vers.Levels[0][j]) - } + pc = newPickedCompactionFromL0(lcf, opts, vers, 0, false) + if !pc.setupInputs() { + return nil } - if len(c.inputs[0]) == 0 { + if pc.startLevel.files.Empty() { opts.Logger.Fatalf("empty compaction chosen") } - c.smallest, c.largest = manifest.KeyRange(c.cmp, c.inputs[0], nil) - c.setupInuseKeyRanges() - // Output only a single sstable for intra-L0 compactions. - // Now that we have the ability to split flushes, we could conceivably - // split the output of intra-L0 compactions too. This may be unnecessary - // complexity -- the inputs to intra-L0 should be narrow in the key space - // (unlike flushes), so writing a single sstable should be ok. - c.maxOutputFileSize = math.MaxUint64 - c.maxOverlapBytes = math.MaxUint64 - c.maxExpandedBytes = math.MaxUint64 - } - return c -} - -func pickIntraL0(env compactionEnv, opts *Options, vers *version) (c *compaction) { - l0Files := vers.Levels[0] - end := len(l0Files) - for ; end >= 1; end-- { - m := l0Files[end-1] + { + iter := pc.startLevel.files.Iter() + if iter.First() == nil || iter.Next() == nil { + // A single-file intra-L0 compaction is unproductive. + return nil + } + } + + pc.smallest, pc.largest = manifest.KeyRange(pc.cmp, pc.startLevel.files.Iter()) + } + return pc +} + +func pickIntraL0(env compactionEnv, opts *Options, vers *version) (pc *pickedCompaction) { + l0Files := vers.Levels[0].Slice() + end := l0Files.Iter() + remaining := l0Files.Len() + for m := end.Last(); m != nil; m = end.Prev() { if m.Compacting { return nil } @@ -711,7 +1210,7 @@ func pickIntraL0(env compactionEnv, opts *Options, vers *version) (c *compaction break } // Don't compact an L0 file which contains a seqnum greater than the - // earliest unflushed seqnum (we continue the loop, rather than existing, + // earliest unflushed seqnum (we continue the loop, rather than exiting, // see conditional above). This can happen when a file is ingested into L0 // yet doesn't overlap with the memtable. Consider the scenario: // @@ -747,52 +1246,60 @@ func pickIntraL0(env compactionEnv, opts *Options, vers *version) (c *compaction // And now everything is copacetic. // // See https://github.com/facebook/rocksdb/pull/5958. + remaining-- } - if end < minIntraL0Count { + if remaining < minIntraL0Count { return nil } - compactTotalSize := l0Files[end-1].Size + compactTotalSize := end.Current().Size + compactTotalCount := 1 compactSizePerFile := uint64(math.MaxUint64) - // The compaction will be in the range [begin,end). We add files to the - // compaction until the amount of compaction work per file begins increasing. - begin := end - 1 - for ; begin >= 1; begin-- { - m := l0Files[begin-1] - if m.Compacting { - break - } - newCompactTotalSize := compactTotalSize + m.Size - newCompactSizePerFile := newCompactTotalSize / uint64(end-(begin-1)) - if newCompactSizePerFile > compactSizePerFile { - break - } - compactTotalSize = newCompactTotalSize - compactSizePerFile = newCompactSizePerFile - } + // The compaction will be a subslice of l0Files ending with the file + // currently positioned beneath end. We add files to the compaction from + // the left until the amount of compaction work per file begins + // increasing. + compactFiles := end.Take().Slice().Reslice(func(start, end *manifest.LevelIterator) { + for m := start.Prev(); m != nil; m = start.Prev() { + if m.Compacting { + break + } - if end-begin < minIntraL0Count { + newCompactTotalSize := compactTotalSize + m.Size + newCompactSizePerFile := newCompactTotalSize / uint64(compactTotalCount+1) + if newCompactSizePerFile > compactSizePerFile { + break + } + compactTotalCount++ + compactTotalSize = newCompactTotalSize + compactSizePerFile = newCompactSizePerFile + } + // We advanced the iterator one too far, either beyond the beginning + // of the level's files or to a file that triggered an early break. + // Move the start back over the last file that was okay. + start.Next() + }) + if compactTotalCount < minIntraL0Count { return nil } - c = newCompaction(opts, vers, 0, 0, env.bytesCompacted) - c.inputs[0] = l0Files[begin:end] - c.smallest, c.largest = manifest.KeyRange(c.cmp, c.inputs[0], nil) - c.setupInuseKeyRanges() + pc = newPickedCompaction(opts, vers, 0, 0) + pc.startLevel.files = compactFiles + pc.smallest, pc.largest = manifest.KeyRange(pc.cmp, compactFiles.Iter()) // Output only a single sstable for intra-L0 compactions. There is no current // benefit to outputting multiple tables, because other parts of the code // (i.e. iterators and comapction) expect L0 sstables to overlap and will // thus read all of the L0 sstables anyways, even if they are partitioned. - c.maxOutputFileSize = math.MaxUint64 - c.maxOverlapBytes = math.MaxUint64 - c.maxExpandedBytes = math.MaxUint64 - return c + pc.maxOutputFileSize = math.MaxUint64 + pc.maxOverlapBytes = math.MaxUint64 + pc.maxExpandedBytes = math.MaxUint64 + return pc } func (p *compactionPickerByScore) pickManual( env compactionEnv, manual *manualCompaction, -) (c *compaction, retryLater bool) { +) (pc *pickedCompaction, retryLater bool) { if p == nil { return nil, false } @@ -814,47 +1321,95 @@ func (p *compactionPickerByScore) pickManual( if conflictsWithInProgress(manual.level, outputLevel, env.inProgressCompactions) { return nil, true } - c = pickManualHelper(env, p.opts, manual, p.vers, p.baseLevel) - if c == nil { + pc = pickManualHelper(p.opts, manual, p.vers, p.baseLevel) + if pc == nil { return nil, false } - if c.outputLevel != outputLevel { + if pc.outputLevel.level != outputLevel { panic("pebble: compaction picked unexpected output level") } // Fail-safe to protect against compacting the same sstable concurrently. - if inputAlreadyCompacting(c) { + if inputRangeAlreadyCompacting(env, pc) { return nil, true } - return c, false + return pc, false } func pickManualHelper( - env compactionEnv, opts *Options, manual *manualCompaction, vers *version, baseLevel int, -) (c *compaction) { - c = newCompaction(opts, vers, manual.level, baseLevel, env.bytesCompacted) - manual.outputLevel = c.outputLevel + opts *Options, manual *manualCompaction, vers *version, baseLevel int, +) (pc *pickedCompaction) { + pc = newPickedCompaction(opts, vers, manual.level, baseLevel) + manual.outputLevel = pc.outputLevel.level cmp := opts.Comparer.Compare - c.inputs[0] = vers.Overlaps(manual.level, cmp, manual.start.UserKey, manual.end.UserKey) - if len(c.inputs[0]) == 0 { + pc.startLevel.files = vers.Overlaps(manual.level, cmp, manual.start.UserKey, manual.end.UserKey) + if pc.startLevel.files.Empty() { // Nothing to do return nil } - c.setupInputs() - return c + if !pc.setupInputs() { + return nil + } + return pc } func (p *compactionPickerByScore) forceBaseLevel1() { p.baseLevel = 1 } -func inputAlreadyCompacting(c *compaction) bool { - for _, inputs := range c.inputs { - for _, f := range inputs { +func inputRangeAlreadyCompacting(env compactionEnv, pc *pickedCompaction) bool { + for _, cl := range pc.inputs { + iter := cl.files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { if f.Compacting { return true } } } + + // Look for active compactions outputting to the same region of the key + // space in the same output level. Two potential compactions may conflict + // without sharing input files if there are no files in the output level + // that overlap with the intersection of the compactions' key spaces. + // + // Consider an active L0->Lbase compaction compacting two L0 files one + // [a-f] and the other [t-z] into Lbase. + // + // L0 + // ↦ 000100 ↤ ↦ 000101 ↤ + // L1 + // ↦ 000004 ↤ + // a b c d e f g h i j k l m n o p q r s t u v w x y z + // + // If a new file 000102 [j-p] is flushed while the existing compaction is + // still ongoing, new file would not be in any compacting sublevel + // intervals and would not overlap with any Lbase files that are also + // compacting. However, this compaction cannot be picked because the + // compaction's output key space [j-p] would overlap the existing + // compaction's output key space [a-z]. + // + // L0 + // ↦ 000100* ↤ ↦ 000102 ↤ ↦ 000101* ↤ + // L1 + // ↦ 000004* ↤ + // a b c d e f g h i j k l m n o p q r s t u v w x y z + // + // * - currently compacting + if pc.outputLevel != nil && pc.outputLevel.level != 0 { + for _, c := range env.inProgressCompactions { + if pc.outputLevel.level != c.outputLevel { + continue + } + if base.InternalCompare(pc.cmp, c.largest, pc.smallest) < 0 || + base.InternalCompare(pc.cmp, c.smallest, pc.largest) > 0 { + continue + } + + // The picked compaction and the in-progress compaction c are + // outputting to the same region of the key space of the same + // level. + return true + } + } return false } @@ -862,10 +1417,12 @@ func conflictsWithInProgress( level int, outputLevel int, inProgressCompactions []compactionInfo, ) bool { for _, c := range inProgressCompactions { - if level == c.startLevel || - level == c.outputLevel || - outputLevel == c.startLevel || - outputLevel == c.outputLevel { + for _, in := range c.inputs { + if in.level == level || in.level == outputLevel { + return true + } + } + if c.outputLevel == level || c.outputLevel == outputLevel { return true } } diff --git a/github.com/cockroachdb/pebble/db.go b/github.com/cockroachdb/pebble/db.go index fecb58f1d0..9cc9590589 100644 --- a/github.com/cockroachdb/pebble/db.go +++ b/github.com/cockroachdb/pebble/db.go @@ -38,7 +38,7 @@ var ( // key. ErrNotFound = base.ErrNotFound // ErrClosed is returned when an operation is performed on a closed snapshot - // or DB. + // or DB. Use errors.Is(err, ErrClosed) to check for this error. ErrClosed = errors.New("pebble: closed") // ErrReadOnly is returned when a write operation is performed on a read-only // database. @@ -160,6 +160,33 @@ type Writer interface { // Comparer: myComparer, // }) type DB struct { + // WARNING: The following struct `atomic` contains fields which are accessed + // atomically. + // + // Go allocations are guaranteed to be 64-bit aligned which we take advantage + // of by placing the 64-bit fields which we access atomically at the beginning + // of the DB struct. For more information, see https://golang.org/pkg/sync/atomic/#pkg-note-BUG. + atomic struct { + // The count and size of referenced memtables. This includes memtables + // present in DB.mu.mem.queue, as well as memtables that have been flushed + // but are still referenced by an inuse readState. + memTableCount int64 + memTableReserved int64 // number of bytes reserved in the cache for memtables + + // bytesFlushed is the number of bytes flushed in the current flush. This + // must be read/written atomically since it is accessed by both the flush + // and compaction routines. + bytesFlushed uint64 + + // bytesCompacted is the number of bytes compacted in the current compaction. + // This is used as a dummy variable to increment during compaction, and the + // value is not used anywhere. + bytesCompacted uint64 + + // The size of the current log file (i.e. db.mu.log.queue[len(queue)-1]. + logSize uint64 + } + cacheID uint64 dirname string walDirname string @@ -190,33 +217,17 @@ type DB struct { sync.RWMutex val *readState } - // logRecycler holds a set of log file numbers that are available for // reuse. Writing to a recycled log file is faster than to a new log file on // some common filesystems (xfs, and ext3/4) due to avoiding metadata // updates. logRecycler logRecycler - closed int32 // updated atomically - - // The count and size of referenced memtables. This includes memtables - // present in DB.mu.mem.queue, as well as memtables that have been flushed - // but are still referenced by an inuse readState. - memTableCount int64 - memTableReserved int64 // number of bytes reserved in the cache for memtables + closed atomic.Value + closedCh chan struct{} compactionLimiter limiter - - // bytesFlushed is the number of bytes flushed in the current flush. This - // must be read/written atomically since it is accessed by both the flush - // and compaction routines. - bytesFlushed uint64 - // bytesCompacted is the number of bytes compacted in the current compaction. - // This is used as a dummy variable to increment during compaction, and the - // value is not used anywhere. - bytesCompacted uint64 - - flushLimiter limiter + flushLimiter limiter // The main mutex protecting internal DB state. This mutex encompasses many // fields because those fields need to be accessed and updated atomically. In @@ -238,8 +249,9 @@ type DB struct { nextJobID int // The collection of immutable versions and state about the log and visible - // sequence numbers. - versions versionSet + // sequence numbers. Use the pointer here to ensure the atomic fields in + // version set are aligned properly. + versions *versionSet log struct { // The queue of logs, containing both flushed and unflushed logs. The @@ -247,8 +259,6 @@ type DB struct { // delimeter between flushed and unflushed logs is // versionSet.minUnflushedLogNum. queue []FileNum - // The size of the current log file (i.e. queue[len(queue)-1]. - size uint64 // The number of input bytes to the log. This is the raw size of the // batches written to the WAL, without the overhead of the record // envelopes. @@ -291,6 +301,9 @@ type DB struct { flushing bool // The number of ongoing compactions. compactingCount int + // The list of deletion hints, suggesting ranges for delete-only + // compactions. + deletionHints []deleteCompactionHint // The list of manual compactions. The next manual compaction to perform // is at the start of the list. New entries are added to the end. manual []*manualCompaction @@ -353,8 +366,8 @@ func (d *DB) Get(key []byte) ([]byte, io.Closer, error) { } func (d *DB) getInternal(key []byte, b *Batch, s *Snapshot) ([]byte, io.Closer, error) { - if atomic.LoadInt32(&d.closed) != 0 { - panic(ErrClosed) + if err := d.closed.Load(); err != nil { + panic(err) } // Grab and reference the current readState. This prevents the underlying @@ -368,7 +381,7 @@ func (d *DB) getInternal(key []byte, b *Batch, s *Snapshot) ([]byte, io.Closer, if s != nil { seqNum = s.seqNum } else { - seqNum = atomic.LoadUint64(&d.mu.versions.visibleSeqNum) + seqNum = atomic.LoadUint64(&d.mu.versions.atomic.visibleSeqNum) } var buf struct { @@ -516,8 +529,11 @@ func (d *DB) LogData(data []byte, opts *WriteOptions) error { // // It is safe to modify the contents of the arguments after Apply returns. func (d *DB) Apply(batch *Batch, opts *WriteOptions) error { - if atomic.LoadInt32(&d.closed) != 0 { - panic(ErrClosed) + if err := d.closed.Load(); err != nil { + panic(err) + } + if atomic.LoadUint32(&batch.applied) != 0 { + panic("pebble: batch already applied") } if d.opts.ReadOnly { return ErrReadOnly @@ -564,6 +580,16 @@ func (d *DB) commitApply(b *Batch, mem *memTable) error { if err != nil { return err } + + // If the batch contains range tombstones and the database is configured + // to flush range deletions, schedule a delayed flush so that disk space + // may be reclaimed without additional writes or an explicit flush. + if b.countRangeDels > 0 && d.opts.Experimental.DeleteRangeFlushDelay > 0 { + d.mu.Lock() + d.maybeScheduleDelayedFlush(mem) + d.mu.Unlock() + } + if mem.writerUnref() { d.mu.Lock() d.maybeScheduleFlush() @@ -627,7 +653,7 @@ func (d *DB) commitWrite(b *Batch, syncWG *sync.WaitGroup, syncErr *error) (*mem } } - atomic.StoreUint64(&d.mu.log.size, uint64(size)) + atomic.StoreUint64(&d.atomic.logSize, uint64(size)) return mem, err } @@ -650,8 +676,8 @@ var iterAllocPool = sync.Pool{ func (d *DB) newIterInternal( batchIter internalIterator, batchRangeDelIter internalIterator, s *Snapshot, o *IterOptions, ) *Iterator { - if atomic.LoadInt32(&d.closed) != 0 { - panic(ErrClosed) + if err := d.closed.Load(); err != nil { + panic(err) } // Grab and reference the current readState. This prevents the underlying @@ -665,7 +691,7 @@ func (d *DB) newIterInternal( if s != nil { seqNum = s.seqNum } else { - seqNum = atomic.LoadUint64(&d.mu.versions.visibleSeqNum) + seqNum = atomic.LoadUint64(&d.mu.versions.atomic.visibleSeqNum) } // Bundle various structures under a single umbrella in order to allocate @@ -720,7 +746,7 @@ func (d *DB) newIterInternal( } } for level := 1; level < len(current.Levels); level++ { - if len(current.Levels[level]) == 0 { + if current.Levels[level].Empty() { continue } mlevels = append(mlevels, mergingIterLevel{}) @@ -729,8 +755,8 @@ func (d *DB) newIterInternal( mlevels = mlevels[start:] levels := buf.levels[:] - addLevelIterForFiles := func(files []*manifest.FileMetadata, level manifest.Level) { - if len(files) == 0 { + addLevelIterForFiles := func(files manifest.LevelSlice, level manifest.Level) { + if files.Empty() { return } var li *levelIter @@ -741,10 +767,11 @@ func (d *DB) newIterInternal( li = &levelIter{} } - li.init(dbi.opts, d.cmp, d.newIters, files, level, nil) + li.init(dbi.opts, d.cmp, d.newIters, files.Iter(), level, nil) li.initRangeDel(&mlevels[0].rangeDelIter) li.initSmallestLargestUserKey(&mlevels[0].smallestUserKey, &mlevels[0].largestUserKey, &mlevels[0].isLargestUserKeyRangeDelSentinel) + li.initIsSyntheticIterBoundsKey(&mlevels[0].isSyntheticIterBoundsKey) mlevels[0].iter = li mlevels = mlevels[1:] } @@ -752,12 +779,13 @@ func (d *DB) newIterInternal( // Add level iterators for the L0 sublevels, iterating from newest to // oldest. for i := len(current.L0Sublevels.Levels) - 1; i >= 0; i-- { - addLevelIterForFiles(current.L0Sublevels.Levels[i], manifest.L0Sublevel(i)) + slice := manifest.NewLevelSlice(current.L0Sublevels.Levels[i]) + addLevelIterForFiles(slice, manifest.L0Sublevel(i)) } // Add level iterators for the non-empty non-L0 levels. for level := 1; level < len(current.Levels); level++ { - addLevelIterForFiles(current.Levels[level], manifest.Level(level)) + addLevelIterForFiles(current.Levels[level].Slice(), manifest.Level(level)) } buf.merging.init(&dbi.opts, d.cmp, finalMLevels...) @@ -803,14 +831,14 @@ func (d *DB) NewIter(o *IterOptions) *Iterator { // deleted. Instead, a snapshot prevents deletion of sequence numbers // referenced by the snapshot. func (d *DB) NewSnapshot() *Snapshot { - if atomic.LoadInt32(&d.closed) != 0 { - panic(ErrClosed) + if err := d.closed.Load(); err != nil { + panic(err) } d.mu.Lock() s := &Snapshot{ db: d, - seqNum: atomic.LoadUint64(&d.mu.versions.visibleSeqNum), + seqNum: atomic.LoadUint64(&d.mu.versions.atomic.visibleSeqNum), } d.mu.snapshots.pushBack(s) d.mu.Unlock() @@ -825,10 +853,11 @@ func (d *DB) NewSnapshot() *Snapshot { func (d *DB) Close() error { d.mu.Lock() defer d.mu.Unlock() - if atomic.LoadInt32(&d.closed) != 0 { - panic(ErrClosed) + if err := d.closed.Load(); err != nil { + panic(err) } - atomic.StoreInt32(&d.closed, 1) + d.closed.Store(errors.WithStack(ErrClosed)) + close(d.closedCh) defer d.opts.Cache.Unref() @@ -880,7 +909,7 @@ func (d *DB) Close() error { for _, mem := range d.mu.mem.queue { mem.readerUnref() } - if reserved := atomic.LoadInt64(&d.memTableReserved); reserved != 0 { + if reserved := atomic.LoadInt64(&d.atomic.memTableReserved); reserved != 0 { return errors.Errorf("leaked memtable reservation: %d", errors.Safe(reserved)) } } @@ -902,8 +931,8 @@ func (d *DB) Close() error { func (d *DB) Compact( start, end []byte, /* CompactionOptions */ ) error { - if atomic.LoadInt32(&d.closed) != 0 { - panic(ErrClosed) + if err := d.closed.Load(); err != nil { + panic(err) } if d.opts.ReadOnly { return ErrReadOnly @@ -911,13 +940,13 @@ func (d *DB) Compact( iStart := base.MakeInternalKey(start, InternalKeySeqNumMax, InternalKeyKindMax) iEnd := base.MakeInternalKey(end, 0, 0) - meta := []*fileMetadata{&fileMetadata{Smallest: iStart, Largest: iEnd}} + meta := []*fileMetadata{{Smallest: iStart, Largest: iEnd}} d.mu.Lock() maxLevelWithFiles := 1 cur := d.mu.versions.currentVersion() for level := 0; level < numLevels; level++ { - if len(cur.Overlaps(level, d.cmp, start, end)) > 0 { + if !cur.Overlaps(level, d.cmp, start, end).Empty() { maxLevelWithFiles = level + 1 } } @@ -1006,8 +1035,8 @@ func (d *DB) Flush() error { // If no error is returned, the caller can receive from the returned channel in // order to wait for the flush to complete. func (d *DB) AsyncFlush() (<-chan struct{}, error) { - if atomic.LoadInt32(&d.closed) != 0 { - panic(ErrClosed) + if err := d.closed.Load(); err != nil { + panic(err) } if d.opts.ReadOnly { return nil, ErrReadOnly @@ -1033,14 +1062,15 @@ func (d *DB) Metrics() *Metrics { d.mu.Lock() *metrics = d.mu.versions.metrics metrics.Compact.EstimatedDebt = d.mu.versions.picker.estimatedCompactionDebt(0) + metrics.Compact.InProgressBytes = atomic.LoadInt64(&d.mu.versions.atomic.atomicInProgressBytes) for _, m := range d.mu.mem.queue { metrics.MemTable.Size += m.totalBytes() } metrics.MemTable.Count = int64(len(d.mu.mem.queue)) - metrics.MemTable.ZombieCount = atomic.LoadInt64(&d.memTableCount) - metrics.MemTable.Count - metrics.MemTable.ZombieSize = uint64(atomic.LoadInt64(&d.memTableReserved)) - metrics.MemTable.Size + metrics.MemTable.ZombieCount = atomic.LoadInt64(&d.atomic.memTableCount) - metrics.MemTable.Count + metrics.MemTable.ZombieSize = uint64(atomic.LoadInt64(&d.atomic.memTableReserved)) - metrics.MemTable.Size metrics.WAL.ObsoleteFiles = int64(recycledLogs) - metrics.WAL.Size = atomic.LoadUint64(&d.mu.log.size) + metrics.WAL.Size = atomic.LoadUint64(&d.atomic.logSize) metrics.WAL.BytesIn = d.mu.log.bytesIn // protected by d.mu for i, n := 0, len(d.mu.mem.queue)-1; i < n; i++ { metrics.WAL.Size += d.mu.mem.queue[i].logSize @@ -1080,19 +1110,22 @@ func (d *DB) SSTables() [][]TableInfo { srcLevels := readState.current.Levels var totalTables int for i := range srcLevels { - totalTables += len(srcLevels[i]) + // TODO(jackson): Use metrics on the LevelMetadata once available + // rather than Slice().Len(). + totalTables += srcLevels[i].Slice().Len() } destTables := make([]TableInfo, totalTables) destLevels := make([][]TableInfo, len(srcLevels)) for i := range destLevels { - srcLevel := srcLevels[i] - destLevel := destTables[:len(srcLevel):len(srcLevel)] - destTables = destTables[len(srcLevel):] - for j := range destLevel { - destLevel[j] = srcLevel[j].TableInfo() + iter := srcLevels[i].Iter() + j := 0 + for m := iter.First(); m != nil; m = iter.Next() { + destTables[j] = m.TableInfo() + j++ } - destLevels[i] = destLevel + destLevels[i] = destTables[:j] + destTables = destTables[j:] } return destLevels } @@ -1109,8 +1142,8 @@ func (d *DB) SSTables() [][]TableInfo { // - There may also exist WAL entries for unflushed keys in this range. This // estimation currently excludes space used for the range in the WAL. func (d *DB) EstimateDiskUsage(start, end []byte) (uint64, error) { - if atomic.LoadInt32(&d.closed) != 0 { - panic(ErrClosed) + if err := d.closed.Load(); err != nil { + panic(err) } if d.opts.Comparer.Compare(start, end) > 0 { return 0, errors.New("invalid key-range specified (start > end)") @@ -1124,19 +1157,15 @@ func (d *DB) EstimateDiskUsage(start, end []byte) (uint64, error) { var totalSize uint64 for level, files := range readState.current.Levels { + iter := files.Iter() if level > 0 { // We can only use `Overlaps` to restrict `files` at L1+ since at L0 it // expands the range iteratively until it has found a set of files that // do not overlap any other L0 files outside that set. - files = readState.current.Overlaps(level, d.opts.Comparer.Compare, start, end) + iter = readState.current.Overlaps(level, d.opts.Comparer.Compare, start, end).Iter() } - for fileIdx, file := range files { - if level > 0 && fileIdx > 0 && fileIdx < len(files)-1 { - // The files to the left and the right at least partially overlap - // with `file`, which means `file` is fully contained within the - // range specified by `[start, end]`. - totalSize += file.Size - } else if d.opts.Comparer.Compare(start, file.Smallest.UserKey) <= 0 && + for file := iter.First(); file != nil; file = iter.Next() { + if d.opts.Comparer.Compare(start, file.Smallest.UserKey) <= 0 && d.opts.Comparer.Compare(file.Largest.UserKey, end) <= 0 { // The range fully contains the file, so skip looking it up in // table cache/looking at its indexes, and add the full file size. @@ -1181,8 +1210,8 @@ func (d *DB) newMemTable(logNum FileNum, logSeqNum uint64) (*memTable, *flushabl } } - atomic.AddInt64(&d.memTableCount, 1) - atomic.AddInt64(&d.memTableReserved, int64(size)) + atomic.AddInt64(&d.atomic.memTableCount, 1) + atomic.AddInt64(&d.atomic.memTableReserved, int64(size)) releaseAccountingReservation := d.opts.Cache.Reserve(size) mem := newMemTable(memTableOptions{ @@ -1198,8 +1227,8 @@ func (d *DB) newMemTable(logNum FileNum, logSeqNum uint64) (*memTable, *flushabl entry.releaseMemAccounting = func() { manual.Free(mem.arenaBuf) mem.arenaBuf = nil - atomic.AddInt64(&d.memTableCount, -1) - atomic.AddInt64(&d.memTableReserved, -int64(size)) + atomic.AddInt64(&d.atomic.memTableCount, -1) + atomic.AddInt64(&d.atomic.memTableReserved, -int64(size)) releaseAccountingReservation() } return mem, entry @@ -1236,14 +1265,12 @@ func (d *DB) makeRoomForWrite(b *Batch) error { err := d.mu.mem.mutable.prepare(b) if err != arenaskl.ErrArenaFull { if stalled { - stalled = false d.opts.EventListener.WriteStallEnd() } return err } } else if !force { if stalled { - stalled = false d.opts.EventListener.WriteStallEnd() } return nil @@ -1267,11 +1294,13 @@ func (d *DB) makeRoomForWrite(b *Batch) error { continue } } - l0FileCount := len(d.mu.versions.currentVersion().Levels[0]) + var l0ReadAmp int if d.opts.Experimental.L0SublevelCompactions { - l0FileCount = d.mu.versions.currentVersion().L0Sublevels.ReadAmplification() + l0ReadAmp = d.mu.versions.currentVersion().L0Sublevels.ReadAmplification() + } else { + l0ReadAmp = d.mu.versions.currentVersion().Levels[0].Slice().Len() } - if l0FileCount >= d.opts.L0StopWritesThreshold { + if l0ReadAmp >= d.opts.L0StopWritesThreshold { // There are too many level-0 files, so we wait. if !stalled { stalled = true @@ -1295,6 +1324,14 @@ func (d *DB) makeRoomForWrite(b *Batch) error { d.mu.mem.switching = true d.mu.Unlock() + // Close the previous log first. This writes an EOF trailer + // signifying the end of the file and syncs it to disk. We must + // close the previous log before linking the new log file, + // otherwise a crash could leave both logs with unclean tails, and + // Open will treat the previous log as corrupt. + prevLogSize = uint64(d.mu.log.Size()) + err = d.mu.log.Close() + newLogName := base.MakeFilename(d.opts.FS, d.walDirname, fileTypeLog, newLogNum) // Try to use a recycled log file. Recycling log files is an important @@ -1303,12 +1340,15 @@ func (d *DB) makeRoomForWrite(b *Batch) error { // time. This is due to the need to sync file metadata when a file is // being written for the first time. Note this is true even if file // preallocation is performed (e.g. fallocate). - recycleLogNum := d.logRecycler.peek() - if recycleLogNum > 0 { - recycleLogName := base.MakeFilename(d.opts.FS, d.walDirname, fileTypeLog, recycleLogNum) - newLogFile, err = d.opts.FS.ReuseForWrite(recycleLogName, newLogName) - } else { - newLogFile, err = d.opts.FS.Create(newLogName) + var recycleLogNum base.FileNum + if err == nil { + recycleLogNum = d.logRecycler.peek() + if recycleLogNum > 0 { + recycleLogName := base.MakeFilename(d.opts.FS, d.walDirname, fileTypeLog, recycleLogNum) + newLogFile, err = d.opts.FS.ReuseForWrite(recycleLogName, newLogName) + } else { + newLogFile, err = d.opts.FS.Create(newLogName) + } } if err == nil { @@ -1317,17 +1357,13 @@ func (d *DB) makeRoomForWrite(b *Batch) error { err = d.walDir.Sync() } - if err == nil { - prevLogSize = uint64(d.mu.log.Size()) - err = d.mu.log.Close() - if err != nil { - newLogFile.Close() - } else { - newLogFile = vfs.NewSyncingFile(newLogFile, vfs.SyncingFileOptions{ - BytesPerSync: d.opts.BytesPerSync, - PreallocateSize: d.walPreallocateSize(), - }) - } + if err != nil && newLogFile != nil { + newLogFile.Close() + } else if err == nil { + newLogFile = vfs.NewSyncingFile(newLogFile, vfs.SyncingFileOptions{ + BytesPerSync: d.opts.WALBytesPerSync, + PreallocateSize: d.walPreallocateSize(), + }) } if recycleLogNum > 0 { @@ -1404,7 +1440,7 @@ func (d *DB) makeRoomForWrite(b *Batch) error { logSeqNum += uint64(b.Count()) } } else { - logSeqNum = atomic.LoadUint64(&d.mu.versions.logSeqNum) + logSeqNum = atomic.LoadUint64(&d.mu.versions.atomic.logSeqNum) } // Create a new memtable, scheduling the previous one for flushing. We do @@ -1445,16 +1481,40 @@ func (d *DB) getEarliestUnflushedSeqNumLocked() uint64 { func (d *DB) getInProgressCompactionInfoLocked(finishing *compaction) (rv []compactionInfo) { for c := range d.mu.compact.inProgress { if len(c.flushing) == 0 && (finishing == nil || c != finishing) { - rv = append(rv, compactionInfo{ - startLevel: c.startLevel, - outputLevel: c.outputLevel, + info := compactionInfo{ inputs: c.inputs, - }) + smallest: c.smallest, + largest: c.largest, + outputLevel: -1, + } + if c.outputLevel != nil { + info.outputLevel = c.outputLevel.level + } + rv = append(rv, info) } } return } +func inProgressL0Compactions(inProgress []compactionInfo) []manifest.L0Compaction { + var compactions []manifest.L0Compaction + for _, info := range inProgress { + l0 := false + for _, cl := range info.inputs { + l0 = l0 || cl.level == 0 + } + if !l0 { + continue + } + compactions = append(compactions, manifest.L0Compaction{ + Smallest: info.smallest, + Largest: info.largest, + IsIntraL0: info.outputLevel == 0, + }) + } + return compactions +} + // firstError returns the first non-nil error of err0 and err1, or nil if both // are nil. func firstError(err0, err1 error) error { diff --git a/github.com/cockroachdb/pebble/error_iter.go b/github.com/cockroachdb/pebble/error_iter.go index e272675757..ca120acca7 100644 --- a/github.com/cockroachdb/pebble/error_iter.go +++ b/github.com/cockroachdb/pebble/error_iter.go @@ -4,9 +4,7 @@ package pebble -import ( - "github.com/cockroachdb/pebble/internal/base" -) +import "github.com/cockroachdb/pebble/internal/base" type errorIter struct { err error diff --git a/github.com/cockroachdb/pebble/event.go b/github.com/cockroachdb/pebble/event.go index 7489422536..14de07ce24 100644 --- a/github.com/cockroachdb/pebble/event.go +++ b/github.com/cockroachdb/pebble/event.go @@ -5,13 +5,12 @@ package pebble import ( - "bytes" - "fmt" "strings" "time" "github.com/cockroachdb/pebble/internal/humanize" "github.com/cockroachdb/pebble/internal/manifest" + "github.com/cockroachdb/redact" ) // TableInfo exports the manifest.TableInfo type. @@ -36,62 +35,94 @@ func formatFileNums(tables []TableInfo) string { return buf.String() } +// LevelInfo contains info pertaining to a partificular level. +type LevelInfo struct { + Level int + Tables []TableInfo +} + +func (i LevelInfo) String() string { + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i LevelInfo) SafeFormat(w redact.SafePrinter, _ rune) { + w.Printf("L%d [%s] (%s)", redact.Safe(i.Level), redact.Safe(formatFileNums(i.Tables)), + redact.Safe(humanize.Uint64(tablesTotalSize(i.Tables)))) +} + // CompactionInfo contains the info for a compaction event. type CompactionInfo struct { // JobID is the ID of the compaction job. JobID int // Reason is the reason for the compaction. Reason string - // Input contains the input tables for the compaction. A compaction is - // performed from Input.Level to Output.Level. Input.Tables[0] contains the - // inputs from Input.Level and Input.Tables[1] contains the inputs from - // Output.Level. - Input struct { - Level int - Tables [2][]TableInfo - } + // Input contains the input tables for the compaction organized by level. + Input []LevelInfo // Output contains the output tables generated by the compaction. The output // tables are empty for the compaction begin event. - Output struct { - Level int - Tables []TableInfo - } + Output LevelInfo Duration time.Duration Done bool Err error } func (i CompactionInfo) String() string { + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i CompactionInfo) SafeFormat(w redact.SafePrinter, _ rune) { if i.Err != nil { - return fmt.Sprintf("[JOB %d] compaction to L%d error: %s", - i.JobID, i.Output.Level, i.Err) + w.Printf("[JOB %d] compaction to L%d error: %s", + redact.Safe(i.JobID), redact.Safe(i.Output.Level), i.Err) + return } if !i.Done { - return fmt.Sprintf("[JOB %d] compacting L%d [%s] (%s) + L%d [%s] (%s)", - i.JobID, - i.Input.Level, - formatFileNums(i.Input.Tables[0]), - humanize.Uint64(tablesTotalSize(i.Input.Tables[0])), - i.Output.Level, - formatFileNums(i.Input.Tables[1]), - humanize.Uint64(tablesTotalSize(i.Input.Tables[1]))) + w.Printf("[JOB %d] compacting ", redact.Safe(i.JobID)) + w.Print(levelInfos(i.Input)) + return } - outputSize := tablesTotalSize(i.Output.Tables) - return fmt.Sprintf("[JOB %d] compacted L%d [%s] (%s) + L%d [%s] (%s) -> L%d [%s] (%s), in %.1fs, output rate %s/s", - i.JobID, - i.Input.Level, - formatFileNums(i.Input.Tables[0]), - humanize.Uint64(tablesTotalSize(i.Input.Tables[0])), - i.Output.Level, - formatFileNums(i.Input.Tables[1]), - humanize.Uint64(tablesTotalSize(i.Input.Tables[1])), - i.Output.Level, - formatFileNums(i.Output.Tables), - humanize.Uint64(outputSize), - i.Duration.Seconds(), - humanize.Uint64(uint64(float64(outputSize)/i.Duration.Seconds()))) + w.Printf("[JOB %d] compacted ", redact.Safe(i.JobID)) + w.Print(levelInfos(i.Input)) + w.Printf(" -> L%d [%s] (%s), in %.1fs, output rate %s/s", + redact.Safe(i.Output.Level), + redact.Safe(formatFileNums(i.Output.Tables)), + redact.Safe(humanize.Uint64(outputSize)), + redact.Safe(i.Duration.Seconds()), + redact.Safe(humanize.Uint64(uint64(float64(outputSize)/i.Duration.Seconds())))) +} + +type levelInfos []LevelInfo + +func (i levelInfos) SafeFormat(w redact.SafePrinter, _ rune) { + for j, levelInfo := range i { + if j > 0 { + w.Printf(" + ") + } + w.Print(levelInfo) + } +} + +// DiskSlowInfo contains the info for a disk slowness event when writing to a +// file. +type DiskSlowInfo struct { + // Path of file being written to. + Path string + // Duration that has elapsed since this disk operation started. + Duration time.Duration +} + +func (i DiskSlowInfo) String() string { + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i DiskSlowInfo) SafeFormat(w redact.SafePrinter, _ rune) { + w.Printf("disk slowness detected: write to file %s has been ongoing for %0.1fs", + i.Path, redact.Safe(i.Duration.Seconds())) } // FlushInfo contains the info for a flush event. @@ -100,26 +131,45 @@ type FlushInfo struct { JobID int // Reason is the reason for the flush. Reason string + // Input contains the count of input memtables that were flushed. + Input int // Output contains the ouptut table generated by the flush. The output info // is empty for the flush begin event. - Output []TableInfo - Done bool - Err error + Output []TableInfo + Duration time.Duration + Done bool + Err error } func (i FlushInfo) String() string { + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i FlushInfo) SafeFormat(w redact.SafePrinter, _ rune) { if i.Err != nil { - return fmt.Sprintf("[JOB %d] flush error: %s", i.JobID, i.Err) + w.Printf("[JOB %d] flush error: %s", redact.Safe(i.JobID), i.Err) + return } - if !i.Done { - return fmt.Sprintf("[JOB %d] flushing to L0", i.JobID) + plural := redact.SafeString("s") + if i.Input == 1 { + plural = "" } - - return fmt.Sprintf("[JOB %d] flushed to L0 [%s] (%s)", - i.JobID, - formatFileNums(i.Output), - humanize.Uint64(tablesTotalSize(i.Output))) + if !i.Done { + w.Printf("[JOB %d] flushing %d memtable", redact.Safe(i.JobID), redact.Safe(i.Input)) + w.SafeString(plural) + w.Printf(" to L0") + return + } + + outputSize := tablesTotalSize(i.Output) + w.Printf("[JOB %d] flushed %d memtable%s to L0 [%s] (%s), in %.1fs, output rate %s/s", + redact.Safe(i.JobID), redact.Safe(i.Input), plural, + redact.Safe(formatFileNums(i.Output)), + redact.Safe(humanize.Uint64(outputSize)), + redact.Safe(i.Duration.Seconds()), + redact.Safe(humanize.Uint64(uint64(float64(outputSize)/i.Duration.Seconds())))) } // ManifestCreateInfo contains info about a manifest creation event. @@ -133,10 +183,16 @@ type ManifestCreateInfo struct { } func (i ManifestCreateInfo) String() string { + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i ManifestCreateInfo) SafeFormat(w redact.SafePrinter, _ rune) { if i.Err != nil { - return fmt.Sprintf("[JOB %d] MANIFEST create error: %s", i.JobID, i.Err) + w.Printf("[JOB %d] MANIFEST create error: %s", redact.Safe(i.JobID), i.Err) + return } - return fmt.Sprintf("[JOB %d] MANIFEST created %s", i.JobID, i.FileNum) + w.Printf("[JOB %d] MANIFEST created %s", redact.Safe(i.JobID), redact.Safe(i.FileNum)) } // ManifestDeleteInfo contains the info for a Manifest deletion event. @@ -149,10 +205,16 @@ type ManifestDeleteInfo struct { } func (i ManifestDeleteInfo) String() string { + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i ManifestDeleteInfo) SafeFormat(w redact.SafePrinter, _ rune) { if i.Err != nil { - return fmt.Sprintf("[JOB %d] MANIFEST delete error: %s", i.JobID, i.Err) + w.Printf("[JOB %d] MANIFEST delete error: %s", redact.Safe(i.JobID), i.Err) + return } - return fmt.Sprintf("[JOB %d] MANIFEST deleted %s", i.JobID, i.FileNum) + w.Printf("[JOB %d] MANIFEST deleted %s", redact.Safe(i.JobID), redact.Safe(i.FileNum)) } // TableCreateInfo contains the info for a table creation event. @@ -166,7 +228,13 @@ type TableCreateInfo struct { } func (i TableCreateInfo) String() string { - return fmt.Sprintf("[JOB %d] %s: sstable created %s", i.JobID, i.Reason, i.FileNum) + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i TableCreateInfo) SafeFormat(w redact.SafePrinter, _ rune) { + w.Printf("[JOB %d] %s: sstable created %s", + redact.Safe(i.JobID), redact.Safe(i.Reason), redact.Safe(i.FileNum)) } // TableDeleteInfo contains the info for a table deletion event. @@ -178,11 +246,17 @@ type TableDeleteInfo struct { } func (i TableDeleteInfo) String() string { + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i TableDeleteInfo) SafeFormat(w redact.SafePrinter, _ rune) { if i.Err != nil { - return fmt.Sprintf("[JOB %d] sstable delete error %s: %s", - i.JobID, i.FileNum, i.Err) + w.Printf("[JOB %d] sstable delete error %s: %s", + redact.Safe(i.JobID), redact.Safe(i.FileNum), i.Err) + return } - return fmt.Sprintf("[JOB %d] sstable deleted %s", i.JobID, i.FileNum) + w.Printf("[JOB %d] sstable deleted %s", redact.Safe(i.JobID), redact.Safe(i.FileNum)) } // TableIngestInfo contains the info for a table ingestion event. @@ -200,20 +274,25 @@ type TableIngestInfo struct { } func (i TableIngestInfo) String() string { + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i TableIngestInfo) SafeFormat(w redact.SafePrinter, _ rune) { if i.Err != nil { - return fmt.Sprintf("[JOB %d] ingest error: %s", i.JobID, i.Err) + w.Printf("[JOB %d] ingest error: %s", redact.Safe(i.JobID), i.Err) + return } - var buf bytes.Buffer - fmt.Fprintf(&buf, "[JOB %d] ingested", i.JobID) + w.Printf("[JOB %d] ingested", redact.Safe(i.JobID)) for j := range i.Tables { t := &i.Tables[j] if j > 0 { - fmt.Fprintf(&buf, ",") + w.Printf(",") } - fmt.Fprintf(&buf, " L%d:%s (%s)", t.Level, t.FileNum, humanize.Uint64(t.Size)) + w.Printf(" L%d:%s (%s)", redact.Safe(t.Level), redact.Safe(t.FileNum), + redact.Safe(humanize.Uint64(t.Size))) } - return buf.String() } // TableStatsInfo contains the info for a table stats loaded event. @@ -224,7 +303,12 @@ type TableStatsInfo struct { } func (i TableStatsInfo) String() string { - return fmt.Sprintf("[JOB %d] all initial table stats loaded", i.JobID) + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i TableStatsInfo) SafeFormat(w redact.SafePrinter, _ rune) { + w.Printf("[JOB %d] all initial table stats loaded", redact.Safe(i.JobID)) } // WALCreateInfo contains info about a WAL creation event. @@ -241,16 +325,23 @@ type WALCreateInfo struct { } func (i WALCreateInfo) String() string { + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i WALCreateInfo) SafeFormat(w redact.SafePrinter, _ rune) { if i.Err != nil { - return fmt.Sprintf("[JOB %d] WAL create error: %s", i.JobID, i.Err) + w.Printf("[JOB %d] WAL create error: %s", redact.Safe(i.JobID), i.Err) + return } if i.RecycledFileNum == 0 { - return fmt.Sprintf("[JOB %d] WAL created %s", i.JobID, i.FileNum) + w.Printf("[JOB %d] WAL created %s", redact.Safe(i.JobID), redact.Safe(i.FileNum)) + return } - return fmt.Sprintf("[JOB %d] WAL created %s (recycled %s)", - i.JobID, i.FileNum, i.RecycledFileNum) + w.Printf("[JOB %d] WAL created %s (recycled %s)", + redact.Safe(i.JobID), redact.Safe(i.FileNum), redact.Safe(i.RecycledFileNum)) } // WALDeleteInfo contains the info for a WAL deletion event. @@ -263,10 +354,16 @@ type WALDeleteInfo struct { } func (i WALDeleteInfo) String() string { + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i WALDeleteInfo) SafeFormat(w redact.SafePrinter, _ rune) { if i.Err != nil { - return fmt.Sprintf("[JOB %d] WAL delete error: %s", i.JobID, i.Err) + w.Printf("[JOB %d] WAL delete error: %s", redact.Safe(i.JobID), i.Err) + return } - return fmt.Sprintf("[JOB %d] WAL deleted %s", i.JobID, i.FileNum) + w.Printf("[JOB %d] WAL deleted %s", redact.Safe(i.JobID), redact.Safe(i.FileNum)) } // WriteStallBeginInfo contains the info for a write stall begin event. @@ -275,7 +372,12 @@ type WriteStallBeginInfo struct { } func (i WriteStallBeginInfo) String() string { - return fmt.Sprintf("write stall beginning: %s", i.Reason) + return redact.StringWithoutMarkers(i) +} + +// SafeFormat implements redact.SafeFormatter. +func (i WriteStallBeginInfo) SafeFormat(w redact.SafePrinter, _ rune) { + w.Printf("write stall beginning: %s", redact.Safe(i.Reason)) } // EventListener contains a set of functions that will be invoked when various @@ -296,6 +398,11 @@ type EventListener struct { // has been installed. CompactionEnd func(CompactionInfo) + // DiskSlow is invoked after a disk write operation on a file created + // with a disk health checking vfs.FS (see vfs.DefaultWithDiskHealthChecks) + // is observed to exceed the specified disk slowness threshold duration. + DiskSlow func(DiskSlowInfo) + // FlushBegin is invoked after the inputs to a flush have been determined, // but before the flush has produced any output. FlushBegin func(FlushInfo) @@ -353,6 +460,9 @@ func (l *EventListener) EnsureDefaults(logger Logger) { if l.CompactionEnd == nil { l.CompactionEnd = func(info CompactionInfo) {} } + if l.DiskSlow == nil { + l.DiskSlow = func(info DiskSlowInfo) {} + } if l.FlushBegin == nil { l.FlushBegin = func(info FlushInfo) {} } @@ -408,6 +518,9 @@ func MakeLoggingEventListener(logger Logger) EventListener { CompactionEnd: func(info CompactionInfo) { logger.Infof("%s", info) }, + DiskSlow: func(info DiskSlowInfo) { + logger.Infof("%s", info) + }, FlushBegin: func(info FlushInfo) { logger.Infof("%s", info) }, diff --git a/github.com/cockroachdb/pebble/flush_external.go b/github.com/cockroachdb/pebble/flush_external.go index a5a5fb6d21..9836065097 100644 --- a/github.com/cockroachdb/pebble/flush_external.go +++ b/github.com/cockroachdb/pebble/flush_external.go @@ -5,7 +5,6 @@ package pebble import ( - "sync/atomic" "time" "github.com/cockroachdb/pebble/internal/private" @@ -17,8 +16,8 @@ import ( // replay package rather than calling this private hook directly. func flushExternalTable(untypedDB interface{}, path string, originalMeta *fileMetadata) error { d := untypedDB.(*DB) - if atomic.LoadInt32(&d.closed) != 0 { - panic(ErrClosed) + if err := d.closed.Load(); err != nil { + panic(err) } if d.opts.ReadOnly { return ErrReadOnly @@ -64,7 +63,12 @@ func flushExternalTable(untypedDB interface{}, path string, originalMeta *fileMe NewFiles: []newFileEntry{newFileEntry{Level: 0, Meta: m}}, } metrics := map[int]*LevelMetrics{ - 0: &LevelMetrics{BytesIngested: m.Size, TablesIngested: 1}, + 0: &LevelMetrics{ + NumFiles: 1, + Size: int64(m.Size), + BytesIngested: m.Size, + TablesIngested: 1, + }, } err := d.mu.versions.logAndApply(jobID, ve, metrics, d.dataDir, func() []compactionInfo { return d.getInProgressCompactionInfoLocked(nil) diff --git a/github.com/cockroachdb/pebble/flushable.go b/github.com/cockroachdb/pebble/flushable.go index 6e56fad18a..e7eba129d9 100644 --- a/github.com/cockroachdb/pebble/flushable.go +++ b/github.com/cockroachdb/pebble/flushable.go @@ -33,6 +33,9 @@ type flushableEntry struct { // flushForced indicates whether a flush was forced on this memtable (either // manual, or due to ingestion). Protected by DB.mu. flushForced bool + // delayedFlushForced indicates whether a timer has been set to force a flush + // on this memtable at some point in the future. Protected by DB.mu + delayedFlushForced bool // logNum corresponds to the WAL that contains the records present in the // receiver. logNum FileNum diff --git a/github.com/cockroachdb/pebble/get_iter.go b/github.com/cockroachdb/pebble/get_iter.go index 9d7f0d7164..045fdbde49 100644 --- a/github.com/cockroachdb/pebble/get_iter.go +++ b/github.com/cockroachdb/pebble/get_iter.go @@ -140,7 +140,7 @@ func (g *getIter) Next() (*InternalKey, []byte) { if g.level == 0 { // Create iterators from L0 from newest to oldest. if n := len(g.l0); n > 0 { - files := g.l0[n-1] + files := manifest.NewLevelSlice(g.l0[n-1]).Iter() g.l0 = g.l0[:n-1] iterOpts := IterOptions{logger: g.logger} g.levelIter.init(iterOpts, g.cmp, g.newIters, files, manifest.L0Sublevel(n), nil) @@ -155,14 +155,14 @@ func (g *getIter) Next() (*InternalKey, []byte) { if g.level >= numLevels { return nil, nil } - if len(g.version.Levels[g.level]) == 0 { + if g.version.Levels[g.level].Empty() { g.level++ continue } iterOpts := IterOptions{logger: g.logger} g.levelIter.init(iterOpts, g.cmp, g.newIters, - g.version.Levels[g.level], manifest.Level(g.level), nil) + g.version.Levels[g.level].Iter(), manifest.Level(g.level), nil) g.levelIter.initRangeDel(&g.rangeDelIter) g.level++ g.iter = &g.levelIter diff --git a/github.com/cockroachdb/pebble/go.mod b/github.com/cockroachdb/pebble/go.mod index baa617e878..12f0d1d9c6 100644 --- a/github.com/cockroachdb/pebble/go.mod +++ b/github.com/cockroachdb/pebble/go.mod @@ -4,6 +4,7 @@ require ( github.com/certifi/gocertifi v0.0.0-20200211180108-c7c1fbc02894 // indirect github.com/cockroachdb/errors v1.2.4 github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f // indirect + github.com/cockroachdb/redact v0.0.0-20200622112456-cd282804bbd3 github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd github.com/davecgh/go-spew v1.1.1 // indirect github.com/getsentry/raven-go v0.2.0 // indirect @@ -16,7 +17,7 @@ require ( github.com/pmezard/go-difflib v1.0.0 github.com/spf13/cobra v0.0.3 github.com/spf13/pflag v1.0.3 // indirect - github.com/stretchr/testify v1.2.2 + github.com/stretchr/testify v1.6.1 golang.org/x/exp v0.0.0-20200513190911-00229845015e golang.org/x/sys v0.0.0-20200519105757-fe76b779f299 ) diff --git a/github.com/cockroachdb/pebble/go.sum b/github.com/cockroachdb/pebble/go.sum index b067e9e17c..c9b86596ec 100644 --- a/github.com/cockroachdb/pebble/go.sum +++ b/github.com/cockroachdb/pebble/go.sum @@ -6,8 +6,11 @@ github.com/cockroachdb/errors v1.2.4 h1:Lap807SXTH5tri2TivECb/4abUkMZC9zRoLarvcK github.com/cockroachdb/errors v1.2.4/go.mod h1:rQD95gz6FARkaKkQXUksEje/d9a6wBJoCr5oaCLELYA= github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f h1:o/kfcElHqOiXqcou5a3rIlMc7oJbMQkeLk0VQJ7zgqY= github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f/go.mod h1:i/u985jwjWRlyHXQbwatDASoW0RMlZ/3i9yJHE2xLkI= +github.com/cockroachdb/redact v0.0.0-20200622112456-cd282804bbd3 h1:2+dpIJzYMSbLi0587YXpi8tOJT52qCOI/1I0UNThc/I= +github.com/cockroachdb/redact v0.0.0-20200622112456-cd282804bbd3/go.mod h1:BVNblN9mBWFyMyqK1k3AAiSxhvhfK2oOZZ2lK+dpvRg= github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd h1:qMd81Ts1T2OTKmB4acZcyKaMtRnY5Y44NuXGX2GFJ1w= github.com/codahale/hdrhistogram v0.0.0-20161010025455-3a0bb77429bd/go.mod h1:sE/e/2PUdi/liOCUjSTXgM1o87ZssimdTWN964YiIeI= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/getsentry/raven-go v0.2.0 h1:no+xWJRb5ZI7eE8TWgIq1jLulQiIoLG0IfYxv5JYMGs= @@ -17,10 +20,6 @@ github.com/ghemawat/stream v0.0.0-20171120220530-696b145b53b9/go.mod h1:106OIgoo github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8= github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= -github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db h1:woRePGFeVFfLKN/pOkfl+p/TAqKOfFu+7KPlMVpok/w= -github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= -github.com/golang/snappy v0.0.1 h1:Qgr9rKW7uDUkrbSmQeiDsGa8SjGyCOGtuasMWwvp2P4= -github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/golang/snappy v0.0.2-0.20190904063534-ff6b7dc882cf h1:gFVkHXmVAhEbxZVDln5V9GKrLaluNoFHDbrZwAWZgws= github.com/golang/snappy v0.0.2-0.20190904063534-ff6b7dc882cf/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q= github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM= @@ -40,35 +39,35 @@ github.com/spf13/cobra v0.0.3 h1:ZlrZ4XsMRm04Fr5pSFxBgfND2EBVa1nLpiy1stUsX/8= github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ= github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg= github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +github.com/stretchr/objx v0.1.0 h1:4G4v2dO3VZwixGIRoQ5Lfboy6nUhCyYzaqnIAPPhYs4= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= +github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0= +github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= -golang.org/x/exp v0.0.0-20190426190305-956cc1757749 h1:Bduxdpx1O6126WsH6F6NwKywZ/FPncphlTduoPxFG78= -golang.org/x/exp v0.0.0-20190426190305-956cc1757749/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= golang.org/x/exp v0.0.0-20200513190911-00229845015e h1:rMqLP+9XLy+LdbCXHjJHAmTfXCr93W7oruWA6Hq1Alc= golang.org/x/exp v0.0.0-20200513190911-00229845015e/go.mod h1:4M0jN8W1tt0AVLNr8HDosyJCDCDuyL9N9+3m7wDWgKw= golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= -golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= -golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20190804053845-51ab0e2deafa h1:KIDDMLT1O0Nr7TSxp8xM5tJcdn8tgyAONntO829og1M= -golang.org/x/sys v0.0.0-20190804053845-51ab0e2deafa/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20200519105757-fe76b779f299 h1:DYfZAGf2WMFjMxbgTjaC+2HC7NkNAQs+6Q8b9WEB/F4= golang.org/x/sys v0.0.0-20200519105757-fe76b779f299/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/tools v0.0.0-20181030221726-6c7e314b6563/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo= +gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/github.com/cockroachdb/pebble/ingest.go b/github.com/cockroachdb/pebble/ingest.go index 20f2d70569..fc4f1feadb 100644 --- a/github.com/cockroachdb/pebble/ingest.go +++ b/github.com/cockroachdb/pebble/ingest.go @@ -6,7 +6,6 @@ package pebble import ( "sort" - "sync/atomic" "time" "github.com/cockroachdb/errors" @@ -34,11 +33,11 @@ func sstableKeyCompare(userCmp Compare, a, b InternalKey) int { func ingestValidateKey(opts *Options, key *InternalKey) error { if key.Kind() == InternalKeyKindInvalid { - return errors.Errorf("pebble: external sstable has corrupted key: %s", + return base.CorruptionErrorf("pebble: external sstable has corrupted key: %s", key.Pretty(opts.Comparer.FormatKey)) } if key.SeqNum() != 0 { - return errors.Errorf("pebble: external sstable has non-zero seqnum: %s", + return base.CorruptionErrorf("pebble: external sstable has non-zero seqnum: %s", key.Pretty(opts.Comparer.FormatKey)) } return nil @@ -80,10 +79,7 @@ func ingestLoad1( // disallowing removal of an open file. Under MemFS, if we don't populate // meta.Stats here, the file will be loaded into the table cache for // calculating stats before we can remove the original link. - if r.Properties.NumRangeDeletions == 0 { - meta.Stats.Valid = true - meta.Stats.RangeDeletionsBytesEstimate = 0 - } + maybeSetStatsFromProperties(meta, &r.Properties) smallestSet, largestSet := false, false empty := true @@ -118,7 +114,7 @@ func ingestLoad1( } } - iter, err := r.NewRangeDelIter() + iter, err := r.NewRawRangeDelIter() if err != nil { return nil, err } @@ -397,15 +393,15 @@ func ingestTargetLevel( targetLevel := 0 // Do we overlap with keys in L0? - for i := 0; i < len(v.Levels[0]); i++ { - meta0 := v.Levels[0][i] + iter := v.Levels[0].Iter() + for meta0 := iter.First(); meta0 != nil; meta0 = iter.Next() { c1 := sstableKeyCompare(cmp, meta.Smallest, meta0.Largest) c2 := sstableKeyCompare(cmp, meta.Largest, meta0.Smallest) if c1 > 0 || c2 < 0 { continue } - iter, rangeDelIter, err := newIters(meta0, nil, nil) + iter, rangeDelIter, err := newIters(iter.Take(), nil, nil) if err != nil { return 0, err } @@ -421,7 +417,7 @@ func ingestTargetLevel( level := baseLevel for ; level < numLevels; level++ { - levelIter := newLevelIter(iterOps, cmp, newIters, v.Levels[level], manifest.Level(level), nil) + levelIter := newLevelIter(iterOps, cmp, newIters, v.Levels[level].Iter(), manifest.Level(level), nil) var rangeDelIter internalIterator // Pass in a non-nil pointer to rangeDelIter so that levelIter.findFileGE sets it up for the target file. levelIter.initRangeDel(&rangeDelIter) @@ -432,7 +428,7 @@ func ingestTargetLevel( } // Check boundary overlap. - if len(v.Overlaps(level, cmp, meta.Smallest.UserKey, meta.Largest.UserKey)) != 0 { + if !v.Overlaps(level, cmp, meta.Smallest.UserKey, meta.Largest.UserKey).Empty() { continue } @@ -445,7 +441,7 @@ func ingestTargetLevel( // negative (else we'd have returned earlier). overlaps := false for c := range compactions { - if level != c.outputLevel { + if c.outputLevel == nil || level != c.outputLevel.level { continue } if cmp(meta.Smallest.UserKey, c.largest.UserKey) <= 0 && @@ -468,6 +464,14 @@ func ingestTargetLevel( // the same filesystem as the DB. Sstables can be created for ingestion using // sstable.Writer. On success, Ingest removes the input paths. // +// All sstables *must* be Sync()'d by the caller after all bytes are written +// and before its file handle is closed; failure to do so could violate +// durability or lead to corrupted on-disk state. This method cannot, in a +// platform-and-FS-agnostic way, ensure that all sstables in the input are +// properly synced to disk. Opening new file handles and Sync()-ing them +// does not always guarantee durability; see the discussion here on that: +// https://github.com/cockroachdb/pebble/pull/835#issuecomment-663075379 +// // Ingestion loads each sstable into the lowest level of the LSM which it // doesn't overlap (see ingestTargetLevel). If an sstable overlaps a memtable, // ingestion forces the memtable to flush, and then waits for the flush to @@ -496,8 +500,8 @@ func ingestTargetLevel( // https://github.com/cockroachdb/pebble/issues/25 for an idea for how to fix // this hiccup. func (d *DB) Ingest(paths []string) error { - if atomic.LoadInt32(&d.closed) != 0 { - panic(ErrClosed) + if err := d.closed.Load(); err != nil { + panic(err) } if d.opts.ReadOnly { return ErrReadOnly @@ -669,6 +673,8 @@ func (d *DB) ingestApply(jobID int, meta []*fileMetadata) (*versionEdit, error) levelMetrics = &LevelMetrics{} metrics[f.Level] = levelMetrics } + levelMetrics.NumFiles++ + levelMetrics.Size += int64(m.Size) levelMetrics.BytesIngested += m.Size levelMetrics.TablesIngested++ } diff --git a/github.com/cockroachdb/pebble/internal.go b/github.com/cockroachdb/pebble/internal.go index 604832fca8..00e38b33b8 100644 --- a/github.com/cockroachdb/pebble/internal.go +++ b/github.com/cockroachdb/pebble/internal.go @@ -4,9 +4,7 @@ package pebble -import ( - "github.com/cockroachdb/pebble/internal/base" -) +import "github.com/cockroachdb/pebble/internal/base" // InternalKeyKind exports the base.InternalKeyKind type. type InternalKeyKind = base.InternalKeyKind diff --git a/github.com/cockroachdb/pebble/internal/arenaskl/flush_iterator.go b/github.com/cockroachdb/pebble/internal/arenaskl/flush_iterator.go index c5305b00cc..3384793ace 100644 --- a/github.com/cockroachdb/pebble/internal/arenaskl/flush_iterator.go +++ b/github.com/cockroachdb/pebble/internal/arenaskl/flush_iterator.go @@ -17,9 +17,7 @@ package arenaskl -import ( - "github.com/cockroachdb/pebble/internal/base" -) +import "github.com/cockroachdb/pebble/internal/base" // flushIterator is an iterator over the skiplist object. Use Skiplist.NewFlushIter // to construct an iterator. The current state of the iterator can be cloned by diff --git a/github.com/cockroachdb/pebble/internal/base/error.go b/github.com/cockroachdb/pebble/internal/base/error.go index 1b1937fe13..6ef7783e7f 100644 --- a/github.com/cockroachdb/pebble/internal/base/error.go +++ b/github.com/cockroachdb/pebble/internal/base/error.go @@ -8,3 +8,21 @@ import "github.com/cockroachdb/errors" // ErrNotFound means that a get or delete call did not find the requested key. var ErrNotFound = errors.New("pebble: not found") + +// ErrCorruption is a marker to indicate that data in a file (WAL, MANIFEST, +// sstable) isn't in the expected format. +var ErrCorruption = errors.New("pebble: corruption") + +// MarkCorruptionError marks given error as a corruption error. +func MarkCorruptionError(err error) error { + if errors.Is(err, ErrCorruption) { + return err + } + return errors.Mark(err, ErrCorruption) +} + +// CorruptionErrorf formats according to a format specifier and returns +// the string as an error value that is marked as a corruption error. +func CorruptionErrorf(format string, args ...interface{}) error { + return errors.Mark(errors.Newf(format, args...), ErrCorruption) +} diff --git a/github.com/cockroachdb/pebble/internal/base/iterator.go b/github.com/cockroachdb/pebble/internal/base/iterator.go index 23612f6f79..ab3cdcd6ab 100644 --- a/github.com/cockroachdb/pebble/internal/base/iterator.go +++ b/github.com/cockroachdb/pebble/internal/base/iterator.go @@ -29,7 +29,8 @@ import "fmt" // // The relative positioning methods can be used in conjunction with any of the // absolute positioning methods with one exception: SeekPrefixGE does not -// support reverse iteration via Prev. +// support reverse iteration via Prev. It is undefined to call relative +// positioning methods without ever calling an absolute positioning method. // // InternalIterators can optionally implement a prefix iteration mode. This // mode is entered by calling SeekPrefixGE and exited by any other absolute @@ -52,11 +53,15 @@ import "fmt" // Last if there is an upper bound). This imposition is done in order to // elevate that enforcement to the caller (generally pebble.Iterator or // pebble.mergingIter) rather than having it duplicated in every -// InternalIterator implementation. InternalIterator implementations are -// required to respect the iterator bounds, never returning records outside of -// the bounds with one exception: an iterator may generate synthetic RANGEDEL -// marker records. See levelIter.syntheticBoundary for the sole existing -// example of this behavior. [TODO(peter): can we eliminate this exception?] +// InternalIterator implementation. Additionally, the caller needs to ensure +// that SeekGE/SeekPrefixGE are not called with a key > the upper bound, and +// SeekLT is not called with a key < the lower bound. +// InternalIterator implementations are required to respect the iterator +// bounds, never returning records outside of the bounds with one exception: +// an iterator may generate synthetic RANGEDEL marker records. See +// levelIter.syntheticBoundary for the sole existing example of this behavior. +// Specifically, levelIter can return synthetic keys whose user key is equal +// to the lower/upper bound. // // An iterator must be closed after use, but it is not necessary to read an // iterator until exhaustion. diff --git a/github.com/cockroachdb/pebble/internal/base/merger.go b/github.com/cockroachdb/pebble/internal/base/merger.go index 0cffed5e9c..b602925ed5 100644 --- a/github.com/cockroachdb/pebble/internal/base/merger.go +++ b/github.com/cockroachdb/pebble/internal/base/merger.go @@ -48,9 +48,16 @@ type ValueMerger interface { // Finish must be the last function called on the ValueMerger. The caller // must not call any other ValueMerger functions after calling Finish. // + // If `includesBase` is true, the oldest merge operand was part of the + // merge. This will always be the true during normal iteration, but may be + // false during compaction when only a subset of operands may be + // available. Note that `includesBase` is set to true conservatively: a false + // value means that we could not definitely determine that the base merge + // operand was included. + // // If a Closer is returned, the returned slice will remain valid until it is // closed. The caller must arrange for the closer to be eventually closed. - Finish() ([]byte, io.Closer, error) + Finish(includesBase bool) ([]byte, io.Closer, error) } // Merger defines an associative merge operation. The merge operation merges @@ -93,7 +100,7 @@ func (a *AppendValueMerger) MergeOlder(value []byte) error { } // Finish returns the buffer that was constructed on-demand in `Merge{OlderNewer}()` calls. -func (a *AppendValueMerger) Finish() ([]byte, io.Closer, error) { +func (a *AppendValueMerger) Finish(includesBase bool) ([]byte, io.Closer, error) { return a.buf, nil, nil } diff --git a/github.com/cockroachdb/pebble/internal/batchskl/iterator.go b/github.com/cockroachdb/pebble/internal/batchskl/iterator.go index 2f3d8dcbe4..9a2d9c68c8 100644 --- a/github.com/cockroachdb/pebble/internal/batchskl/iterator.go +++ b/github.com/cockroachdb/pebble/internal/batchskl/iterator.go @@ -17,9 +17,7 @@ package batchskl -import ( - "github.com/cockroachdb/pebble/internal/base" -) +import "github.com/cockroachdb/pebble/internal/base" type splice struct { prev uint32 @@ -188,9 +186,7 @@ func (it *Iterator) SetBounds(lower, upper []byte) { it.upper = upper } -func (it *Iterator) seekForBaseSplice( - key []byte, abbreviatedKey uint64, -) (prev, next uint32) { +func (it *Iterator) seekForBaseSplice(key []byte, abbreviatedKey uint64) (prev, next uint32) { prev = it.list.head for level := it.list.height - 1; ; level-- { prev, next = it.list.findSpliceForLevel(key, abbreviatedKey, level, prev) diff --git a/github.com/cockroachdb/pebble/internal/crc/crc.go b/github.com/cockroachdb/pebble/internal/crc/crc.go index c867c71937..990952ea52 100644 --- a/github.com/cockroachdb/pebble/internal/crc/crc.go +++ b/github.com/cockroachdb/pebble/internal/crc/crc.go @@ -14,9 +14,7 @@ // In pebble, the uint32 value is then stored in little-endian format. package crc // import "github.com/cockroachdb/pebble/internal/crc" -import ( - "hash/crc32" -) +import "hash/crc32" var table = crc32.MakeTable(crc32.Castagnoli) diff --git a/github.com/cockroachdb/pebble/internal/fastrand/fastrand.go b/github.com/cockroachdb/pebble/internal/fastrand/fastrand.go index c9d88f5ef8..8c1d52f24a 100644 --- a/github.com/cockroachdb/pebble/internal/fastrand/fastrand.go +++ b/github.com/cockroachdb/pebble/internal/fastrand/fastrand.go @@ -4,9 +4,7 @@ package fastrand -import ( - _ "unsafe" // required by go:linkname -) +import _ "unsafe" // required by go:linkname // Uint32 returns a lock free uint32 value. //go:linkname Uint32 runtime.fastrand diff --git a/github.com/cockroachdb/pebble/internal/manifest/btree.go b/github.com/cockroachdb/pebble/internal/manifest/btree.go new file mode 100644 index 0000000000..f0ae1ea7e7 --- /dev/null +++ b/github.com/cockroachdb/pebble/internal/manifest/btree.go @@ -0,0 +1,988 @@ +// Copyright 2020 The LevelDB-Go and Pebble Authors. All rights reserved. Use +// of this source code is governed by a BSD-style license that can be found in +// the LICENSE file. + +package manifest + +import ( + "bytes" + "fmt" + "strings" + "sync/atomic" + "unsafe" + + "github.com/cockroachdb/pebble/internal/base" +) + +const ( + degree = 16 + maxItems = 2*degree - 1 + minItems = degree - 1 +) + +type leafNode struct { + ref int32 + count int16 + leaf bool + items [maxItems]*FileMetadata +} + +type node struct { + leafNode + children [maxItems + 1]*node +} + +//go:nocheckptr casts a ptr to a smaller struct to a ptr to a larger struct. +func leafToNode(ln *leafNode) *node { + return (*node)(unsafe.Pointer(ln)) +} + +func nodeToLeaf(n *node) *leafNode { + return (*leafNode)(unsafe.Pointer(n)) +} + +func newLeafNode() *node { + n := leafToNode(new(leafNode)) + n.leaf = true + n.ref = 1 + return n +} + +func newNode() *node { + n := new(node) + n.ref = 1 + return n +} + +// mut creates and returns a mutable node reference. If the node is not shared +// with any other trees then it can be modified in place. Otherwise, it must be +// cloned to ensure unique ownership. In this way, we enforce a copy-on-write +// policy which transparently incorporates the idea of local mutations, like +// Clojure's transients or Haskell's ST monad, where nodes are only copied +// during the first time that they are modified between Clone operations. +// +// When a node is cloned, the provided pointer will be redirected to the new +// mutable node. +func mut(n **node) *node { + if atomic.LoadInt32(&(*n).ref) == 1 { + // Exclusive ownership. Can mutate in place. + return *n + } + // If we do not have unique ownership over the node then we + // clone it to gain unique ownership. After doing so, we can + // release our reference to the old node. We pass recursive + // as true because even though we just observed the node's + // reference count to be greater than 1, we might be racing + // with another call to decRef on this node. + c := (*n).clone() + (*n).decRef(true /* recursive */, nil) + *n = c + return *n +} + +// incRef acquires a reference to the node. +func (n *node) incRef() { + atomic.AddInt32(&n.ref, 1) +} + +// decRef releases a reference to the node. If requested, the method +// will recurse into child nodes and decrease their refcounts as well. +// When a node is released, its contained files are dereferenced. +func (n *node) decRef(recursive bool, obsolete *[]base.FileNum) { + if atomic.AddInt32(&n.ref, -1) > 0 { + // Other references remain. Can't free. + return + } + + // Dereference the node's metadata and release child references. + if recursive { + for _, f := range n.items[:n.count] { + if atomic.AddInt32(&f.refs, -1) == 0 { + // There are two sources of node dereferences: tree mutations + // and Version dereferences. Files should only be made obsolete + // during Version dereferences, during which `obsolete` will be + // non-nil. + if obsolete == nil { + panic(fmt.Sprintf("file metadata %s dereferenced to zero during tree mutation", f.FileNum)) + } + *obsolete = append(*obsolete, f.FileNum) + } + } + if !n.leaf { + for i := int16(0); i <= n.count; i++ { + n.children[i].decRef(true /* recursive */, obsolete) + } + } + } +} + +// clone creates a clone of the receiver with a single reference count. +func (n *node) clone() *node { + var c *node + if n.leaf { + c = newLeafNode() + } else { + c = newNode() + } + // NB: copy field-by-field without touching n.ref to avoid + // triggering the race detector and looking like a data race. + c.count = n.count + c.items = n.items + // Increase the refcount of each contained item. + for _, f := range n.items[:n.count] { + atomic.AddInt32(&f.refs, 1) + } + if !c.leaf { + // Copy children and increase each refcount. + c.children = n.children + for i := int16(0); i <= c.count; i++ { + c.children[i].incRef() + } + } + return c +} + +func (n *node) insertAt(index int, item *FileMetadata, nd *node) { + if index < int(n.count) { + copy(n.items[index+1:n.count+1], n.items[index:n.count]) + if !n.leaf { + copy(n.children[index+2:n.count+2], n.children[index+1:n.count+1]) + } + } + n.items[index] = item + if !n.leaf { + n.children[index+1] = nd + } + n.count++ +} + +func (n *node) pushBack(item *FileMetadata, nd *node) { + n.items[n.count] = item + if !n.leaf { + n.children[n.count+1] = nd + } + n.count++ +} + +func (n *node) pushFront(item *FileMetadata, nd *node) { + if !n.leaf { + copy(n.children[1:n.count+2], n.children[:n.count+1]) + n.children[0] = nd + } + copy(n.items[1:n.count+1], n.items[:n.count]) + n.items[0] = item + n.count++ +} + +// removeAt removes a value at a given index, pulling all subsequent values +// back. +func (n *node) removeAt(index int) (*FileMetadata, *node) { + var child *node + if !n.leaf { + child = n.children[index+1] + copy(n.children[index+1:n.count], n.children[index+2:n.count+1]) + n.children[n.count] = nil + } + n.count-- + out := n.items[index] + copy(n.items[index:n.count], n.items[index+1:n.count+1]) + n.items[n.count] = nil + return out, child +} + +// popBack removes and returns the last element in the list. +func (n *node) popBack() (*FileMetadata, *node) { + n.count-- + out := n.items[n.count] + n.items[n.count] = nil + if n.leaf { + return out, nil + } + child := n.children[n.count+1] + n.children[n.count+1] = nil + return out, child +} + +// popFront removes and returns the first element in the list. +func (n *node) popFront() (*FileMetadata, *node) { + n.count-- + var child *node + if !n.leaf { + child = n.children[0] + copy(n.children[:n.count+1], n.children[1:n.count+2]) + n.children[n.count+1] = nil + } + out := n.items[0] + copy(n.items[:n.count], n.items[1:n.count+1]) + n.items[n.count] = nil + return out, child +} + +// find returns the index where the given item should be inserted into this +// list. 'found' is true if the item already exists in the list at the given +// index. +func (n *node) find( + cmp func(*FileMetadata, *FileMetadata) int, item *FileMetadata, +) (index int, found bool) { + // Logic copied from sort.Search. Inlining this gave + // an 11% speedup on BenchmarkBTreeDeleteInsert. + i, j := 0, int(n.count) + for i < j { + h := int(uint(i+j) >> 1) // avoid overflow when computing h + // i ≤ h < j + v := cmp(item, n.items[h]) + if v == 0 { + return h, true + } else if v > 0 { + i = h + 1 + } else { + j = h + } + } + return i, false +} + +// split splits the given node at the given index. The current node shrinks, +// and this function returns the item that existed at that index and a new +// node containing all items/children after it. +// +// Before: +// +// +-----------+ +// | x y z | +// +--/-/-\-\--+ +// +// After: +// +// +-----------+ +// | y | +// +----/-\----+ +// / \ +// v v +// +-----------+ +-----------+ +// | x | | z | +// +-----------+ +-----------+ +// +func (n *node) split(i int) (*FileMetadata, *node) { + out := n.items[i] + var next *node + if n.leaf { + next = newLeafNode() + } else { + next = newNode() + } + next.count = n.count - int16(i+1) + copy(next.items[:], n.items[i+1:n.count]) + for j := int16(i); j < n.count; j++ { + n.items[j] = nil + } + if !n.leaf { + copy(next.children[:], n.children[i+1:n.count+1]) + for j := int16(i + 1); j <= n.count; j++ { + n.children[j] = nil + } + } + n.count = int16(i) + return out, next +} + +// insert inserts a item into the subtree rooted at this node, making sure no +// nodes in the subtree exceed maxItems items. +func (n *node) insert(cmp func(*FileMetadata, *FileMetadata) int, item *FileMetadata) { + i, found := n.find(cmp, item) + if found { + // cmp provides a total ordering of the files within a level. + // If we're inserting a metadata that's equal to an existing item + // in the tree, we're inserting a file into a level twice. + panic(fmt.Sprintf("file key collision: existing metadata %s, inserting %s", + n.items[i].FileNum, item.FileNum)) + } + if n.leaf { + n.insertAt(i, item, nil) + return + } + if n.children[i].count >= maxItems { + splitLa, splitNode := mut(&n.children[i]).split(maxItems / 2) + n.insertAt(i, splitLa, splitNode) + + switch cmp := cmp(item, n.items[i]); { + case cmp < 0: + // no change, we want first split node + case cmp > 0: + i++ // we want second split node + default: + // cmp provides a total ordering of the files within a level. + // If we're inserting a metadata that's equal to an existing item + // in the tree, we're inserting a file into a level twice. + panic(fmt.Sprintf("file key collision: existing metadata %s, inserting %s", + n.items[i].FileNum, item.FileNum)) + } + } + mut(&n.children[i]).insert(cmp, item) +} + +// removeMax removes and returns the maximum item from the subtree rooted +// at this node. +func (n *node) removeMax() *FileMetadata { + if n.leaf { + n.count-- + out := n.items[n.count] + n.items[n.count] = nil + return out + } + child := mut(&n.children[n.count]) + if child.count <= minItems { + n.rebalanceOrMerge(int(n.count)) + return n.removeMax() + } + return child.removeMax() +} + +// remove removes a item from the subtree rooted at this node. Returns +// the item that was removed or nil if no matching item was found. +func (n *node) remove( + cmp func(*FileMetadata, *FileMetadata) int, item *FileMetadata, +) (out *FileMetadata) { + i, found := n.find(cmp, item) + if n.leaf { + if found { + out, _ = n.removeAt(i) + return out + } + return nil + } + if n.children[i].count <= minItems { + // Child not large enough to remove from. + n.rebalanceOrMerge(i) + return n.remove(cmp, item) + } + child := mut(&n.children[i]) + if found { + // Replace the item being removed with the max item in our left child. + out = n.items[i] + n.items[i] = child.removeMax() + return out + } + // Latch is not in this node and child is large enough to remove from. + out = child.remove(cmp, item) + return out +} + +// rebalanceOrMerge grows child 'i' to ensure it has sufficient room to remove +// a item from it while keeping it at or above minItems. +func (n *node) rebalanceOrMerge(i int) { + switch { + case i > 0 && n.children[i-1].count > minItems: + // Rebalance from left sibling. + // + // +-----------+ + // | y | + // +----/-\----+ + // / \ + // v v + // +-----------+ +-----------+ + // | x | | | + // +----------\+ +-----------+ + // \ + // v + // a + // + // After: + // + // +-----------+ + // | x | + // +----/-\----+ + // / \ + // v v + // +-----------+ +-----------+ + // | | | y | + // +-----------+ +/----------+ + // / + // v + // a + // + left := mut(&n.children[i-1]) + child := mut(&n.children[i]) + xLa, grandChild := left.popBack() + yLa := n.items[i-1] + child.pushFront(yLa, grandChild) + n.items[i-1] = xLa + + case i < int(n.count) && n.children[i+1].count > minItems: + // Rebalance from right sibling. + // + // +-----------+ + // | y | + // +----/-\----+ + // / \ + // v v + // +-----------+ +-----------+ + // | | | x | + // +-----------+ +/----------+ + // / + // v + // a + // + // After: + // + // +-----------+ + // | x | + // +----/-\----+ + // / \ + // v v + // +-----------+ +-----------+ + // | y | | | + // +----------\+ +-----------+ + // \ + // v + // a + // + right := mut(&n.children[i+1]) + child := mut(&n.children[i]) + xLa, grandChild := right.popFront() + yLa := n.items[i] + child.pushBack(yLa, grandChild) + n.items[i] = xLa + + default: + // Merge with either the left or right sibling. + // + // +-----------+ + // | u y v | + // +----/-\----+ + // / \ + // v v + // +-----------+ +-----------+ + // | x | | z | + // +-----------+ +-----------+ + // + // After: + // + // +-----------+ + // | u v | + // +-----|-----+ + // | + // v + // +-----------+ + // | x y z | + // +-----------+ + // + if i >= int(n.count) { + i = int(n.count - 1) + } + child := mut(&n.children[i]) + // Make mergeChild mutable, bumping the refcounts on its children if necessary. + _ = mut(&n.children[i+1]) + mergeLa, mergeChild := n.removeAt(i) + child.items[child.count] = mergeLa + copy(child.items[child.count+1:], mergeChild.items[:mergeChild.count]) + if !child.leaf { + copy(child.children[child.count+1:], mergeChild.children[:mergeChild.count+1]) + } + child.count += mergeChild.count + 1 + + mergeChild.decRef(false /* recursive */, nil) + } +} + +// btree is an implementation of a B-Tree. +// +// btree stores FileMetadata in an ordered structure, allowing easy insertion, +// removal, and iteration. The B-Tree stores items in order based on cmp. The +// first level of the LSM uses a cmp function that compares sequence numbers. +// All other levels compare using the FileMetadata.Smallest. +// +// Write operations are not safe for concurrent mutation by multiple +// goroutines, but Read operations are. +type btree struct { + root *node + length int + cmp func(*FileMetadata, *FileMetadata) int +} + +// release dereferences and clears the root node of the btree, removing all +// items from the btree. In doing so, it decrements contained file counts. +// It returns a slice of file numbers of newly obsolete files, if any. +func (t *btree) release() (obsolete []base.FileNum) { + if t.root != nil { + t.root.decRef(true /* recursive */, &obsolete) + t.root = nil + } + t.length = 0 + return obsolete +} + +// clone clones the btree, lazily. It does so in constant time. +func (t *btree) clone() btree { + c := *t + if c.root != nil { + // Incrementing the reference count on the root node is sufficient to + // ensure that no node in the cloned tree can be mutated by an actor + // holding a reference to the original tree and vice versa. This + // property is upheld because the root node in the receiver btree and + // the returned btree will both necessarily have a reference count of at + // least 2 when this method returns. All tree mutations recursively + // acquire mutable node references (see mut) as they traverse down the + // tree. The act of acquiring a mutable node reference performs a clone + // if a node's reference count is greater than one. Cloning a node (see + // clone) increases the reference count on each of its children, + // ensuring that they have a reference count of at least 2. This, in + // turn, ensures that any of the child nodes that are modified will also + // be copied-on-write, recursively ensuring the immutability property + // over the entire tree. + c.root.incRef() + } + return c +} + +// delete removes the provided file from the tree. +// It returns true if the file now has a zero reference count. +func (t *btree) delete(item *FileMetadata) (obsolete bool) { + if t.root == nil || t.root.count == 0 { + return false + } + if out := mut(&t.root).remove(t.cmp, item); out != nil { + t.length-- + obsolete = atomic.AddInt32(&out.refs, -1) == 0 + } + if t.root.count == 0 { + old := t.root + if t.root.leaf { + t.root = nil + } else { + t.root = t.root.children[0] + } + old.decRef(false /* recursive */, nil) + } + return obsolete +} + +// insert adds the given item to the tree. If a item in the tree already +// equals the given one, insert panics. +func (t *btree) insert(item *FileMetadata) { + if t.root == nil { + t.root = newLeafNode() + } else if t.root.count >= maxItems { + splitLa, splitNode := mut(&t.root).split(maxItems / 2) + newRoot := newNode() + newRoot.count = 1 + newRoot.items[0] = splitLa + newRoot.children[0] = t.root + newRoot.children[1] = splitNode + t.root = newRoot + } + atomic.AddInt32(&item.refs, 1) + mut(&t.root).insert(t.cmp, item) + t.length++ +} + +// iter returns a new iterator object. It is not safe to continue using an +// iterator after modifications are made to the tree. If modifications are made, +// create a new iterator. +func (t *btree) iter() iterator { + return iterator{r: t.root, pos: -1, cmp: t.cmp} +} + +// height returns the height of the tree. +func (t *btree) height() int { + if t.root == nil { + return 0 + } + h := 1 + n := t.root + for !n.leaf { + n = n.children[0] + h++ + } + return h +} + +// String returns a string description of the tree. The format is +// similar to the https://en.wikipedia.org/wiki/Newick_format. +func (t *btree) String() string { + if t.length == 0 { + return ";" + } + var b strings.Builder + t.root.writeString(&b) + return b.String() +} + +func (n *node) writeString(b *strings.Builder) { + if n.leaf { + for i := int16(0); i < n.count; i++ { + if i != 0 { + b.WriteString(",") + } + b.WriteString(n.items[i].String()) + } + return + } + for i := int16(0); i <= n.count; i++ { + b.WriteString("(") + n.children[i].writeString(b) + b.WriteString(")") + if i < n.count { + b.WriteString(n.items[i].String()) + } + } +} + +// iterStack represents a stack of (node, pos) tuples, which captures +// iteration state as an iterator descends a btree. +type iterStack struct { + a iterStackArr + aLen int16 // -1 when using s + s []iterFrame +} + +// Used to avoid allocations for stacks below a certain size. +type iterStackArr [3]iterFrame + +type iterFrame struct { + n *node + pos int16 +} + +func (is *iterStack) push(f iterFrame) { + if is.aLen == -1 { + is.s = append(is.s, f) + } else if int(is.aLen) == len(is.a) { + is.s = make([]iterFrame, int(is.aLen)+1, 2*int(is.aLen)) + copy(is.s, is.a[:]) + is.s[int(is.aLen)] = f + is.aLen = -1 + } else { + is.a[is.aLen] = f + is.aLen++ + } +} + +func (is *iterStack) pop() iterFrame { + if is.aLen == -1 { + f := is.s[len(is.s)-1] + is.s = is.s[:len(is.s)-1] + return f + } + is.aLen-- + return is.a[is.aLen] +} + +func (is *iterStack) len() int { + if is.aLen == -1 { + return len(is.s) + } + return int(is.aLen) +} + +func (is *iterStack) clone() iterStack { + // If the iterator is using the embedded iterStackArr, we only need to + // copy the struct itself. + if is.s == nil { + return *is + } + clone := *is + clone.s = make([]iterFrame, len(is.s)) + copy(clone.s, is.s) + return clone +} + +func (is *iterStack) nth(n int) (f iterFrame, ok bool) { + if is.aLen == -1 { + if n >= len(is.s) { + return f, false + } + return is.s[n], true + } + if int16(n) >= is.aLen { + return f, false + } + return is.a[n], true +} + +func (is *iterStack) reset() { + if is.aLen == -1 { + is.s = is.s[:0] + } else { + is.aLen = 0 + } +} + +// iterator is responsible for search and traversal within a btree. +type iterator struct { + r *node + n *node + pos int16 + cmp func(*FileMetadata, *FileMetadata) int + s iterStack +} + +func (i *iterator) clone() iterator { + c := *i + c.s = i.s.clone() + return c +} + +func (i *iterator) reset() { + i.n = i.r + i.pos = -1 + i.s.reset() +} + +func (i iterator) String() string { + var buf bytes.Buffer + for n := 0; ; n++ { + f, ok := i.s.nth(n) + if !ok { + break + } + fmt.Fprintf(&buf, "%p: %02d/%02d\n", f.n, f.pos, f.n.count) + } + if i.n == nil { + fmt.Fprintf(&buf, ": %02d", i.pos) + } else { + fmt.Fprintf(&buf, "%p: %02d/%02d", i.n, i.pos, i.n.count) + } + return buf.String() +} + +func cmpIter(a, b iterator) int { + if a.r != b.r { + panic("compared iterators from different btrees") + } + + // Each iterator has a stack of frames marking the path from the root node + // to the current iterator position. We walk both paths formed by the + // iterators' stacks simulatenously, descending from the shared root node, + // always comparing nodes at the same level in the tree. + // + // If the iterators' paths ever diverge and point to different nodes, the + // iterators are not equal and we use the node positions to evaluate the + // comparison. + // + // If an iterator's stack ends, we stop descending and use its current + // node and position for the final comparison. One iterator's stack may + // end before another's if one iterator is positioned deeper in the tree. + // + // a b + // +------------------------+ +--------------------------+ - + // | Root pos:5 | = | Root pos:5 | | + // +------------------------+ +--------------------------+ | stack + // | Root/5 pos:3 | = | Root/5 pos:3 | | frames + // +------------------------+ +--------------------------+ | + // | Root/5/3 pos:9 | > | Root/5/3 pos:1 | | + // +========================+ +==========================+ - + // | | | | + // | a.n: Root/5/3/9 a.pos:2| | b.n: Root/5/3/1, b.pos:5 | + // +------------------------+ +--------------------------+ + + // Initialize with the iterator's current node and position. These are + // conceptually the most-recent/current frame of the iterator stack. + an, apos := a.n, a.pos + bn, bpos := b.n, b.pos + + // aok, bok are set while traversing the iterator's path down the B-Tree. + // They're declared in the outer scope because they help distinguish the + // sentinel case when both iterators' first frame points to the last child + // of the root. If an iterator has no other frames in its stack, it's the + // end sentinel state which sorts after everything else. + var aok, bok bool + for i := 0; ; i++ { + var af, bf iterFrame + af, aok = a.s.nth(i) + bf, bok = b.s.nth(i) + if !aok || !bok { + if aok { + // Iterator a, unlike iterator b, still has a frame. Set an, + // apos so we compare using the frame from the stack. + an, apos = af.n, af.pos + } + if bok { + // Iterator b, unlike iterator a, still has a frame. Set bn, + // bpos so we compare using the frame from the stack. + bn, bpos = bf.n, bf.pos + } + break + } + + // aok && bok + if af.n != bf.n { + panic("nonmatching nodes during btree iterator comparison") + } + switch { + case af.pos < bf.pos: + return -1 + case af.pos > bf.pos: + return +1 + default: + // Continue up both iterators' stacks (equivalently, down the + // B-Tree away from the root). + } + } + + if aok && bok { + panic("expected one or more stacks to have been exhausted") + } + if an != bn { + panic("nonmatching nodes during btree iterator comparison") + } + switch { + case apos < bpos: + return -1 + case apos > bpos: + return +1 + default: + switch { + case aok: + // a is positioned at a leaf child at this position and b is at an + // end sentinel state. + return -1 + case bok: + // b is positioned at a leaf child at this position and a is at an + // end sentinel state. + return +1 + default: + return 0 + } + } +} + +func (i *iterator) descend(n *node, pos int16) { + i.s.push(iterFrame{n: n, pos: pos}) + i.n = n.children[pos] + i.pos = 0 +} + +// ascend ascends up to the current node's parent and resets the position +// to the one previously set for this parent node. +func (i *iterator) ascend() { + f := i.s.pop() + i.n = f.n + i.pos = f.pos +} + +// seekGE seeks to the first item greater-than or equal to the provided +// item. +func (i *iterator) seekGE(item *FileMetadata) { + i.reset() + if i.n == nil { + return + } + for { + pos, found := i.n.find(i.cmp, item) + i.pos = int16(pos) + if found { + return + } + if i.n.leaf { + if i.pos == i.n.count { + i.next() + } + return + } + i.descend(i.n, i.pos) + } +} + +// seekLT seeks to the first item less-than the provided item. +func (i *iterator) seekLT(item *FileMetadata) { + i.reset() + if i.n == nil { + return + } + for { + pos, found := i.n.find(i.cmp, item) + i.pos = int16(pos) + if found || i.n.leaf { + i.prev() + return + } + i.descend(i.n, i.pos) + } +} + +// first seeks to the first item in the btree. +func (i *iterator) first() { + i.reset() + if i.n == nil { + return + } + for !i.n.leaf { + i.descend(i.n, 0) + } + i.pos = 0 +} + +// last seeks to the last item in the btree. +func (i *iterator) last() { + i.reset() + if i.n == nil { + return + } + for !i.n.leaf { + i.descend(i.n, i.n.count) + } + i.pos = i.n.count - 1 +} + +// next positions the iterator to the item immediately following +// its current position. +func (i *iterator) next() { + if i.n == nil { + return + } + + if i.n.leaf { + i.pos++ + if i.pos < i.n.count { + return + } + for i.s.len() > 0 && i.pos >= i.n.count { + i.ascend() + } + return + } + + i.descend(i.n, i.pos+1) + for !i.n.leaf { + i.descend(i.n, 0) + } + i.pos = 0 +} + +// prev positions the iterator to the item immediately preceding +// its current position. +func (i *iterator) prev() { + if i.n == nil { + return + } + + if i.n.leaf { + i.pos-- + if i.pos >= 0 { + return + } + for i.s.len() > 0 && i.pos < 0 { + i.ascend() + i.pos-- + } + return + } + + i.descend(i.n, i.pos) + for !i.n.leaf { + i.descend(i.n, i.n.count) + } + i.pos = i.n.count - 1 +} + +// valid returns whether the iterator is positioned at a valid position. +func (i *iterator) valid() bool { + return i.pos >= 0 && i.pos < i.n.count +} + +// cur returns the item at the iterator's current position. It is illegal +// to call cur if the iterator is not valid. +func (i *iterator) cur() *FileMetadata { + return i.n.items[i.pos] +} diff --git a/github.com/cockroachdb/pebble/internal/manifest/l0_sublevels.go b/github.com/cockroachdb/pebble/internal/manifest/l0_sublevels.go index 47cfd55682..76c4c1dbe3 100644 --- a/github.com/cockroachdb/pebble/internal/manifest/l0_sublevels.go +++ b/github.com/cockroachdb/pebble/internal/manifest/l0_sublevels.go @@ -15,12 +15,6 @@ import ( "github.com/cockroachdb/pebble/internal/base" ) -// TODO(bilal): -// - Integrate compaction picking logic with the rest of pebble. -// - Refactor away slicing and indexing for simplicity and stronger correctness -// guarantees, especially in extendCandidateToRectangle and -// Pick{Base,IntraL0}Compactions. - // Intervals are of the form [start, end) with no gap between intervals. Each // file overlaps perfectly with a sequence of intervals. This perfect overlap // occurs because the union of file boundary keys is used to pick intervals. @@ -32,6 +26,10 @@ import ( // - intervalKey{k, false} < intervalKey{k, true} // - k1 < k2 -> intervalKey{k1, _} < intervalKey{k2, _}. // +// Note that the file's largest key is exclusive if the internal key +// has a trailer matching the rangedel sentinel key. In this case, we set +// isLargest to false for end interval computation. +// // For example, consider three files with bounds [a,e], [b,g], and [e,j]. The // interval keys produced would be intervalKey{a, false}, intervalKey{b, false}, // intervalKey{e, false}, intervalKey{e, true}, intervalKey{g, true} and @@ -96,11 +94,6 @@ func sortAndDedup(keys []intervalKey, cmp Compare) []intervalKey { return keys[:j+1] } -type subLevelAndFile struct { - subLevel int - fileIndex int -} - // A key interval of the form [start, end). The end is not represented here // since it is implicit in the start of the next interval. The last interval is // an exception but we don't need to ever lookup the end of that interval; the @@ -144,10 +137,8 @@ type fileInterval struct { fileCount int compactingFileCount int - // All files in this interval, in increasing sublevel order. Indexes map - // to s.filesByAge. Note that the indexes are not sublevel indexes; the - // sublevel lives at s.filesByAge[files[i]].subLevel. - files []int + // All files in this interval, in increasing sublevel order. + files []*FileMetadata // Interpolated from files in this interval. For files spanning multiple // intervals, we assume an equal distribution of bytes across all those @@ -197,8 +188,8 @@ type L0Sublevels struct { formatKey base.FormatKey fileBytes uint64 - // Contains all files in one slice, ordered from oldest to youngest. - filesByAge []*FileMetadata + // All the L0 files, ordered from oldest to youngest. + levelMetadata *LevelMetadata // The file intervals in increasing key order. orderedIntervals []fileInterval @@ -232,15 +223,22 @@ func insertIntoSubLevel(files []*FileMetadata, f *FileMetadata) []*FileMetadata // IsIntraL0Compacting. Those fields are accessed in InitCompactingFileInfo // instead. func NewL0Sublevels( - files []*FileMetadata, cmp Compare, formatKey base.FormatKey, flushSplitMaxBytes int64, + levelMetadata *LevelMetadata, + cmp Compare, + formatKey base.FormatKey, + flushSplitMaxBytes int64, ) (*L0Sublevels, error) { s := &L0Sublevels{cmp: cmp, formatKey: formatKey} - s.filesByAge = files - keys := make([]intervalKey, 0, 2*len(files)) - for i := range s.filesByAge { - files[i].l0Index = i - keys = append(keys, intervalKey{key: files[i].Smallest.UserKey}) - keys = append(keys, intervalKey{key: files[i].Largest.UserKey, isLargest: true}) + s.levelMetadata = levelMetadata + keys := make([]intervalKey, 0, 2*s.levelMetadata.Len()) + iter := levelMetadata.Iter() + for i, f := 0, iter.First(); f != nil; i, f = i+1, iter.Next() { + f.l0Index = i + keys = append(keys, intervalKey{key: f.Smallest.UserKey}) + keys = append(keys, intervalKey{ + key: f.Largest.UserKey, + isLargest: f.Largest.Trailer != base.InternalKeyRangeDeleteSentinel, + }) } keys = sortAndDedup(keys, cmp) // All interval indices reference s.orderedIntervals. @@ -255,7 +253,7 @@ func NewL0Sublevels( } // Initialize minIntervalIndex and maxIntervalIndex for each file, and use that // to update intervals. - for fileIndex, f := range s.filesByAge { + for f := iter.First(); f != nil; f = iter.Next() { // Set f.minIntervalIndex and f.maxIntervalIndex. f.minIntervalIndex = sort.Search(len(keys), func(index int) bool { return intervalKeyCompare(cmp, intervalKey{key: f.Smallest.UserKey}, keys[index]) <= 0 @@ -265,7 +263,7 @@ func NewL0Sublevels( } f.maxIntervalIndex = sort.Search(len(keys), func(index int) bool { return intervalKeyCompare( - cmp, intervalKey{key: f.Largest.UserKey, isLargest: true}, keys[index]) <= 0 + cmp, intervalKey{key: f.Largest.UserKey, isLargest: f.Largest.Trailer != base.InternalKeyRangeDeleteSentinel}, keys[index]) <= 0 }) if f.maxIntervalIndex == len(keys) { return nil, errors.Errorf("expected sstable bound to be in interval keys: %s", f.Largest.UserKey) @@ -283,8 +281,8 @@ func NewL0Sublevels( for i := f.minIntervalIndex; i <= f.maxIntervalIndex; i++ { interval := &s.orderedIntervals[i] if len(interval.files) > 0 && - subLevel <= s.filesByAge[interval.files[len(interval.files)-1]].subLevel { - subLevel = s.filesByAge[interval.files[len(interval.files)-1]].subLevel + 1 + subLevel <= interval.files[len(interval.files)-1].subLevel { + subLevel = interval.files[len(interval.files)-1].subLevel + 1 } s.orderedIntervals[i].fileCount++ interval.estimatedBytes += interpolatedBytes @@ -297,7 +295,7 @@ func NewL0Sublevels( } for i := f.minIntervalIndex; i <= f.maxIntervalIndex; i++ { interval := &s.orderedIntervals[i] - interval.files = append(interval.files, fileIndex) + interval.files = append(interval.files, f) } f.subLevel = subLevel if subLevel > len(s.Levels) { @@ -310,6 +308,9 @@ func NewL0Sublevels( } } var cumulativeBytes uint64 + // Multiply flushSplitMaxBytes by the number of sublevels. This prevents + // excessive flush splitting when the number of sublevels increases. + flushSplitMaxBytes *= int64(len(s.Levels)) for i := 0; i < len(s.orderedIntervals); i++ { interval := &s.orderedIntervals[i] if flushSplitMaxBytes > 0 && cumulativeBytes > uint64(flushSplitMaxBytes) && @@ -323,17 +324,26 @@ func NewL0Sublevels( return s, nil } +// L0Compaction describes an active compaction with inputs from L0. +type L0Compaction struct { + Smallest InternalKey + Largest InternalKey + IsIntraL0 bool +} + // InitCompactingFileInfo initializes internal flags relating to compacting // files. Must be called after sublevel initialization. // // Requires DB.mu to be held. -func (s *L0Sublevels) InitCompactingFileInfo() { +func (s *L0Sublevels) InitCompactingFileInfo(inProgress []L0Compaction) { for i := range s.orderedIntervals { s.orderedIntervals[i].compactingFileCount = 0 s.orderedIntervals[i].isBaseCompacting = false s.orderedIntervals[i].intervalRangeIsBaseCompacting = false } - for _, f := range s.filesByAge { + + iter := s.levelMetadata.Iter() + for f := iter.First(); f != nil; f = iter.Next() { if !f.Compacting { continue } @@ -347,6 +357,29 @@ func (s *L0Sublevels) InitCompactingFileInfo() { } } } + + // Some intervals may be base compacting without the files contained + // within those intervals being marked as compacting. This is possible if + // the files were added after the compaction initiated, and the active + // compaction files straddle the input file. Mark these intervals as base + // compacting. + for _, c := range inProgress { + startIK := intervalKey{key: c.Smallest.UserKey, isLargest: false} + endIK := intervalKey{key: c.Largest.UserKey, isLargest: c.Largest.Trailer != base.InternalKeyRangeDeleteSentinel} + start := sort.Search(len(s.orderedIntervals), func(i int) bool { + return intervalKeyCompare(s.cmp, s.orderedIntervals[i].startKey, startIK) >= 0 + }) + end := sort.Search(len(s.orderedIntervals), func(i int) bool { + return intervalKeyCompare(s.cmp, s.orderedIntervals[i].startKey, endIK) >= 0 + }) + for i := start; i < end && i < len(s.orderedIntervals); i++ { + interval := &s.orderedIntervals[i] + if !c.IsIntraL0 { + interval.isBaseCompacting = true + } + } + } + min := 0 for i := range s.orderedIntervals { interval := &s.orderedIntervals[i] @@ -372,7 +405,7 @@ func (s *L0Sublevels) String() string { func (s *L0Sublevels) describe(verbose bool) string { var buf strings.Builder fmt.Fprintf(&buf, "file count: %d, sublevels: %d, intervals: %d\nflush split keys(%d): [", - len(s.filesByAge), len(s.Levels), len(s.orderedIntervals), len(s.flushSplitUserKeys)) + s.levelMetadata.Len(), len(s.Levels), len(s.orderedIntervals), len(s.flushSplitUserKeys)) for i := range s.flushSplitUserKeys { fmt.Fprintf(&buf, "%s", s.formatKey(s.flushSplitUserKeys[i])) if i < len(s.flushSplitUserKeys)-1 { @@ -404,7 +437,7 @@ func (s *L0Sublevels) describe(verbose bool) string { if verbose { fmt.Fprintf(&buf, "\t%s\n", f) } - if len(s.filesByAge) > 50 && intervals*3 > len(s.orderedIntervals) { + if s.levelMetadata.Len() > 50 && intervals*3 > len(s.orderedIntervals) { var intervalsBytes uint64 for k := f.minIntervalIndex; k <= f.maxIntervalIndex; k++ { intervalsBytes += s.orderedIntervals[k].estimatedBytes @@ -499,7 +532,7 @@ func (s *L0Sublevels) MaxDepthAfterOngoingCompactions() int { // TODO(bilal): Simplify away the debugging statements in this method, and make // this a pure sanity checker. func (s *L0Sublevels) checkCompaction(c *L0CompactionFiles) error { - includedFiles := newBitSet(len(s.filesByAge)) + includedFiles := newBitSet(s.levelMetadata.Len()) fileIntervalsByLevel := make([]struct { min int max int @@ -595,15 +628,15 @@ func (s *L0Sublevels) checkCompaction(c *L0CompactionFiles) error { // compaction must be involving L0 SSTables. It's assumed that the Compacting // and IsIntraL0Compacting fields are already set on all FileMetadatas passed // in. -func (s *L0Sublevels) UpdateStateForStartedCompaction(inputs [][]*FileMetadata, isBase bool) error { +func (s *L0Sublevels) UpdateStateForStartedCompaction(inputs []LevelSlice, isBase bool) error { minIntervalIndex := -1 maxIntervalIndex := 0 for i := range inputs { - for _, f := range inputs[i] { + iter := inputs[i].Iter() + for f := iter.First(); f != nil; f = iter.Next() { for i := f.minIntervalIndex; i <= f.maxIntervalIndex; i++ { interval := &s.orderedIntervals[i] interval.compactingFileCount++ - interval.isBaseCompacting = isBase } if f.minIntervalIndex < minIntervalIndex || minIntervalIndex == -1 { minIntervalIndex = f.minIntervalIndex @@ -616,6 +649,7 @@ func (s *L0Sublevels) UpdateStateForStartedCompaction(inputs [][]*FileMetadata, if isBase { for i := minIntervalIndex; i <= maxIntervalIndex; i++ { interval := &s.orderedIntervals[i] + interval.isBaseCompacting = isBase for j := interval.filesMinIntervalIndex; j <= interval.filesMaxIntervalIndex; j++ { s.orderedIntervals[j].intervalRangeIsBaseCompacting = true } @@ -663,7 +697,7 @@ type L0CompactionFiles struct { filesAdded []*FileMetadata } -// Adds the specified file to the LCF. +// addFile adds the specified file to the LCF. func (l *L0CompactionFiles) addFile(f *FileMetadata) { if l.FilesIncluded[f.l0Index] { return @@ -836,7 +870,7 @@ func (is intervalSorterByDecreasingScore) Swap(i, j int) { // files that can be selected for compaction. Returns nil if no compaction is // possible. func (s *L0Sublevels) PickBaseCompaction( - minCompactionDepth int, baseFiles []*FileMetadata, + minCompactionDepth int, baseFiles LevelSlice, ) (*L0CompactionFiles, error) { // For LBase compactions, we consider intervals in a greedy manner in the // following order: @@ -880,7 +914,7 @@ func (s *L0Sublevels) PickBaseCompaction( // Pick the seed file for the interval as the file // in the lowest sub-level. - f := s.filesByAge[interval.files[0]] + f := interval.files[0] // Don't bother considering the intervals that are // covered by the seed file since they are likely // nearby. Note that it is possible that those intervals @@ -901,7 +935,7 @@ func (s *L0Sublevels) PickBaseCompaction( // compacting. Usually means the score is not accurately // accounting for files already compacting, or internal state is // inconsistent. - return nil, errors.Errorf("file %d chosen as seed file for compaction should not be compacting", f.FileNum) + return nil, errors.Errorf("file %s chosen as seed file for compaction should not be compacting", f.FileNum) } c := s.baseCompactionUsingSeed(f, interval.index, minCompactionDepth) @@ -909,27 +943,20 @@ func (s *L0Sublevels) PickBaseCompaction( // Check if the chosen compaction overlaps with any files // in Lbase that have Compacting = true. If that's the case, // this compaction cannot be chosen. - firstBaseIndex := sort.Search(len(baseFiles), func(i int) bool { - // An interval starting at ImmediateSuccessor(key) can never be the - // first interval of a compaction since no file can start at that - // interval. - return s.cmp(baseFiles[i].Largest.UserKey, s.orderedIntervals[c.minIntervalIndex].startKey.key) >= 0 - }) - // Exclusive - lastBaseIndex := sort.Search(len(baseFiles), func(i int) bool { - cmp := s.cmp(baseFiles[i].Smallest.UserKey, s.orderedIntervals[c.maxIntervalIndex+1].startKey.key) + baseIter := baseFiles.Iter() + // An interval starting at ImmediateSuccessor(key) can never be the + // first interval of a compaction since no file can start at that + // interval. + m := baseIter.SeekGE(s.cmp, s.orderedIntervals[c.minIntervalIndex].startKey.key) + + var baseCompacting bool + for ; m != nil && !baseCompacting; m = baseIter.Next() { + cmp := s.cmp(m.Smallest.UserKey, s.orderedIntervals[c.maxIntervalIndex+1].startKey.key) // Compaction is ending at exclusive bound of c.maxIntervalIndex+1 if cmp > 0 || (cmp == 0 && !s.orderedIntervals[c.maxIntervalIndex+1].startKey.isLargest) { - return true - } - return false - }) - baseCompacting := false - for j := firstBaseIndex; j < lastBaseIndex; j++ { - if baseFiles[j].Compacting { - baseCompacting = true break } + baseCompacting = baseCompacting || m.Compacting } if baseCompacting { continue @@ -946,7 +973,7 @@ func (s *L0Sublevels) baseCompactionUsingSeed( f *FileMetadata, intervalIndex int, minCompactionDepth int, ) *L0CompactionFiles { c := &L0CompactionFiles{ - FilesIncluded: newBitSet(len(s.filesByAge)), + FilesIncluded: newBitSet(s.levelMetadata.Len()), seedInterval: intervalIndex, seedIntervalMinLevel: 0, minIntervalIndex: f.minIntervalIndex, @@ -962,7 +989,7 @@ func (s *L0Sublevels) baseCompactionUsingSeed( var lastCandidate *L0CompactionFiles interval := &s.orderedIntervals[intervalIndex] for i := 0; i < len(interval.files); i++ { - f2 := s.filesByAge[interval.files[i]] + f2 := interval.files[i] sl := f2.subLevel c.seedIntervalStackDepthReduction++ c.seedIntervalMaxLevel = sl @@ -1099,7 +1126,7 @@ func (s *L0Sublevels) PickIntraL0Compaction( // in the highest sub-level. stackDepthReduction := scoredInterval.score for i := len(interval.files) - 1; i >= 0; i-- { - f = s.filesByAge[interval.files[i]] + f = interval.files[i] if f.Compacting { break } @@ -1148,7 +1175,7 @@ func (s *L0Sublevels) intraL0CompactionUsingSeed( // we need to exclude files >= earliestUnflushedSeqNum c := &L0CompactionFiles{ - FilesIncluded: newBitSet(len(s.filesByAge)), + FilesIncluded: newBitSet(s.levelMetadata.Len()), seedInterval: intervalIndex, seedIntervalMaxLevel: len(s.Levels) - 1, minIntervalIndex: f.minIntervalIndex, @@ -1162,7 +1189,7 @@ func (s *L0Sublevels) intraL0CompactionUsingSeed( interval := &s.orderedIntervals[intervalIndex] slIndex := len(interval.files) - 1 for { - if interval.files[slIndex] == f.l0Index { + if interval.files[slIndex] == f { break } slIndex-- @@ -1174,7 +1201,7 @@ func (s *L0Sublevels) intraL0CompactionUsingSeed( // successful candidate. The code stops adding when it can't add more, or // when fileBytes grows too large. for ; slIndex >= 0; slIndex-- { - f2 := s.filesByAge[interval.files[slIndex]] + f2 := interval.files[slIndex] sl := f2.subLevel if f2.Compacting { break @@ -1234,33 +1261,49 @@ func (s *L0Sublevels) intraL0CompactionUsingSeed( } // ExtendL0ForBaseCompactionTo extends the specified base compaction candidate -// L0CompactionFiles to cover all L0 files in the specified key interval, -// by calling extendCandidateToRectangle. +// L0CompactionFiles to optionally cover more files in L0 without "touching" +// any of the passed-in keys (i.e. the smallest/largest bounds are exclusive), +// as including any user keys for those internal keys +// could require choosing more files in LBase which is undesirable. Unbounded +// start/end keys are indicated by passing in the InvalidInternalKey. func (s *L0Sublevels) ExtendL0ForBaseCompactionTo( - smallest []byte, largest []byte, candidate *L0CompactionFiles, + smallest, largest InternalKey, candidate *L0CompactionFiles, ) bool { - firstIntervalIndex := sort.Search(len(s.orderedIntervals), func(i int) bool { - // Need to start at >= smallest since if we widen too much we may miss - // an Lbase file that overlaps with an L0 file that will get picked in - // this widening, which would be bad. This interval will not start with - // an immediate successor key. - return s.cmp(smallest, s.orderedIntervals[i].startKey.key) <= 0 - }) - // First interval that starts at or beyond the largest. This interval will not - // start with an immediate successor key. - lastIntervalIndex := sort.Search(len(s.orderedIntervals), func(i int) bool { - return s.cmp(largest, s.orderedIntervals[i].startKey.key) < 0 - }) - // Right now, lastIntervalIndex has a start that's higher than largest. - // The previous interval, by definition, has an end key higher than largest. - // Iterate back twice to get the last interval that's completely within - // [smallest, largest]. Except in the case where we went past the end of the - // list; in that case, the last interval to include is the very last - // interval in the list. - if lastIntervalIndex < len(s.orderedIntervals) { + firstIntervalIndex := 0 + lastIntervalIndex := len(s.orderedIntervals) - 1 + if smallest.Kind() != base.InternalKeyKindInvalid { + if smallest.Trailer == base.InternalKeyRangeDeleteSentinel { + // Starting at smallest.UserKey == interval.startKey is okay. + firstIntervalIndex = sort.Search(len(s.orderedIntervals), func(i int) bool { + return s.cmp(smallest.UserKey, s.orderedIntervals[i].startKey.key) <= 0 + }) + } else { + firstIntervalIndex = sort.Search(len(s.orderedIntervals), func(i int) bool { + // Need to start at >= smallest since if we widen too much we may miss + // an Lbase file that overlaps with an L0 file that will get picked in + // this widening, which would be bad. This interval will not start with + // an immediate successor key. + return s.cmp(smallest.UserKey, s.orderedIntervals[i].startKey.key) < 0 + }) + } + } + if largest.Kind() != base.InternalKeyKindInvalid { + // First interval that starts at or beyond the largest. This interval will not + // start with an immediate successor key. + lastIntervalIndex = sort.Search(len(s.orderedIntervals), func(i int) bool { + return s.cmp(largest.UserKey, s.orderedIntervals[i].startKey.key) <= 0 + }) + // Right now, lastIntervalIndex has a startKey that extends beyond largest. + // The previous interval, by definition, has an end key higher than largest. + // Iterate back twice to get the last interval that's completely within + // (smallest, largest). Except in the case where we went past the end of the + // list; in that case, the last interval to include is the very last + // interval in the list. + if lastIntervalIndex < len(s.orderedIntervals) { + lastIntervalIndex-- + } lastIntervalIndex-- } - lastIntervalIndex-- if lastIntervalIndex < firstIntervalIndex { return false } @@ -1290,10 +1333,10 @@ func (s *L0Sublevels) ExtendL0ForBaseCompactionTo( // it's in the bounds, then add b-d, then a--d, and so on, to produce this: // // _____________ -// L0.3 |a--d g-j| _________ -// L0.2 | f--j| | r-t | -// L0.1 | b-d e---j| | | -// L0.0 |a--d f--j| l--o |p-----x| +// L0.3 |a--d g-j| +// L0.2 | f--j| r-t +// L0.1 | b-d e---j| +// L0.0 |a--d f--j| l--o p-----x // // Lbase a-------i m---------w // diff --git a/github.com/cockroachdb/pebble/internal/manifest/level_metadata.go b/github.com/cockroachdb/pebble/internal/manifest/level_metadata.go new file mode 100644 index 0000000000..246fbc60b0 --- /dev/null +++ b/github.com/cockroachdb/pebble/internal/manifest/level_metadata.go @@ -0,0 +1,232 @@ +// Copyright 2020 The LevelDB-Go and Pebble Authors. All rights reserved. Use +// of this source code is governed by a BSD-style license that can be found in +// the LICENSE file. + +package manifest + +import "sort" + +// LevelMetadata contains metadata for all of the files within +// a level of the LSM. +type LevelMetadata struct { + files []*FileMetadata +} + +// Empty indicates whether there are any files in the level. +func (lm *LevelMetadata) Empty() bool { + return lm.Len() == 0 +} + +// Len returns the number of files within the level. +func (lm *LevelMetadata) Len() int { + return len(lm.files) +} + +// Iter constructs a LevelIterator over the entire level. +func (lm *LevelMetadata) Iter() LevelIterator { + return LevelIterator{files: lm.files, end: len(lm.files)} +} + +// Slice constructs a slice containing the entire level. +func (lm *LevelMetadata) Slice() LevelSlice { + return LevelSlice{files: lm.files, end: len(lm.files)} +} + +// LevelFile holds a file's metadata along with its position +// within a level of the LSM. +type LevelFile struct { + *FileMetadata + slice LevelSlice +} + +// Slice constructs a LevelSlice containing only this file. +func (lf LevelFile) Slice() LevelSlice { + return lf.slice +} + +// NewLevelSlice constructs a LevelSlice over the provided files. This +// function is expected to be a temporary adapter between interfaces. +// TODO(jackson): Revisit once the conversion of Version.Files to a btree is +// complete. +func NewLevelSlice(files []*FileMetadata) LevelSlice { + return LevelSlice{files: files, start: 0, end: len(files)} +} + +// LevelSlice contains a slice of the files within a level of the LSM. +type LevelSlice struct { + files []*FileMetadata + start int + end int +} + +// Each invokes fn for each element in the slice. +func (ls LevelSlice) Each(fn func(*FileMetadata)) { + iter := ls.Iter() + for f := iter.First(); f != nil; f = iter.Next() { + fn(f) + } +} + +// Empty indicates whether the slice contains any files. +func (ls LevelSlice) Empty() bool { + return ls.start >= ls.end +} + +// Iter constructs a LevelIterator that iterates over the slice. +func (ls LevelSlice) Iter() LevelIterator { + return LevelIterator{ + files: ls.files, + start: ls.start, + end: ls.end, + } +} + +// Len returns the number of files in the slice. +func (ls LevelSlice) Len() int { + return ls.end - ls.start +} + +// SizeSum sums the size of all files in the slice. +func (ls LevelSlice) SizeSum() uint64 { + var sum uint64 + for _, f := range ls.files[ls.start:ls.end] { + sum += f.Size + } + return sum +} + +// Reslice constructs a new slice backed by the same underlying level, with +// new start and end positions. Reslice invokes the provided function, passing +// two LevelIterators: one positioned to i's inclusive start and one +// positioned to i's inclusive end. The resliceFunc may move either iterator +// forward or backwards, including beyond the callee's original bounds to +// capture additional files from the underlying level. Reslice constructs and +// returns a new LevelSlice with the final bounds of the iterators after +// calling resliceFunc. +func (ls LevelSlice) Reslice(resliceFunc func(start, end *LevelIterator)) LevelSlice { + start := LevelIterator{ + files: ls.files, + cur: ls.start, + start: 0, + end: len(ls.files), + } + end := LevelIterator{ + files: ls.files, + cur: ls.end - 1, + start: 0, + end: len(ls.files), + } + resliceFunc(&start, &end) + return LevelSlice{ + files: ls.files, + start: start.cur, + end: end.cur + 1, + } +} + +// LevelIterator iterates over a set of files' metadata. Its zero value is an +// empty iterator. +type LevelIterator struct { + files []*FileMetadata + cur int + start int + end int +} + +// Clone copies the iterator, returning an independent iterator at the same +// position. +func (i *LevelIterator) Clone() LevelIterator { + return *i +} + +// Current returns the item at the current iterator position. +func (i LevelIterator) Current() *FileMetadata { + if i.cur < i.start || i.cur >= i.end { + return nil + } + return i.files[i.cur] +} + +// First seeks to the first file in the iterator and returns it. +func (i *LevelIterator) First() *FileMetadata { + i.cur = i.start + if i.cur < i.start || i.cur >= i.end { + return nil + } + return i.files[i.cur] +} + +// Last seeks to the last file in the iterator and returns it. +func (i *LevelIterator) Last() *FileMetadata { + i.cur = i.end - 1 + if i.cur < i.start || i.cur >= i.end { + return nil + } + return i.files[i.cur] +} + +// Next advances the iterator to the next file and returns it. +func (i *LevelIterator) Next() *FileMetadata { + i.cur++ + if i.cur < i.start || i.cur >= i.end { + return nil + } + return i.files[i.cur] +} + +// Prev moves the iterator the previous file and returns it. +func (i *LevelIterator) Prev() *FileMetadata { + i.cur-- + if i.cur < i.start || i.cur >= i.end { + return nil + } + return i.files[i.cur] +} + +// SeekGE seeks to the first file in the iterator's file set with a largest +// user key less than or equal to the provided user key. The iterator must +// have been constructed from L1+, because it requires the underlying files to +// be sorted by user keys and non-overlapping. +func (i *LevelIterator) SeekGE(cmp Compare, userKey []byte) *FileMetadata { + files := i.files[i.start:i.end] + i.cur = i.start + sort.Search(len(files), func(j int) bool { + return cmp(userKey, files[j].Largest.UserKey) <= 0 + }) + if i.cur >= i.end { + return nil + } + return i.files[i.cur] +} + +// SeekLT seeks to the last file in the iterator's file set with a smallest +// user key less than the provided user key. The iterator must have been +// constructed from L1+, because it requries the underlying files to be sorted +// by user keys and non-overlapping. +func (i *LevelIterator) SeekLT(cmp Compare, userKey []byte) *FileMetadata { + files := i.files[i.start:i.end] + i.cur = i.start + sort.Search(len(files), func(j int) bool { + return cmp(files[j].Smallest.UserKey, userKey) >= 0 + }) + if i.cur < i.start { + return nil + } + return i.Prev() +} + +// Take constructs a LevelFile containing the file at the iterator's current +// position. Take panics if the iterator is not currently positioned over a +// file. +func (i LevelIterator) Take() LevelFile { + m := i.Current() + if m == nil { + panic("Take called on invalid LevelIterator") + } + return LevelFile{ + FileMetadata: m, + slice: LevelSlice{ + files: i.files, + start: i.cur, + end: i.cur + 1, + }, + } +} diff --git a/github.com/cockroachdb/pebble/internal/manifest/version.go b/github.com/cockroachdb/pebble/internal/manifest/version.go index c67ecc3c52..ee17bbe71f 100644 --- a/github.com/cockroachdb/pebble/internal/manifest/version.go +++ b/github.com/cockroachdb/pebble/internal/manifest/version.go @@ -43,12 +43,20 @@ type TableStats struct { // Valid true if stats have been loaded for the table. The rest of the // structure is populated only if true. Valid bool - // Estimate of the total disk space that may be reclaimed by compacting - // this table's range deletions to the bottom of the LSM. This estimate is - // at data-block granularity and is not updated if compactions beneath the - // table reduce the amount of reclaimable disk space. It also does not - // account for overlapping data in L0 and ignores L0 sublevels, but the - // error that introduces is expected to be small. + // The total number of entries in the table. + NumEntries uint64 + // The number of point and range deletion entries in the table. + NumDeletions uint64 + // Estimate of the total disk space that may be dropped by this table's + // range deletions by compacting them. This estimate is at data-block + // granularity and is not updated if compactions beneath the table reduce + // the amount of reclaimable disk space. It also does not account for + // overlapping data in L0 and ignores L0 sublevels, but the error that + // introduces is expected to be small. + // + // Tables in the bottommost level of the LSM may have a nonzero estimate + // if snapshots or move compactions prevented the elision of their range + // tombstones. RangeDeletionsBytesEstimate uint64 } @@ -93,10 +101,25 @@ type FileMetadata struct { maxIntervalIndex int } -func (m FileMetadata) String() string { +func (m *FileMetadata) String() string { return fmt.Sprintf("%s:%s-%s", m.FileNum, m.Smallest, m.Largest) } +// Validate validates the metadata for consistency with itself, returning an +// error if inconsistent. +func (m *FileMetadata) Validate(cmp Compare, formatKey base.FormatKey) error { + if base.InternalCompare(cmp, m.Smallest, m.Largest) > 0 { + return base.CorruptionErrorf("file %s has inconsistent bounds: %s vs %s", + errors.Safe(m.FileNum), m.Smallest.Pretty(formatKey), + m.Largest.Pretty(formatKey)) + } + if m.SmallestSeqNum > m.LargestSeqNum { + return base.CorruptionErrorf("file %s has inconsistent seqnum bounds: %d vs %d", + errors.Safe(m.FileNum), m.SmallestSeqNum, m.LargestSeqNum) + } + return nil +} + // TableInfo returns a subset of the FileMetadata state formatted as a // TableInfo. func (m *FileMetadata) TableInfo() TableInfo { @@ -130,20 +153,20 @@ func (m *FileMetadata) lessSmallestKey(b *FileMetadata, cmp Compare) bool { } // KeyRange returns the minimum smallest and maximum largest internalKey for -// all the fileMetadata in f0 and f1. -func KeyRange(ucmp Compare, f0, f1 []*FileMetadata) (smallest, largest InternalKey) { +// all the FileMetadata in iters. +func KeyRange(ucmp Compare, iters ...LevelIterator) (smallest, largest InternalKey) { first := true - for _, f := range [2][]*FileMetadata{f0, f1} { - for _, meta := range f { + for _, iter := range iters { + for meta := iter.First(); meta != nil; meta = iter.Next() { if first { first = false smallest, largest = meta.Smallest, meta.Largest continue } - if base.InternalCompare(ucmp, meta.Smallest, smallest) < 0 { + if base.InternalCompare(ucmp, smallest, meta.Smallest) >= 0 { smallest = meta.Smallest } - if base.InternalCompare(ucmp, meta.Largest, largest) > 0 { + if base.InternalCompare(ucmp, largest, meta.Largest) <= 0 { largest = meta.Largest } } @@ -196,6 +219,24 @@ func overlaps(files []*FileMetadata, cmp Compare, start, end []byte) (lower, upp // NumLevels is the number of levels a Version contains. const NumLevels = 7 +// NewVersion constructs a new Version with the provided files. It assumes +// the provided files are already well-ordered. It's intended for testing. +func NewVersion( + cmp Compare, + formatKey base.FormatKey, + flushSplitBytes int64, + files [NumLevels][]*FileMetadata, +) *Version { + var v Version + for i := range files { + v.Levels[i].files = files[i] + } + if err := v.InitL0Sublevels(cmp, formatKey, flushSplitBytes); err != nil { + panic(err) + } + return &v +} + // Version is a collection of file metadata for on-disk tables at various // levels. In-memory DBs are written to level-0 tables, and compactions // migrate data from level N to level N+1. The tables map internal keys (which @@ -235,7 +276,7 @@ type Version struct { // in Files[0] are in L0Sublevels.Levels. L0Sublevels *L0Sublevels - Levels [NumLevels][]*FileMetadata + Levels [NumLevels]LevelMetadata // The callback to invoke when the last reference to a version is // removed. Will be called with list.mu held. @@ -255,25 +296,22 @@ func (v *Version) String() string { // Pretty returns a string representation of the version. func (v *Version) Pretty(format base.FormatKey) string { var buf bytes.Buffer - for level := 0; level < NumLevels; level++ { - if len(v.Levels[level]) == 0 { - continue - } - - if level == 0 { - for sublevel := len(v.L0Sublevels.Levels) - 1; sublevel >= 0; sublevel-- { - fmt.Fprintf(&buf, "0.%d:\n", sublevel) - for _, f := range v.L0Sublevels.Levels[sublevel] { - fmt.Fprintf(&buf, " %06d:[%s-%s]\n", f.FileNum, - format(f.Smallest.UserKey), format(f.Largest.UserKey)) - } + if v.L0Sublevels != nil { + for sublevel := len(v.L0Sublevels.Levels) - 1; sublevel >= 0; sublevel-- { + fmt.Fprintf(&buf, "0.%d:\n", sublevel) + for _, f := range v.L0Sublevels.Levels[sublevel] { + fmt.Fprintf(&buf, " %06d:[%s-%s]\n", f.FileNum, + format(f.Smallest.UserKey), format(f.Largest.UserKey)) } + } + } + for level := 1; level < NumLevels; level++ { + if v.Levels[level].Empty() { continue } - fmt.Fprintf(&buf, "%d:\n", level) - for j := range v.Levels[level] { - f := v.Levels[level][j] + iter := v.Levels[level].Iter() + for f := iter.First(); f != nil; f = iter.Next() { fmt.Fprintf(&buf, " %s:[%s-%s]\n", f.FileNum, format(f.Smallest.UserKey), format(f.Largest.UserKey)) } @@ -285,25 +323,23 @@ func (v *Version) Pretty(format base.FormatKey) string { // sequence number and kind information for the sstable boundaries. func (v *Version) DebugString(format base.FormatKey) string { var buf bytes.Buffer - for level := 0; level < NumLevels; level++ { - if len(v.Levels[level]) == 0 { - continue - } - if level == 0 { - for sublevel := len(v.L0Sublevels.Levels) - 1; sublevel >= 0; sublevel-- { - fmt.Fprintf(&buf, "0.%d:\n", sublevel) - for _, f := range v.L0Sublevels.Levels[sublevel] { - fmt.Fprintf(&buf, " %06d:[%s-%s]\n", f.FileNum, - f.Smallest.Pretty(format), f.Largest.Pretty(format)) - } + if v.L0Sublevels != nil { + for sublevel := len(v.L0Sublevels.Levels) - 1; sublevel >= 0; sublevel-- { + fmt.Fprintf(&buf, "0.%d:\n", sublevel) + for _, f := range v.L0Sublevels.Levels[sublevel] { + fmt.Fprintf(&buf, " %06d:[%s-%s]\n", f.FileNum, + f.Smallest.Pretty(format), f.Largest.Pretty(format)) } + } + } + for level := 1; level < NumLevels; level++ { + if v.Levels[level].Empty() { continue } - fmt.Fprintf(&buf, "%d:\n", level) - for j := range v.Levels[level] { - f := v.Levels[level][j] + iter := v.Levels[level].Iter() + for f := iter.First(); f != nil; f = iter.Next() { fmt.Fprintf(&buf, " %s:[%s-%s]\n", f.FileNum, f.Smallest.Pretty(format), f.Largest.Pretty(format)) } @@ -348,10 +384,12 @@ func (v *Version) UnrefLocked() { } func (v *Version) unrefFiles() []base.FileNum { + // TODO(jackson): Move responsibility of ref-ing of individual files into + // the LevelMetadata type. var obsolete []base.FileNum for _, files := range v.Levels { - for i := range files { - f := files[i] + iter := files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { if atomic.AddInt32(&f.refs, -1) == 0 { obsolete = append(obsolete, f.FileNum) } @@ -370,7 +408,7 @@ func (v *Version) InitL0Sublevels( cmp Compare, formatKey base.FormatKey, flushSplitBytes int64, ) error { var err error - v.L0Sublevels, err = NewL0Sublevels(v.Levels[0], cmp, formatKey, flushSplitBytes) + v.L0Sublevels, err = NewL0Sublevels(&v.Levels[0], cmp, formatKey, flushSplitBytes) return err } @@ -379,12 +417,11 @@ func (v *Version) InitL0Sublevels( // searches among the files. If level is zero, Contains scans the entire // level. func (v *Version) Contains(level int, cmp Compare, m *FileMetadata) bool { - files := v.Levels[level] + iter := v.Levels[level].Slice().Iter() if level > 0 { - files = v.Overlaps(level, cmp, m.Smallest.UserKey, m.Largest.UserKey) + iter = v.Overlaps(level, cmp, m.Smallest.UserKey, m.Largest.UserKey).Iter() } - - for _, f := range files { + for f := iter.First(); f != nil; f = iter.Next() { if f == m { return true } @@ -400,18 +437,21 @@ func (v *Version) Contains(level int, cmp Compare, m *FileMetadata) bool { // and the computation is repeated until [start, end] stabilizes. // The returned files are a subsequence of the input files, i.e., the ordering // is not changed. -func (v *Version) Overlaps(level int, cmp Compare, start, end []byte) (ret []*FileMetadata) { +func (v *Version) Overlaps(level int, cmp Compare, start, end []byte) LevelSlice { if level == 0 { // Indices that have been selected as overlapping. - selectedIndices := make([]bool, len(v.Levels[level])) + l0 := v.Levels[level].Slice() + l0Iter := l0.Iter() + selectedIndices := make([]bool, l0.Len()) numSelected := 0 + var slice LevelSlice for { restart := false - for i, selected := range selectedIndices { + for i, meta := 0, l0Iter.First(); meta != nil; i, meta = i+1, l0Iter.Next() { + selected := selectedIndices[i] if selected { continue } - meta := v.Levels[level][i] smallest := meta.Smallest.UserKey largest := meta.Largest.UserKey if cmp(largest, start) < 0 { @@ -442,25 +482,28 @@ func (v *Version) Overlaps(level int, cmp Compare, start, end []byte) (ret []*Fi } if !restart { - ret = make([]*FileMetadata, 0, numSelected) - for i, selected := range selectedIndices { - if selected { - ret = append(ret, v.Levels[level][i]) + slice.files = make([]*FileMetadata, 0, numSelected) + for i, meta := 0, l0Iter.First(); meta != nil; i, meta = i+1, l0Iter.Next() { + if selectedIndices[i] { + slice.files = append(slice.files, meta) } } + slice.end = len(slice.files) break } // Continue looping to retry the files that were not selected. } - return + return slice } - files := v.Levels[level] - lower, upper := overlaps(files, cmp, start, end) - if lower >= upper { - return nil + var slice LevelSlice + lower, upper := overlaps(v.Levels[level].files, cmp, start, end) + if lower < upper { + slice.files = v.Levels[level].files + slice.start = lower + slice.end = upper } - return files[lower:upper] + return slice } // CheckOrdering checks that the files are consistent with respect to @@ -468,14 +511,15 @@ func (v *Version) Overlaps(level int, cmp Compare, start, end []byte) (ret []*Fi // overlapping internal key ranges (for level non-0 files). func (v *Version) CheckOrdering(cmp Compare, format base.FormatKey) error { for sublevel := len(v.L0Sublevels.Levels) - 1; sublevel >= 0; sublevel-- { - if err := CheckOrdering(cmp, format, L0Sublevel(sublevel), v.L0Sublevels.Levels[sublevel]); err != nil { - return errors.Errorf("%s\n%s", err, v.DebugString(format)) + iter := NewLevelSlice(v.L0Sublevels.Levels[sublevel]).Iter() + if err := CheckOrdering(cmp, format, L0Sublevel(sublevel), iter); err != nil { + return base.CorruptionErrorf("%s\n%s", err, v.DebugString(format)) } } - for level, files := range v.Levels { - if err := CheckOrdering(cmp, format, Level(level), files); err != nil { - return errors.Errorf("%s\n%s", err, v.DebugString(format)) + for level, lm := range v.Levels { + if err := CheckOrdering(cmp, format, Level(level), lm.Iter()); err != nil { + return base.CorruptionErrorf("%s\n%s", err, v.DebugString(format)) } } return nil @@ -488,7 +532,8 @@ func (v *Version) CheckConsistency(dirname string, fs vfs.FS) error { var args []interface{} for level, files := range v.Levels { - for _, f := range files { + iter := files.Iter() + for f := iter.First(); f != nil; f = iter.Next() { path := base.MakeFilename(fs, dirname, base.FileTypeTable, f.FileNum) info, err := fs.Stat(path) if err != nil { @@ -573,7 +618,7 @@ func (l *VersionList) Remove(v *Version) { // CheckOrdering checks that the files are consistent with respect to // seqnums (for level 0 files -- see detailed comment below) and increasing and non- // overlapping internal key ranges (for non-level 0 files). -func CheckOrdering(cmp Compare, format base.FormatKey, level Level, files []*FileMetadata) error { +func CheckOrdering(cmp Compare, format base.FormatKey, level Level, files LevelIterator) error { // The invariants to check for L0 sublevels are the same as the ones to // check for all other levels. However, if L0 is not organized into // sublevels, or if all L0 files are being passed in, we do the legacy L0 @@ -623,39 +668,36 @@ func CheckOrdering(cmp Compare, format base.FormatKey, level Level, files []*Fil // with future versions of pebble, this method relaxes most L0 invariant // checks. - for i := range files { - f := files[i] - if i > 0 { - // Validate that the sorting is sane. - prev := files[i-1] - if prev.LargestSeqNum == 0 && f.LargestSeqNum == prev.LargestSeqNum { - // Multiple files satisfying case 2 mentioned above. - } else if !prev.lessSeqNum(f) { - return errors.Errorf("L0 files %s and %s are not properly ordered: <#%d-#%d> vs <#%d-#%d>", - errors.Safe(prev.FileNum), errors.Safe(f.FileNum), - errors.Safe(prev.SmallestSeqNum), errors.Safe(prev.LargestSeqNum), - errors.Safe(f.SmallestSeqNum), errors.Safe(f.LargestSeqNum)) - } + var prev *FileMetadata + for f := files.First(); f != nil; f, prev = files.Next(), f { + if prev == nil { + continue + } + // Validate that the sorting is sane. + if prev.LargestSeqNum == 0 && f.LargestSeqNum == prev.LargestSeqNum { + // Multiple files satisfying case 2 mentioned above. + } else if !prev.lessSeqNum(f) { + return base.CorruptionErrorf("L0 files %s and %s are not properly ordered: <#%d-#%d> vs <#%d-#%d>", + errors.Safe(prev.FileNum), errors.Safe(f.FileNum), + errors.Safe(prev.SmallestSeqNum), errors.Safe(prev.LargestSeqNum), + errors.Safe(f.SmallestSeqNum), errors.Safe(f.LargestSeqNum)) } } } else { - for i := range files { - f := files[i] - if base.InternalCompare(cmp, f.Smallest, f.Largest) > 0 { - return errors.Errorf("%s file %s has inconsistent bounds: %s vs %s", - errors.Safe(level), errors.Safe(f.FileNum), - f.Smallest.Pretty(format), f.Largest.Pretty(format)) + var prev *FileMetadata + for f := files.First(); f != nil; f, prev = files.Next(), f { + if err := f.Validate(cmp, format); err != nil { + return errors.Wrapf(err, "%s ", level) } - if i > 0 { - prev := files[i-1] + if prev != nil { if !prev.lessSmallestKey(f, cmp) { - return errors.Errorf("%s files %s and %s are not properly ordered: [%s-%s] vs [%s-%s]", + return base.CorruptionErrorf("%s files %s and %s are not properly ordered: [%s-%s] vs [%s-%s]", errors.Safe(level), errors.Safe(prev.FileNum), errors.Safe(f.FileNum), prev.Smallest.Pretty(format), prev.Largest.Pretty(format), f.Smallest.Pretty(format), f.Largest.Pretty(format)) } if base.InternalCompare(cmp, prev.Largest, f.Smallest) >= 0 { - return errors.Errorf("%s files %s and %s have overlapping ranges: [%s-%s] vs [%s-%s]", + return base.CorruptionErrorf("%s files %s and %s have overlapping ranges: [%s-%s] vs [%s-%s]", errors.Safe(level), errors.Safe(prev.FileNum), errors.Safe(f.FileNum), prev.Smallest.Pretty(format), prev.Largest.Pretty(format), f.Smallest.Pretty(format), f.Largest.Pretty(format)) diff --git a/github.com/cockroachdb/pebble/internal/manifest/version_edit.go b/github.com/cockroachdb/pebble/internal/manifest/version_edit.go index 37b31f7cb7..3a992ec050 100644 --- a/github.com/cockroachdb/pebble/internal/manifest/version_edit.go +++ b/github.com/cockroachdb/pebble/internal/manifest/version_edit.go @@ -8,7 +8,6 @@ import ( "bufio" "bytes" "encoding/binary" - "fmt" "io" "sort" "sync/atomic" @@ -20,7 +19,7 @@ import ( // TODO(peter): describe the MANIFEST file format, independently of the C++ // project. -var errCorruptManifest = errors.New("pebble: corrupt manifest") +var errCorruptManifest = base.CorruptionErrorf("pebble: corrupt manifest") type byteReader interface { io.ByteReader @@ -233,7 +232,7 @@ func (v *VersionEdit) Decode(r io.Reader) error { switch customTag { case customTagNeedsCompaction: if len(field) != 1 { - return errors.New("new-file4: need-compaction field wrong size") + return base.CorruptionErrorf("new-file4: need-compaction field wrong size") } markedForCompaction = (field[0] == 1) @@ -241,15 +240,15 @@ func (v *VersionEdit) Decode(r io.Reader) error { var n int creationTime, n = binary.Uvarint(field) if n != len(field) { - return errors.New("new-file4: invalid file creation time") + return base.CorruptionErrorf("new-file4: invalid file creation time") } case customTagPathID: - return errors.New("new-file4: path-id field not supported") + return base.CorruptionErrorf("new-file4: path-id field not supported") default: if (customTag & customTagNonSafeIgnoreMask) != 0 { - return errors.Errorf("new-file4: custom field not supported: %d", customTag) + return base.CorruptionErrorf("new-file4: custom field not supported: %d", customTag) } } } @@ -276,7 +275,7 @@ func (v *VersionEdit) Decode(r io.Reader) error { v.ObsoletePrevLogNum = n case tagColumnFamily, tagColumnFamilyAdd, tagColumnFamilyDrop, tagMaxColumnFamily: - return errors.New("column families are not supported") + return base.CorruptionErrorf("column families are not supported") default: return errCorruptManifest @@ -436,7 +435,7 @@ type BulkVersionEdit struct { // Accumulate adds the file addition and deletions in the specified version // edit to the bulk edit's internal state. -func (b *BulkVersionEdit) Accumulate(ve *VersionEdit) { +func (b *BulkVersionEdit) Accumulate(ve *VersionEdit) error { for df := range ve.DeletedFiles { dmap := b.Deleted[df.Level] if dmap == nil { @@ -451,11 +450,12 @@ func (b *BulkVersionEdit) Accumulate(ve *VersionEdit) { // VersionEdit at the same level (though files can move across levels). if dmap := b.Deleted[nf.Level]; dmap != nil { if _, ok := dmap[nf.Meta.FileNum]; ok { - panic(fmt.Sprintf("file deleted %d before it was inserted\n", nf.Meta.FileNum)) + return base.CorruptionErrorf("file deleted %d before it was inserted\n", nf.Meta.FileNum) } } b.Added[nf.Level] = append(b.Added[nf.Level], nf.Meta) } + return nil } // Apply applies the delta b to the current version to produce a new @@ -501,11 +501,11 @@ func (b *BulkVersionEdit) Apply( if curr == nil { continue } - files := curr.Levels[level] - v.Levels[level] = files + levelMetadata := curr.Levels[level] + v.Levels[level] = levelMetadata // We still have to bump the ref count for all files. - for i := range files { - atomic.AddInt32(&files[i].refs, 1) + for i := range levelMetadata.files { + atomic.AddInt32(&levelMetadata.files[i].refs, 1) } continue } @@ -513,17 +513,17 @@ func (b *BulkVersionEdit) Apply( // Some edits on this level. var currFiles []*FileMetadata if curr != nil { - currFiles = curr.Levels[level] + currFiles = curr.Levels[level].files } addedFiles := b.Added[level] deletedMap := b.Deleted[level] n := len(currFiles) + len(addedFiles) if n == 0 { - return nil, nil, errors.Errorf( + return nil, nil, base.CorruptionErrorf( "pebble: internal error: No current or added files but have deleted files: %d", errors.Safe(len(deletedMap))) } - v.Levels[level] = make([]*FileMetadata, 0, n) + v.Levels[level].files = make([]*FileMetadata, 0, n) // We have 2 lists of files, currFiles and addedFiles either of which (but not both) can // be empty. // - currFiles is internally consistent, since it comes from curr. @@ -553,14 +553,14 @@ func (b *BulkVersionEdit) Apply( continue } atomic.AddInt32(&f.refs, 1) - v.Levels[level] = append(v.Levels[level], f) + v.Levels[level].files = append(v.Levels[level].files, f) } } - SortBySeqNum(v.Levels[level]) + SortBySeqNum(v.Levels[level].files) if err := v.InitL0Sublevels(cmp, formatKey, flushSplitBytes); err != nil { return nil, nil, errors.Wrap(err, "pebble: internal error") } - if err := CheckOrdering(cmp, formatKey, Level(0), v.Levels[level]); err != nil { + if err := CheckOrdering(cmp, formatKey, Level(0), v.Levels[level].Iter()); err != nil { return nil, nil, errors.Wrap(err, "pebble: internal error") } continue @@ -599,10 +599,10 @@ func (b *BulkVersionEdit) Apply( } removeZombie(cf.FileNum) atomic.AddInt32(&cf.refs, 1) - v.Levels[level] = append(v.Levels[level], cf) + v.Levels[level].files = append(v.Levels[level].files, cf) } currFiles = currFiles[j:] - numFiles := len(v.Levels[level]) + numFiles := len(v.Levels[level].files) if numFiles > 0 { // We expect k to typically be large, and we can avoid doing consistency // checks of the files within that set of k, since they are already mutually @@ -612,16 +612,16 @@ func (b *BulkVersionEdit) Apply( // its predecessor either came from currFiles or addedFiles, and both are ones // which we need to check against f for consistency (since we have not checked // addedFiles for internal consistency). - if base.InternalCompare(cmp, v.Levels[level][numFiles-1].Largest, f.Smallest) >= 0 { - cf := v.Levels[level][numFiles-1] - return nil, nil, errors.Errorf( + if base.InternalCompare(cmp, v.Levels[level].files[numFiles-1].Largest, f.Smallest) >= 0 { + cf := v.Levels[level].files[numFiles-1] + return nil, nil, base.CorruptionErrorf( "pebble: internal error: L%d files %s and %s have overlapping ranges: [%s-%s] vs [%s-%s]", errors.Safe(level), errors.Safe(cf.FileNum), errors.Safe(f.FileNum), cf.Smallest.Pretty(formatKey), cf.Largest.Pretty(formatKey), f.Smallest.Pretty(formatKey), f.Largest.Pretty(formatKey)) } } - v.Levels[level] = append(v.Levels[level], f) + v.Levels[level].files = append(v.Levels[level].files, f) } // Add any remaining files in currFiles that are after all the added files. for i := range currFiles { @@ -632,7 +632,7 @@ func (b *BulkVersionEdit) Apply( } removeZombie(f.FileNum) atomic.AddInt32(&f.refs, 1) - v.Levels[level] = append(v.Levels[level], f) + v.Levels[level].files = append(v.Levels[level].files, f) } } return v, zombies, nil diff --git a/github.com/cockroachdb/pebble/internal/private/flush_external.go b/github.com/cockroachdb/pebble/internal/private/flush_external.go index 25deeff1dc..398b4d3cc0 100644 --- a/github.com/cockroachdb/pebble/internal/private/flush_external.go +++ b/github.com/cockroachdb/pebble/internal/private/flush_external.go @@ -4,9 +4,7 @@ package private -import ( - "github.com/cockroachdb/pebble/internal/manifest" -) +import "github.com/cockroachdb/pebble/internal/manifest" // FlushExternalTable is a hook for linking files into L0 without assigning a // global sequence number, mimicking a flush. FlushExternalTable takes a diff --git a/github.com/cockroachdb/pebble/internal/rangedel/fragmenter.go b/github.com/cockroachdb/pebble/internal/rangedel/fragmenter.go index 9d1e258ad9..dec8898d35 100644 --- a/github.com/cockroachdb/pebble/internal/rangedel/fragmenter.go +++ b/github.com/cockroachdb/pebble/internal/rangedel/fragmenter.go @@ -63,7 +63,8 @@ func Sort(cmp base.Compare, tombstones []Tombstone) { // tombstones are split at their overlap points. The fragmented tombstones are // output to the supplied Output function. type Fragmenter struct { - Cmp base.Compare + Cmp base.Compare + Format base.FormatKey // Emit is called to emit a chunk of tombstone fragments. Every tombstone // within the chunk has the same start and end key and are in decreasing // order of their sequence numbers. @@ -95,7 +96,7 @@ func (f *Fragmenter) checkInvariants(buf []Tombstone) { } if f.Cmp(buf[i-1].Start.UserKey, buf[i].Start.UserKey) != 0 { panic(fmt.Sprintf("pebble: pending tombstone invariant violated: %s %s", - buf[i-1].Start, buf[i].Start)) + buf[i-1].Start.Pretty(f.Format), buf[i].Start.Pretty(f.Format))) } } } @@ -182,7 +183,7 @@ func (f *Fragmenter) Add(start base.InternalKey, end []byte) { switch c := f.Cmp(start.UserKey, f.flushedKey); { case c < 0: panic(fmt.Sprintf("pebble: start key (%s) < flushed key (%s)", - start.UserKey, f.flushedKey)) + f.Format(start.UserKey), f.Format(f.flushedKey))) } } if f.Cmp(start.UserKey, end) >= 0 { @@ -200,7 +201,7 @@ func (f *Fragmenter) Add(start base.InternalKey, end []byte) { switch c := f.Cmp(f.pending[0].Start.UserKey, start.UserKey); { case c > 0: panic(fmt.Sprintf("pebble: keys must be added in order: %s > %s", - f.pending[0].Start, start)) + f.pending[0].Start.Pretty(f.Format), start.Pretty(f.Format))) case c == 0: // The new tombstone has the same start key as the existing pending // tombstones. Add it to the pending buffer. @@ -236,7 +237,7 @@ func (f *Fragmenter) Deleted(key base.InternalKey, snapshot uint64) bool { if f.Cmp(f.pending[0].Start.UserKey, key.UserKey) > 0 { panic(fmt.Sprintf("pebble: keys must be in order: %s > %s", - f.pending[0].Start, key)) + f.pending[0].Start.Pretty(f.Format), key.Pretty(f.Format))) } seqNum := key.SeqNum() @@ -283,7 +284,7 @@ func (f *Fragmenter) FlushTo(key []byte) { switch c := f.Cmp(key, f.flushedKey); { case c < 0: panic(fmt.Sprintf("pebble: flush-to key (%s) < flushed key (%s)", - key, f.flushedKey)) + f.Format(key), f.Format(f.flushedKey))) } } f.flushedKey = append(f.flushedKey[:0], key...) @@ -294,7 +295,7 @@ func (f *Fragmenter) FlushTo(key []byte) { switch c := f.Cmp(f.pending[0].Start.UserKey, key); { case c > 0: panic(fmt.Sprintf("pebble: keys must be in order: %s > %s", - f.pending[0].Start, key)) + f.Format(f.pending[0].Start.UserKey), f.Format(key))) } // Note that we explicitly do not return early here if Start.UserKey == // key. Similar to the scenario described above, consider: @@ -360,7 +361,7 @@ func (f *Fragmenter) TruncateAndFlushTo(key []byte) { switch c := f.Cmp(key, f.flushedKey); { case c < 0: panic(fmt.Sprintf("pebble: start key (%s) < flushed key (%s)", - key, f.flushedKey)) + f.Format(key), f.Format(f.flushedKey))) } } if invariants.RaceEnabled { @@ -373,7 +374,7 @@ func (f *Fragmenter) TruncateAndFlushTo(key []byte) { switch c := f.Cmp(f.pending[0].Start.UserKey, key); { case c > 0: panic(fmt.Sprintf("pebble: keys must be added in order: %s > %s", - f.pending[0].Start, key)) + f.Format(f.pending[0].Start.UserKey), f.Format(key))) case c == 0: return } diff --git a/github.com/cockroachdb/pebble/internal/rangedel/iter.go b/github.com/cockroachdb/pebble/internal/rangedel/iter.go index 37a3209152..c139b338d8 100644 --- a/github.com/cockroachdb/pebble/internal/rangedel/iter.go +++ b/github.com/cockroachdb/pebble/internal/rangedel/iter.go @@ -4,9 +4,7 @@ package rangedel -import ( - "github.com/cockroachdb/pebble/internal/base" -) +import "github.com/cockroachdb/pebble/internal/base" // Iter is an iterator over a set of fragmented tombstones. type Iter struct { diff --git a/github.com/cockroachdb/pebble/internal/rangedel/truncate.go b/github.com/cockroachdb/pebble/internal/rangedel/truncate.go index d328dce7b9..28bd5cccff 100644 --- a/github.com/cockroachdb/pebble/internal/rangedel/truncate.go +++ b/github.com/cockroachdb/pebble/internal/rangedel/truncate.go @@ -8,13 +8,32 @@ import "github.com/cockroachdb/pebble/internal/base" // Truncate creates a new iterator where every tombstone in the supplied // iterator is truncated to be contained within the range [lower, upper). -func Truncate(cmp base.Compare, iter base.InternalIterator, lower, upper []byte) *Iter { +// If start and end are specified, filter out any range tombstones that +// are completely outside those bounds. +func Truncate( + cmp base.Compare, + iter base.InternalIterator, + lower, upper []byte, + start, end *base.InternalKey, +) *Iter { var tombstones []Tombstone for key, value := iter.First(); key != nil; key, value = iter.Next() { t := Tombstone{ Start: *key, End: value, } + // Ignore this tombstone if it lies completely outside [start, end]. + // The comparison between t.End and start is by user key only, as + // the range tombstone is exclusive at t.End, so comparing by user keys + // is sufficient. Alternatively, the below comparison can be seen to + // be logically equivalent to: + // InternalKey{UserKey: t.End, SeqNum: SeqNumMax} < start + if start != nil && cmp(t.End, start.UserKey) <= 0 { + continue + } + if end != nil && base.InternalCompare(cmp, t.Start, *end) > 0 { + continue + } if cmp(t.Start.UserKey, lower) < 0 { t.Start.UserKey = lower } diff --git a/github.com/cockroachdb/pebble/internal/record/log_writer.go b/github.com/cockroachdb/pebble/internal/record/log_writer.go index 7ce8dc1d6d..91e9409df8 100644 --- a/github.com/cockroachdb/pebble/internal/record/log_writer.go +++ b/github.com/cockroachdb/pebble/internal/record/log_writer.go @@ -255,6 +255,7 @@ type LogWriter struct { block *block free struct { sync.Mutex + // Condition variable used to signal a block is freed. cond sync.Cond blocks []*block allocated int @@ -415,11 +416,20 @@ func (w *LogWriter) flushLoop(context.Context) { data := w.block.buf[w.block.flushed:written] w.block.flushed = written + // If flusher has an error, we propagate it to waiters. Note in spite of + // error we consume the pending list above to free blocks for writers. + if f.err != nil { + f.syncQ.pop(head, tail, f.err) + continue + } f.Unlock() - synced, err := w.flushPending(data, pending, head, tail) - f.Lock() + f.err = err + if f.err != nil { + f.syncQ.clearBlocked() + continue + } if synced && f.minSyncInterval != nil { // A sync was performed. Make sure we've waited for the min sync @@ -436,14 +446,6 @@ func (w *LogWriter) flushLoop(context.Context) { } } } - - f.err = err - if f.err != nil { - // TODO(peter): There might be new waiters that we should propagate f.err - // to. Because f.err is now set, we only have to perform a single extra - // clearing of those waiters as no new ones can arrive via SyncRecord(). - return - } } } @@ -528,9 +530,15 @@ func (w *LogWriter) queueBlock() { } // Close flushes and syncs any unwritten data and closes the writer. +// Where required, external synchronisation is provided by commitPipeline.mu. func (w *LogWriter) Close() error { f := &w.flusher + // Emit an EOF trailer signifying the end of this log. This helps readers + // differentiate between a corrupted entry in the middle of a log from + // garbage at the tail from a recycled log file. + w.emitEOFTrailer() + // Signal the flush loop to close. f.Lock() f.close = true @@ -562,6 +570,7 @@ func (w *LogWriter) Close() error { // WriteRecord writes a complete record. Returns the offset just past the end // of the record. +// External synchronisation provided by commitPipeline.mu. func (w *LogWriter) WriteRecord(p []byte) (int64, error) { return w.SyncRecord(p, nil, nil) } @@ -570,6 +579,7 @@ func (w *LogWriter) WriteRecord(p []byte) (int64, error) { // asynchronously persisted to the underlying writer and done will be called on // the wait group upon completion. Returns the offset just past the end of the // record. +// External synchronisation provided by commitPipeline.mu. func (w *LogWriter) SyncRecord(p []byte, wg *sync.WaitGroup, err *error) (int64, error) { if w.err != nil { return -1, w.err @@ -603,10 +613,23 @@ func (w *LogWriter) SyncRecord(p []byte, wg *sync.WaitGroup, err *error) (int64, } // Size returns the current size of the file. +// External synchronisation provided by commitPipeline.mu. func (w *LogWriter) Size() int64 { return w.blockNum*blockSize + int64(w.block.written) } +func (w *LogWriter) emitEOFTrailer() { + // Write a recyclable chunk header with a different log number. Readers + // will treat the header as EOF when the log number does not match. + b := w.block + i := b.written + binary.LittleEndian.PutUint32(b.buf[i+0:i+4], 0) // CRC + binary.LittleEndian.PutUint16(b.buf[i+4:i+6], 0) // Size + b.buf[i+6] = recyclableFullChunkType + binary.LittleEndian.PutUint32(b.buf[i+7:i+11], w.logNum+1) // Log number + atomic.StoreInt32(&b.written, i+int32(recyclableHeaderSize)) +} + func (w *LogWriter) emitFragment(n int, p []byte) []byte { b := w.block i := b.written diff --git a/github.com/cockroachdb/pebble/internal/record/record.go b/github.com/cockroachdb/pebble/internal/record/record.go index e223c34ee5..2e9058f3ad 100644 --- a/github.com/cockroachdb/pebble/internal/record/record.go +++ b/github.com/cockroachdb/pebble/internal/record/record.go @@ -141,12 +141,12 @@ var ( // ErrZeroedChunk is returned if a chunk is encountered that is zeroed. This // usually occurs due to log file preallocation. - ErrZeroedChunk = errors.New("pebble/record: zeroed chunk") + ErrZeroedChunk = base.CorruptionErrorf("pebble/record: zeroed chunk") // ErrInvalidChunk is returned if a chunk is encountered with an invalid // header, length, or checksum. This usually occurs when a log is recycled, // but can also occur due to corruption. - ErrInvalidChunk = errors.New("pebble/record: invalid chunk") + ErrInvalidChunk = base.CorruptionErrorf("pebble/record: invalid chunk") ) // IsInvalidRecord returns true if the error matches one of the error types @@ -185,7 +185,7 @@ type Reader struct { // NewReader returns a new reader. If the file contains records encoded using // the recyclable record format, then the log number in those records must -// match the specifed logNum. +// match the specified logNum. func NewReader(r io.Reader, logNum base.FileNum) *Reader { return &Reader{ r: r, diff --git a/github.com/cockroachdb/pebble/iterator.go b/github.com/cockroachdb/pebble/iterator.go index 8839e9ea33..51a2d26197 100644 --- a/github.com/cockroachdb/pebble/iterator.go +++ b/github.com/cockroachdb/pebble/iterator.go @@ -9,6 +9,7 @@ import ( "io" "github.com/cockroachdb/errors" + "github.com/cockroachdb/pebble/internal/base" ) type iterPos int8 @@ -21,6 +22,15 @@ const ( var errReversePrefixIteration = errors.New("pebble: unsupported reverse prefix iteration") +// IteratorMetrics holds per-iterator metrics. +type IteratorMetrics struct { + // The read amplification experienced by this iterator. This is the sum of + // the memtables, the L0 sublevels and the non-empty Ln levels. Higher read + // amplification generally results in slower reads, though allowing higher + // read amplification can also result in faster writes. + ReadAmp int +} + // Iterator iterates over a DB's key/value pairs in key order. // // An iterator must be closed after use, but it is not necessary to read an @@ -103,12 +113,12 @@ func (i *Iterator) findNextEntry() bool { i.mergeNext(key, valueMerger) } if i.err == nil { - i.value, i.valueCloser, i.err = valueMerger.Finish() + i.value, i.valueCloser, i.err = valueMerger.Finish(true /* includesBase */) } return i.err == nil default: - i.err = errors.Errorf("pebble: invalid internal key kind: %d", errors.Safe(key.Kind())) + i.err = base.CorruptionErrorf("pebble: invalid internal key kind: %d", errors.Safe(key.Kind())) return false } } @@ -159,7 +169,7 @@ func (i *Iterator) findPrevEntry() bool { // We've iterated to the previous user key. i.pos = iterPosPrev if valueMerger != nil { - i.value, i.valueCloser, i.err = valueMerger.Finish() + i.value, i.valueCloser, i.err = valueMerger.Finish(true /* includesBase */) } return i.err == nil } @@ -214,7 +224,7 @@ func (i *Iterator) findPrevEntry() bool { continue default: - i.err = errors.Errorf("pebble: invalid internal key kind: %d", errors.Safe(key.Kind())) + i.err = base.CorruptionErrorf("pebble: invalid internal key kind: %d", errors.Safe(key.Kind())) return false } } @@ -222,7 +232,7 @@ func (i *Iterator) findPrevEntry() bool { if i.valid { i.pos = iterPosPrev if valueMerger != nil { - i.value, i.valueCloser, i.err = valueMerger.Finish() + i.value, i.valueCloser, i.err = valueMerger.Finish(true /* includesBase */) } return i.err == nil } @@ -291,7 +301,7 @@ func (i *Iterator) mergeNext(key InternalKey, valueMerger ValueMerger) { continue default: - i.err = errors.Errorf("pebble: invalid internal key kind: %d", errors.Safe(key.Kind())) + i.err = base.CorruptionErrorf("pebble: invalid internal key kind: %d", errors.Safe(key.Kind())) return } } @@ -305,6 +315,8 @@ func (i *Iterator) SeekGE(key []byte) bool { i.prefix = nil if lowerBound := i.opts.GetLowerBound(); lowerBound != nil && i.cmp(key, lowerBound) < 0 { key = lowerBound + } else if upperBound := i.opts.GetUpperBound(); upperBound != nil && i.cmp(key, upperBound) > 0 { + key = upperBound } i.iterKey, i.iterValue = i.iter.SeekGE(key) @@ -341,6 +353,19 @@ func (i *Iterator) SeekGE(key []byte) bool { // SeekPrefixGE("a@0") -> "a@1" // Next() -> "a@2" // Next() -> EOF +// +// If you're just looking to iterate over keys with a shared prefix, as +// defined by the configured comparer, set iterator bounds instead: +// +// iter := db.NewIter(&pebble.IterOptions{ +// LowerBound: []byte("prefix"), +// UpperBound: []byte("prefiy"), +// }) +// for iter.First(); iter.Valid(); iter.Next() { +// // Only keys beginning with "prefix" will be visited. +// } +// +// See Example_prefixiteration for a working example. func (i *Iterator) SeekPrefixGE(key []byte) bool { i.err = nil // clear cached iteration error @@ -360,6 +385,12 @@ func (i *Iterator) SeekPrefixGE(key []byte) bool { return false } key = lowerBound + } else if upperBound := i.opts.GetUpperBound(); upperBound != nil && i.cmp(key, upperBound) > 0 { + if n := i.split(upperBound); !bytes.Equal(i.prefix, upperBound[:n]) { + i.err = errors.New("pebble: SeekPrefixGE supplied with key outside of upper bound") + return false + } + key = upperBound } i.iterKey, i.iterValue = i.iter.SeekPrefixGE(i.prefix, key) @@ -372,8 +403,10 @@ func (i *Iterator) SeekPrefixGE(key []byte) bool { func (i *Iterator) SeekLT(key []byte) bool { i.err = nil // clear cached iteration error i.prefix = nil - if upperBound := i.opts.GetUpperBound(); upperBound != nil && i.cmp(key, upperBound) >= 0 { + if upperBound := i.opts.GetUpperBound(); upperBound != nil && i.cmp(key, upperBound) > 0 { key = upperBound + } else if lowerBound := i.opts.GetLowerBound(); lowerBound != nil && i.cmp(key, lowerBound) < 0 { + key = lowerBound } i.iterKey, i.iterValue = i.iter.SeekLT(key) @@ -557,3 +590,14 @@ func (i *Iterator) SetBounds(lower, upper []byte) { i.opts.UpperBound = upper i.iter.SetBounds(lower, upper) } + +// Metrics returns per-iterator metrics. +func (i *Iterator) Metrics() IteratorMetrics { + m := IteratorMetrics{ + ReadAmp: 1, + } + if mi, ok := i.iter.(*mergingIter); ok { + m.ReadAmp = len(mi.levels) + } + return m +} diff --git a/github.com/cockroachdb/pebble/level_checker.go b/github.com/cockroachdb/pebble/level_checker.go index 7a3b097ff2..817100670e 100644 --- a/github.com/cockroachdb/pebble/level_checker.go +++ b/github.com/cockroachdb/pebble/level_checker.go @@ -151,7 +151,7 @@ func (m *simpleMergingIter) step() bool { // Ongoing series of MERGE records ends with a MERGE record. if keyChanged && m.valueMerger != nil { var closer io.Closer - _, closer, m.err = m.valueMerger.Finish() + _, closer, m.err = m.valueMerger.Finish(true /* includesBase */) if m.err == nil && closer != nil { m.err = closer.Close() } @@ -162,7 +162,7 @@ func (m *simpleMergingIter) step() bool { switch item.key.Kind() { case InternalKeyKindSingleDelete, InternalKeyKindDelete: var closer io.Closer - _, closer, m.err = m.valueMerger.Finish() + _, closer, m.err = m.valueMerger.Finish(true /* includesBase */) if m.err == nil && closer != nil { m.err = closer.Close() } @@ -171,7 +171,7 @@ func (m *simpleMergingIter) step() bool { m.err = m.valueMerger.MergeOlder(item.value) if m.err == nil { var closer io.Closer - _, closer, m.err = m.valueMerger.Finish() + _, closer, m.err = m.valueMerger.Finish(true /* includesBase */) if m.err == nil && closer != nil { m.err = closer.Close() } @@ -252,7 +252,7 @@ func (m *simpleMergingIter) step() bool { // Last record was a MERGE record. if m.valueMerger != nil { var closer io.Closer - _, closer, m.err = m.valueMerger.Finish() + _, closer, m.err = m.valueMerger.Finish(true /* includesBase */) if m.err == nil && closer != nil { m.err = closer.Close() } @@ -368,11 +368,12 @@ func checkRangeTombstones(c *checkConfig) error { } current := c.readState.current - addTombstonesFromLevel := func(files *[]*fileMetadata, lsmLevel int) error { - for j := 0; j < len(*files); j++ { - lower, upper := getAtomicUnitBounds(c.cmp, *files, j) - f := (*files)[j] - iterToClose, iter, err := c.newIters(f, nil, nil) + addTombstonesFromLevel := func(files manifest.LevelIterator, lsmLevel int) error { + for f := files.First(); f != nil; f = files.Next() { + lf := files.Take() + atomicUnit, _ := expandToAtomicUnit(c.cmp, lf.Slice()) + lower, upper := manifest.KeyRange(c.cmp, atomicUnit.Iter()) + iterToClose, iter, err := c.newIters(lf, nil, nil) if err != nil { return err } @@ -382,11 +383,11 @@ func checkRangeTombstones(c *checkConfig) error { } truncate := func(t rangedel.Tombstone) rangedel.Tombstone { // Same checks as in rangedel.Truncate. - if c.cmp(t.Start.UserKey, lower) < 0 { - t.Start.UserKey = lower + if c.cmp(t.Start.UserKey, lower.UserKey) < 0 { + t.Start.UserKey = lower.UserKey } - if c.cmp(t.End, upper) > 0 { - t.End = upper + if c.cmp(t.End, upper.UserKey) > 0 { + t.End = upper.UserKey } if c.cmp(t.Start.UserKey, t.End) >= 0 { t.Start.SetKind(InternalKeyKindInvalid) @@ -405,17 +406,14 @@ func checkRangeTombstones(c *checkConfig) error { if len(current.L0Sublevels.Levels[i]) == 0 { continue } - if err := addTombstonesFromLevel(¤t.L0Sublevels.Levels[i], 0); err != nil { + err := addTombstonesFromLevel(manifest.NewLevelSlice(current.L0Sublevels.Levels[i]).Iter(), 0) + if err != nil { return err } level++ } for i := 1; i < len(current.Levels); i++ { - if len(current.Levels[i]) == 0 { - continue - } - files := ¤t.Levels[i] - if err := addTombstonesFromLevel(files, i); err != nil { + if err := addTombstonesFromLevel(current.Levels[i].Iter(), i); err != nil { return err } level++ @@ -431,35 +429,6 @@ func checkRangeTombstones(c *checkConfig) error { return iterateAndCheckTombstones(c.cmp, c.formatKey, tombstones) } -// TODO(sbhola): mostly copied from compaction.go: refactor and reuse? -func getAtomicUnitBounds(cmp Compare, files []*fileMetadata, index int) (lower, upper []byte) { - lower = files[index].Smallest.UserKey - upper = files[index].Largest.UserKey - for i := index; i > 0; i-- { - cur := files[i] - prev := files[i-1] - if cmp(prev.Largest.UserKey, cur.Smallest.UserKey) < 0 { - break - } - if prev.Largest.Trailer == InternalKeyRangeDeleteSentinel { - break - } - lower = prev.Smallest.UserKey - } - for i := index + 1; i < len(files); i++ { - cur := files[i-1] - next := files[i] - if cmp(cur.Largest.UserKey, next.Smallest.UserKey) < 0 { - break - } - if cur.Largest.Trailer == InternalKeyRangeDeleteSentinel { - break - } - upper = next.Largest.UserKey - } - return -} - func levelOrMemtable(lsmLevel int, fileNum FileNum) string { if lsmLevel == -1 { return "memtable" @@ -596,7 +565,7 @@ func (d *DB) CheckLevels(stats *CheckLevelsStats) error { // Determine the seqnum to read at after grabbing the read state (current and // memtables) above. - seqNum := atomic.LoadUint64(&d.mu.versions.visibleSeqNum) + seqNum := atomic.LoadUint64(&d.mu.versions.atomic.visibleSeqNum) checkConfig := &checkConfig{ logger: d.opts.Logger, @@ -654,7 +623,7 @@ func checkLevelsInternal(c *checkConfig) (err error) { mlevels = append(mlevels, simpleMergingIterLevel{}) } for level := 1; level < len(current.Levels); level++ { - if len(current.Levels[level]) == 0 { + if current.Levels[level].Slice().Empty() { continue } mlevels = append(mlevels, simpleMergingIterLevel{}) @@ -665,9 +634,10 @@ func checkLevelsInternal(c *checkConfig) (err error) { if len(current.L0Sublevels.Levels[sublevel]) == 0 { continue } + manifestIter := manifest.NewLevelSlice(current.L0Sublevels.Levels[sublevel]).Iter() iterOpts := IterOptions{logger: c.logger} li := &levelIter{} - li.init(iterOpts, c.cmp, c.newIters, current.L0Sublevels.Levels[sublevel], + li.init(iterOpts, c.cmp, c.newIters, manifestIter, manifest.L0Sublevel(sublevel), nil) li.initRangeDel(&mlevelAlloc[0].rangeDelIter) li.initSmallestLargestUserKey(&mlevelAlloc[0].smallestUserKey, nil, nil) @@ -675,12 +645,13 @@ func checkLevelsInternal(c *checkConfig) (err error) { mlevelAlloc = mlevelAlloc[1:] } for level := 1; level < len(current.Levels); level++ { - if len(current.Levels[level]) == 0 { + if current.Levels[level].Empty() { continue } + iterOpts := IterOptions{logger: c.logger} li := &levelIter{} - li.init(iterOpts, c.cmp, c.newIters, current.Levels[level], manifest.Level(level), nil) + li.init(iterOpts, c.cmp, c.newIters, current.Levels[level].Iter(), manifest.Level(level), nil) li.initRangeDel(&mlevelAlloc[0].rangeDelIter) li.initSmallestLargestUserKey(&mlevelAlloc[0].smallestUserKey, nil, nil) mlevelAlloc[0].iter = li diff --git a/github.com/cockroachdb/pebble/level_iter.go b/github.com/cockroachdb/pebble/level_iter.go index cb83586032..62df6e2226 100644 --- a/github.com/cockroachdb/pebble/level_iter.go +++ b/github.com/cockroachdb/pebble/level_iter.go @@ -7,7 +7,6 @@ package pebble import ( "fmt" "runtime/debug" - "sort" "github.com/cockroachdb/pebble/internal/base" "github.com/cockroachdb/pebble/internal/invariants" @@ -18,7 +17,7 @@ import ( // number. If bytesIterated is specified, it is incremented as the given file is // iterated through. type tableNewIters func( - meta *fileMetadata, opts *IterOptions, bytesIterated *uint64, + file manifest.LevelFile, opts *IterOptions, bytesIterated *uint64, ) (internalIterator, internalIterator, error) // levelIter provides a merged view of the sstables in a level. @@ -56,8 +55,6 @@ type levelIter struct { tableOpts IterOptions // The LSM level this levelIter is initialized for. level manifest.Level - // The current file wrt the iterator position. - index int // The keys to return when iterating past an sstable boundary and that // boundary is a range deletion tombstone. The boundary could be smallest // (i.e. arrived at with Prev), or largest (arrived at with Next). @@ -67,16 +64,27 @@ type levelIter struct { // which doesn't contain the search key, but which does contain range // tombstones. syntheticBoundary InternalKey - // The iter for the current index. It is nil under any of the following conditions: - // - index < 0 or index > len(files) + // The iter for the current file. It is nil under any of the following conditions: + // - files.Current() == nil // - err != nil // - some other constraint, like the bounds in opts, caused the file at index to not // be relevant to the iteration. - iter internalIterator - newIters tableNewIters - rangeDelIter *internalIterator - files []*fileMetadata - err error + iter internalIterator + iterFile *fileMetadata + newIters tableNewIters + // When rangeDelIterPtr != nil, the caller requires that *rangeDelIterPtr must + // point to a range del iterator corresponding to the current file. When this + // iterator returns nil, *rangeDelIterPtr should also be set to nil. Whenever + // a non-nil internalIterator is placed in rangeDelIterPtr, a copy is placed + // in rangeDelIterCopy. This is done for the following special case: + // when this iterator returns nil because of exceeding the bounds, we don't + // close iter and *rangeDelIterPtr since we could reuse it in the next seek. But + // we need to set *rangeDelIterPtr to nil because of the aforementioned contract. + // This copy is used to revive the *rangeDelIterPtr in the case of reuse. + rangeDelIterPtr *internalIterator + rangeDelIterCopy internalIterator + files manifest.LevelIterator + err error // Pointer into this level's entry in `mergingIterLevel::smallestUserKey,largestUserKey`. // We populate it with the corresponding bounds for the currently opened file. It is used for @@ -127,6 +135,12 @@ type levelIter struct { smallestUserKey, largestUserKey *[]byte isLargestUserKeyRangeDelSentinel *bool + // Set to true iff the key returned by this iterator is a synthetic key + // derived from the iterator bounds. This is used to prevent the + // mergingIter from being stuck at such a synthetic key if it becomes the + // top element of the heap. + isSyntheticIterBoundsKey *bool + // bytesIterated keeps track of the number of bytes iterated during compaction. bytesIterated *uint64 @@ -143,7 +157,7 @@ func newLevelIter( opts IterOptions, cmp Compare, newIters tableNewIters, - files []*fileMetadata, + files manifest.LevelIterator, level manifest.Level, bytesIterated *uint64, ) *levelIter { @@ -156,7 +170,7 @@ func (l *levelIter) init( opts IterOptions, cmp Compare, newIters tableNewIters, - files []*fileMetadata, + files manifest.LevelIterator, level manifest.Level, bytesIterated *uint64, ) { @@ -167,14 +181,14 @@ func (l *levelIter) init( l.upper = opts.UpperBound l.tableOpts.TableFilter = opts.TableFilter l.cmp = cmp - l.index = -1 + l.iterFile = nil l.newIters = newIters l.files = files l.bytesIterated = bytesIterated } func (l *levelIter) initRangeDel(rangeDelIter *internalIterator) { - l.rangeDelIter = rangeDelIter + l.rangeDelIterPtr = rangeDelIter } func (l *levelIter) initSmallestLargestUserKey( @@ -185,7 +199,11 @@ func (l *levelIter) initSmallestLargestUserKey( l.isLargestUserKeyRangeDelSentinel = isLargestUserKeyRangeDelSentinel } -func (l *levelIter) findFileGE(key []byte) int { +func (l *levelIter) initIsSyntheticIterBoundsKey(isSyntheticIterBoundsKey *bool) { + l.isSyntheticIterBoundsKey = isSyntheticIterBoundsKey +} + +func (l *levelIter) findFileGE(key []byte) *fileMetadata { // Find the earliest file whose largest key is >= ikey. // // If the earliest file has its largest key == ikey and that largest key is a @@ -198,24 +216,18 @@ func (l *levelIter) findFileGE(key []byte) int { // Additionally, this prevents loading untruncated range deletions from a table which can't // possibly contain the target key and is required for correctness by mergingIter.SeekGE // (see the comment in that function). - // - // TODO(peter): inline the binary search. - return sort.Search(len(l.files), func(i int) bool { - largest := &l.files[i].Largest - c := l.cmp(largest.UserKey, key) - if c > 0 { - return true - } - return c == 0 && largest.Trailer != InternalKeyRangeDeleteSentinel - }) + + m := l.files.SeekGE(l.cmp, key) + for m != nil && m.Largest.Trailer == InternalKeyRangeDeleteSentinel && + l.cmp(m.Largest.UserKey, key) == 0 { + m = l.files.Next() + } + return m } -func (l *levelIter) findFileLT(key []byte) int { +func (l *levelIter) findFileLT(key []byte) *fileMetadata { // Find the last file whose smallest key is < ikey. - index := sort.Search(len(l.files), func(i int) bool { - return l.cmp(l.files[i].Smallest.UserKey, key) >= 0 - }) - return index - 1 + return l.files.SeekLT(l.cmp, key) } // Init the iteration bounds for the current table. Returns -1 if the table @@ -252,10 +264,13 @@ func (l *levelIter) initTableBounds(f *fileMetadata) int { return 0 } -func (l *levelIter) loadFile(index, dir int) bool { +func (l *levelIter) loadFile(file *fileMetadata, dir int) bool { l.smallestBoundary = nil l.largestBoundary = nil - if l.index == index { + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = false + } + if l.iterFile == file { if l.err != nil { return false } @@ -263,34 +278,37 @@ func (l *levelIter) loadFile(index, dir int) bool { // We don't bother comparing the file bounds with the iteration bounds when we have // an already open iterator. It is possible that the iter may not be relevant given the // current iteration bounds, but it knows those bounds, so it will enforce them. + if l.rangeDelIterPtr != nil { + *l.rangeDelIterPtr = l.rangeDelIterCopy + } return true } - // We were already at index, but don't have an iterator, probably because the file was + // We were already at file, but don't have an iterator, probably because the file was // beyond the iteration bounds. It may still be, but it is also possible that the bounds // have changed. We handle that below. } - // Close both iter and rangeDelIter. While mergingIter knows about - // rangeDelIter, it can't call Close() on it because it does not know when - // the levelIter will switch it. Note that levelIter.Close() can be called - // multiple times. + // Close both iter and rangeDelIterPtr. While mergingIter knows about + // rangeDelIterPtr, it can't call Close() on it because it does not know + // when the levelIter will switch it. Note that levelIter.Close() can be + // called multiple times. if err := l.Close(); err != nil { return false } - for ; ; index += dir { - l.index = index - if l.index < 0 || l.index >= len(l.files) { + for { + l.iterFile = file + if file == nil { return false } - f := l.files[l.index] - switch l.initTableBounds(f) { + switch l.initTableBounds(file) { case -1: // The largest key in the sstable is smaller than the lower bound. if dir < 0 { return false } + file = l.files.Next() continue case +1: // The smallest key in the sstable is greater than or equal to the upper @@ -298,27 +316,29 @@ func (l *levelIter) loadFile(index, dir int) bool { if dir > 0 { return false } + file = l.files.Prev() continue } var rangeDelIter internalIterator - l.iter, rangeDelIter, l.err = l.newIters(f, &l.tableOpts, l.bytesIterated) + l.iter, rangeDelIter, l.err = l.newIters(l.files.Take(), &l.tableOpts, l.bytesIterated) if l.err != nil { return false } - if l.rangeDelIter != nil { - *l.rangeDelIter = rangeDelIter + if l.rangeDelIterPtr != nil { + *l.rangeDelIterPtr = rangeDelIter + l.rangeDelIterCopy = rangeDelIter } else if rangeDelIter != nil { rangeDelIter.Close() } if l.smallestUserKey != nil { - *l.smallestUserKey = f.Smallest.UserKey + *l.smallestUserKey = file.Smallest.UserKey } if l.largestUserKey != nil { - *l.largestUserKey = f.Largest.UserKey + *l.largestUserKey = file.Largest.UserKey } if l.isLargestUserKeyRangeDelSentinel != nil { - *l.isLargestUserKeyRangeDelSentinel = f.Largest.Trailer == InternalKeyRangeDeleteSentinel + *l.isLargestUserKeyRangeDelSentinel = file.Largest.Trailer == InternalKeyRangeDeleteSentinel } return true } @@ -346,10 +366,13 @@ func (l *levelIter) verify(key *InternalKey, val []byte) (*InternalKey, []byte) func (l *levelIter) SeekGE(key []byte) (*InternalKey, []byte) { l.err = nil // clear cached iteration error + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = false + } // NB: the top-level Iterator has already adjusted key based on // IterOptions.LowerBound. - if !l.loadFile(l.findFileGE(key), 1) { + if !l.loadFile(l.findFileGE(key), +1) { return nil, nil } if ikey, val := l.iter.SeekGE(key); ikey != nil { @@ -360,10 +383,13 @@ func (l *levelIter) SeekGE(key []byte) (*InternalKey, []byte) { func (l *levelIter) SeekPrefixGE(prefix, key []byte) (*InternalKey, []byte) { l.err = nil // clear cached iteration error + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = false + } // NB: the top-level Iterator has already adjusted key based on // IterOptions.LowerBound. - if !l.loadFile(l.findFileGE(key), 1) { + if !l.loadFile(l.findFileGE(key), +1) { return nil, nil } if key, val := l.iter.SeekPrefixGE(prefix, key); key != nil { @@ -375,11 +401,22 @@ func (l *levelIter) SeekPrefixGE(prefix, key []byte) (*InternalKey, []byte) { // table as findFileGE found the table where key <= meta.Largest. We treat // this case the same as SeekGE where an upper-bound resides within the // sstable and generate a synthetic boundary key. - if l.rangeDelIter != nil { - f := l.files[l.index] - l.syntheticBoundary = f.Largest + if l.rangeDelIterPtr != nil && *l.rangeDelIterPtr != nil { + if l.tableOpts.UpperBound != nil { + l.syntheticBoundary.UserKey = l.tableOpts.UpperBound + l.syntheticBoundary.Trailer = InternalKeyRangeDeleteSentinel + l.largestBoundary = &l.syntheticBoundary + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = true + } + return l.verify(l.largestBoundary, nil) + } + l.syntheticBoundary = l.iterFile.Largest l.syntheticBoundary.SetKind(InternalKeyKindRangeDelete) l.largestBoundary = &l.syntheticBoundary + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = false + } return l.verify(l.largestBoundary, nil) } return l.verify(l.skipEmptyFileForward()) @@ -387,6 +424,9 @@ func (l *levelIter) SeekPrefixGE(prefix, key []byte) (*InternalKey, []byte) { func (l *levelIter) SeekLT(key []byte) (*InternalKey, []byte) { l.err = nil // clear cached iteration error + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = false + } // NB: the top-level Iterator has already adjusted key based on // IterOptions.UpperBound. @@ -401,10 +441,13 @@ func (l *levelIter) SeekLT(key []byte) (*InternalKey, []byte) { func (l *levelIter) First() (*InternalKey, []byte) { l.err = nil // clear cached iteration error + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = false + } // NB: the top-level Iterator will call SeekGE if IterOptions.LowerBound is // set. - if !l.loadFile(0, 1) { + if !l.loadFile(l.files.First(), +1) { return nil, nil } if key, val := l.iter.First(); key != nil { @@ -415,10 +458,13 @@ func (l *levelIter) First() (*InternalKey, []byte) { func (l *levelIter) Last() (*InternalKey, []byte) { l.err = nil // clear cached iteration error + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = false + } // NB: the top-level Iterator will call SeekLT if IterOptions.UpperBound is // set. - if !l.loadFile(len(l.files)-1, -1) { + if !l.loadFile(l.files.Last(), -1) { return nil, nil } if key, val := l.iter.Last(); key != nil { @@ -431,11 +477,25 @@ func (l *levelIter) Next() (*InternalKey, []byte) { if l.err != nil || l.iter == nil { return nil, nil } + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = false + } switch { case l.largestBoundary != nil: + if l.tableOpts.UpperBound != nil { + // The UpperBound was within this file, so don't load the next + // file. We leave the largestBoundary unchanged so that subsequent + // calls to Next() stay at this file. If a Seek/First/Last call is + // made and this file continues to be relevant, loadFile() will + // set the largestBoundary to nil. + if l.rangeDelIterPtr != nil { + *l.rangeDelIterPtr = nil + } + return nil, nil + } // We're stepping past the boundary key, so now we can load the next file. - if l.loadFile(l.index+1, 1) { + if l.loadFile(l.files.Next(), +1) { if key, val := l.iter.First(); key != nil { return l.verify(key, val) } @@ -457,11 +517,25 @@ func (l *levelIter) Prev() (*InternalKey, []byte) { if l.err != nil || l.iter == nil { return nil, nil } + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = false + } switch { case l.smallestBoundary != nil: + if l.tableOpts.LowerBound != nil { + // The LowerBound was within this file, so don't load the previous + // file. We leave the smallestBoundary unchanged so that + // subsequent calls to Prev() stay at this file. If a + // Seek/First/Last call is made and this file continues to be + // relevant, loadFile() will set the smallestBoundary to nil. + if l.rangeDelIterPtr != nil { + *l.rangeDelIterPtr = nil + } + return nil, nil + } // We're stepping past the boundary key, so now we can load the prev file. - if l.loadFile(l.index-1, -1) { + if l.loadFile(l.files.Prev(), -1) { if key, val := l.iter.Last(); key != nil { return l.verify(key, val) } @@ -485,19 +559,20 @@ func (l *levelIter) skipEmptyFileForward() (*InternalKey, []byte) { // The first iteration of this loop starts with an already exhausted // l.iter. The reason for the exhaustion is either that we iterated to the // end of the sstable, or our iteration was terminated early due to the - // presence of an upper-bound or the use of SeekPrefixGE. If l.rangeDelIter - // is non-nil, we may need to pretend the iterator is not exhausted to allow - // for the merging to finish consuming the l.rangeDelIter before levelIter - // switches the rangeDelIter from under it. This pretense is done by either - // generating a synthetic boundary key or returning the largest key of the - // file, depending on the exhaustion reason. + // presence of an upper-bound or the use of SeekPrefixGE. If + // l.rangeDelIterPtr is non-nil, we may need to pretend the iterator is + // not exhausted to allow for the merging to finish consuming the + // l.rangeDelIterPtr before levelIter switches the rangeDelIter from + // under it. This pretense is done by either generating a synthetic + // boundary key or returning the largest key of the file, depending on the + // exhaustion reason. // Subsequent iterations will examine consecutive files such that the first // file that does not have an exhausted iterator causes the code to return // that key, else the behavior described above if there is a corresponding - // rangeDelIter. + // rangeDelIterPtr. for ; key == nil; key, val = l.iter.First() { - if l.rangeDelIter != nil { + if l.rangeDelIterPtr != nil { // We're being used as part of a mergingIter and we've exhausted the // current sstable. If an upper bound is present and the upper bound lies // within the current sstable, then we will have reached the upper bound @@ -505,31 +580,35 @@ func (l *levelIter) skipEmptyFileForward() (*InternalKey, []byte) { // boundary key so that mergingIter can use the range tombstone iterator // until the other levels have reached this boundary. // - // It is safe to set the boundary key kind to RANGEDEL because we're - // never going to look at subsequent sstables (we've reached the upper - // bound). - f := l.files[l.index] + // It is safe to set the boundary key to the UpperBound user key + // with the RANGEDEL sentinel since it is the smallest InternalKey + // that matches the exclusive upper bound, and does not represent + // a real key. if l.tableOpts.UpperBound != nil { - // TODO(peter): Rather than using f.Largest, can we use - // l.tableOpts.UpperBound and set the seqnum to 0? We know the upper - // bound resides within the table boundaries. Not clear if this is - // kosher with respect to the invariant that only one record for a - // given user key will have seqnum 0. See Iterator.nextUserKey for an - // optimization that requires this. - l.syntheticBoundary = f.Largest - l.syntheticBoundary.SetKind(InternalKeyKindRangeDelete) - l.largestBoundary = &l.syntheticBoundary - return l.largestBoundary, nil + if *l.rangeDelIterPtr != nil { + l.syntheticBoundary.UserKey = l.tableOpts.UpperBound + l.syntheticBoundary.Trailer = InternalKeyRangeDeleteSentinel + l.largestBoundary = &l.syntheticBoundary + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = true + } + return l.largestBoundary, nil + } + // Else there are no range deletions in this sstable. This + // helps with performance when many levels are populated with + // sstables and most don't have any actual keys within the + // bounds. + return nil, nil } // If the boundary is a range deletion tombstone, return that key. - if f.Largest.Kind() == InternalKeyKindRangeDelete { - l.largestBoundary = &f.Largest + if l.iterFile.Largest.Kind() == InternalKeyKindRangeDelete { + l.largestBoundary = &l.iterFile.Largest return l.largestBoundary, nil } } // Current file was exhausted. Move to the next file. - if !l.loadFile(l.index+1, 1) { + if !l.loadFile(l.files.Next(), +1) { return nil, nil } } @@ -542,19 +621,19 @@ func (l *levelIter) skipEmptyFileBackward() (*InternalKey, []byte) { // The first iteration of this loop starts with an already exhausted // l.iter. The reason for the exhaustion is either that we iterated to the // end of the sstable, or our iteration was terminated early due to the - // presence of a lower-bound. If l.rangeDelIter is non-nil, we may need to - // pretend the iterator is not exhausted to allow for the merging to finish - // consuming the l.rangeDelIter before levelIter switches the rangeDelIter - // from under it. This pretense is done by either generating a synthetic - // boundary key or returning the smallest key of the file, depending on the - // exhaustion reason. + // presence of a lower-bound. If l.rangeDelIterPtr is non-nil, we may need + // to pretend the iterator is not exhausted to allow for the merging to + // finish consuming the l.rangeDelIterPtr before levelIter switches the + // rangeDelIter from under it. This pretense is done by either generating + // a synthetic boundary key or returning the smallest key of the file, + // depending on the exhaustion reason. // Subsequent iterations will examine consecutive files such that the first // file that does not have an exhausted iterator causes the code to return // that key, else the behavior described above if there is a corresponding - // rangeDelIter. + // rangeDelIterPtr. for ; key == nil; key, val = l.iter.Last() { - if l.rangeDelIter != nil { + if l.rangeDelIterPtr != nil { // We're being used as part of a mergingIter and we've exhausted the // current sstable. If a lower bound is present and the lower bound lies // within the current sstable, then we will have reached the lower bound @@ -562,28 +641,35 @@ func (l *levelIter) skipEmptyFileBackward() (*InternalKey, []byte) { // synthetic boundary key so that mergingIter can use the range tombstone // iterator until the other levels have reached this boundary. // - // It is safe to set the boundary key kind to RANGEDEL because we're - // never going to look at earlier sstables (we've reached the lower - // bound). - f := l.files[l.index] + // It is safe to set the boundary key to the LowerBound user key + // with the RANGEDEL sentinel since it is the smallest InternalKey + // that is within the inclusive lower bound, and does not + // represent a real key. if l.tableOpts.LowerBound != nil { - // TODO(peter): Rather than using f.Smallest, can we use - // l.tableOpts.LowerBound and set the seqnum to InternalKeySeqNumMax? - // We know the lower bound resides within the table boundaries. - l.syntheticBoundary = f.Smallest - l.syntheticBoundary.SetKind(InternalKeyKindRangeDelete) - l.smallestBoundary = &l.syntheticBoundary - return l.smallestBoundary, nil + if *l.rangeDelIterPtr != nil { + l.syntheticBoundary.UserKey = l.tableOpts.LowerBound + l.syntheticBoundary.Trailer = InternalKeyRangeDeleteSentinel + l.smallestBoundary = &l.syntheticBoundary + if l.isSyntheticIterBoundsKey != nil { + *l.isSyntheticIterBoundsKey = true + } + return l.smallestBoundary, nil + } + // Else there are no range deletions in this sstable. This + // helps with performance when many levels are populated with + // sstables and most don't have any actual keys within the + // bounds. + return nil, nil } // If the boundary is a range deletion tombstone, return that key. - if f.Smallest.Kind() == InternalKeyKindRangeDelete { - l.smallestBoundary = &f.Smallest + if l.iterFile.Smallest.Kind() == InternalKeyKindRangeDelete { + l.smallestBoundary = &l.iterFile.Smallest return l.smallestBoundary, nil } } // Current file was exhausted. Move to the previous file. - if !l.loadFile(l.index-1, -1) { + if !l.loadFile(l.files.Prev(), -1) { return nil, nil } } @@ -602,11 +688,12 @@ func (l *levelIter) Close() error { l.err = l.iter.Close() l.iter = nil } - if l.rangeDelIter != nil { - if t := *l.rangeDelIter; t != nil { + if l.rangeDelIterPtr != nil { + if t := l.rangeDelIterCopy; t != nil { l.err = firstError(l.err, t.Close()) } - *l.rangeDelIter = nil + *l.rangeDelIterPtr = nil + l.rangeDelIterCopy = nil } return l.err } @@ -621,8 +708,7 @@ func (l *levelIter) SetBounds(lower, upper []byte) { // Update tableOpts.{Lower,Upper}Bound in case the new boundaries fall within // the boundaries of the current table. - f := l.files[l.index] - if l.initTableBounds(f) != 0 { + if l.initTableBounds(l.iterFile) != 0 { // The table does not overlap the bounds. Close() will set levelIter.err if // an error occurs. _ = l.Close() @@ -633,7 +719,7 @@ func (l *levelIter) SetBounds(lower, upper []byte) { } func (l *levelIter) String() string { - if l.index >= 0 && l.index < len(l.files) { + if l.iterFile != nil { return fmt.Sprintf("%s: fileNum=%s", l.level, l.iter.String()) } return fmt.Sprintf("%s: fileNum=", l.level) diff --git a/github.com/cockroachdb/pebble/mem_table.go b/github.com/cockroachdb/pebble/mem_table.go index 92542012ac..aa4df8b103 100644 --- a/github.com/cockroachdb/pebble/mem_table.go +++ b/github.com/cockroachdb/pebble/mem_table.go @@ -60,6 +60,7 @@ var memTableEmptySize = func() uint32 { // It is safe to call get, apply, newIter, and newRangeDelIter concurrently. type memTable struct { cmp Compare + formatKey base.FormatKey equal Equal arenaBuf []byte skl arenaskl.Skiplist @@ -109,6 +110,7 @@ func newMemTable(opts memTableOptions) *memTable { m := &memTable{ cmp: opts.Comparer.Compare, + formatKey: opts.Comparer.FormatKey, equal: opts.Comparer.Equal, arenaBuf: opts.arenaBuf, writerRefs: 1, @@ -183,7 +185,7 @@ func (m *memTable) prepare(batch *Batch) error { func (m *memTable) apply(batch *Batch, seqNum uint64) error { if seqNum < m.logSeqNum { - return errors.Errorf("pebble: batch seqnum %d is less than memtable creation seqnum %d", + return base.CorruptionErrorf("pebble: batch seqnum %d is less than memtable creation seqnum %d", errors.Safe(seqNum), errors.Safe(m.logSeqNum)) } @@ -213,8 +215,8 @@ func (m *memTable) apply(batch *Batch, seqNum uint64) error { } } if seqNum != startSeqNum+uint64(batch.Count()) { - panic(errors.Errorf("pebble: inconsistent batch count: %d vs %d", - errors.Safe(seqNum), errors.Safe(startSeqNum+uint64(batch.Count())))) + return base.CorruptionErrorf("pebble: inconsistent batch count: %d vs %d", + errors.Safe(seqNum), errors.Safe(startSeqNum+uint64(batch.Count()))) } if tombstoneCount != 0 { m.tombstones.invalidate(tombstoneCount) @@ -291,7 +293,8 @@ type rangeTombstoneFrags struct { func (f *rangeTombstoneFrags) get(m *memTable) []rangedel.Tombstone { f.once.Do(func() { frag := &rangedel.Fragmenter{ - Cmp: m.cmp, + Cmp: m.cmp, + Format: m.formatKey, Emit: func(fragmented []rangedel.Tombstone) { f.tombstones = append(f.tombstones, fragmented...) }, diff --git a/github.com/cockroachdb/pebble/merging_iter.go b/github.com/cockroachdb/pebble/merging_iter.go index 8f7d3e19d9..937c7b2216 100644 --- a/github.com/cockroachdb/pebble/merging_iter.go +++ b/github.com/cockroachdb/pebble/merging_iter.go @@ -31,6 +31,7 @@ type mergingIterLevel struct { // below. smallestUserKey, largestUserKey []byte isLargestUserKeyRangeDelSentinel bool + isSyntheticIterBoundsKey bool // tombstone caches the tombstone rangeDelIter is currently pointed at. If // tombstone.Empty() is true, there are no further tombstones within the @@ -366,7 +367,33 @@ func (m *mergingIter) switchToMinHeap() { if l == cur { continue } - if l.iterKey == nil || l.iterKey.Kind() == base.InternalKeyKindRangeDelete { + + // If the iterator is exhausted, it may be out of bounds if range + // deletions modified our search key as we descended. we need to + // reposition it within the search bounds. If the current key is a + // range tombstone, the iterator might still be exhausted but at a + // sstable boundary sentinel. It would be okay to reposition an + // interator like this only through successive Next calls, except that + // it would violate the levelIter's invariants by causing it to return + // a key before the lower bound. + // + // bounds = [ f, _ ) + // L0: [ b ] [ f* z ] + // L1: [ a |----| k y ] + // L2: [ c (d) ] [ e g m ] + // L3: [ x ] + // + // * - current key [] - table bounds () - heap item + // + // In the above diagram, the L2 iterator is positioned at a sstable + // boundary (d) outside the lower bound (f). It arrived here from a + // seek whose seek-key was modified by a range tombstone. If we called + // Next on the L2 iterator, it would return e, violating its lower + // bound. Instead, we seek it to >= f and Next from there. + + if l.iterKey == nil || (m.lower != nil && l.isSyntheticIterBoundsKey && + l.iterKey.Trailer == InternalKeyRangeDeleteSentinel && + m.heap.cmp(l.iterKey.UserKey, m.lower) <= 0) { if m.lower != nil { l.iterKey, l.iterValue = l.iter.SeekGE(m.lower) } else { @@ -415,7 +442,33 @@ func (m *mergingIter) switchToMaxHeap() { if l == cur { continue } - if l.iterKey == nil || l.iterKey.Kind() == base.InternalKeyKindRangeDelete { + + // If the iterator is exhausted, it may be out of bounds if range + // deletions modified our search key as we descended. we need to + // reposition it within the search bounds. If the current key is a + // range tombstone, the iterator might still be exhausted but at a + // sstable boundary sentinel. It would be okay to reposition an + // interator like this only through successive Prev calls, except that + // it would violate the levelIter's invariants by causing it to return + // a key beyond the upper bound. + // + // bounds = [ _, g ) + // L0: [ b ] [ f* z ] + // L1: [ a |-------| k y ] + // L2: [ c d ] h [(i) m ] + // L3: [ e x ] + // + // * - current key [] - table bounds () - heap item + // + // In the above diagram, the L2 iterator is positioned at a sstable + // boundary (i) outside the upper bound (g). It arrived here from a + // seek whose seek-key was modified by a range tombstone. If we called + // Prev on the L2 iterator, it would return h, violating its upper + // bound. Instead, we seek it to < g, and Prev from there. + + if l.iterKey == nil || (m.upper != nil && l.isSyntheticIterBoundsKey && + l.iterKey.Trailer == InternalKeyRangeDeleteSentinel && + m.heap.cmp(l.iterKey.UserKey, m.upper) >= 0) { if m.upper != nil { l.iterKey, l.iterValue = l.iter.SeekLT(m.upper) } else { @@ -560,6 +613,9 @@ func (m *mergingIter) isNextEntryDeleted(item *mergingIterItem) bool { func (m *mergingIter) findNextEntry() (*InternalKey, []byte) { for m.heap.len() > 0 && m.err == nil { item := &m.heap.items[0] + if m.levels[item.index].isSyntheticIterBoundsKey { + break + } if m.isNextEntryDeleted(item) { continue } @@ -702,6 +758,9 @@ func (m *mergingIter) isPrevEntryDeleted(item *mergingIterItem) bool { func (m *mergingIter) findPrevEntry() (*InternalKey, []byte) { for m.heap.len() > 0 && m.err == nil { item := &m.heap.items[0] + if m.levels[item.index].isSyntheticIterBoundsKey { + break + } if m.isPrevEntryDeleted(item) { continue } diff --git a/github.com/cockroachdb/pebble/metrics.go b/github.com/cockroachdb/pebble/metrics.go index ada77c7b33..582e27d74a 100644 --- a/github.com/cockroachdb/pebble/metrics.go +++ b/github.com/cockroachdb/pebble/metrics.go @@ -38,7 +38,7 @@ type LevelMetrics struct { // The total number of files in the level. NumFiles int64 // The total size in bytes of the files in the level. - Size uint64 + Size int64 // The level's compaction score. Score float64 // The number of incoming bytes from other levels read during @@ -74,6 +74,8 @@ type LevelMetrics struct { // Add updates the counter metrics for the level. func (m *LevelMetrics) Add(u *LevelMetrics) { + m.NumFiles += u.NumFiles + m.Size += u.Size m.BytesIn += u.BytesIn m.BytesIngested += u.BytesIngested m.BytesMoved += u.BytesMoved @@ -100,7 +102,7 @@ func (m *LevelMetrics) WriteAmp() float64 { func (m *LevelMetrics) format(buf *bytes.Buffer, score string) { fmt.Fprintf(buf, "%9d %7s %7s %7s %7s %7s %7s %7s %7s %7s %7s %7d %7.1f\n", m.NumFiles, - humanize.IEC.Uint64(m.Size), + humanize.IEC.Int64(m.Size), score, humanize.IEC.Uint64(m.BytesIn), humanize.IEC.Uint64(m.BytesIngested), @@ -129,6 +131,10 @@ type Metrics struct { // An estimate of the number of bytes that need to be compacted for the LSM // to reach a stable state. EstimatedDebt uint64 + // Number of bytes present in sstables being written by in-progress + // compactions. This value will be zero if there are no in-progress + // compactions. + InProgressBytes int64 } Flush struct { @@ -181,6 +187,14 @@ type Metrics struct { } } +func (m *Metrics) levelSizes() [numLevels]int64 { + var sizes [numLevels]int64 + for i := 0; i < len(sizes); i++ { + sizes[i] = m.Levels[i].Size + } + return sizes +} + // ReadAmp returns the current read amplification of the database. // It's computed as the number of sublevels in L0 + the number of non-empty // levels below L0. @@ -192,6 +206,23 @@ func (m *Metrics) ReadAmp() int { return int(ramp) } +// Total returns the sum of the per-level metrics and WAL metrics. +func (m *Metrics) Total() LevelMetrics { + var total LevelMetrics + for level := 0; level < numLevels; level++ { + l := &m.Levels[level] + total.Add(l) + total.Sublevels += l.Sublevels + } + // Compute total bytes-in as the bytes written to the WAL + bytes ingested. + total.BytesIn = m.WAL.BytesWritten + total.BytesIngested + // Add the total bytes-in to the total bytes-flushed. This is to account for + // the bytes written to the log and bytes written externally and then + // ingested. + total.BytesFlushed += total.BytesIn + return total +} + const notApplicable = "-" func (m *Metrics) formatWAL(buf *bytes.Buffer) { @@ -259,10 +290,8 @@ func (m *Metrics) String() string { l.format(&buf, score) total.Add(l) total.Sublevels += l.Sublevels - total.NumFiles += l.NumFiles - total.Size += l.Size } - // Compute total bytes-in as the bytes written to the WAL + bytes ingested + // Compute total bytes-in as the bytes written to the WAL + bytes ingested. total.BytesIn = m.WAL.BytesWritten + total.BytesIngested // Add the total bytes-in to the total bytes-flushed. This is to account for // the bytes written to the log and bytes written externally and then @@ -272,10 +301,11 @@ func (m *Metrics) String() string { total.format(&buf, "-") fmt.Fprintf(&buf, " flush %9d\n", m.Flush.Count) - fmt.Fprintf(&buf, "compact %9d %7s %7s (size == estimated-debt)\n", + fmt.Fprintf(&buf, "compact %9d %7s %7s %7s (size == estimated-debt, in = in-progress-bytes)\n", m.Compact.Count, humanize.IEC.Uint64(m.Compact.EstimatedDebt), - "") + "", + humanize.IEC.Int64(m.Compact.InProgressBytes)) fmt.Fprintf(&buf, " memtbl %9d %7s\n", m.MemTable.Count, humanize.IEC.Uint64(m.MemTable.Size)) diff --git a/github.com/cockroachdb/pebble/open.go b/github.com/cockroachdb/pebble/open.go index 2d4fa707cb..7bc69f303d 100644 --- a/github.com/cockroachdb/pebble/open.go +++ b/github.com/cockroachdb/pebble/open.go @@ -12,7 +12,6 @@ import ( "os" "runtime" "sort" - "sync/atomic" "time" "github.com/cockroachdb/errors" @@ -65,7 +64,9 @@ func Open(dirname string, opts *Options) (db *DB, _ error) { abbreviatedKey: opts.Comparer.AbbreviatedKey, largeBatchThreshold: (opts.MemTableSize - int(memTableEmptySize)) / 2, logRecycler: logRecycler{limit: opts.MemTableStopWritesThreshold + 1}, + closedCh: make(chan struct{}), } + d.mu.versions = &versionSet{} defer func() { // If an error or panic occurs during open, attempt to release the manually @@ -100,13 +101,17 @@ func Open(dirname string, opts *Options) (db *DB, _ error) { d.tableCache.init(d.cacheID, dirname, opts.FS, d.opts, tableCacheSize) d.newIters = d.tableCache.newIters d.commit = newCommitPipeline(commitEnv{ - logSeqNum: &d.mu.versions.logSeqNum, - visibleSeqNum: &d.mu.versions.visibleSeqNum, + logSeqNum: &d.mu.versions.atomic.logSeqNum, + visibleSeqNum: &d.mu.versions.atomic.visibleSeqNum, apply: d.commitApply, write: d.commitWrite, }) - d.compactionLimiter = rate.NewLimiter(rate.Limit(d.opts.MinCompactionRate), d.opts.MinCompactionRate) - d.flushLimiter = rate.NewLimiter(rate.Limit(d.opts.MinFlushRate), d.opts.MinFlushRate) + d.compactionLimiter = rate.NewLimiter( + rate.Limit(d.opts.private.minCompactionRate), + d.opts.private.minCompactionRate) + d.flushLimiter = rate.NewLimiter( + rate.Limit(d.opts.private.minFlushRate), + d.opts.private.minFlushRate) d.mu.nextJobID = 1 d.mu.mem.nextSize = opts.MemTableSize if d.mu.mem.nextSize > initialMemTableSize { @@ -119,7 +124,7 @@ func Open(dirname string, opts *Options) (db *DB, _ error) { d.mu.snapshots.init() // logSeqNum is the next sequence number that will be assigned. Start // assigning sequence numbers from 1 to match rocksdb. - d.mu.versions.logSeqNum = 1 + d.mu.versions.atomic.logSeqNum = 1 d.timeNow = time.Now @@ -202,7 +207,7 @@ func Open(dirname string, opts *Options) (db *DB, _ error) { // sequence number of the first batch that will be inserted. if !d.opts.ReadOnly { var entry *flushableEntry - d.mu.mem.mutable, entry = d.newMemTable(0 /* logNum */, d.mu.versions.logSeqNum) + d.mu.mem.mutable, entry = d.newMemTable(0 /* logNum */, d.mu.versions.atomic.logSeqNum) d.mu.mem.queue = append(d.mu.mem.queue, entry) } @@ -224,11 +229,19 @@ func Open(dirname string, opts *Options) (db *DB, _ error) { name string } var logFiles []fileNumAndName + var strictWALTail bool for _, filename := range ls { ft, fn, ok := base.ParseFilename(opts.FS, filename) if !ok { continue } + + // Don't reuse any obsolete file numbers to avoid modifying an + // ingested sstable's original external file. + if d.mu.versions.nextFileNum <= fn { + d.mu.versions.nextFileNum = fn + 1 + } + switch ft { case fileTypeLog: if fn >= d.mu.versions.minUnflushedLogNum { @@ -238,7 +251,8 @@ func Open(dirname string, opts *Options) (db *DB, _ error) { d.logRecycler.minRecycleLogNum = fn + 1 } case fileTypeOptions: - if err := checkOptions(opts, opts.FS.PathJoin(dirname, filename)); err != nil { + strictWALTail, err = checkOptions(opts, opts.FS.PathJoin(dirname, filename)) + if err != nil { return nil, err } case fileTypeTemp: @@ -257,17 +271,19 @@ func Open(dirname string, opts *Options) (db *DB, _ error) { }) var ve versionEdit - for _, lf := range logFiles { - maxSeqNum, err := d.replayWAL(jobID, &ve, opts.FS, opts.FS.PathJoin(d.walDirname, lf.name), lf.num) + for i, lf := range logFiles { + lastWAL := i == len(logFiles)-1 + maxSeqNum, err := d.replayWAL(jobID, &ve, opts.FS, + opts.FS.PathJoin(d.walDirname, lf.name), lf.num, strictWALTail && !lastWAL) if err != nil { return nil, err } d.mu.versions.markFileNumUsed(lf.num) - if d.mu.versions.logSeqNum < maxSeqNum { - d.mu.versions.logSeqNum = maxSeqNum + if d.mu.versions.atomic.logSeqNum < maxSeqNum { + d.mu.versions.atomic.logSeqNum = maxSeqNum } } - d.mu.versions.visibleSeqNum = d.mu.versions.logSeqNum + d.mu.versions.atomic.visibleSeqNum = d.mu.versions.atomic.logSeqNum if !d.opts.ReadOnly { // Create an empty .log file. @@ -291,7 +307,7 @@ func Open(dirname string, opts *Options) (db *DB, _ error) { d.mu.mem.queue[len(d.mu.mem.queue)-1].logNum = newLogNum logFile = vfs.NewSyncingFile(logFile, vfs.SyncingFileOptions{ - BytesPerSync: d.opts.BytesPerSync, + BytesPerSync: d.opts.WALBytesPerSync, PreallocateSize: d.walPreallocateSize(), }) d.mu.log.LogWriter = record.NewLogWriter(logFile, newLogNum) @@ -303,7 +319,7 @@ func Open(dirname string, opts *Options) (db *DB, _ error) { // newLogNum. There should be no difference in using either value. ve.MinUnflushedLogNum = newLogNum d.mu.versions.logLock() - if err := d.mu.versions.logAndApply(jobID, &ve, nil, d.dataDir, func() []compactionInfo { + if err := d.mu.versions.logAndApply(jobID, &ve, newFileMetrics(ve.NewFiles), d.dataDir, func() []compactionInfo { return nil }); err != nil { return nil, err @@ -346,7 +362,7 @@ func Open(dirname string, opts *Options) (db *DB, _ error) { if invariants.Enabled { runtime.SetFinalizer(d, func(obj interface{}) { d := obj.(*DB) - if atomic.LoadInt32(&d.closed) == 0 { + if err := d.closed.Load(); err == nil { fmt.Fprintf(os.Stderr, "%p: unreferenced DB not closed\n", d) os.Exit(1) } @@ -420,7 +436,7 @@ func GetVersion(dir string, fs vfs.FS) (string, error) { // d.mu must be held when calling this, but the mutex may be dropped and // re-acquired during the course of this method. func (d *DB) replayWAL( - jobID int, ve *versionEdit, fs vfs.FS, filename string, logNum FileNum, + jobID int, ve *versionEdit, fs vfs.FS, filename string, logNum FileNum, strictWALTail bool, ) (maxSeqNum uint64, err error) { file, err := fs.Open(filename) if err != nil { @@ -485,17 +501,20 @@ func (d *DB) replayWAL( } if err != nil { // It is common to encounter a zeroed or invalid chunk due to WAL - // preallocation and WAL recycling. We need to distinguish these errors - // from EOF in order to recognize that the record was truncated, but want + // preallocation and WAL recycling. We need to distinguish these + // errors from EOF in order to recognize that the record was + // truncated and to avoid replaying subsequent WALs, but want // to otherwise treat them like EOF. - if err == io.EOF || record.IsInvalidRecord(err) { + if err == io.EOF { + break + } else if record.IsInvalidRecord(err) && !strictWALTail { break } return 0, errors.Wrap(err, "pebble: error when replaying WAL") } if buf.Len() < batchHeaderLen { - return 0, errors.Errorf("pebble: corrupt log file %q (num %s)", + return 0, base.CorruptionErrorf("pebble: corrupt log file %q (num %s)", filename, errors.Safe(logNum)) } @@ -548,7 +567,7 @@ func (d *DB) replayWAL( // mem is nil here. if !d.opts.ReadOnly { c := newFlush(d.opts, d.mu.versions.currentVersion(), - 1 /* base level */, toFlush, &d.bytesFlushed) + 1 /* base level */, toFlush, &d.atomic.bytesFlushed) newVE, _, err := d.runCompaction(jobID, c, nilPacer) if err != nil { return 0, err @@ -558,19 +577,19 @@ func (d *DB) replayWAL( toFlush[i].readerUnref() } } - return maxSeqNum, nil + return maxSeqNum, err } -func checkOptions(opts *Options, path string) error { +func checkOptions(opts *Options, path string) (strictWALTail bool, err error) { f, err := opts.FS.Open(path) if err != nil { - return err + return false, err } defer f.Close() data, err := ioutil.ReadAll(f) if err != nil { - return err + return false, err } - return opts.Check(string(data)) + return opts.checkOptions(string(data)) } diff --git a/github.com/cockroachdb/pebble/options.go b/github.com/cockroachdb/pebble/options.go index 9f9b272e0c..ae0ca624f2 100644 --- a/github.com/cockroachdb/pebble/options.go +++ b/github.com/cockroachdb/pebble/options.go @@ -47,15 +47,6 @@ type FilterWriter = base.FilterWriter // FilterPolicy exports the base.FilterPolicy type. type FilterPolicy = base.FilterPolicy -// TableFormat exports the base.TableFormat type. -type TableFormat = sstable.TableFormat - -// Exported TableFormat constants. -const ( - TableFormatRocksDBv2 = sstable.TableFormatRocksDBv2 - TableFormatLevelDB = sstable.TableFormatLevelDB -) - // TablePropertyCollector exports the sstable.TablePropertyCollector type. type TablePropertyCollector = sstable.TablePropertyCollector @@ -231,10 +222,11 @@ func (o *LevelOptions) EnsureDefaults() *LevelOptions { // apply to the DB at large; per-query options are defined by the IterOptions // and WriteOptions types. type Options struct { - // Sync sstables and the WAL periodically in order to smooth out writes to - // disk. This option does not provide any persistency guarantee, but is used - // to avoid latency spikes if the OS automatically decides to write out a - // large chunk of dirty filesystem buffers. + // Sync sstables periodically in order to smooth out writes to disk. This + // option does not provide any persistency guarantee, but is used to avoid + // latency spikes if the OS automatically decides to write out a large chunk + // of dirty filesystem buffers. This option only controls SSTable syncs; WAL + // syncs are controlled by WALBytesPerSync. // // The default value is 512KB. BytesPerSync int @@ -289,20 +281,34 @@ type Options struct { // out of the experimental group, or made the non-adjustable default. These // options may change at any time, so do not rely on them. Experimental struct { - // FlushSplitBytes denotes the target number of bytes in each - // flush split interval (i.e. range between two flush split keys) in - // L0 sstables. When set to zero, only a single sstable is generated + // FlushSplitBytes denotes the target number of bytes per sublevel in + // each flush split interval (i.e. range between two flush split keys) + // in L0 sstables. When set to zero, only a single sstable is generated // by each flush. When set to a non-zero value, flushes are split at // points to meet L0's TargetFileSize, any grandparent-related overlap - // options, and at boundary keys of L0 flush split intervals (each of - // which are targeted to contain around FlushSplitBytes bytes across - // all L0 sstables). Splitting sstables during flush allows increased - // compaction flexibility and concurrency when those tables are - // compacted to lower levels. + // options, and at boundary keys of L0 flush split intervals (which are + // targeted to contain around FlushSplitBytes bytes in each sublevel + // between pairs of boundary keys). Splitting sstables during flush + // allows increased compaction flexibility and concurrency when those + // tables are compacted to lower levels. // // TODO(bilal): Experiment with this option to pick a good value. FlushSplitBytes int64 + // The threshold of L0 read-amplification at which compaction concurrency + // is enabled (if CompactionDebtConcurrency was not already exceeded). + // Every multiple of this value enables another concurrent + // compaction up to MaxConcurrentCompactions. + L0CompactionConcurrency int + + // CompactionDebtConcurrency controls the threshold of compaction debt + // at which additional compaction concurrency slots are added. For every + // multiple of this value in compaction debt bytes, an additional + // concurrent compaction is added. This works "on top" of + // L0CompactionConcurrency, so the higher of the count of compaction + // concurrency slots as determined by the two options is chosen. + CompactionDebtConcurrency int + // L0SublevelCompactions enables the use of L0 sublevel-based compaction // picking logic. Defaults to false for now. This logic will become // a non-configurable default once it's better tuned. @@ -311,6 +317,12 @@ type Options struct { // L0CompactionThreshold and L0StopWritesThreshold to refer to L0 // read amplification as opposed to the count of L0 files. L0SublevelCompactions bool + + // DeleteRangeFlushDelay configures how long the database should wait + // before forcing a flush of a memtable that contains a range + // deletion. Disk space cannot be reclaimed until the range deletion + // is flushed. No automatic flush occurs if zero. + DeleteRangeFlushDelay time.Duration } // Filters is a map from filter policy name to filter policy. It is used for @@ -324,11 +336,13 @@ type Options struct { // The default value uses the underlying operating system's file system. FS vfs.FS - // The number of files necessary to trigger an L0 compaction. + // The amount of L0 read-amplification necessary to trigger an L0 compaction. L0CompactionThreshold int - // Hard limit on the number of L0 files. Writes are stopped when this - // threshold is reached. + // Hard limit on L0 read-amplification. Writes are stopped when this + // threshold is reached. If Experimental.L0SublevelCompactions is enabled + // this threshold is measured against the number of L0 sublevels. Otherwise + // it is measured against the number of files in L0. L0StopWritesThreshold int // The maximum number of bytes for LBase. The base level is the level which @@ -380,16 +394,9 @@ type Options struct { // The default merger concatenates values. Merger *Merger - // MinCompactionRate sets the minimum rate at which compactions occur. The - // default is 4 MB/s. - MinCompactionRate int - - // MinFlushRate sets the minimum rate at which the MemTables are flushed. The - // default is 1 MB/s. - MinFlushRate int - - // MaxConcurrentCompactions specifies the maximum number of concurrent compactions. The - // default is 1. + // MaxConcurrentCompactions specifies the maximum number of concurrent + // compactions. The default is 1. Concurrent compactions are only performed + // when L0 read-amplification passes the L0CompactionConcurrency threshold. MaxConcurrentCompactions int // ReadOnly indicates that the DB should be opened in read-only mode. Writes @@ -398,17 +405,22 @@ type Options struct { // disabled. ReadOnly bool - // TableFormat specifies the format version for writing sstables. The default - // is TableFormatRocksDBv2 which creates RocksDB compatible sstables. Use - // TableFormatLevelDB to create LevelDB compatible sstable which can be used - // by a wider range of tools and libraries. - TableFormat TableFormat - // TablePropertyCollectors is a list of TablePropertyCollector creation // functions. A new TablePropertyCollector is created for each sstable built // and lives for the lifetime of the table. TablePropertyCollectors []func() TablePropertyCollector + // WALBytesPerSync sets the number of bytes to write to a WAL before calling + // Sync on it in the background. Just like with BytesPerSync above, this + // helps smooth out disk write latencies, and avoids cases where the OS + // writes a lot of buffered data to disk at once. However, this is less + // necessary with WALs, as many write operations already pass in + // Sync = true. + // + // The default value is 0, i.e. no background syncing. This matches the + // default behaviour in RocksDB. + WALBytesPerSync int + // WALDir specifies the directory to store write-ahead logs (WALs) in. If // empty (the default), WALs will be stored in the same directory as sstables // (i.e. the directory passed to pebble.Open). @@ -426,8 +438,19 @@ type Options struct { // changing options dynamically? WALMinSyncInterval func() time.Duration - // private options are only used by internal tests. + // private options are only used by internal tests or are used internally + // for facilitating upgrade paths of unconfigurable functionality. private struct { + // strictWALTail configures whether or not a database's WALs created + // prior to the most recent one should be interpreted strictly, + // requiring a clean EOF. RocksDB 6.2.1 and the version of Pebble + // included in CockroachDB 20.1 do not guarantee that closed WALs end + // cleanly. If this option is set within an OPTIONS file, Pebble + // interprets previous WALs strictly, requiring a clean EOF. + // Otherwise, it interprets them permissively in the same manner as + // RocksDB 6.2.1. + strictWALTail bool + // TODO(peter): A private option to enable flush/compaction pacing. Only used // by tests. Compaction/flush pacing is disabled until we fix the impact on // throughput. @@ -438,6 +461,16 @@ type Options struct { // A private option disable automatic compactions. disableAutomaticCompactions bool + + // minCompactionRate sets the minimum rate at which compactions occur. The + // default is 4 MB/s. Currently disabled as this option has no effect while + // private.enablePacing is false. + minCompactionRate int + + // minFlushRate sets the minimum rate at which the MemTables are flushed. The + // default is 1 MB/s. Currently disabled as this option has no effect while + // private.enablePacing is false. + minFlushRate int } } @@ -463,8 +496,11 @@ func (o *Options) EnsureDefaults() *Options { if o.Comparer == nil { o.Comparer = DefaultComparer } - if o.FS == nil { - o.FS = vfs.Default + if o.Experimental.L0CompactionConcurrency <= 0 { + o.Experimental.L0CompactionConcurrency = 10 + } + if o.Experimental.CompactionDebtConcurrency <= 0 { + o.Experimental.CompactionDebtConcurrency = 1 << 30 // 1 GB } if o.L0CompactionThreshold <= 0 { o.L0CompactionThreshold = 4 @@ -510,15 +546,25 @@ func (o *Options) EnsureDefaults() *Options { if o.Merger == nil { o.Merger = DefaultMerger } - if o.MinCompactionRate == 0 { - o.MinCompactionRate = 4 << 20 // 4 MB/s + o.private.strictWALTail = true + if o.private.minCompactionRate == 0 { + o.private.minCompactionRate = 4 << 20 // 4 MB/s } - if o.MinFlushRate == 0 { - o.MinFlushRate = 1 << 20 // 1 MB/s + if o.private.minFlushRate == 0 { + o.private.minFlushRate = 1 << 20 // 1 MB/s } if o.MaxConcurrentCompactions <= 0 { o.MaxConcurrentCompactions = 1 } + if o.FS == nil { + o.FS = vfs.WithDiskHealthChecks(vfs.Default, 5*time.Second, + func(name string, duration time.Duration) { + o.EventListener.DiskSlow(DiskSlowInfo{ + Path: name, + Duration: duration, + }) + }) + } o.initMaps() return o @@ -585,19 +631,23 @@ func (o *Options) String() string { fmt.Fprintf(&buf, " cache_size=%d\n", cacheSize) fmt.Fprintf(&buf, " cleaner=%s\n", o.Cleaner) fmt.Fprintf(&buf, " comparer=%s\n", o.Comparer.Name) + fmt.Fprintf(&buf, " delete_range_flush_delay=%s\n", o.Experimental.DeleteRangeFlushDelay) fmt.Fprintf(&buf, " disable_wal=%t\n", o.DisableWAL) fmt.Fprintf(&buf, " flush_split_bytes=%d\n", o.Experimental.FlushSplitBytes) + fmt.Fprintf(&buf, " l0_compaction_concurrency=%d\n", o.Experimental.L0CompactionConcurrency) fmt.Fprintf(&buf, " l0_compaction_threshold=%d\n", o.L0CompactionThreshold) fmt.Fprintf(&buf, " l0_stop_writes_threshold=%d\n", o.L0StopWritesThreshold) + fmt.Fprintf(&buf, " l0_sublevel_compactions=%t\n", o.Experimental.L0SublevelCompactions) fmt.Fprintf(&buf, " lbase_max_bytes=%d\n", o.LBaseMaxBytes) fmt.Fprintf(&buf, " max_concurrent_compactions=%d\n", o.MaxConcurrentCompactions) fmt.Fprintf(&buf, " max_manifest_file_size=%d\n", o.MaxManifestFileSize) fmt.Fprintf(&buf, " max_open_files=%d\n", o.MaxOpenFiles) fmt.Fprintf(&buf, " mem_table_size=%d\n", o.MemTableSize) fmt.Fprintf(&buf, " mem_table_stop_writes_threshold=%d\n", o.MemTableStopWritesThreshold) - fmt.Fprintf(&buf, " min_compaction_rate=%d\n", o.MinCompactionRate) - fmt.Fprintf(&buf, " min_flush_rate=%d\n", o.MinFlushRate) + fmt.Fprintf(&buf, " min_compaction_rate=%d\n", o.private.minCompactionRate) + fmt.Fprintf(&buf, " min_flush_rate=%d\n", o.private.minFlushRate) fmt.Fprintf(&buf, " merger=%s\n", o.Merger.Name) + fmt.Fprintf(&buf, " strict_wal_tail=%t\n", o.private.strictWALTail) fmt.Fprintf(&buf, " table_property_collectors=[") for i := range o.TablePropertyCollectors { if i > 0 { @@ -609,6 +659,7 @@ func (o *Options) String() string { } fmt.Fprintf(&buf, "]\n") fmt.Fprintf(&buf, " wal_dir=%s\n", o.WALDir) + fmt.Fprintf(&buf, " wal_bytes_per_sync=%d\n", o.WALBytesPerSync) for i := range o.Levels { l := &o.Levels[i] @@ -709,13 +760,16 @@ func (o *Options) Parse(s string, hooks *ParseHooks) error { case "bytes_per_sync": o.BytesPerSync, err = strconv.Atoi(value) case "cache_size": - n, err := strconv.ParseInt(value, 10, 64) - if err == nil && hooks.NewCache != nil { + var n int64 + n, err = strconv.ParseInt(value, 10, 64) + if err == nil && hooks != nil && hooks.NewCache != nil { if o.Cache != nil { o.Cache.Unref() } o.Cache = hooks.NewCache(n) } + // We avoid calling cache.New in parsing because it makes it + // too easy to leak a cache. case "cleaner": switch value { case "archive": @@ -736,14 +790,20 @@ func (o *Options) Parse(s string, hooks *ParseHooks) error { o.Comparer, err = hooks.NewComparer(value) } } + case "delete_range_flush_delay": + o.Experimental.DeleteRangeFlushDelay, err = time.ParseDuration(value) case "disable_wal": o.DisableWAL, err = strconv.ParseBool(value) case "flush_split_bytes": o.Experimental.FlushSplitBytes, err = strconv.ParseInt(value, 10, 64) + case "l0_compaction_concurrency": + o.Experimental.L0CompactionConcurrency, err = strconv.Atoi(value) case "l0_compaction_threshold": o.L0CompactionThreshold, err = strconv.Atoi(value) case "l0_stop_writes_threshold": o.L0StopWritesThreshold, err = strconv.Atoi(value) + case "l0_sublevel_compactions": + o.Experimental.L0SublevelCompactions, err = strconv.ParseBool(value) case "lbase_max_bytes": o.LBaseMaxBytes, err = strconv.ParseInt(value, 10, 64) case "max_concurrent_compactions": @@ -757,9 +817,11 @@ func (o *Options) Parse(s string, hooks *ParseHooks) error { case "mem_table_stop_writes_threshold": o.MemTableStopWritesThreshold, err = strconv.Atoi(value) case "min_compaction_rate": - o.MinCompactionRate, err = strconv.Atoi(value) + o.private.minCompactionRate, err = strconv.Atoi(value) case "min_flush_rate": - o.MinFlushRate, err = strconv.Atoi(value) + o.private.minFlushRate, err = strconv.Atoi(value) + case "strict_wal_tail": + o.private.strictWALTail, err = strconv.ParseBool(value) case "merger": switch value { case "nullptr": @@ -774,9 +836,7 @@ func (o *Options) Parse(s string, hooks *ParseHooks) error { case "table_format": switch value { case "leveldb": - o.TableFormat = TableFormatLevelDB case "rocksdbv2": - o.TableFormat = TableFormatRocksDBv2 default: return errors.Errorf("pebble: unknown table format: %q", errors.Safe(value)) } @@ -784,6 +844,8 @@ func (o *Options) Parse(s string, hooks *ParseHooks) error { // TODO(peter): set o.TablePropertyCollectors case "wal_dir": o.WALDir = value + case "wal_bytes_per_sync": + o.WALBytesPerSync, err = strconv.Atoi(value) default: if hooks != nil && hooks.SkipUnknown != nil && hooks.SkipUnknown(section+"."+key) { return nil @@ -858,11 +920,9 @@ func (o *Options) Parse(s string, hooks *ParseHooks) error { }) } -// Check verifies the options are compatible with the previous options -// serialized by Options.String(). For example, the Comparer and Merger must be -// the same, or data will not be able to be properly read from the DB. -func (o *Options) Check(s string) error { - return parseOptions(s, func(section, key, value string) error { +func (o *Options) checkOptions(s string) (strictWALTail bool, err error) { + // TODO(jackson): Refactor to avoid awkwardness of the strictWALTail return value. + return strictWALTail, parseOptions(s, func(section, key, value string) error { switch section + "." + key { case "Options.comparer": if value != o.Comparer.Name { @@ -876,11 +936,24 @@ func (o *Options) Check(s string) error { return errors.Errorf("pebble: merger name from file %q != merger name from options %q", errors.Safe(value), errors.Safe(o.Merger.Name)) } + case "Options.strict_wal_tail": + strictWALTail, err = strconv.ParseBool(value) + if err != nil { + return errors.Errorf("pebble: error parsing strict_wal_tail value %q: %w", value, err) + } } return nil }) } +// Check verifies the options are compatible with the previous options +// serialized by Options.String(). For example, the Comparer and Merger must be +// the same, or data will not be able to be properly read from the DB. +func (o *Options) Check(s string) error { + _, err := o.checkOptions(s) + return err +} + // Validate verifies that the options are mutually consistent. For example, // L0StopWritesThreshold must be >= L0CompactionThreshold, otherwise a write // stall would persist indefinitely. @@ -889,11 +962,15 @@ func (o *Options) Validate() error { // is no need to check for zero values. var buf strings.Builder + if o.Experimental.L0CompactionConcurrency < 1 { + fmt.Fprintf(&buf, "L0CompactionConcurrency (%d) must be >= 1\n", + o.Experimental.L0CompactionConcurrency) + } if o.L0StopWritesThreshold < o.L0CompactionThreshold { fmt.Fprintf(&buf, "L0StopWritesThreshold (%d) must be >= L0CompactionThreshold (%d)\n", o.L0StopWritesThreshold, o.L0CompactionThreshold) } - if o.MemTableSize >= maxMemTableSize { + if uint64(o.MemTableSize) >= maxMemTableSize { fmt.Fprintf(&buf, "MemTableSize (%s) must be < %s\n", humanize.Uint64(uint64(o.MemTableSize)), humanize.Uint64(maxMemTableSize)) } @@ -901,10 +978,6 @@ func (o *Options) Validate() error { fmt.Fprintf(&buf, "MemTableStopWritesThreshold (%d) must be >= 2\n", o.MemTableStopWritesThreshold) } - switch o.TableFormat { - case TableFormatLevelDB: - fmt.Fprintf(&buf, "TableFormatLevelDB not supported for DB\n") - } if buf.Len() == 0 { return nil } @@ -936,7 +1009,7 @@ func (o *Options) MakeWriterOptions(level int) sstable.WriterOptions { if o.Merger != nil { writerOpts.MergerName = o.Merger.Name } - writerOpts.TableFormat = o.TableFormat + writerOpts.TableFormat = sstable.TableFormatRocksDBv2 writerOpts.TablePropertyCollectors = o.TablePropertyCollectors } levelOpts := o.Level(level) diff --git a/github.com/cockroachdb/pebble/snapshot.go b/github.com/cockroachdb/pebble/snapshot.go index 09ffd6b010..76aae30c6b 100644 --- a/github.com/cockroachdb/pebble/snapshot.go +++ b/github.com/cockroachdb/pebble/snapshot.go @@ -4,7 +4,10 @@ package pebble -import "io" +import ( + "io" + "math" +) // Snapshot provides a read-only point-in-time view of the DB state. type Snapshot struct { @@ -45,16 +48,22 @@ func (s *Snapshot) NewIter(o *IterOptions) *Iterator { return s.db.newIterInternal(nil /* batchIter */, nil /* batchRangeDelIter */, s, o) } -// Close closes the snapshot, releasing its resources. Close must be -// called. Failure to do so while result in a tiny memory leak, and a large -// leak of resources on disk due to the entries the snapshot is preventing from -// being deleted. +// Close closes the snapshot, releasing its resources. Close must be called. +// Failure to do so will result in a tiny memory leak and a large leak of +// resources on disk due to the entries the snapshot is preventing from being +// deleted. func (s *Snapshot) Close() error { if s.db == nil { panic(ErrClosed) } s.db.mu.Lock() s.db.mu.snapshots.remove(s) + + // If s was the previous earliest snapshot, we might be able to reclaim + // disk space by dropping obsolete records that were pinned by s. + if e := s.db.mu.snapshots.earliest(); e > s.seqNum { + s.db.maybeScheduleCompactionPicker(pickElisionOnly) + } s.db.mu.Unlock() s.db = nil return nil @@ -73,6 +82,14 @@ func (l *snapshotList) empty() bool { return l.root.next == &l.root } +func (l *snapshotList) earliest() uint64 { + v := uint64(math.MaxUint64) + if !l.empty() { + v = l.root.next.seqNum + } + return v +} + func (l *snapshotList) toSlice() []uint64 { if l.empty() { return nil diff --git a/github.com/cockroachdb/pebble/sstable/block.go b/github.com/cockroachdb/pebble/sstable/block.go index 42dfb355a5..846056b4f6 100644 --- a/github.com/cockroachdb/pebble/sstable/block.go +++ b/github.com/cockroachdb/pebble/sstable/block.go @@ -8,7 +8,6 @@ import ( "encoding/binary" "unsafe" - "github.com/cockroachdb/errors" "github.com/cockroachdb/pebble/internal/base" "github.com/cockroachdb/pebble/internal/cache" ) @@ -256,7 +255,7 @@ func (i *blockIter) String() string { func (i *blockIter) init(cmp Compare, block block, globalSeqNum uint64) error { numRestarts := int32(binary.LittleEndian.Uint32(block[len(block)-4:])) if numRestarts == 0 { - return errors.New("pebble/table: invalid table (block has no restart points)") + return base.CorruptionErrorf("pebble/table: invalid table (block has no restart points)") } i.cmp = cmp i.restarts = int32(len(block)) - 4*(1+numRestarts) @@ -305,61 +304,58 @@ func (i *blockIter) readEntry() { // TODO(peter): remove this hack if go:inline is ever supported. var shared uint32 - src := (*[5]uint8)(ptr) - if a := (*src)[0]; a < 128 { + if a := *((*uint8)(ptr)); a < 128 { shared = uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 1) - } else if a, b := a&0x7f, (*src)[1]; b < 128 { + } else if a, b := a&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 1))); b < 128 { shared = uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 2) - } else if b, c := b&0x7f, (*src)[2]; c < 128 { + } else if b, c := b&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 2))); c < 128 { shared = uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 3) - } else if c, d := c&0x7f, (*src)[3]; d < 128 { + } else if c, d := c&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 3))); d < 128 { shared = uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 4) } else { - d, e := d&0x7f, (*src)[4] + d, e := d&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 4))) shared = uint32(e)<<28 | uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 5) } var unshared uint32 - src = (*[5]uint8)(ptr) - if a := (*src)[0]; a < 128 { + if a := *((*uint8)(ptr)); a < 128 { unshared = uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 1) - } else if a, b := a&0x7f, (*src)[1]; b < 128 { + } else if a, b := a&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 1))); b < 128 { unshared = uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 2) - } else if b, c := b&0x7f, (*src)[2]; c < 128 { + } else if b, c := b&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 2))); c < 128 { unshared = uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 3) - } else if c, d := c&0x7f, (*src)[3]; d < 128 { + } else if c, d := c&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 3))); d < 128 { unshared = uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 4) } else { - d, e := d&0x7f, (*src)[4] + d, e := d&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 4))) unshared = uint32(e)<<28 | uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 5) } var value uint32 - src = (*[5]uint8)(ptr) - if a := (*src)[0]; a < 128 { + if a := *((*uint8)(ptr)); a < 128 { value = uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 1) - } else if a, b := a&0x7f, (*src)[1]; b < 128 { + } else if a, b := a&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 1))); b < 128 { value = uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 2) - } else if b, c := b&0x7f, (*src)[2]; c < 128 { + } else if b, c := b&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 2))); c < 128 { value = uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 3) - } else if c, d := c&0x7f, (*src)[3]; d < 128 { + } else if c, d := c&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 3))); d < 128 { value = uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 4) } else { - d, e := d&0x7f, (*src)[4] + d, e := d&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 4))) value = uint32(e)<<28 | uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 5) } @@ -447,32 +443,31 @@ func (i *blockIter) SeekGE(key []byte) (*InternalKey, []byte) { // sought. See the comment in readEntry for why we manually inline the // varint decoding. var v1 uint32 - src := (*[5]uint8)(ptr) - if a := (*src)[0]; a < 128 { + if a := *((*uint8)(ptr)); a < 128 { v1 = uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 1) - } else if a, b := a&0x7f, (*src)[1]; b < 128 { + } else if a, b := a&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 1))); b < 128 { v1 = uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 2) - } else if b, c := b&0x7f, (*src)[2]; c < 128 { + } else if b, c := b&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 2))); c < 128 { v1 = uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 3) - } else if c, d := c&0x7f, (*src)[3]; d < 128 { + } else if c, d := c&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 3))); d < 128 { v1 = uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 4) } else { - d, e := d&0x7f, (*src)[4] + d, e := d&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 4))) v1 = uint32(e)<<28 | uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 5) } - if src := (*[5]uint8)(ptr); (*src)[0] < 128 { + if *((*uint8)(ptr)) < 128 { ptr = unsafe.Pointer(uintptr(ptr) + 1) - } else if (*src)[1] < 128 { + } else if *((*uint8)(unsafe.Pointer(uintptr(ptr) + 1))) < 128 { ptr = unsafe.Pointer(uintptr(ptr) + 2) - } else if (*src)[2] < 128 { + } else if *((*uint8)(unsafe.Pointer(uintptr(ptr) + 2))) < 128 { ptr = unsafe.Pointer(uintptr(ptr) + 3) - } else if (*src)[3] < 128 { + } else if *((*uint8)(unsafe.Pointer(uintptr(ptr) + 3))) < 128 { ptr = unsafe.Pointer(uintptr(ptr) + 4) } else { ptr = unsafe.Pointer(uintptr(ptr) + 5) @@ -559,32 +554,31 @@ func (i *blockIter) SeekLT(key []byte) (*InternalKey, []byte) { // sought. See the comment in readEntry for why we manually inline the // varint decoding. var v1 uint32 - src := (*[5]uint8)(ptr) - if a := (*src)[0]; a < 128 { + if a := *((*uint8)(ptr)); a < 128 { v1 = uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 1) - } else if a, b := a&0x7f, (*src)[1]; b < 128 { + } else if a, b := a&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 1))); b < 128 { v1 = uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 2) - } else if b, c := b&0x7f, (*src)[2]; c < 128 { + } else if b, c := b&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 2))); c < 128 { v1 = uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 3) - } else if c, d := c&0x7f, (*src)[3]; d < 128 { + } else if c, d := c&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 3))); d < 128 { v1 = uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 4) } else { - d, e := d&0x7f, (*src)[4] + d, e := d&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 4))) v1 = uint32(e)<<28 | uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a) ptr = unsafe.Pointer(uintptr(ptr) + 5) } - if src := (*[5]uint8)(ptr); (*src)[0] < 128 { + if *((*uint8)(ptr)) < 128 { ptr = unsafe.Pointer(uintptr(ptr) + 1) - } else if (*src)[1] < 128 { + } else if *((*uint8)(unsafe.Pointer(uintptr(ptr) + 1))) < 128 { ptr = unsafe.Pointer(uintptr(ptr) + 2) - } else if (*src)[2] < 128 { + } else if *((*uint8)(unsafe.Pointer(uintptr(ptr) + 2))) < 128 { ptr = unsafe.Pointer(uintptr(ptr) + 3) - } else if (*src)[3] < 128 { + } else if *((*uint8)(unsafe.Pointer(uintptr(ptr) + 3))) < 128 { ptr = unsafe.Pointer(uintptr(ptr) + 4) } else { ptr = unsafe.Pointer(uintptr(ptr) + 5) diff --git a/github.com/cockroachdb/pebble/sstable/internal.go b/github.com/cockroachdb/pebble/sstable/internal.go index 8ec23d9131..20ab1f6946 100644 --- a/github.com/cockroachdb/pebble/sstable/internal.go +++ b/github.com/cockroachdb/pebble/sstable/internal.go @@ -4,9 +4,7 @@ package sstable -import ( - "github.com/cockroachdb/pebble/internal/base" -) +import "github.com/cockroachdb/pebble/internal/base" // InternalKeyKind exports the base.InternalKeyKind type. type InternalKeyKind = base.InternalKeyKind diff --git a/github.com/cockroachdb/pebble/sstable/properties.go b/github.com/cockroachdb/pebble/sstable/properties.go index 611c13e7d3..3457dca97c 100644 --- a/github.com/cockroachdb/pebble/sstable/properties.go +++ b/github.com/cockroachdb/pebble/sstable/properties.go @@ -105,7 +105,8 @@ type Properties struct { MergerName string `prop:"rocksdb.merge.operator"` // The number of blocks in this table. NumDataBlocks uint64 `prop:"rocksdb.num.data.blocks"` - // The number of deletion entries in this table. + // The number of deletion entries in this table, including both point and + // range deletions. NumDeletions uint64 `prop:"rocksdb.deleted.keys"` // The number of entries in this table. NumEntries uint64 `prop:"rocksdb.num.entries"` diff --git a/github.com/cockroachdb/pebble/sstable/raw_block.go b/github.com/cockroachdb/pebble/sstable/raw_block.go index 1c9bd2cd09..82584dcbe0 100644 --- a/github.com/cockroachdb/pebble/sstable/raw_block.go +++ b/github.com/cockroachdb/pebble/sstable/raw_block.go @@ -9,7 +9,7 @@ import ( "sort" "unsafe" - "github.com/cockroachdb/errors" + "github.com/cockroachdb/pebble/internal/base" ) type rawBlockWriter struct { @@ -57,7 +57,7 @@ func newRawBlockIter(cmp Compare, block block) (*rawBlockIter, error) { func (i *rawBlockIter) init(cmp Compare, block block) error { numRestarts := int32(binary.LittleEndian.Uint32(block[len(block)-4:])) if numRestarts == 0 { - return errors.New("pebble/table: invalid table (block has no restart points)") + return base.CorruptionErrorf("pebble/table: invalid table (block has no restart points)") } i.cmp = cmp i.restarts = int32(len(block)) - 4*(1+numRestarts) diff --git a/github.com/cockroachdb/pebble/sstable/reader.go b/github.com/cockroachdb/pebble/sstable/reader.go index 0ae5973267..7e1aa3aac6 100644 --- a/github.com/cockroachdb/pebble/sstable/reader.go +++ b/github.com/cockroachdb/pebble/sstable/reader.go @@ -26,7 +26,7 @@ import ( "github.com/golang/snappy" ) -var errCorruptIndexEntry = errors.New("pebble/table: corrupt index entry") +var errCorruptIndexEntry = base.CorruptionErrorf("pebble/table: corrupt index entry") const ( // Constants for dynamic readahead of data blocks. Note that the size values @@ -35,8 +35,8 @@ const ( minFileReadsForReadahead = 2 // TODO(bilal): Have the initial size value be a factor of the block size, // as opposed to a hardcoded value. - initialReadaheadSize = 64 << 10 /* 64KB */ - maxReadaheadSize = 512 << 10 /* 512KB */ + initialReadaheadSize = 64 << 10 /* 64KB */ + maxReadaheadSize = 256 << 10 /* 256KB */ ) // decodeBlockHandle returns the block handle encoded at the start of src, as @@ -162,6 +162,19 @@ func (i *singleLevelIterator) init(r *Reader, lower, upper []byte) error { return nil } +// setupForCompaction sets up the singleLevelIterator for use with compactionIter. +// Currently, it skips readahead ramp-up. It should be called after init is called. +func (i *singleLevelIterator) setupForCompaction() { + if i.reader.fs != nil { + f, err := i.reader.fs.Open(i.reader.filename, vfs.SequentialReadsOption) + if err == nil { + // Given that this iterator is for a compaction, we can assume that it + // will be read sequentially and we can skip the readahead ramp-up. + i.dataRS.sequentialFile = f + } + } +} + func (i *singleLevelIterator) resetForReuse() singleLevelIterator { return singleLevelIterator{ index: i.index.resetForReuse(), @@ -419,13 +432,17 @@ func (i *singleLevelIterator) skipForward() (*InternalKey, []byte) { i.data.invalidate() break } - if i.loadBlock() { - if key, val := i.data.First(); key != nil { - if i.blockUpper != nil && i.cmp(key.UserKey, i.blockUpper) >= 0 { - return nil, nil - } - return key, val + if !i.loadBlock() { + if i.err != nil { + break + } + continue + } + if key, val := i.data.First(); key != nil { + if i.blockUpper != nil && i.cmp(key.UserKey, i.blockUpper) >= 0 { + return nil, nil } + return key, val } } return nil, nil @@ -437,16 +454,20 @@ func (i *singleLevelIterator) skipBackward() (*InternalKey, []byte) { i.data.invalidate() break } - if i.loadBlock() { - key, val := i.data.Last() - if key == nil { - return nil, nil - } - if i.blockLower != nil && i.cmp(key.UserKey, i.blockLower) < 0 { - return nil, nil + if !i.loadBlock() { + if i.err != nil { + break } - return key, val + continue + } + key, val := i.data.Last() + if key == nil { + return nil, nil } + if i.blockLower != nil && i.cmp(key.UserKey, i.blockLower) < 0 { + return nil, nil + } + return key, val } return nil, nil } @@ -490,6 +511,10 @@ func (i *singleLevelIterator) Close() error { } err = firstError(err, i.data.Close()) err = firstError(err, i.index.Close()) + if i.dataRS.sequentialFile != nil { + err = firstError(err, i.dataRS.sequentialFile.Close()) + i.dataRS.sequentialFile = nil + } err = firstError(err, i.err) *i = i.resetForReuse() singleLevelIterPool.Put(i) @@ -564,10 +589,14 @@ func (i *compactionIterator) skipForward(key *InternalKey, val []byte) (*Interna if key, _ := i.index.Next(); key == nil { break } - if i.loadBlock() { - if key, val = i.data.First(); key != nil { + if !i.loadBlock() { + if i.err != nil { break } + continue + } + if key, val = i.data.First(); key != nil { + break } } } @@ -601,7 +630,7 @@ func (i *twoLevelIterator) loadIndex() bool { } h, n := decodeBlockHandle(i.topLevelIndex.Value()) if n == 0 || n != len(i.topLevelIndex.Value()) { - i.err = errors.New("pebble/table: corrupt top level index entry") + i.err = base.CorruptionErrorf("pebble/table: corrupt top level index entry") return false } indexBlock, err := i.reader.readBlock(h, nil /* transform */, nil /* readaheadState */) @@ -783,6 +812,9 @@ func (i *twoLevelIterator) Prev() (*InternalKey, []byte) { func (i *twoLevelIterator) skipForward() (*InternalKey, []byte) { for { + if i.err != nil { + return nil, nil + } if i.singleLevelIterator.valid() { // The iterator is positioned at valid record in the current data block // which implies the previous positioning call reached the upper bound. @@ -804,6 +836,9 @@ func (i *twoLevelIterator) skipForward() (*InternalKey, []byte) { func (i *twoLevelIterator) skipBackward() (*InternalKey, []byte) { for { + if i.err != nil { + return nil, nil + } if i.singleLevelIterator.valid() { // The iterator is positioned at valid record in the current data block // which implies the previous positioning call reached the lower bound. @@ -833,6 +868,10 @@ func (i *twoLevelIterator) Close() error { err = firstError(err, i.data.Close()) err = firstError(err, i.index.Close()) err = firstError(err, i.topLevelIndex.Close()) + if i.dataRS.sequentialFile != nil { + err = firstError(err, i.dataRS.sequentialFile.Close()) + i.dataRS.sequentialFile = nil + } err = firstError(err, i.err) *i = twoLevelIterator{ singleLevelIterator: i.singleLevelIterator.resetForReuse(), @@ -934,23 +973,66 @@ type readaheadState struct { // operation. Reads after this limit can benefit from a new call to // Prefetch. limit int64 + // sequentialFile holds a file descriptor to the same underlying File, + // except with fadvise(FADV_SEQUENTIAL) called on it to take advantage of + // OS-level readahead. Initialized when the iterator has been consistently + // reading blocks in a sequential access pattern. Once this is non-nil, + // the other variables in readaheadState don't matter much as we defer + // to OS-level readahead. + sequentialFile vfs.File +} + +func (rs *readaheadState) recordCacheHit(offset, blockLength int64) { + currentReadEnd := offset + blockLength + if rs.sequentialFile != nil { + // Using OS-level readahead instead, so do nothing. + return + } + if rs.numReads >= minFileReadsForReadahead { + if currentReadEnd >= rs.limit && offset <= rs.limit+maxReadaheadSize { + // This is a read that would have resulted in a readahead, had it + // not been a cache hit. + rs.limit = currentReadEnd + return + } + if currentReadEnd < rs.limit-rs.prevSize || offset > rs.limit+maxReadaheadSize { + // We read too far away from rs.limit to benefit from readahead in + // any scenario. Reset all variables. + rs.numReads = 1 + rs.limit = currentReadEnd + rs.size = initialReadaheadSize + rs.prevSize = 0 + return + } + // Reads in the range [rs.limit - rs.prevSize, rs.limit] end up + // here. This is a read that is potentially benefitting from a past + // readahead. + return + } + if currentReadEnd >= rs.limit && offset <= rs.limit+maxReadaheadSize { + // Blocks are being read sequentially and would benefit from readahead + // down the line. + rs.numReads++ + return + } + // We read too far ahead of the last read, or before it. This indicates + // a random read, where readahead is not desirable. Reset all variables. + rs.numReads = 1 + rs.limit = currentReadEnd + rs.size = initialReadaheadSize + rs.prevSize = 0 } // maybeReadahead updates state and determines whether to issue a readahead / // prefetch call for a block read at offset for blockLength bytes. // Returns a size value (greater than 0) that should be prefetched if readahead // would be beneficial. -// -// TODO(bilal): This method is only called when there's a cache miss. Reads -// from the cache are completely invisible to the logic here and the state -// variables in readaheadState. This is a big reason why the -// readahead window in this method needs to be optimistic (that reading ahead -// will generally help) by always extending the window to -// rs.limit + maxReadaheadSize, to adjust for blocks that appear to be skipped -// but were actually read from the cache. Update this method or add another -// method to properly account for cache hits. func (rs *readaheadState) maybeReadahead(offset, blockLength int64) int64 { currentReadEnd := offset + blockLength + if rs.sequentialFile != nil { + // Using OS-level readahead instead, so do nothing. + return 0 + } if rs.numReads >= minFileReadsForReadahead { // The minimum threshold of sequential reads to justify reading ahead // has been reached. @@ -963,7 +1045,7 @@ func (rs *readaheadState) maybeReadahead(offset, blockLength int64) int64 { // readahead may not be beneficial with a small readahead size, but over // time the readahead size would increase exponentially to make it // beneficial. - if currentReadEnd >= rs.limit && offset <= rs.limit + maxReadaheadSize { + if currentReadEnd >= rs.limit && offset <= rs.limit+maxReadaheadSize { // We are doing a read in the interval ahead of // the last readahead range. In the diagrams below, ++++ is the last // readahead range, ==== is the range represented by @@ -997,7 +1079,7 @@ func (rs *readaheadState) maybeReadahead(offset, blockLength int64) int64 { } return rs.prevSize } - if currentReadEnd < rs.limit - rs.prevSize || offset > rs.limit + maxReadaheadSize { + if currentReadEnd < rs.limit-rs.prevSize || offset > rs.limit+maxReadaheadSize { // The above conditional has rs.limit > rs.prevSize to confirm that // rs.limit - rs.prevSize would not underflow. // We read too far away from rs.limit to benefit from readahead in @@ -1039,7 +1121,7 @@ func (rs *readaheadState) maybeReadahead(offset, blockLength int64) int64 { rs.numReads++ return 0 } - if currentReadEnd >= rs.limit && offset <= rs.limit + maxReadaheadSize { + if currentReadEnd >= rs.limit && offset <= rs.limit+maxReadaheadSize { // Blocks are being read sequentially and would benefit from readahead // down the line. // @@ -1088,6 +1170,7 @@ func (c Comparers) readerApply(r *Reader) { } if comparer, ok := c[r.Properties.ComparerName]; ok { r.Compare = comparer.Compare + r.FormatKey = comparer.FormatKey r.Split = comparer.Split } } @@ -1135,6 +1218,20 @@ func (c *cacheOpts) writerApply(w *Writer) { } } +// FileReopenOpt is specified if this reader is allowed to reopen additional +// file descriptors for this file. Used to take advantage of OS-level readahead. +type FileReopenOpt struct { + FS vfs.FS + Filename string +} + +func (f FileReopenOpt) readerApply(r *Reader) { + if r.fs == nil { + r.fs = f.FS + r.filename = f.Filename + } +} + // rawTombstonesOpt is a Reader open option for specifying that range // tombstones returned by Reader.NewRangeDelIter() should not be // fragmented. Used by debug tools to get a raw view of the tombstones @@ -1157,6 +1254,8 @@ func init() { // Reader is a table reader. type Reader struct { file vfs.File + fs vfs.FS + filename string cacheID uint64 fileNum base.FileNum rawTombstones bool @@ -1170,6 +1269,7 @@ type Reader struct { footerBH BlockHandle opts ReaderOptions Compare Compare + FormatKey base.FormatKey Split Split mergerOK bool tableFilter *tableFilterReader @@ -1281,6 +1381,7 @@ func (r *Reader) NewCompactionIter(bytesIterated *uint64) (Iterator, error) { if err != nil { return nil, err } + i.setupForCompaction() return &twoLevelCompactionIterator{ twoLevelIterator: i, bytesIterated: bytesIterated, @@ -1291,16 +1392,17 @@ func (r *Reader) NewCompactionIter(bytesIterated *uint64) (Iterator, error) { if err != nil { return nil, err } + i.setupForCompaction() return &compactionIterator{ singleLevelIterator: i, bytesIterated: bytesIterated, }, nil } -// NewRangeDelIter returns an internal iterator for the contents of the -// range-del block for the table. Returns nil if the table does not contain any -// range deletions. -func (r *Reader) NewRangeDelIter() (base.InternalIterator, error) { +// NewRawRangeDelIter returns an internal iterator for the contents of the +// range-del block for the table. Returns nil if the table does not contain +// any range deletions. +func (r *Reader) NewRawRangeDelIter() (base.InternalIterator, error) { if r.rangeDelBH.Length == 0 { return nil, nil } @@ -1328,20 +1430,46 @@ func (r *Reader) readRangeDel() (cache.Handle, error) { } // readBlock reads and decompresses a block from disk into memory. -func (r *Reader) readBlock(bh BlockHandle, transform blockTransform, raState *readaheadState) (cache.Handle, error) { +func (r *Reader) readBlock( + bh BlockHandle, transform blockTransform, raState *readaheadState, +) (cache.Handle, error) { if h := r.opts.Cache.Get(r.cacheID, r.fileNum, bh.Offset); h.Get() != nil { + if raState != nil { + raState.recordCacheHit(int64(bh.Offset), int64(bh.Length+blockTrailerLen)) + } return h, nil } + file := r.file if raState != nil { - if readaheadSize := raState.maybeReadahead(int64(bh.Offset), int64(bh.Length + blockTrailerLen)); readaheadSize > 0 { - _ = vfs.Prefetch(r.file, bh.Offset, uint64(readaheadSize)) + if raState.sequentialFile != nil { + file = raState.sequentialFile + } else if readaheadSize := raState.maybeReadahead(int64(bh.Offset), int64(bh.Length+blockTrailerLen)); readaheadSize > 0 { + if readaheadSize >= maxReadaheadSize { + // We've reached the maximum readahead size. Beyond this + // point, rely on OS-level readahead. Note that we can only + // reopen a new file handle with this optimization if + // r.fs != nil. This reader must have been created with the + // FileReopenOpt for this field to be set. + if r.fs != nil { + f, err := r.fs.Open(r.filename, vfs.SequentialReadsOption) + if err == nil { + // Use this new file handle for all sequential reads by + // this iterator going forward. + raState.sequentialFile = f + file = f + } + } + } + if raState.sequentialFile != nil { + _ = vfs.Prefetch(r.file, bh.Offset, uint64(readaheadSize)) + } } } v := r.opts.Cache.Alloc(int(bh.Length + blockTrailerLen)) b := v.Buf() - if _, err := r.file.ReadAt(b, int64(bh.Offset)); err != nil { + if _, err := file.ReadAt(b, int64(bh.Offset)); err != nil { r.opts.Cache.Free(v) return cache.Handle{}, err } @@ -1350,7 +1478,9 @@ func (r *Reader) readBlock(bh BlockHandle, transform blockTransform, raState *re checksum1 := crc.New(b[:bh.Length+1]).Value() if checksum0 != checksum1 { r.opts.Cache.Free(v) - return cache.Handle{}, errors.New("pebble/table: invalid table (checksum mismatch)") + return cache.Handle{}, base.CorruptionErrorf( + "pebble/table: invalid table %s (checksum mismatch at %d/%d)", + errors.Safe(r.fileNum), errors.Safe(bh.Offset), errors.Safe(bh.Length)) } typ := b[bh.Length] @@ -1364,7 +1494,7 @@ func (r *Reader) readBlock(bh BlockHandle, transform blockTransform, raState *re decodedLen, err := snappy.DecodedLen(b) if err != nil { r.opts.Cache.Free(v) - return cache.Handle{}, err + return cache.Handle{}, base.MarkCorruptionError(err) } decoded := r.opts.Cache.Alloc(decodedLen) decodedBuf := decoded.Buf() @@ -1372,18 +1502,18 @@ func (r *Reader) readBlock(bh BlockHandle, transform blockTransform, raState *re r.opts.Cache.Free(v) if err != nil { r.opts.Cache.Free(decoded) - return cache.Handle{}, err + return cache.Handle{}, base.MarkCorruptionError(err) } if len(result) != 0 && (len(result) != len(decodedBuf) || &result[0] != &decodedBuf[0]) { r.opts.Cache.Free(decoded) - return cache.Handle{}, errors.Errorf("pebble/table: snappy decoded into unexpected buffer: %p != %p", + return cache.Handle{}, base.CorruptionErrorf("pebble/table: snappy decoded into unexpected buffer: %p != %p", errors.Safe(result), errors.Safe(decodedBuf)) } v, b = decoded, decodedBuf default: r.opts.Cache.Free(v) - return cache.Handle{}, errors.Errorf("pebble/table: unknown block compression: %d", errors.Safe(typ)) + return cache.Handle{}, base.CorruptionErrorf("pebble/table: unknown block compression: %d", errors.Safe(typ)) } if transform != nil { @@ -1429,7 +1559,8 @@ func (r *Reader) transformRangeDelV1(b []byte) ([]byte, error) { restartInterval: 1, } frag := rangedel.Fragmenter{ - Cmp: r.Compare, + Cmp: r.Compare, + Format: r.FormatKey, Emit: func(fragmented []rangedel.Tombstone) { for i := range fragmented { t := &fragmented[i] @@ -1456,7 +1587,7 @@ func (r *Reader) readMetaindex(metaindexBH BlockHandle) error { defer b.Release() if uint64(len(data)) != metaindexBH.Length { - return errors.Errorf("pebble/table: unexpected metaindex block size: %d vs %d", + return base.CorruptionErrorf("pebble/table: unexpected metaindex block size: %d vs %d", errors.Safe(len(data)), errors.Safe(metaindexBH.Length)) } @@ -1469,7 +1600,7 @@ func (r *Reader) readMetaindex(metaindexBH BlockHandle) error { for valid := i.First(); valid; valid = i.Next() { bh, n := decodeBlockHandle(i.Value()) if n == 0 { - return errors.New("pebble/table: invalid table (bad filter block handle)") + return base.CorruptionErrorf("pebble/table: invalid table (bad filter block handle)") } meta[string(i.Key().UserKey)] = bh } @@ -1515,7 +1646,7 @@ func (r *Reader) readMetaindex(metaindexBH BlockHandle) error { case TableFilter: r.tableFilter = newTableFilterReader(fp) default: - return errors.Errorf("unknown filter type: %v", errors.Safe(t.ftype)) + return base.CorruptionErrorf("unknown filter type: %v", errors.Safe(t.ftype)) } done = true @@ -1747,6 +1878,7 @@ func NewReader(f vfs.File, o ReaderOptions, extraOpts ...ReaderOption) (*Reader, if r.Properties.ComparerName == "" || o.Comparer.Name == r.Properties.ComparerName { r.Compare = o.Comparer.Compare + r.FormatKey = o.Comparer.FormatKey r.Split = o.Comparer.Split } diff --git a/github.com/cockroachdb/pebble/sstable/table.go b/github.com/cockroachdb/pebble/sstable/table.go index 861d1d7b80..843bf19d69 100644 --- a/github.com/cockroachdb/pebble/sstable/table.go +++ b/github.com/cockroachdb/pebble/sstable/table.go @@ -65,6 +65,7 @@ import ( "io" "github.com/cockroachdb/errors" + "github.com/cockroachdb/pebble/internal/base" "github.com/cockroachdb/pebble/vfs" ) @@ -201,7 +202,7 @@ func readFooter(f vfs.File) (footer, error) { return footer, errors.Wrap(err, "pebble/table: invalid table (could not stat file)") } if stat.Size() < minFooterLen { - return footer, errors.New("pebble/table: invalid table (file size is too small)") + return footer, base.CorruptionErrorf("pebble/table: invalid table (file size is too small)") } buf := make([]byte, maxFooterLen) @@ -218,7 +219,8 @@ func readFooter(f vfs.File) (footer, error) { switch string(buf[len(buf)-len(rocksDBMagic):]) { case levelDBMagic: if len(buf) < levelDBFooterLen { - return footer, errors.Errorf("pebble/table: invalid table (footer too short): %d", errors.Safe(len(buf))) + return footer, base.CorruptionErrorf( + "pebble/table: invalid table (footer too short): %d", errors.Safe(len(buf))) } footer.footerBH.Offset = uint64(off+int64(len(buf))) - levelDBFooterLen buf = buf[len(buf)-levelDBFooterLen:] @@ -228,37 +230,37 @@ func readFooter(f vfs.File) (footer, error) { case rocksDBMagic: if len(buf) < rocksDBFooterLen { - return footer, errors.Errorf("pebble/table: invalid table (footer too short): %d", errors.Safe(len(buf))) + return footer, base.CorruptionErrorf("pebble/table: invalid table (footer too short): %d", errors.Safe(len(buf))) } footer.footerBH.Offset = uint64(off+int64(len(buf))) - rocksDBFooterLen buf = buf[len(buf)-rocksDBFooterLen:] footer.footerBH.Length = uint64(len(buf)) version := binary.LittleEndian.Uint32(buf[rocksDBVersionOffset:rocksDBMagicOffset]) if version != rocksDBFormatVersion2 { - return footer, errors.Errorf("pebble/table: unsupported format version %d", errors.Safe(version)) + return footer, base.CorruptionErrorf("pebble/table: unsupported format version %d", errors.Safe(version)) } footer.format = TableFormatRocksDBv2 footer.checksum = uint8(buf[0]) if footer.checksum != checksumCRC32c { - return footer, errors.Errorf("pebble/table: unsupported checksum type %d", errors.Safe(footer.checksum)) + return footer, base.CorruptionErrorf("pebble/table: unsupported checksum type %d", errors.Safe(footer.checksum)) } buf = buf[1:] default: - return footer, errors.New("pebble/table: invalid table (bad magic number)") + return footer, base.CorruptionErrorf("pebble/table: invalid table (bad magic number)") } { var n int footer.metaindexBH, n = decodeBlockHandle(buf) if n == 0 { - return footer, errors.New("pebble/table: invalid table (bad metaindex block handle)") + return footer, base.CorruptionErrorf("pebble/table: invalid table (bad metaindex block handle)") } buf = buf[n:] footer.indexBH, n = decodeBlockHandle(buf) if n == 0 { - return footer, errors.New("pebble/table: invalid table (bad index block handle)") + return footer, base.CorruptionErrorf("pebble/table: invalid table (bad index block handle)") } } diff --git a/github.com/cockroachdb/pebble/sstable/unsafe.go b/github.com/cockroachdb/pebble/sstable/unsafe.go index da9ef351a8..11ec068914 100644 --- a/github.com/cockroachdb/pebble/sstable/unsafe.go +++ b/github.com/cockroachdb/pebble/sstable/unsafe.go @@ -4,28 +4,31 @@ package sstable -import "unsafe" +import ( + "unsafe" + + "github.com/cockroachdb/pebble/internal/manual" +) func getBytes(ptr unsafe.Pointer, length int) []byte { - return (*[1 << 30]byte)(ptr)[:length:length] + return (*[manual.MaxArrayLen]byte)(ptr)[:length:length] } func decodeVarint(ptr unsafe.Pointer) (uint32, unsafe.Pointer) { - src := (*[5]uint8)(ptr) - if a := (*src)[0]; a < 128 { + if a := *((*uint8)(ptr)); a < 128 { return uint32(a), unsafe.Pointer(uintptr(ptr) + 1) - } else if a, b := a&0x7f, (*src)[1]; b < 128 { + } else if a, b := a&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 1))); b < 128 { return uint32(b)<<7 | uint32(a), unsafe.Pointer(uintptr(ptr) + 2) - } else if b, c := b&0x7f, (*src)[2]; c < 128 { + } else if b, c := b&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 2))); c < 128 { return uint32(c)<<14 | uint32(b)<<7 | uint32(a), unsafe.Pointer(uintptr(ptr) + 3) - } else if c, d := c&0x7f, (*src)[3]; d < 128 { + } else if c, d := c&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 3))); d < 128 { return uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a), unsafe.Pointer(uintptr(ptr) + 4) } else { - d, e := d&0x7f, (*src)[4] + d, e := d&0x7f, *((*uint8)(unsafe.Pointer(uintptr(ptr) + 4))) return uint32(e)<<28 | uint32(d)<<21 | uint32(c)<<14 | uint32(b)<<7 | uint32(a), unsafe.Pointer(uintptr(ptr) + 5) } diff --git a/github.com/cockroachdb/pebble/table_cache.go b/github.com/cockroachdb/pebble/table_cache.go index 49c4003e6a..d35d8f631e 100644 --- a/github.com/cockroachdb/pebble/table_cache.go +++ b/github.com/cockroachdb/pebble/table_cache.go @@ -18,6 +18,7 @@ import ( "github.com/cockroachdb/errors" "github.com/cockroachdb/pebble/internal/base" "github.com/cockroachdb/pebble/internal/invariants" + "github.com/cockroachdb/pebble/internal/manifest" "github.com/cockroachdb/pebble/internal/private" "github.com/cockroachdb/pebble/sstable" "github.com/cockroachdb/pebble/vfs" @@ -29,7 +30,7 @@ var tableCacheLabels = pprof.Labels("pebble", "table-cache") type tableCache struct { cache *Cache - shards []tableCacheShard + shards []*tableCacheShard filterMetrics FilterMetrics } @@ -37,21 +38,22 @@ func (c *tableCache) init(cacheID uint64, dirname string, fs vfs.FS, opts *Optio c.cache = opts.Cache c.cache.Ref() - c.shards = make([]tableCacheShard, runtime.NumCPU()) + c.shards = make([]*tableCacheShard, runtime.NumCPU()) for i := range c.shards { + c.shards[i] = &tableCacheShard{} c.shards[i].init(cacheID, dirname, fs, opts, size/len(c.shards)) c.shards[i].filterMetrics = &c.filterMetrics } } func (c *tableCache) getShard(fileNum FileNum) *tableCacheShard { - return &c.shards[uint64(fileNum)%uint64(len(c.shards))] + return c.shards[uint64(fileNum)%uint64(len(c.shards))] } func (c *tableCache) newIters( - meta *fileMetadata, opts *IterOptions, bytesIterated *uint64, + file manifest.LevelFile, opts *IterOptions, bytesIterated *uint64, ) (internalIterator, internalIterator, error) { - return c.getShard(meta.FileNum).newIters(meta, opts, bytesIterated) + return c.getShard(file.FileNum).newIters(file, opts, bytesIterated) } func (c *tableCache) evict(fileNum FileNum) { @@ -61,12 +63,12 @@ func (c *tableCache) evict(fileNum FileNum) { func (c *tableCache) metrics() (CacheMetrics, FilterMetrics) { var m CacheMetrics for i := range c.shards { - s := &c.shards[i] + s := c.shards[i] s.mu.RLock() m.Count += int64(len(s.mu.nodes)) s.mu.RUnlock() - m.Hits += atomic.LoadInt64(&s.hits) - m.Misses += atomic.LoadInt64(&s.misses) + m.Hits += atomic.LoadInt64(&s.atomic.hits) + m.Misses += atomic.LoadInt64(&s.atomic.misses) } m.Size = m.Count * int64(unsafe.Sizeof(sstable.Reader{})) f := FilterMetrics{ @@ -89,7 +91,7 @@ func (c *tableCache) withReader(meta *fileMetadata, fn func(*sstable.Reader) err func (c *tableCache) iterCount() int64 { var n int64 for i := range c.shards { - n += int64(atomic.LoadInt32(&c.shards[i].iterCount)) + n += int64(atomic.LoadInt32(&c.shards[i].atomic.iterCount)) } return n } @@ -97,6 +99,10 @@ func (c *tableCache) iterCount() int64 { func (c *tableCache) Close() error { var err error for i := range c.shards { + // The cache shard is not allocated yet, nothing to close + if c.shards[i] == nil { + continue + } err = firstError(err, c.shards[i].Close()) } c.cache.Unref() @@ -104,6 +110,17 @@ func (c *tableCache) Close() error { } type tableCacheShard struct { + // WARNING: The following struct `atomic` contains fields are accessed atomically. + // + // Go allocations are guaranteed to be 64-bit aligned which we take advantage + // of by placing the 64-bit fields which we access atomically at the beginning + // of the DB struct. For more information, see https://golang.org/pkg/sync/atomic/#pkg-note-BUG. + atomic struct { + hits int64 + misses int64 + iterCount int32 + } + logger Logger cacheID uint64 dirname string @@ -126,10 +143,6 @@ type tableCacheShard struct { sizeCold int sizeTest int } - - hits int64 - misses int64 - iterCount int32 releasing sync.WaitGroup releasingCh chan *tableCacheValue filterMetrics *FilterMetrics @@ -162,13 +175,13 @@ func (c *tableCacheShard) releaseLoop() { } func (c *tableCacheShard) newIters( - meta *fileMetadata, opts *IterOptions, bytesIterated *uint64, + file manifest.LevelFile, opts *IterOptions, bytesIterated *uint64, ) (internalIterator, internalIterator, error) { // Calling findNode gives us the responsibility of decrementing v's // refCount. If opening the underlying table resulted in error, then we // decrement this straight away. Otherwise, we pass that responsibility to // the sstable iterator, which decrements when it is closed. - v := c.findNode(meta) + v := c.findNode(file.FileMetadata) if v.err != nil { c.unrefValue(v) return nil, nil, v.err @@ -197,7 +210,7 @@ func (c *tableCacheShard) newIters( // NB: v.closeHook takes responsibility for calling unrefValue(v) here. iter.SetCloseHook(v.closeHook) - atomic.AddInt32(&c.iterCount, 1) + atomic.AddInt32(&c.atomic.iterCount, 1) if invariants.RaceEnabled { c.mu.Lock() c.mu.iters[iter] = debug.Stack() @@ -206,7 +219,7 @@ func (c *tableCacheShard) newIters( // NB: range-del iterator does not maintain a reference to the table, nor // does it need to read from it after creation. - rangeDelIter, err := v.reader.NewRangeDelIter() + rangeDelIter, err := v.reader.NewRawRangeDelIter() if err != nil { _ = iter.Close() return nil, nil, err @@ -296,7 +309,7 @@ func (c *tableCacheShard) findNode(meta *fileMetadata) *tableCacheValue { atomic.AddInt32(&v.refCount, 1) c.mu.RUnlock() atomic.StoreInt32(&n.referenced, 1) - atomic.AddInt64(&c.hits, 1) + atomic.AddInt64(&c.atomic.hits, 1) <-v.loaded return v } @@ -322,7 +335,7 @@ func (c *tableCacheShard) findNode(meta *fileMetadata) *tableCacheValue { v := n.value atomic.AddInt32(&v.refCount, 1) atomic.StoreInt32(&n.referenced, 1) - atomic.AddInt64(&c.hits, 1) + atomic.AddInt64(&c.atomic.hits, 1) c.mu.Unlock() <-v.loaded return v @@ -341,7 +354,7 @@ func (c *tableCacheShard) findNode(meta *fileMetadata) *tableCacheValue { c.mu.sizeHot++ } - atomic.AddInt64(&c.misses, 1) + atomic.AddInt64(&c.atomic.misses, 1) v := &tableCacheValue{ loaded: make(chan struct{}), @@ -356,7 +369,7 @@ func (c *tableCacheShard) findNode(meta *fileMetadata) *tableCacheValue { c.mu.Unlock() } c.unrefValue(v) - atomic.AddInt32(&c.iterCount, -1) + atomic.AddInt32(&c.atomic.iterCount, -1) return nil } n.value = v @@ -503,7 +516,7 @@ func (c *tableCacheShard) Close() error { // Check for leaked iterators. Note that we'll still perform cleanup below in // the case that there are leaked iterators. var err error - if v := atomic.LoadInt32(&c.iterCount); v > 0 { + if v := atomic.LoadInt32(&c.atomic.iterCount); v > 0 { if !invariants.RaceEnabled { err = errors.Errorf("leaked iterators: %d", errors.Safe(v)) } else { @@ -556,11 +569,12 @@ type tableCacheValue struct { func (v *tableCacheValue) load(meta *fileMetadata, c *tableCacheShard) { // Try opening the fileTypeTable first. var f vfs.File - f, v.err = c.fs.Open(base.MakeFilename(c.fs, c.dirname, fileTypeTable, meta.FileNum), - vfs.RandomReadsOption) + filename := base.MakeFilename(c.fs, c.dirname, fileTypeTable, meta.FileNum) + f, v.err = c.fs.Open(filename, vfs.RandomReadsOption) if v.err == nil { cacheOpts := private.SSTableCacheOpts(c.cacheID, meta.FileNum).(sstable.ReaderOption) - v.reader, v.err = sstable.NewReader(f, c.opts, cacheOpts, c.filterMetrics) + reopenOpt := sstable.FileReopenOpt{FS: c.fs, Filename: filename} + v.reader, v.err = sstable.NewReader(f, c.opts, cacheOpts, c.filterMetrics, reopenOpt) } if v.err == nil { if meta.SmallestSeqNum == meta.LargestSeqNum { diff --git a/github.com/cockroachdb/pebble/table_stats.go b/github.com/cockroachdb/pebble/table_stats.go index 78458448e2..b677ca5edd 100644 --- a/github.com/cockroachdb/pebble/table_stats.go +++ b/github.com/cockroachdb/pebble/table_stats.go @@ -5,10 +5,11 @@ package pebble import ( - "sync/atomic" + "math" "github.com/cockroachdb/pebble/internal/base" "github.com/cockroachdb/pebble/internal/manifest" + "github.com/cockroachdb/pebble/internal/rangedel" "github.com/cockroachdb/pebble/sstable" ) @@ -66,7 +67,7 @@ func (d *DB) updateTableStatsLocked(newFiles []manifest.NewFileEntry) { func (d *DB) shouldCollectTableStats() bool { ok := !d.mu.tableStats.loading - ok = ok && atomic.LoadInt32(&d.closed) == 0 + ok = ok && d.closed.Load() == nil ok = ok && !d.opts.private.disableTableStats ok = ok && (len(d.mu.tableStats.pending) > 0 || !d.mu.tableStats.loadedInitial) return ok @@ -97,12 +98,13 @@ func (d *DB) collectTableStats() { // Grab a read state to scan for tables. rs := d.loadReadState() var collected []collectedStats + var hints []deleteCompactionHint if len(pending) > 0 { - collected = d.loadNewFileStats(rs, pending) + collected, hints = d.loadNewFileStats(rs, pending) } else { var moreRemain bool var buf [maxTableStatsPerScan]collectedStats - collected, moreRemain = d.scanReadStateTableStats(rs, buf[:0]) + collected, hints, moreRemain = d.scanReadStateTableStats(rs, buf[:0]) loadedInitial = !moreRemain } rs.unref() @@ -125,6 +127,27 @@ func (d *DB) collectTableStats() { } d.mu.tableStats.cond.Broadcast() d.maybeCollectTableStats() + if len(hints) > 0 { + // Verify that all of the hint tombstones' files still exist in the + // current version. Otherwise, the tombstone itself may have been + // compacted into L6 and more recent keys may have had their sequence + // numbers zeroed. + // + // Note that it's possible that the tombstone file is being compacted + // presently. In that case, the file will be present in v. When the + // compaction finishes compacting the tombstone file, it will detect + // and clear the hint. + // + // See DB.maybeUpdateDeleteCompactionHints. + v := d.mu.versions.currentVersion() + keepHints := hints[:0] + for _, h := range hints { + if v.Contains(h.tombstoneLevel, d.cmp, h.tombstoneFile) { + keepHints = append(keepHints, h) + } + } + d.mu.compact.deletionHints = append(d.mu.compact.deletionHints, keepHints...) + } if maybeCompact { d.maybeScheduleCompaction() } @@ -135,7 +158,10 @@ type collectedStats struct { manifest.TableStats } -func (d *DB) loadNewFileStats(rs *readState, pending []manifest.NewFileEntry) []collectedStats { +func (d *DB) loadNewFileStats( + rs *readState, pending []manifest.NewFileEntry, +) ([]collectedStats, []deleteCompactionHint) { + var hints []deleteCompactionHint collected := make([]collectedStats, 0, len(pending)) for _, nf := range pending { // A file's stats might have been populated by an earlier call to @@ -155,7 +181,7 @@ func (d *DB) loadNewFileStats(rs *readState, pending []manifest.NewFileEntry) [] continue } - stats, err := d.loadTableStats(rs.current, nf.Level, nf.Meta) + stats, newHints, err := d.loadTableStats(rs.current, nf.Level, nf.Meta) if err != nil { d.opts.EventListener.BackgroundError(err) continue @@ -167,8 +193,9 @@ func (d *DB) loadNewFileStats(rs *readState, pending []manifest.NewFileEntry) [] fileMetadata: nf.Meta, TableStats: stats, }) + hints = append(hints, newHints...) } - return collected + return collected, hints } // scanReadStateTableStats is run by an active stat collection job when there @@ -176,10 +203,12 @@ func (d *DB) loadNewFileStats(rs *readState, pending []manifest.NewFileEntry) [] // which we haven't loaded table stats. func (d *DB) scanReadStateTableStats( rs *readState, fill []collectedStats, -) ([]collectedStats, bool) { +) ([]collectedStats, []deleteCompactionHint, bool) { moreRemain := false - for l, ff := range rs.current.Levels { - for _, f := range ff { + var hints []deleteCompactionHint + for l, levelMetadata := range rs.current.Levels { + iter := levelMetadata.Iter() + for f := iter.First(); f != nil; f = iter.Next() { // NB: We're not holding d.mu which protects f.Stats, but only the // active stats collection job updates f.Stats for active files, // and we ensure only one goroutine runs it at a time through @@ -196,10 +225,10 @@ func (d *DB) scanReadStateTableStats( // work to do. if len(fill) == cap(fill) { moreRemain = true - return fill, moreRemain + return fill, hints, moreRemain } - stats, err := d.loadTableStats(rs.current, l, f) + stats, newHints, err := d.loadTableStats(rs.current, l, f) if err != nil { // Set `moreRemain` so we'll try again. moreRemain = true @@ -210,16 +239,20 @@ func (d *DB) scanReadStateTableStats( fileMetadata: f, TableStats: stats, }) + hints = append(hints, newHints...) } } - return fill, moreRemain + return fill, hints, moreRemain } func (d *DB) loadTableStats( v *version, level int, meta *fileMetadata, -) (manifest.TableStats, error) { - var totalRangeDeletionEstimate uint64 +) (manifest.TableStats, []deleteCompactionHint, error) { + var stats manifest.TableStats + var compactionHints []deleteCompactionHint err := d.tableCache.withReader(meta, func(r *sstable.Reader) (err error) { + stats.NumEntries = r.Properties.NumEntries + stats.NumDeletions = r.Properties.NumDeletions if r.Properties.NumRangeDeletions == 0 { return nil } @@ -229,33 +262,69 @@ func (d *DB) loadTableStats( // Also, merging abutting tombstones reduces the number of calls to // estimateSizeBeneath which is costly, and improves the accuracy of // our overall estimate. - rangeDelIter, err := r.NewRangeDelIter() + rangeDelIter, err := r.NewRawRangeDelIter() if err != nil { return err } defer rangeDelIter.Close() - err = foreachDefragmentedTombstone(rangeDelIter, d.cmp, func(startUserKey, endUserKey []byte) error { - estimate, err := d.estimateSizeBeneath(v, level, meta, startUserKey, endUserKey) - if err != nil { - return err - } - totalRangeDeletionEstimate += estimate - return nil - }) + // Truncate tombstones to the containing file's bounds if necessary. + // See docs/range_deletions.md for why this is necessary. + rangeDelIter = rangedel.Truncate( + d.cmp, rangeDelIter, meta.Smallest.UserKey, meta.Largest.UserKey, nil, nil) + err = foreachDefragmentedTombstone(rangeDelIter, d.cmp, + func(startUserKey, endUserKey []byte, smallestSeqNum, largestSeqNum uint64) error { + // If the file is in the last level of the LSM, there is no + // data beneath it. The fact that there is still a range + // tombstone in a bottomost file suggests that an open + // snapshot kept the tombstone around. Estimate disk usage + // within the file itself. + if level == numLevels-1 { + size, err := r.EstimateDiskUsage(startUserKey, endUserKey) + if err != nil { + return err + } + stats.RangeDeletionsBytesEstimate += size + return nil + } + + estimate, hintSeqNum, err := d.estimateSizeBeneath(v, level, meta, startUserKey, endUserKey) + if err != nil { + return err + } + stats.RangeDeletionsBytesEstimate += estimate + + // If any files were completely contained with the range, + // hintSeqNum is the smallest sequence number contained in any + // such file. + if hintSeqNum == math.MaxUint64 { + return nil + } + hint := deleteCompactionHint{ + start: make([]byte, len(startUserKey)), + end: make([]byte, len(endUserKey)), + tombstoneFile: meta, + tombstoneLevel: level, + tombstoneLargestSeqNum: largestSeqNum, + tombstoneSmallestSeqNum: smallestSeqNum, + fileSmallestSeqNum: hintSeqNum, + } + copy(hint.start, startUserKey) + copy(hint.end, endUserKey) + compactionHints = append(compactionHints, hint) + return nil + }) return err }) - var stats manifest.TableStats if err != nil { - return stats, err + return stats, nil, err } stats.Valid = true - stats.RangeDeletionsBytesEstimate = totalRangeDeletionEstimate - return stats, nil + return stats, compactionHints, nil } func (d *DB) estimateSizeBeneath( v *version, level int, meta *fileMetadata, start, end []byte, -) (uint64, error) { +) (estimate uint64, hintSeqNum uint64, err error) { // Find all files in lower levels that overlap with the deleted range. // // An overlapping file might be completely contained by the range @@ -264,21 +333,18 @@ func (d *DB) estimateSizeBeneath( // // Otherwise, estimating the range for the file requires // additional I/O to read the file's index blocks. - var estimate uint64 + hintSeqNum = math.MaxUint64 for l := level + 1; l < numLevels; l++ { - overlaps := v.Overlaps(l, d.cmp, start, end) - - for i, file := range overlaps { - if i > 0 && i < len(overlaps)-1 { - // The files to the left and the right at least - // partially overlap with the range tombstone, which - // means `file` is fully contained within the range. - estimate += file.Size - } else if d.cmp(start, file.Smallest.UserKey) <= 0 && + iter := v.Overlaps(l, d.cmp, start, end).Iter() + for file := iter.First(); file != nil; file = iter.Next() { + if d.cmp(start, file.Smallest.UserKey) <= 0 && d.cmp(file.Largest.UserKey, end) <= 0 { // The range fully contains the file, so skip looking it up in // table cache/looking at its indexes and add the full file size. estimate += file.Size + if hintSeqNum > file.SmallestSeqNum { + hintSeqNum = file.SmallestSeqNum + } } else if d.cmp(file.Smallest.UserKey, end) <= 0 && d.cmp(start, file.Largest.UserKey) <= 0 { var size uint64 err := d.tableCache.withReader(file, func(r *sstable.Reader) (err error) { @@ -286,27 +352,37 @@ func (d *DB) estimateSizeBeneath( return err }) if err != nil { - return 0, err + return 0, hintSeqNum, err } estimate += size } } } - return estimate, nil + return estimate, hintSeqNum, nil } func foreachDefragmentedTombstone( - rangeDelIter base.InternalIterator, cmp base.Compare, fn func([]byte, []byte) error, + rangeDelIter base.InternalIterator, + cmp base.Compare, + fn func([]byte, []byte, uint64, uint64) error, ) error { var startUserKey, endUserKey []byte + var smallestSeqNum, largestSeqNum uint64 var initialized bool for start, end := rangeDelIter.First(); start != nil; start, end = rangeDelIter.Next() { + // Range tombstones are fragmented such that any two tombstones // that share the same start key also share the same end key. // Multiple tombstones may exist at different sequence numbers. // If this tombstone starts or ends at the same point, it's a fragment // of the previous one. if cmp(startUserKey, start.UserKey) == 0 || cmp(endUserKey, end) == 0 { + if smallestSeqNum > start.SeqNum() { + smallestSeqNum = start.SeqNum() + } + if largestSeqNum < start.SeqNum() { + largestSeqNum = start.SeqNum() + } continue } @@ -314,6 +390,12 @@ func foreachDefragmentedTombstone( // tombstone ended, merge it and continue. if cmp(endUserKey, start.UserKey) == 0 { endUserKey = append(endUserKey[:0], end...) + if smallestSeqNum > start.SeqNum() { + smallestSeqNum = start.SeqNum() + } + if largestSeqNum < start.SeqNum() { + largestSeqNum = start.SeqNum() + } continue } @@ -322,18 +404,20 @@ func foreachDefragmentedTombstone( if !initialized { startUserKey = append(startUserKey[:0], start.UserKey...) endUserKey = append(endUserKey[:0], end...) + smallestSeqNum, largestSeqNum = start.SeqNum(), start.SeqNum() initialized = true continue } - if err := fn(startUserKey, endUserKey); err != nil { + if err := fn(startUserKey, endUserKey, smallestSeqNum, largestSeqNum); err != nil { return err } startUserKey = append(startUserKey[:0], start.UserKey...) endUserKey = append(endUserKey[:0], end...) + smallestSeqNum, largestSeqNum = start.SeqNum(), start.SeqNum() } if initialized { - if err := fn(startUserKey, endUserKey); err != nil { + if err := fn(startUserKey, endUserKey, smallestSeqNum, largestSeqNum); err != nil { return err } } @@ -343,3 +427,20 @@ func foreachDefragmentedTombstone( } return rangeDelIter.Close() } + +func maybeSetStatsFromProperties(meta *fileMetadata, props *sstable.Properties) bool { + // If a table has range deletions, we can't calculate the + // RangeDeletionsBytesEstimate statistic and can't populate table stats + // from just the properties. The table stats collector goroutine will + // populate the stats. + if props.NumRangeDeletions != 0 { + return false + } + meta.Stats = manifest.TableStats{ + Valid: true, + NumEntries: props.NumEntries, + NumDeletions: props.NumDeletions, + RangeDeletionsBytesEstimate: 0, + } + return true +} diff --git a/github.com/cockroachdb/pebble/tool/db.go b/github.com/cockroachdb/pebble/tool/db.go index 40830fec55..56c2979060 100644 --- a/github.com/cockroachdb/pebble/tool/db.go +++ b/github.com/cockroachdb/pebble/tool/db.go @@ -372,7 +372,9 @@ func (d *dbT) runProperties(cmd *cobra.Command, args []string) { if err != nil { return err } - bve.Accumulate(&ve) + if err := bve.Accumulate(&ve); err != nil { + return err + } if ve.ComparerName != "" { cmp = d.comparers[ve.ComparerName] d.fmtKey.setForComparer(ve.ComparerName, d.comparers) @@ -389,8 +391,9 @@ func (d *dbT) runProperties(cmd *cobra.Command, args []string) { var total props var all []props for _, l := range v.Levels { + iter := l.Iter() var level props - for _, t := range l { + for t := iter.First(); t != nil; t = iter.Next() { err := d.addProps(dirname, t, &level) if err != nil { return err diff --git a/github.com/cockroachdb/pebble/tool/find.go b/github.com/cockroachdb/pebble/tool/find.go index d0c43b79d9..76a1ea1528 100644 --- a/github.com/cockroachdb/pebble/tool/find.go +++ b/github.com/cockroachdb/pebble/tool/find.go @@ -423,7 +423,7 @@ func (f *findT) searchTables(searchKey []byte, refs []findRef) []findRef { // bit more work here to put them in a form that can be iterated in // parallel with the point records. rangeDelIter, err := func() (base.InternalIterator, error) { - iter, err := r.NewRangeDelIter() + iter, err := r.NewRawRangeDelIter() if err != nil { return nil, err } diff --git a/github.com/cockroachdb/pebble/tool/lsm.go b/github.com/cockroachdb/pebble/tool/lsm.go index 7f58d58ef8..7c1690e4c3 100644 --- a/github.com/cockroachdb/pebble/tool/lsm.go +++ b/github.com/cockroachdb/pebble/tool/lsm.go @@ -36,7 +36,7 @@ type lsmVersionEdit struct { Added map[int][]base.FileNum `json:",omitempty"` // Map from level to files deleted from the level. Deleted map[int][]base.FileNum `json:",omitempty"` - // L0 sublevel structure for all files at this point. + // L0 sublevels for any files with changed sublevels so far. Sublevels map[base.FileNum]int `json:",omitempty"` } @@ -240,13 +240,16 @@ func (l *lsmT) buildEdits(edits []*manifest.VersionEdit) { edit.Added[nf.Level] = append(edit.Added[nf.Level], nf.Meta.FileNum) currentFiles[nf.Level] = append(currentFiles[nf.Level], nf.Meta) } - sublevels, err := manifest.NewL0Sublevels(currentFiles[0], l.cmp.Compare, l.fmtKey.fn, 0) - if err != nil { - panic(err) - } + v := manifest.NewVersion(l.cmp.Compare, l.fmtKey.fn, 0, currentFiles) edit.Sublevels = make(map[base.FileNum]int) - for sublevel, files := range sublevels.Levels { + for sublevel, files := range v.L0Sublevels.Levels { for _, f := range files { + if len(l.state.Edits) > 0 { + lastEdit := l.state.Edits[len(l.state.Edits)-1] + if sublevel2, ok := lastEdit.Sublevels[f.FileNum]; ok && sublevel == sublevel2 { + continue + } + } edit.Sublevels[f.FileNum] = sublevel } } diff --git a/github.com/cockroachdb/pebble/tool/lsm_data.go b/github.com/cockroachdb/pebble/tool/lsm_data.go index babe898a6d..91e0c2a458 100644 --- a/github.com/cockroachdb/pebble/tool/lsm_data.go +++ b/github.com/cockroachdb/pebble/tool/lsm_data.go @@ -65,6 +65,7 @@ body { #container { height: 100vh; width: 100%; + overflow-y: scroll; } #header { @@ -248,6 +249,7 @@ let version = { levelHeights[0] = sublevelHeight; } levelOffsets = generateLevelOffsets(); + vis.style("height", levelOffsets[6] + 100); }, onCheckboxChange: function(value) { @@ -308,7 +310,10 @@ let version = { this.sublevels.push([]); } for (let file of this.levels[0]) { - let sublevel = data.Edits[this.index].Sublevels[file]; + let sublevel = null; + for (let i = index; i >= 0 && (sublevel === null || sublevel === undefined); i--) { + sublevel = data.Edits[i].Sublevels[file]; + } this.sublevels[sublevel].push(file); } diff --git a/github.com/cockroachdb/pebble/tool/make_test_find_db.go b/github.com/cockroachdb/pebble/tool/make_test_find_db.go index ecfc3506d4..a83e88e9de 100644 --- a/github.com/cockroachdb/pebble/tool/make_test_find_db.go +++ b/github.com/cockroachdb/pebble/tool/make_test_find_db.go @@ -98,11 +98,6 @@ func (d *db) ingest(keyVals ...string) { if err := d.db.Ingest([]string{path}); err != nil { log.Fatal(err) } - - if err := fs.Remove(path); err != nil { - // TODO(peter): why is the remove sometimes failing under CI? - log.Print(err) - } } func (d *db) flush() { diff --git a/github.com/cockroachdb/pebble/tool/manifest.go b/github.com/cockroachdb/pebble/tool/manifest.go index 571e7335c3..5d9c9cd9e1 100644 --- a/github.com/cockroachdb/pebble/tool/manifest.go +++ b/github.com/cockroachdb/pebble/tool/manifest.go @@ -79,7 +79,7 @@ Check the contents of the MANIFEST files. func (m *manifestT) printLevels(v *manifest.Version) { for level := range v.Levels { - if level == 0 && v.L0Sublevels != nil && len(v.Levels[level]) > 0 { + if level == 0 && v.L0Sublevels != nil && !v.Levels[level].Empty() { for sublevel := len(v.L0Sublevels.Levels) - 1; sublevel >= 0; sublevel-- { fmt.Fprintf(stdout, "--- L0.%d ---\n", sublevel) for _, f := range v.L0Sublevels.Levels[sublevel] { @@ -92,8 +92,8 @@ func (m *manifestT) printLevels(v *manifest.Version) { continue } fmt.Fprintf(stdout, "--- L%d ---\n", level) - for j := range v.Levels[level] { - f := v.Levels[level][j] + iter := v.Levels[level].Iter() + for f := iter.First(); f != nil; f = iter.Next() { fmt.Fprintf(stdout, " %s:%d", f.FileNum, f.Size) formatSeqNumRange(stdout, f.SmallestSeqNum, f.LargestSeqNum) formatKeyRange(stdout, m.fmtKey, &f.Smallest, &f.Largest) @@ -131,7 +131,10 @@ func (m *manifestT) runDump(cmd *cobra.Command, args []string) { fmt.Fprintf(stdout, "%s\n", err) break } - bve.Accumulate(&ve) + if err := bve.Accumulate(&ve); err != nil { + fmt.Fprintf(stdout, "%s\n", err) + break + } empty := true fmt.Fprintf(stdout, "%d\n", offset) @@ -242,7 +245,11 @@ func (m *manifestT) runCheck(cmd *cobra.Command, args []string) { break } var bve manifest.BulkVersionEdit - bve.Accumulate(&ve) + if err := bve.Accumulate(&ve); err != nil { + fmt.Fprintf(stderr, "%s\n", err) + ok = false + return + } empty := true if ve.ComparerName != "" { diff --git a/github.com/cockroachdb/pebble/tool/sstable.go b/github.com/cockroachdb/pebble/tool/sstable.go index 7b178e5ebd..096a4f65e9 100644 --- a/github.com/cockroachdb/pebble/tool/sstable.go +++ b/github.com/cockroachdb/pebble/tool/sstable.go @@ -389,7 +389,7 @@ func (s *sstableT) runScan(cmd *cobra.Command, args []string) { // bit more work here to put them in a form that can be iterated in // parallel with the point records. rangeDelIter, err := func() (base.InternalIterator, error) { - iter, err := r.NewRangeDelIter() + iter, err := r.NewRawRangeDelIter() if err != nil { return nil, err } diff --git a/github.com/cockroachdb/pebble/tool/util.go b/github.com/cockroachdb/pebble/tool/util.go index d083124f44..48838808c6 100644 --- a/github.com/cockroachdb/pebble/tool/util.go +++ b/github.com/cockroachdb/pebble/tool/util.go @@ -269,7 +269,11 @@ func formatKeyRange(w io.Writer, fmtKey keyFormatter, start, end *base.InternalK } func formatKeyValue( - w io.Writer, fmtKey keyFormatter, fmtValue valueFormatter, key *base.InternalKey, value []byte, + w io.Writer, + fmtKey keyFormatter, + fmtValue valueFormatter, + key *base.InternalKey, + value []byte, ) { if key.Kind() == base.InternalKeyKindRangeDelete { if fmtKey.spec != "null" { diff --git a/github.com/cockroachdb/pebble/version_set.go b/github.com/cockroachdb/pebble/version_set.go index bce21e4ac5..be556119b9 100644 --- a/github.com/cockroachdb/pebble/version_set.go +++ b/github.com/cockroachdb/pebble/version_set.go @@ -13,6 +13,7 @@ import ( "github.com/cockroachdb/errors" "github.com/cockroachdb/pebble/internal/base" + "github.com/cockroachdb/pebble/internal/invariants" "github.com/cockroachdb/pebble/internal/manifest" "github.com/cockroachdb/pebble/internal/record" "github.com/cockroachdb/pebble/vfs" @@ -35,8 +36,30 @@ type versionList = manifest.VersionList // like it sounds: a delta from the previous version. Version edits are logged // to the MANIFEST file, which is replayed at startup. type versionSet struct { + // WARNING: The following struct `atomic` contains fields are accessed atomically. + // + // Go allocations are guaranteed to be 64-bit aligned which we take advantage + // of by placing the 64-bit fields which we access atomically at the beginning + // of the versionSet struct. + // For more information, see https://golang.org/pkg/sync/atomic/#pkg-note-BUG. + atomic struct { + logSeqNum uint64 // next seqNum to use for WAL writes + + // The upper bound on sequence numbers that have been assigned so far. + // A suffix of these sequence numbers may not have been written to a + // WAL. Both logSeqNum and visibleSeqNum are atomically updated by the + // commitPipeline. + visibleSeqNum uint64 // visible seqNum (<= logSeqNum) + + // Number of bytes present in sstables being written by in-progress + // compactions. This value will be zero if there are no in-progress + // compactions. Updated and read atomically. + atomicInProgressBytes int64 + } + // Immutable fields. dirname string + // Set to DB.mu. mu *sync.Mutex opts *Options fs vfs.FS @@ -71,13 +94,6 @@ type versionSet struct { // for the WAL, MANIFEST, sstable, and OPTIONS files. nextFileNum FileNum - // The upper bound on sequence numbers that have been assigned so far. - // A suffix of these sequence numbers may not have been written to a - // WAL. Both logSeqNum and visibleSeqNum are atomically updated by the - // commitPipeline. - logSeqNum uint64 // next seqNum to use for WAL writes - visibleSeqNum uint64 // visible seqNum (<= logSeqNum) - // The current manifest file number. manifestFileNum FileNum @@ -110,7 +126,7 @@ func (vs *versionSet) create( vs.init(dirname, opts, mu) newVersion := &version{} vs.append(newVersion) - vs.picker = newCompactionPicker(newVersion, vs.opts, nil) + vs.picker = newCompactionPicker(newVersion, vs.opts, nil, vs.metrics.levelSizes()) // Note that a "snapshot" version edit is written to the manifest when it is // created. @@ -176,13 +192,13 @@ func (vs *versionSet) load(dirname string, opts *Options, mu *sync.Mutex) error return err } if b[n-1] != '\n' { - return errors.Errorf("pebble: CURRENT file for DB %q is malformed", dirname) + return base.CorruptionErrorf("pebble: CURRENT file for DB %q is malformed", dirname) } b = bytes.TrimSpace(b) var ok bool if _, vs.manifestFileNum, ok = base.ParseFilename(vs.fs, string(b)); !ok { - return errors.Errorf("pebble: MANIFEST name %q is malformed", errors.Safe(b)) + return base.CorruptionErrorf("pebble: MANIFEST name %q is malformed", errors.Safe(b)) } // Read the versionEdits in the manifest file. @@ -220,7 +236,9 @@ func (vs *versionSet) load(dirname string, opts *Options, mu *sync.Mutex) error errors.Safe(b), dirname, errors.Safe(ve.ComparerName), errors.Safe(vs.cmpName)) } } - bve.Accumulate(&ve) + if err := bve.Accumulate(&ve); err != nil { + return err + } if ve.MinUnflushedLogNum != 0 { vs.minUnflushedLogNum = ve.MinUnflushedLogNum } @@ -235,7 +253,7 @@ func (vs *versionSet) load(dirname string, opts *Options, mu *sync.Mutex) error // (assuming no WALs contain higher sequence numbers than the // manifest's LastSeqNum). Increment LastSeqNum by 1 to get the // next sequence number that will be assigned. - vs.logSeqNum = ve.LastSeqNum + 1 + vs.atomic.logSeqNum = ve.LastSeqNum + 1 } } // We have already set vs.nextFileNum = 2 at the beginning of the @@ -249,7 +267,7 @@ func (vs *versionSet) load(dirname string, opts *Options, mu *sync.Mutex) error // minUnflushedLogNum, even if WALs with non-zero file numbers are // present in the directory. } else { - return errors.Errorf("pebble: malformed manifest file %q for DB %q", + return base.CorruptionErrorf("pebble: malformed manifest file %q for DB %q", errors.Safe(b), dirname) } } @@ -259,16 +277,15 @@ func (vs *versionSet) load(dirname string, opts *Options, mu *sync.Mutex) error if err != nil { return err } - newVersion.L0Sublevels.InitCompactingFileInfo() + newVersion.L0Sublevels.InitCompactingFileInfo(nil /* in-progress compactions */) vs.append(newVersion) - vs.picker = newCompactionPicker(newVersion, vs.opts, nil) - for i := range vs.metrics.Levels { l := &vs.metrics.Levels[i] - l.NumFiles = int64(len(newVersion.Levels[i])) - l.Size = uint64(totalSize(newVersion.Levels[i])) + l.NumFiles = int64(newVersion.Levels[i].Slice().Len()) + l.Size = int64(newVersion.Levels[i].Slice().SizeSum()) } + vs.picker = newCompactionPicker(newVersion, vs.opts, nil, vs.metrics.levelSizes()) return nil } @@ -354,7 +371,7 @@ func (vs *versionSet) logAndApply( // in an unflushed memtable. logSeqNum is the _next_ sequence number that // will be assigned, so subtract that by 1 to get the upper bound on the // last assigned sequence number. - logSeqNum := atomic.LoadUint64(&vs.logSeqNum) + logSeqNum := atomic.LoadUint64(&vs.atomic.logSeqNum) ve.LastSeqNum = logSeqNum - 1 if logSeqNum == 0 { // logSeqNum is initialized to 1 in Open() if there are no previous WAL @@ -383,7 +400,9 @@ func (vs *versionSet) logAndApply( defer vs.mu.Lock() var bve bulkVersionEdit - bve.Accumulate(ve) + if err := bve.Accumulate(ve); err != nil { + return err + } var err error newVersion, zombies, err = bve.Apply(currentVersion, vs.cmp, vs.opts.Comparer.FormatKey, vs.opts.Experimental.FlushSplitBytes) @@ -446,7 +465,9 @@ func (vs *versionSet) logAndApply( // Now that DB.mu is held again, initialize compacting file info in // L0Sublevels. - newVersion.L0Sublevels.InitCompactingFileInfo() + inProgress := inProgressCompactions() + + newVersion.L0Sublevels.InitCompactingFileInfo(inProgressL0Compactions(inProgress)) // Update the zombie tables set first, as installation of the new version // will unref the previous version which could result in addObsoleteLocked @@ -466,24 +487,32 @@ func (vs *versionSet) logAndApply( } vs.manifestFileNum = newManifestFileNum } - vs.picker = newCompactionPicker(newVersion, vs.opts, inProgressCompactions()) - if !vs.dynamicBaseLevel { - vs.picker.forceBaseLevel1() - } for level, update := range metrics { vs.metrics.Levels[level].Add(update) } for i := range vs.metrics.Levels { l := &vs.metrics.Levels[i] - l.NumFiles = int64(len(newVersion.Levels[i])) - l.Size = uint64(totalSize(newVersion.Levels[i])) l.Sublevels = 0 if l.NumFiles > 0 { l.Sublevels = 1 } + if invariants.Enabled { + if count := int64(newVersion.Levels[i].Slice().Len()); l.NumFiles != count { + vs.opts.Logger.Fatalf("versionSet metrics L%d NumFiles = %d, actual count = %d", i, l.NumFiles, count) + } + if size := int64(newVersion.Levels[i].Slice().SizeSum()); l.Size != size { + vs.opts.Logger.Fatalf("versionSet metrics L%d Size = %d, actual size = %d", i, l.Size, size) + } + } } vs.metrics.Levels[0].Sublevels = int32(len(newVersion.L0Sublevels.Levels)) + + vs.picker = newCompactionPicker(newVersion, vs.opts, inProgress, vs.metrics.levelSizes()) + if !vs.dynamicBaseLevel { + vs.picker.forceBaseLevel1() + } + return nil } @@ -495,6 +524,10 @@ func (vs *versionSet) incrementFlushes() { vs.metrics.Flush.Count++ } +func (vs *versionSet) incrementCompactionBytes(numBytes int64) { + atomic.AddInt64(&vs.atomic.atomicInProgressBytes, numBytes) +} + // createManifest creates a manifest file that contains a snapshot of vs. func (vs *versionSet) createManifest( dirname string, fileNum, minUnflushedLogNum, nextFileNum FileNum, @@ -524,8 +557,9 @@ func (vs *versionSet) createManifest( snapshot := versionEdit{ ComparerName: vs.cmpName, } - for level, fileMetadata := range vs.currentVersion().Levels { - for _, meta := range fileMetadata { + for level, levelMetadata := range vs.currentVersion().Levels { + iter := levelMetadata.Iter() + for meta := iter.First(); meta != nil; meta = iter.Next() { snapshot.NewFiles = append(snapshot.NewFiles, newFileEntry{ Level: level, Meta: meta, @@ -597,8 +631,9 @@ func (vs *versionSet) currentVersion() *version { func (vs *versionSet) addLiveFileNums(m map[FileNum]struct{}) { current := vs.currentVersion() for v := vs.versions.Front(); true; v = v.Next() { - for _, ff := range v.Levels { - for _, f := range ff { + for _, lm := range v.Levels { + iter := lm.Iter() + for f := iter.First(); f != nil; f = iter.Next() { m[f.FileNum] = struct{}{} } } @@ -619,3 +654,17 @@ func (vs *versionSet) addObsoleteLocked(obsolete []FileNum) { } vs.obsoleteTables = append(vs.obsoleteTables, obsolete...) } + +func newFileMetrics(newFiles []manifest.NewFileEntry) map[int]*LevelMetrics { + m := map[int]*LevelMetrics{} + for _, nf := range newFiles { + lm := m[nf.Level] + if lm == nil { + lm = &LevelMetrics{} + m[nf.Level] = lm + } + lm.NumFiles++ + lm.Size += int64(nf.Meta.Size) + } + return m +} diff --git a/github.com/cockroachdb/pebble/vfs/disk_health.go b/github.com/cockroachdb/pebble/vfs/disk_health.go new file mode 100644 index 0000000000..9fee4b4d02 --- /dev/null +++ b/github.com/cockroachdb/pebble/vfs/disk_health.go @@ -0,0 +1,177 @@ +// Copyright 2020 The LevelDB-Go and Pebble Authors. All rights reserved. Use +// of this source code is governed by a BSD-style license that can be found in +// the LICENSE file. + +package vfs + +import ( + "sync/atomic" + "time" +) + +const ( + // defaultTickInterval is the default interval between two ticks of each + // diskHealthCheckingFile loop iteration. + defaultTickInterval = 2 * time.Second +) + +// diskHealthCheckingFile is a File wrapper to detect slow disk operations, and +// call onSlowDisk if a disk operation is seen to exceed diskSlowThreshold. +// +// This struct creates a goroutine (in startTicker()) that, at every tick +// interval, sees if there's a disk operation taking longer than the specified +// duration. This setup is preferable to creating a new timer at every disk +// operation, as it reduces overhead per disk operation. +type diskHealthCheckingFile struct { + File + + onSlowDisk func(time.Duration) + diskSlowThreshold time.Duration + tickInterval time.Duration + + stopper chan struct{} + lastWriteNanos int64 +} + +// newDiskHealthCheckingFile instantiates a new diskHealthCheckingFile, with the +// specified time threshold and event listener. +func newDiskHealthCheckingFile( + file File, diskSlowThreshold time.Duration, onSlowDisk func(time.Duration), +) *diskHealthCheckingFile { + return &diskHealthCheckingFile{ + File: file, + onSlowDisk: onSlowDisk, + diskSlowThreshold: diskSlowThreshold, + tickInterval: defaultTickInterval, + + stopper: make(chan struct{}), + } +} + +// startTicker starts a new goroutine with a ticker to monitor disk operations. +// Can only be called if the ticker goroutine isn't running already. +func (d *diskHealthCheckingFile) startTicker() { + if d.diskSlowThreshold == 0 { + return + } + + go func() { + ticker := time.NewTicker(d.tickInterval) + defer ticker.Stop() + + for { + select { + case <-d.stopper: + return + + case <-ticker.C: + lastWriteNanos := atomic.LoadInt64(&d.lastWriteNanos) + if lastWriteNanos == 0 { + continue + } + lastWrite := time.Unix(0, lastWriteNanos) + now := time.Now() + if lastWrite.Add(d.diskSlowThreshold).Before(now) { + // diskSlowThreshold was exceeded. Call the passed-in + // listener. + d.onSlowDisk(now.Sub(lastWrite)) + } + } + } + }() +} + +// stopTicker stops the goroutine started in startTicker. +func (d *diskHealthCheckingFile) stopTicker() { + close(d.stopper) +} + +// Write implements the io.Writer interface. +func (d *diskHealthCheckingFile) Write(p []byte) (n int, err error) { + d.timeDiskOp(func() { + n, err = d.File.Write(p) + }) + return n, err +} + +// Close implements the io.Closer interface. +func (d *diskHealthCheckingFile) Close() error { + d.stopTicker() + return d.File.Close() +} + +// Sync implements the io.Syncer interface. +func (d *diskHealthCheckingFile) Sync() (err error) { + d.timeDiskOp(func() { + err = d.File.Sync() + }) + return err +} + +// timeDiskOp runs the specified closure and makes its timing visible to the +// monitoring goroutine, in case it exceeds one of the slow disk durations. +func (d *diskHealthCheckingFile) timeDiskOp(op func()) { + if d == nil { + op() + return + } + + atomic.StoreInt64(&d.lastWriteNanos, time.Now().UnixNano()) + defer func() { + atomic.StoreInt64(&d.lastWriteNanos, 0) + }() + op() +} + +type diskHealthCheckingFS struct { + FS + + diskSlowThreshold time.Duration + onSlowDisk func(string, time.Duration) +} + +// WithDiskHealthChecks wraps an FS and ensures that all +// write-oriented created with that FS are wrapped with disk health detection +// checks. Disk operations that are observed to take longer than +// diskSlowThreshold trigger an onSlowDisk call. +func WithDiskHealthChecks( + fs FS, diskSlowThreshold time.Duration, onSlowDisk func(string, time.Duration), +) FS { + return diskHealthCheckingFS{ + FS: fs, + diskSlowThreshold: diskSlowThreshold, + onSlowDisk: onSlowDisk, + } +} + +// Create implements the vfs.FS interface. +func (d diskHealthCheckingFS) Create(name string) (File, error) { + f, err := d.FS.Create(name) + if err != nil { + return f, err + } + if d.diskSlowThreshold == 0 { + return f, nil + } + checkingFile := newDiskHealthCheckingFile(f, d.diskSlowThreshold, func(duration time.Duration) { + d.onSlowDisk(name, duration) + }) + checkingFile.startTicker() + return checkingFile, nil +} + +// ReuseForWrite implements the vfs.FS interface. +func (d diskHealthCheckingFS) ReuseForWrite(oldname, newname string) (File, error) { + f, err := d.FS.ReuseForWrite(oldname, newname) + if err != nil { + return f, err + } + if d.diskSlowThreshold == 0 { + return f, nil + } + checkingFile := newDiskHealthCheckingFile(f, d.diskSlowThreshold, func(duration time.Duration) { + d.onSlowDisk(newname, duration) + }) + checkingFile.startTicker() + return checkingFile, nil +} diff --git a/github.com/cockroachdb/pebble/vfs/disk_usage_unix.go b/github.com/cockroachdb/pebble/vfs/disk_usage_unix.go new file mode 100644 index 0000000000..c011f6bcfe --- /dev/null +++ b/github.com/cockroachdb/pebble/vfs/disk_usage_unix.go @@ -0,0 +1,17 @@ +// Copyright 2020 The LevelDB-Go and Pebble Authors. All rights reserved. Use +// of this source code is governed by a BSD-style license that can be found in +// the LICENSE file. + +// +build darwin linux openbsd dragonfly freebsd + +package vfs + +import "golang.org/x/sys/unix" + +func (defaultFS) GetFreeSpace(path string) (uint64, error) { + stat := unix.Statfs_t{} + if err := unix.Statfs(path, &stat); err != nil { + return 0, err + } + return uint64(stat.Bsize) * stat.Bfree, nil +} diff --git a/github.com/cockroachdb/pebble/vfs/disk_usage_windows.go b/github.com/cockroachdb/pebble/vfs/disk_usage_windows.go new file mode 100644 index 0000000000..b92aa5adb2 --- /dev/null +++ b/github.com/cockroachdb/pebble/vfs/disk_usage_windows.go @@ -0,0 +1,19 @@ +// Copyright 2020 The LevelDB-Go and Pebble Authors. All rights reserved. Use +// of this source code is governed by a BSD-style license that can be found in +// the LICENSE file. + +// +build windows + +package vfs + +import "golang.org/x/sys/windows" + +func (defaultFS) GetFreeSpace(path string) (uint64, error) { + var freeSpace uint64 + p, err := windows.UTF16PtrFromString(path) + if err != nil { + return 0, err + } + err = windows.GetDiskFreeSpaceEx(p, &freeSpace, nil, nil) + return freeSpace, err +} diff --git a/github.com/cockroachdb/pebble/vfs/errors_unix.go b/github.com/cockroachdb/pebble/vfs/errors_unix.go new file mode 100644 index 0000000000..184ec25aff --- /dev/null +++ b/github.com/cockroachdb/pebble/vfs/errors_unix.go @@ -0,0 +1,18 @@ +// Copyright 2020 The LevelDB-Go and Pebble Authors. All rights reserved. Use +// of this source code is governed by a BSD-style license that can be found in +// the LICENSE file. + +// +build darwin dragonfly freebsd linux openbsd + +package vfs + +import ( + "github.com/cockroachdb/errors" + "golang.org/x/sys/unix" +) + +// IsNoSpaceError returns true if the given error indicates that the disk is +// out of space. +func IsNoSpaceError(err error) bool { + return errors.Is(err, unix.ENOSPC) +} diff --git a/github.com/cockroachdb/pebble/vfs/errors_windows.go b/github.com/cockroachdb/pebble/vfs/errors_windows.go new file mode 100644 index 0000000000..3eea5bdf4a --- /dev/null +++ b/github.com/cockroachdb/pebble/vfs/errors_windows.go @@ -0,0 +1,19 @@ +// Copyright 2020 The LevelDB-Go and Pebble Authors. All rights reserved. Use +// of this source code is governed by a BSD-style license that can be found in +// the LICENSE file. + +// +build windows + +package vfs + +import ( + "github.com/cockroachdb/errors" + "golang.org/x/sys/windows" +) + +// IsNoSpaceError returns true if the given error indicates that the disk is +// out of space. +func IsNoSpaceError(err error) bool { + return errors.Is(err, windows.ERROR_DISK_FULL) || + errors.Is(err, windows.ERROR_HANDLE_DISK_FULL) +} diff --git a/github.com/cockroachdb/pebble/vfs/fadvise_generic.go b/github.com/cockroachdb/pebble/vfs/fadvise_generic.go index b7aa995700..22b3de8eff 100644 --- a/github.com/cockroachdb/pebble/vfs/fadvise_generic.go +++ b/github.com/cockroachdb/pebble/vfs/fadvise_generic.go @@ -9,3 +9,7 @@ package vfs func fadviseRandom(f uintptr) error { return nil } + +func fadviseSequential(f uintptr) error { + return nil +} diff --git a/github.com/cockroachdb/pebble/vfs/fadvise_linux.go b/github.com/cockroachdb/pebble/vfs/fadvise_linux.go index 05d611d255..50c6547572 100644 --- a/github.com/cockroachdb/pebble/vfs/fadvise_linux.go +++ b/github.com/cockroachdb/pebble/vfs/fadvise_linux.go @@ -12,3 +12,8 @@ import "golang.org/x/sys/unix" func fadviseRandom(f uintptr) error { return unix.Fadvise(int(f), 0, 0, unix.FADV_RANDOM) } + +// Calls Fadvise with FADV_SEQUENTIAL to enable readahead on a file descriptor. +func fadviseSequential(f uintptr) error { + return unix.Fadvise(int(f), 0, 0, unix.FADV_SEQUENTIAL) +} diff --git a/github.com/cockroachdb/pebble/vfs/mem_fs.go b/github.com/cockroachdb/pebble/vfs/mem_fs.go index 6b496cf632..bcd799b8c4 100644 --- a/github.com/cockroachdb/pebble/vfs/mem_fs.go +++ b/github.com/cockroachdb/pebble/vfs/mem_fs.go @@ -492,6 +492,11 @@ func (*MemFS) PathDir(p string) string { return path.Dir(p) } +// GetFreeSpace implements FS.GetFreeSpace. +func (*MemFS) GetFreeSpace(string) (uint64, error) { + return 0, errors.New("pebble: not supported") +} + // memNode holds a file's data or a directory's children, and implements os.FileInfo. type memNode struct { name string diff --git a/github.com/cockroachdb/pebble/vfs/syncing_file.go b/github.com/cockroachdb/pebble/vfs/syncing_file.go index ab337e75ad..1811de662f 100644 --- a/github.com/cockroachdb/pebble/vfs/syncing_file.go +++ b/github.com/cockroachdb/pebble/vfs/syncing_file.go @@ -31,6 +31,7 @@ type syncingFile struct { preallocatedBlocks int64 syncData func() error syncTo func(offset int64) error + timeDiskOp func(op func()) } // NewSyncingFile wraps a writable file and ensures that data is synced @@ -54,6 +55,16 @@ func NewSyncingFile(f File, opts SyncingFileOptions) File { if d, ok := f.(fd); ok { s.fd = d.Fd() } + type dhChecker interface { + timeDiskOp(op func()) + } + if d, ok := f.(dhChecker); ok { + s.timeDiskOp = d.timeDiskOp + } else { + s.timeDiskOp = func(op func()) { + op() + } + } s.init() diff --git a/github.com/cockroachdb/pebble/vfs/syncing_file_linux.go b/github.com/cockroachdb/pebble/vfs/syncing_file_linux.go index b8e4bfbe57..5393349b0d 100644 --- a/github.com/cockroachdb/pebble/vfs/syncing_file_linux.go +++ b/github.com/cockroachdb/pebble/vfs/syncing_file_linux.go @@ -25,9 +25,9 @@ func isSyncRangeSupported(fd uintptr) bool { return false } - // Whitelist which filesystems we allow using sync_file_range with as some - // filesystems treat that syscall as a noop (notably ZFS). A whitelist is - // used instead of a blacklist in order to have a more graceful failure mode + // Allowlist which filesystems we allow using sync_file_range with as some + // filesystems treat that syscall as a noop (notably ZFS). A allowlist is + // used instead of a denylist in order to have a more graceful failure mode // in case a filesystem we haven't tested is encountered. Currently only // ext2/3/4 are known to work properly. const extMagic = 0xef53 @@ -42,7 +42,9 @@ func (f *syncingFile) init() { if f.fd == 0 { return } - f.useSyncRange = isSyncRangeSupported(f.fd) + f.timeDiskOp(func() { + f.useSyncRange = isSyncRangeSupported(f.fd) + }) if f.useSyncRange { f.syncTo = f.syncToRange } else { @@ -55,7 +57,11 @@ func (f *syncingFile) syncFdatasync() error { if f.fd == 0 { return f.File.Sync() } - return syscall.Fdatasync(int(f.fd)) + var err error + f.timeDiskOp(func() { + err = syscall.Fdatasync(int(f.fd)) + }) + return err } func (f *syncingFile) syncToFdatasync(_ int64) error { @@ -82,5 +88,9 @@ func (f *syncingFile) syncToRange(offset int64) error { // use of `waitBefore` is to limit how much dirty data is allowed to // accumulate. Linux sometimes behaves poorly when a large amount of dirty // data accumulates, impacting other I/O operations. - return syscall.SyncFileRange(int(f.fd), 0, offset, write|waitBefore) + var err error + f.timeDiskOp(func() { + err = syscall.SyncFileRange(int(f.fd), 0, offset, write|waitBefore) + }) + return err } diff --git a/github.com/cockroachdb/pebble/vfs/vfs.go b/github.com/cockroachdb/pebble/vfs/vfs.go index a1c0b5697d..6a0609ccb8 100644 --- a/github.com/cockroachdb/pebble/vfs/vfs.go +++ b/github.com/cockroachdb/pebble/vfs/vfs.go @@ -112,6 +112,10 @@ type FS interface { // PathDir returns all but the last element of path, typically the path's directory. PathDir(path string) string + + // GetFreeSpace returns the amount of free disk space for the filesystem + // where path is any file or directory within that filesystem. + GetFreeSpace(path string) (uint64, error) } // Default is a FS implementation backed by the underlying operating system's @@ -191,13 +195,33 @@ type randomReadsOption struct{} // RandomReadsOption is an OpenOption that optimizes opened file handle for // random reads, by calling fadvise() with POSIX_FADV_RANDOM on Linux systems -// to disable readahead. Only works when specified to defaultFS. +// to disable readahead. var RandomReadsOption OpenOption = &randomReadsOption{} // Apply implements the OpenOption interface. func (randomReadsOption) Apply(f File) { - if osFile, ok := f.(*os.File); ok { - _ = fadviseRandom(osFile.Fd()) + type fd interface { + Fd() uintptr + } + if fdFile, ok := f.(fd); ok { + _ = fadviseRandom(fdFile.Fd()) + } +} + +type sequentialReadsOption struct{} + +// SequentialReadsOption is an OpenOption that optimizes opened file handle for +// sequential reads, by calling fadvise() with POSIX_FADV_SEQUENTIAL on Linux +// systems to enable readahead. +var SequentialReadsOption OpenOption = &sequentialReadsOption{} + +// Apply implements the OpenOption interface. +func (sequentialReadsOption) Apply(f File) { + type fd interface { + Fd() uintptr + } + if fdFile, ok := f.(fd); ok { + _ = fadviseSequential(fdFile.Fd()) } } @@ -251,7 +275,7 @@ func LinkOrCopy(fs FS, oldname, newname string) error { if err == nil { return nil } - // Whitelist a handful of errors which we know won't be fixed by copying the + // Permit a handful of errors which we know won't be fixed by copying the // file. Note that we don't check for the specifics of the error code as it // isn't easy to do so in a portable manner. On Unix we'd have to check for // LinkError.Err == syscall.EXDEV. On Windows we'd have to check for diff --git a/github.com/cockroachdb/redact/README.md b/github.com/cockroachdb/redact/README.md index 293b96871a..38c248af36 100644 --- a/github.com/cockroachdb/redact/README.md +++ b/github.com/cockroachdb/redact/README.md @@ -1,2 +1,5 @@ # redact Utilities to redact Go strings for confidentiality + +Godoc link: https://pkg.go.dev/github.com/cockroachdb/redact?tab=doc + diff --git a/github.com/cockroachdb/redact/api.go b/github.com/cockroachdb/redact/api.go index 68394b3b32..3ed2e486df 100644 --- a/github.com/cockroachdb/redact/api.go +++ b/github.com/cockroachdb/redact/api.go @@ -17,7 +17,8 @@ package redact import "fmt" // SafeFormatter is implemented by object types that want to separate -// safe and non-safe information. +// safe and non-safe information when printed out by a Printf-like +// formatter. type SafeFormatter interface { // SafeFormat is like the Format method of fmt.Formatter, except // that it operates using a SafePrinter instead of a fmt.State for diff --git a/github.com/cockroachdb/redact/builder.go b/github.com/cockroachdb/redact/builder.go new file mode 100644 index 0000000000..a87d89756e --- /dev/null +++ b/github.com/cockroachdb/redact/builder.go @@ -0,0 +1,142 @@ +// Copyright 2020 The Cockroach Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +// implied. See the License for the specific language governing +// permissions and limitations under the License. + +package redact + +import ( + "bytes" + "io" + "unicode/utf8" +) + +// StringBuilder accumulates strings with optional redaction markers. +// +// It implements io.Writer but marks direct writes as redactable. +// To distinguish safe and unsafe bits, it also implements the SafeWriter +// interface. +type StringBuilder struct { + // we use bytes.Buffer internally to simplify the implementation of + // the SafeWriter interface. + buf bytes.Buffer +} + +// String returns the accumulated string, with redaction markers stripped out. +// To obtain the redactable string, call RedactableString(). +func (b *StringBuilder) String() string { return b.RedactableString().StripMarkers() } + +// RedactableString returns the accumulated string, including redaction markers. +func (b *StringBuilder) RedactableString() RedactableString { return RedactableString(b.buf.String()) } + +// RedactableString returns the accumulated bytes, including redaction markers. +func (b *StringBuilder) RedactableBytes() RedactableBytes { return RedactableBytes(b.buf.Bytes()) } + +// SafeFormat implements SafeFormatter. +func (b *StringBuilder) SafeFormat(p SafePrinter, _ rune) { + // We only support the %v / %s natural print here. + // Go supports other formatting verbs for strings: %x/%X/%q. + // + // We don't do this here, keeping in mind that the output + // of a SafeFormat must remain a redactable string. + // + // %x/%X cannot be implemented because they would turn redaction + // markers into hex codes, and the entire result string would + // appear safe for reporting, which would break the semantics + // of this package. + // + // %q cannot be implemented because it replaces non-ASCII characters + // with numeric unicode escapes, which breaks redaction + // markers too. + p.Print(b.RedactableString()) +} + +// Len returns the number of accumulated bytes, including redaction +// markers; b.Len() == len(b.RedactableString()). +func (b *StringBuilder) Len() int { return b.buf.Len() } + +// Cap returns the capacity of the builder's underlying byte slice. It is the +// total space allocated for the string being built and includes any bytes +// already written. +func (b *StringBuilder) Cap() int { return b.buf.Cap() } + +// Reset resets the Builder to be empty. +func (b *StringBuilder) Reset() { b.buf.Reset() } + +// StringBuilder implements io.Writer. +// Direct Write() calls are considered unsafe. +var _ io.Writer = (*StringBuilder)(nil) + +// Write implements the io.Writer interface. +func (b *StringBuilder) Write(s []byte) (int, error) { + b.UnsafeBytes(s) + return len(s), nil +} + +// StringBuilder implements SafeWriter. +var _ SafeWriter = (*StringBuilder)(nil) + +// Print is part of the SafeWriter interface. +func (b *StringBuilder) Print(args ...interface{}) { + _, _ = Fprint(&b.buf, args...) +} + +// Printf is part of the SafeWriter interface. +func (b *StringBuilder) Printf(format string, args ...interface{}) { + _, _ = Fprintf(&b.buf, format, args...) +} + +// SafeString is part of the SafeWriter interface. +func (b *StringBuilder) SafeString(s SafeString) { + w := escapeWriter{w: &b.buf, enclose: false} + _, _ = w.Write([]byte(s)) +} + +// SafeRune is part of the SafeWriter interface. +func (b *StringBuilder) SafeRune(s SafeRune) { + if s == startRedactable || s == endRedactable { + s = escapeMark + } + _, _ = b.buf.WriteRune(rune(s)) +} + +// UnsafeString is part of the SafeWriter interface. +func (b *StringBuilder) UnsafeString(s string) { + w := escapeWriter{w: &b.buf, enclose: true, strip: true} + _, _ = w.Write([]byte(s)) +} + +// UnsafeRune is part of the SafeWriter interface. +func (b *StringBuilder) UnsafeRune(s rune) { + _, _ = b.buf.WriteRune(startRedactable) + b.SafeRune(SafeRune(s)) + _, _ = b.buf.WriteRune(endRedactable) +} + +// UnsafeByte is part of the SafeWriter interface. +func (b *StringBuilder) UnsafeByte(s byte) { + _, _ = b.buf.WriteRune(startRedactable) + if s >= utf8.RuneSelf || + s == startRedactableBytes[0] || s == endRedactableBytes[0] { + // Unsafe byte. Escape it. + _, _ = b.buf.Write(escapeBytes) + } else { + _ = b.buf.WriteByte(s) + } + _, _ = b.buf.WriteRune(endRedactable) +} + +// UnsafeBytes is part of the SafeWriter interface. +func (b *StringBuilder) UnsafeBytes(s []byte) { + w := escapeWriter{w: &b.buf, enclose: true, strip: true} + _, _ = w.Write(s) +} diff --git a/github.com/cockroachdb/redact/doc.go b/github.com/cockroachdb/redact/doc.go index a1b4274000..ff37065e7f 100644 --- a/github.com/cockroachdb/redact/doc.go +++ b/github.com/cockroachdb/redact/doc.go @@ -20,4 +20,32 @@ // the boundaries of the current system, for example via telemetry // or crash reporting. Conversely, data is considered “unsafe†// until/unless it is known to be “safeâ€. +// +// Example use: +// +// redactable := redact.Sprintf("hello %s", "universe") +// +// // At this point, 'redactable' contains "hello ‹universe›". +// +// // This prints "hello universe": +// fmt.Println(redactable.StripMarkers()) +// +// // This reports "hello ‹×›": +// fmt.Println(redactable.Redact()) +// +// +// When defining your own custom types, you can define +// a SafeFormat method, implementing the redact.SafeFormatter +// interface in a way you'd otherwise implement fmt.Formatter. +// This is then recognized by this package's API automatically. +// +// Alternatively: +// +// - you can implement the SafeValue interface, which tells the +// redact package to always the default formatting of a type +// as safe and thus not included inside redaction markers. +// +// - you can include a value within redact.Safe() and redact.Unsafe() +// in redact.Sprintf / redact.Fprintf calls, to force +// the omission or inclusion of redaction markers. package redact diff --git a/github.com/cockroachdb/redact/internal/README.md b/github.com/cockroachdb/redact/internal/README.md new file mode 100644 index 0000000000..768a89545f --- /dev/null +++ b/github.com/cockroachdb/redact/internal/README.md @@ -0,0 +1,52 @@ +Overall code structure +====================== + +This directory imports the "printf" logic as-is from the Go standard +library and instruments it for redaction of sensitive data. + +Overall, the original Go code is structured as follows: + +- the top-level API functions (`Printf` `Fprintf` etc) instantiate a + "printer" struct called `pp`, then call the `doPrint*()` methods + on it. + +- the `pp` contains a byte slice (`pp.buf`, type `buffer`) that + accumulates the result of formatting *regardless of the final output + of the API* - i.e. a buffer is used even when using `Fprint` to an + `io.Writer`. Only after the `doPrint()` method finishes, is the + `buffer` copied to the final `io.Writer` in the `Fprint*` variants. + +- each of the `doPrint` methods does some analysis on the argument + list - quite simple for e.g. `doPrintln`, more intricate for + `doPrintf`. As part of the analysis it emits "spacing" or no-op + bytes directly on the `pp.buf`. For example `doPrint` emits spaces + between arguments directly, `doPrintf` emits the non-formatting + characters from the format string, as well as certain *constant* + error strings (e.g. `%(BADPREC)`). + +At this point **we make the following assumption**: there is no data +from the printed *arguments* that is emitted to `pp.buf` directly by a +`doPrint()` method. This is true of the implementation as of Go 1.14. + +From this point, this package contains a *custom patch* onto the +default implementation: the calls to `p.printArg` performed by the +`doPrint()` methods are redirected to a function callback that is +injected by the `redact` package. + +Refreshing the sources +====================== + +The files in this directory have been imported from + +`$GOROOT/src/fmt/{format,print}.go` + +and + +`$GOROOT/src/internal/fmtsort` + +And patched using the included `.diff` files. + +To upgrade to a newer Go implementation, import the files anew and +re-apply the patches. + +See the script `refresh.sh` for details. diff --git a/github.com/cockroachdb/redact/internal/fmtsort/sort.go b/github.com/cockroachdb/redact/internal/fmtsort/sort.go new file mode 100644 index 0000000000..9d2d580aa8 --- /dev/null +++ b/github.com/cockroachdb/redact/internal/fmtsort/sort.go @@ -0,0 +1,222 @@ +// Code generated from the Go standard library. DO NOT EDIT +// GENERATED FILE DO NOT EDIT +// Copyright 2018 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +// Package fmtsort provides a general stable ordering mechanism +// for maps, on behalf of the fmt and text/template packages. +// It is not guaranteed to be efficient and works only for types +// that are valid map keys. +package fmtsort + +import ( + "reflect" + "sort" +) + +// Note: Throughout this package we avoid calling reflect.Value.Interface as +// it is not always legal to do so and it's easier to avoid the issue than to face it. + +// SortedMap represents a map's keys and values. The keys and values are +// aligned in index order: Value[i] is the value in the map corresponding to Key[i]. +type SortedMap struct { + Key []reflect.Value + Value []reflect.Value +} + +func (o *SortedMap) Len() int { return len(o.Key) } +func (o *SortedMap) Less(i, j int) bool { return compare(o.Key[i], o.Key[j]) < 0 } +func (o *SortedMap) Swap(i, j int) { + o.Key[i], o.Key[j] = o.Key[j], o.Key[i] + o.Value[i], o.Value[j] = o.Value[j], o.Value[i] +} + +// Sort accepts a map and returns a SortedMap that has the same keys and +// values but in a stable sorted order according to the keys, modulo issues +// raised by unorderable key values such as NaNs. +// +// The ordering rules are more general than with Go's < operator: +// +// - when applicable, nil compares low +// - ints, floats, and strings order by < +// - NaN compares less than non-NaN floats +// - bool compares false before true +// - complex compares real, then imag +// - pointers compare by machine address +// - channel values compare by machine address +// - structs compare each field in turn +// - arrays compare each element in turn. +// Otherwise identical arrays compare by length. +// - interface values compare first by reflect.Type describing the concrete type +// and then by concrete value as described in the previous rules. +// +func Sort(mapValue reflect.Value) *SortedMap { + if mapValue.Type().Kind() != reflect.Map { + return nil + } + // Note: this code is arranged to not panic even in the presence + // of a concurrent map update. The runtime is responsible for + // yelling loudly if that happens. See issue 33275. + n := mapValue.Len() + key := make([]reflect.Value, 0, n) + value := make([]reflect.Value, 0, n) + iter := mapValue.MapRange() + for iter.Next() { + key = append(key, iter.Key()) + value = append(value, iter.Value()) + } + sorted := &SortedMap{ + Key: key, + Value: value, + } + sort.Stable(sorted) + return sorted +} + +// compare compares two values of the same type. It returns -1, 0, 1 +// according to whether a > b (1), a == b (0), or a < b (-1). +// If the types differ, it returns -1. +// See the comment on Sort for the comparison rules. +func compare(aVal, bVal reflect.Value) int { + aType, bType := aVal.Type(), bVal.Type() + if aType != bType { + return -1 // No good answer possible, but don't return 0: they're not equal. + } + switch aVal.Kind() { + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + a, b := aVal.Int(), bVal.Int() + switch { + case a < b: + return -1 + case a > b: + return 1 + default: + return 0 + } + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + a, b := aVal.Uint(), bVal.Uint() + switch { + case a < b: + return -1 + case a > b: + return 1 + default: + return 0 + } + case reflect.String: + a, b := aVal.String(), bVal.String() + switch { + case a < b: + return -1 + case a > b: + return 1 + default: + return 0 + } + case reflect.Float32, reflect.Float64: + return floatCompare(aVal.Float(), bVal.Float()) + case reflect.Complex64, reflect.Complex128: + a, b := aVal.Complex(), bVal.Complex() + if c := floatCompare(real(a), real(b)); c != 0 { + return c + } + return floatCompare(imag(a), imag(b)) + case reflect.Bool: + a, b := aVal.Bool(), bVal.Bool() + switch { + case a == b: + return 0 + case a: + return 1 + default: + return -1 + } + case reflect.Ptr: + a, b := aVal.Pointer(), bVal.Pointer() + switch { + case a < b: + return -1 + case a > b: + return 1 + default: + return 0 + } + case reflect.Chan: + if c, ok := nilCompare(aVal, bVal); ok { + return c + } + ap, bp := aVal.Pointer(), bVal.Pointer() + switch { + case ap < bp: + return -1 + case ap > bp: + return 1 + default: + return 0 + } + case reflect.Struct: + for i := 0; i < aVal.NumField(); i++ { + if c := compare(aVal.Field(i), bVal.Field(i)); c != 0 { + return c + } + } + return 0 + case reflect.Array: + for i := 0; i < aVal.Len(); i++ { + if c := compare(aVal.Index(i), bVal.Index(i)); c != 0 { + return c + } + } + return 0 + case reflect.Interface: + if c, ok := nilCompare(aVal, bVal); ok { + return c + } + c := compare(reflect.ValueOf(aVal.Elem().Type()), reflect.ValueOf(bVal.Elem().Type())) + if c != 0 { + return c + } + return compare(aVal.Elem(), bVal.Elem()) + default: + // Certain types cannot appear as keys (maps, funcs, slices), but be explicit. + panic("bad type in compare: " + aType.String()) + } +} + +// nilCompare checks whether either value is nil. If not, the boolean is false. +// If either value is nil, the boolean is true and the integer is the comparison +// value. The comparison is defined to be 0 if both are nil, otherwise the one +// nil value compares low. Both arguments must represent a chan, func, +// interface, map, pointer, or slice. +func nilCompare(aVal, bVal reflect.Value) (int, bool) { + if aVal.IsNil() { + if bVal.IsNil() { + return 0, true + } + return -1, true + } + if bVal.IsNil() { + return 1, true + } + return 0, false +} + +// floatCompare compares two floating-point values. NaNs compare low. +func floatCompare(a, b float64) int { + switch { + case isNaN(a): + return -1 // No good answer if b is a NaN so don't bother checking. + case isNaN(b): + return 1 + case a < b: + return -1 + case a > b: + return 1 + } + return 0 +} + +func isNaN(a float64) bool { + return a != a +} diff --git a/github.com/cockroachdb/redact/internal/format.go b/github.com/cockroachdb/redact/internal/format.go new file mode 100644 index 0000000000..1082879b5b --- /dev/null +++ b/github.com/cockroachdb/redact/internal/format.go @@ -0,0 +1,584 @@ +// Code generated from the Go standard library. DO NOT EDIT +// GENERATED FILE DO NOT EDIT +// Copyright 2009 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package fmt + +import ( + "strconv" + "unicode/utf8" +) + +const ( + ldigits = "0123456789abcdefx" + udigits = "0123456789ABCDEFX" +) + +const ( + signed = true + unsigned = false +) + +// flags placed in a separate struct for easy clearing. +type fmtFlags struct { + widPresent bool + precPresent bool + minus bool + plus bool + sharp bool + space bool + zero bool + + // For the formats %+v %#v, we set the plusV/sharpV flags + // and clear the plus/sharp flags since %+v and %#v are in effect + // different, flagless formats set at the top level. + plusV bool + sharpV bool +} + +// A fmt is the raw formatter used by Printf etc. +// It prints into a buffer that must be set up separately. +type fmt struct { + buf *buffer + + fmtFlags + + wid int // width + prec int // precision + + // intbuf is large enough to store %b of an int64 with a sign and + // avoids padding at the end of the struct on 32 bit architectures. + intbuf [68]byte +} + +func (f *fmt) clearflags() { + f.fmtFlags = fmtFlags{} +} + +func (f *fmt) init(buf *buffer) { + f.buf = buf + f.clearflags() +} + +// writePadding generates n bytes of padding. +func (f *fmt) writePadding(n int) { + if n <= 0 { // No padding bytes needed. + return + } + buf := *f.buf + oldLen := len(buf) + newLen := oldLen + n + // Make enough room for padding. + if newLen > cap(buf) { + buf = make(buffer, cap(buf)*2+n) + copy(buf, *f.buf) + } + // Decide which byte the padding should be filled with. + padByte := byte(' ') + if f.zero { + padByte = byte('0') + } + // Fill padding with padByte. + padding := buf[oldLen:newLen] + for i := range padding { + padding[i] = padByte + } + *f.buf = buf[:newLen] +} + +// pad appends b to f.buf, padded on left (!f.minus) or right (f.minus). +func (f *fmt) pad(b []byte) { + if !f.widPresent || f.wid == 0 { + f.buf.write(b) + return + } + width := f.wid - utf8.RuneCount(b) + if !f.minus { + // left padding + f.writePadding(width) + f.buf.write(b) + } else { + // right padding + f.buf.write(b) + f.writePadding(width) + } +} + +// padString appends s to f.buf, padded on left (!f.minus) or right (f.minus). +func (f *fmt) padString(s string) { + if !f.widPresent || f.wid == 0 { + f.buf.writeString(s) + return + } + width := f.wid - utf8.RuneCountInString(s) + if !f.minus { + // left padding + f.writePadding(width) + f.buf.writeString(s) + } else { + // right padding + f.buf.writeString(s) + f.writePadding(width) + } +} + +// fmtBoolean formats a boolean. +func (f *fmt) fmtBoolean(v bool) { + if v { + f.padString("true") + } else { + f.padString("false") + } +} + +// fmtUnicode formats a uint64 as "U+0078" or with f.sharp set as "U+0078 'x'". +func (f *fmt) fmtUnicode(u uint64) { + buf := f.intbuf[0:] + + // With default precision set the maximum needed buf length is 18 + // for formatting -1 with %#U ("U+FFFFFFFFFFFFFFFF") which fits + // into the already allocated intbuf with a capacity of 68 bytes. + prec := 4 + if f.precPresent && f.prec > 4 { + prec = f.prec + // Compute space needed for "U+" , number, " '", character, "'". + width := 2 + prec + 2 + utf8.UTFMax + 1 + if width > len(buf) { + buf = make([]byte, width) + } + } + + // Format into buf, ending at buf[i]. Formatting numbers is easier right-to-left. + i := len(buf) + + // For %#U we want to add a space and a quoted character at the end of the buffer. + if f.sharp && u <= utf8.MaxRune && strconv.IsPrint(rune(u)) { + i-- + buf[i] = '\'' + i -= utf8.RuneLen(rune(u)) + utf8.EncodeRune(buf[i:], rune(u)) + i-- + buf[i] = '\'' + i-- + buf[i] = ' ' + } + // Format the Unicode code point u as a hexadecimal number. + for u >= 16 { + i-- + buf[i] = udigits[u&0xF] + prec-- + u >>= 4 + } + i-- + buf[i] = udigits[u] + prec-- + // Add zeros in front of the number until requested precision is reached. + for prec > 0 { + i-- + buf[i] = '0' + prec-- + } + // Add a leading "U+". + i-- + buf[i] = '+' + i-- + buf[i] = 'U' + + oldZero := f.zero + f.zero = false + f.pad(buf[i:]) + f.zero = oldZero +} + +// fmtInteger formats signed and unsigned integers. +func (f *fmt) fmtInteger(u uint64, base int, isSigned bool, verb rune, digits string) { + negative := isSigned && int64(u) < 0 + if negative { + u = -u + } + + buf := f.intbuf[0:] + // The already allocated f.intbuf with a capacity of 68 bytes + // is large enough for integer formatting when no precision or width is set. + if f.widPresent || f.precPresent { + // Account 3 extra bytes for possible addition of a sign and "0x". + width := 3 + f.wid + f.prec // wid and prec are always positive. + if width > len(buf) { + // We're going to need a bigger boat. + buf = make([]byte, width) + } + } + + // Two ways to ask for extra leading zero digits: %.3d or %03d. + // If both are specified the f.zero flag is ignored and + // padding with spaces is used instead. + prec := 0 + if f.precPresent { + prec = f.prec + // Precision of 0 and value of 0 means "print nothing" but padding. + if prec == 0 && u == 0 { + oldZero := f.zero + f.zero = false + f.writePadding(f.wid) + f.zero = oldZero + return + } + } else if f.zero && f.widPresent { + prec = f.wid + if negative || f.plus || f.space { + prec-- // leave room for sign + } + } + + // Because printing is easier right-to-left: format u into buf, ending at buf[i]. + // We could make things marginally faster by splitting the 32-bit case out + // into a separate block but it's not worth the duplication, so u has 64 bits. + i := len(buf) + // Use constants for the division and modulo for more efficient code. + // Switch cases ordered by popularity. + switch base { + case 10: + for u >= 10 { + i-- + next := u / 10 + buf[i] = byte('0' + u - next*10) + u = next + } + case 16: + for u >= 16 { + i-- + buf[i] = digits[u&0xF] + u >>= 4 + } + case 8: + for u >= 8 { + i-- + buf[i] = byte('0' + u&7) + u >>= 3 + } + case 2: + for u >= 2 { + i-- + buf[i] = byte('0' + u&1) + u >>= 1 + } + default: + panic("fmt: unknown base; can't happen") + } + i-- + buf[i] = digits[u] + for i > 0 && prec > len(buf)-i { + i-- + buf[i] = '0' + } + + // Various prefixes: 0x, -, etc. + if f.sharp { + switch base { + case 2: + // Add a leading 0b. + i-- + buf[i] = 'b' + i-- + buf[i] = '0' + case 8: + if buf[i] != '0' { + i-- + buf[i] = '0' + } + case 16: + // Add a leading 0x or 0X. + i-- + buf[i] = digits[16] + i-- + buf[i] = '0' + } + } + if verb == 'O' { + i-- + buf[i] = 'o' + i-- + buf[i] = '0' + } + + if negative { + i-- + buf[i] = '-' + } else if f.plus { + i-- + buf[i] = '+' + } else if f.space { + i-- + buf[i] = ' ' + } + + // Left padding with zeros has already been handled like precision earlier + // or the f.zero flag is ignored due to an explicitly set precision. + oldZero := f.zero + f.zero = false + f.pad(buf[i:]) + f.zero = oldZero +} + +// truncate truncates the string s to the specified precision, if present. +func (f *fmt) truncateString(s string) string { + if f.precPresent { + n := f.prec + for i := range s { + n-- + if n < 0 { + return s[:i] + } + } + } + return s +} + +// truncate truncates the byte slice b as a string of the specified precision, if present. +func (f *fmt) truncate(b []byte) []byte { + if f.precPresent { + n := f.prec + for i := 0; i < len(b); { + n-- + if n < 0 { + return b[:i] + } + wid := 1 + if b[i] >= utf8.RuneSelf { + _, wid = utf8.DecodeRune(b[i:]) + } + i += wid + } + } + return b +} + +// fmtS formats a string. +func (f *fmt) fmtS(s string) { + s = f.truncateString(s) + f.padString(s) +} + +// fmtBs formats the byte slice b as if it was formatted as string with fmtS. +func (f *fmt) fmtBs(b []byte) { + b = f.truncate(b) + f.pad(b) +} + +// fmtSbx formats a string or byte slice as a hexadecimal encoding of its bytes. +func (f *fmt) fmtSbx(s string, b []byte, digits string) { + length := len(b) + if b == nil { + // No byte slice present. Assume string s should be encoded. + length = len(s) + } + // Set length to not process more bytes than the precision demands. + if f.precPresent && f.prec < length { + length = f.prec + } + // Compute width of the encoding taking into account the f.sharp and f.space flag. + width := 2 * length + if width > 0 { + if f.space { + // Each element encoded by two hexadecimals will get a leading 0x or 0X. + if f.sharp { + width *= 2 + } + // Elements will be separated by a space. + width += length - 1 + } else if f.sharp { + // Only a leading 0x or 0X will be added for the whole string. + width += 2 + } + } else { // The byte slice or string that should be encoded is empty. + if f.widPresent { + f.writePadding(f.wid) + } + return + } + // Handle padding to the left. + if f.widPresent && f.wid > width && !f.minus { + f.writePadding(f.wid - width) + } + // Write the encoding directly into the output buffer. + buf := *f.buf + if f.sharp { + // Add leading 0x or 0X. + buf = append(buf, '0', digits[16]) + } + var c byte + for i := 0; i < length; i++ { + if f.space && i > 0 { + // Separate elements with a space. + buf = append(buf, ' ') + if f.sharp { + // Add leading 0x or 0X for each element. + buf = append(buf, '0', digits[16]) + } + } + if b != nil { + c = b[i] // Take a byte from the input byte slice. + } else { + c = s[i] // Take a byte from the input string. + } + // Encode each byte as two hexadecimal digits. + buf = append(buf, digits[c>>4], digits[c&0xF]) + } + *f.buf = buf + // Handle padding to the right. + if f.widPresent && f.wid > width && f.minus { + f.writePadding(f.wid - width) + } +} + +// fmtSx formats a string as a hexadecimal encoding of its bytes. +func (f *fmt) fmtSx(s, digits string) { + f.fmtSbx(s, nil, digits) +} + +// fmtBx formats a byte slice as a hexadecimal encoding of its bytes. +func (f *fmt) fmtBx(b []byte, digits string) { + f.fmtSbx("", b, digits) +} + +// fmtQ formats a string as a double-quoted, escaped Go string constant. +// If f.sharp is set a raw (backquoted) string may be returned instead +// if the string does not contain any control characters other than tab. +func (f *fmt) fmtQ(s string) { + s = f.truncateString(s) + if f.sharp && strconv.CanBackquote(s) { + f.padString("`" + s + "`") + return + } + buf := f.intbuf[:0] + if f.plus { + f.pad(strconv.AppendQuoteToASCII(buf, s)) + } else { + f.pad(strconv.AppendQuote(buf, s)) + } +} + +// fmtC formats an integer as a Unicode character. +// If the character is not valid Unicode, it will print '\ufffd'. +func (f *fmt) fmtC(c uint64) { + r := rune(c) + if c > utf8.MaxRune { + r = utf8.RuneError + } + buf := f.intbuf[:0] + w := utf8.EncodeRune(buf[:utf8.UTFMax], r) + f.pad(buf[:w]) +} + +// fmtQc formats an integer as a single-quoted, escaped Go character constant. +// If the character is not valid Unicode, it will print '\ufffd'. +func (f *fmt) fmtQc(c uint64) { + r := rune(c) + if c > utf8.MaxRune { + r = utf8.RuneError + } + buf := f.intbuf[:0] + if f.plus { + f.pad(strconv.AppendQuoteRuneToASCII(buf, r)) + } else { + f.pad(strconv.AppendQuoteRune(buf, r)) + } +} + +// fmtFloat formats a float64. It assumes that verb is a valid format specifier +// for strconv.AppendFloat and therefore fits into a byte. +func (f *fmt) fmtFloat(v float64, size int, verb rune, prec int) { + // Explicit precision in format specifier overrules default precision. + if f.precPresent { + prec = f.prec + } + // Format number, reserving space for leading + sign if needed. + num := strconv.AppendFloat(f.intbuf[:1], v, byte(verb), prec, size) + if num[1] == '-' || num[1] == '+' { + num = num[1:] + } else { + num[0] = '+' + } + // f.space means to add a leading space instead of a "+" sign unless + // the sign is explicitly asked for by f.plus. + if f.space && num[0] == '+' && !f.plus { + num[0] = ' ' + } + // Special handling for infinities and NaN, + // which don't look like a number so shouldn't be padded with zeros. + if num[1] == 'I' || num[1] == 'N' { + oldZero := f.zero + f.zero = false + // Remove sign before NaN if not asked for. + if num[1] == 'N' && !f.space && !f.plus { + num = num[1:] + } + f.pad(num) + f.zero = oldZero + return + } + // The sharp flag forces printing a decimal point for non-binary formats + // and retains trailing zeros, which we may need to restore. + if f.sharp && verb != 'b' { + digits := 0 + switch verb { + case 'v', 'g', 'G', 'x': + digits = prec + // If no precision is set explicitly use a precision of 6. + if digits == -1 { + digits = 6 + } + } + + // Buffer pre-allocated with enough room for + // exponent notations of the form "e+123" or "p-1023". + var tailBuf [6]byte + tail := tailBuf[:0] + + hasDecimalPoint := false + // Starting from i = 1 to skip sign at num[0]. + for i := 1; i < len(num); i++ { + switch num[i] { + case '.': + hasDecimalPoint = true + case 'p', 'P': + tail = append(tail, num[i:]...) + num = num[:i] + case 'e', 'E': + if verb != 'x' && verb != 'X' { + tail = append(tail, num[i:]...) + num = num[:i] + break + } + fallthrough + default: + digits-- + } + } + if !hasDecimalPoint { + num = append(num, '.') + } + for digits > 0 { + num = append(num, '0') + digits-- + } + num = append(num, tail...) + } + // We want a sign if asked for and if the sign is not positive. + if f.plus || num[0] != '+' { + // If we're zero padding to the left we want the sign before the leading zeros. + // Achieve this by writing the sign out and then padding the unsigned number. + if f.zero && f.widPresent && f.wid > len(num) { + f.buf.writeByte(num[0]) + f.writePadding(f.wid - len(num)) + f.buf.write(num[1:]) + return + } + f.pad(num) + return + } + // No sign to show and the number is positive; just print the unsigned number. + f.pad(num[1:]) +} diff --git a/github.com/cockroachdb/redact/internal/hooks.go b/github.com/cockroachdb/redact/internal/hooks.go new file mode 100644 index 0000000000..246e600de6 --- /dev/null +++ b/github.com/cockroachdb/redact/internal/hooks.go @@ -0,0 +1,83 @@ +// Copyright 2020 The Cockroach Authors. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +// implied. See the License for the specific language governing +// permissions and limitations under the License. + +package fmt + +// printArg overrides the original implementation by redirecting the +// call to a substitute function in the printer, if available. +func (p *pp) printArg(a interface{}, verb rune) { + if p.printArgSubstituteFn != nil { + p.substituteState = p.printArgSubstituteFn(p, a, verb) + } else { + p.printArgOrig(a, verb) + } +} + +// InternalPrinter exposes pp to the redact package. +type InternalPrinter = pp + +// InternalBuffer exposes buffer to the redact package. +type InternalBuffer = buffer + +// NewInternalPrinter exposes pp allocation to the redact package. +func NewInternalPrinter() *InternalPrinter { return newPrinter() } + +// SetHook connects an outer printer to the inner printer. +func SetHook( + p *InternalPrinter, fn func(p *InternalPrinter, arg interface{}, verb rune) (newState int), +) { + p.printArgSubstituteFn = fn +} + +// Free exposes pp deallocation to the redact package. +func Free(p *InternalPrinter) { + p.substituteState = 0 + p.printArgSubstituteFn = nil + p.free() +} + +// Buf exposes the string buffer to the redact package. +func Buf(p *InternalPrinter) []byte { return []byte(p.buf) } + +// GetState exposes the state to the redact package. +func GetState(p *InternalPrinter) int { + return p.substituteState +} + +// SetState exposes the string buffer to the redact package. +func SetState(p *InternalPrinter, b []byte) { + p.buf = buffer(b) + p.substituteState = len(p.buf) +} + +// Append adds bytes to the buffer. +func Append(p *InternalPrinter, b []byte) { p.buf.write(b) } + +// PrintArg exposes the printArgOrig() method to the redact package. +func PrintArg(p *InternalPrinter, a interface{}, verb rune) { p.printArgOrig(a, verb) } + +// DoPrint exposes the doPrint() method to the redact package. +func DoPrint(p *InternalPrinter, a []interface{}) { p.doPrint(a) } + +// DoPrintf exposes the doPrintf() method to the redact package. +func DoPrintf(p *InternalPrinter, format string, a []interface{}) { p.doPrintf(format, a) } + +// SetCollectError enables wrapped error collection. +func SetCollectError(p *InternalPrinter) { p.wrapErrs = true } + +// CollectingError exposes wrapped error collection. +func CollectingError(p *InternalPrinter) bool { return p.wrapErrs } + +// WrappedError retrieves the wrapped error if found. +func WrappedError(p *InternalPrinter) error { return p.wrappedErr } diff --git a/github.com/cockroachdb/redact/internal/print.go b/github.com/cockroachdb/redact/internal/print.go new file mode 100644 index 0000000000..b7cdd3fa60 --- /dev/null +++ b/github.com/cockroachdb/redact/internal/print.go @@ -0,0 +1,1189 @@ +// Code generated from print.go.orig. DO NOT EDIT +// GENERATED FILE DO NOT EDIT +// +// Copyright 2009 The Go Authors. All rights reserved. +// Use of this source code is governed by a BSD-style +// license that can be found in the LICENSE file. + +package fmt + +import ( + // CUSTOM: needed to avoid a type mismatch on Formatter. + origFmt "fmt" + "io" + "os" + "reflect" + "sync" + "unicode/utf8" + + // CUSTOM: our own import since we can't use internal from go stdlib. + "github.com/cockroachdb/redact/internal/fmtsort" +) + +// Strings for use with buffer.WriteString. +// This is less overhead than using buffer.Write with byte arrays. +const ( + commaSpaceString = ", " + nilAngleString = "" + nilParenString = "(nil)" + nilString = "nil" + mapString = "map[" + percentBangString = "%!" + missingString = "(MISSING)" + badIndexString = "(BADINDEX)" + panicString = "(PANIC=" + extraString = "%!(EXTRA " + badWidthString = "%!(BADWIDTH)" + badPrecString = "%!(BADPREC)" + noVerbString = "%!(NOVERB)" + invReflectString = "" +) + +// State represents the printer state passed to custom formatters. +// It provides access to the io.Writer interface plus information about +// the flags and options for the operand's format specifier. +type State interface { + // Write is the function to call to emit formatted output to be printed. + Write(b []byte) (n int, err error) + // Width returns the value of the width option and whether it has been set. + Width() (wid int, ok bool) + // Precision returns the value of the precision option and whether it has been set. + Precision() (prec int, ok bool) + + // Flag reports whether the flag c, a character, has been set. + Flag(c int) bool +} + +// Formatter is the interface implemented by values with a custom formatter. +// The implementation of Format may call Sprint(f) or Fprint(f) etc. +// to generate its output. +type Formatter interface { + // CUSTOM: refer to the original type, not the one defined here. + Format(f origFmt.State, c rune) +} + +// Stringer is implemented by any value that has a String method, +// which defines the ``native'' format for that value. +// The String method is used to print values passed as an operand +// to any format that accepts a string or to an unformatted printer +// such as Print. +type Stringer interface { + String() string +} + +// GoStringer is implemented by any value that has a GoString method, +// which defines the Go syntax for that value. +// The GoString method is used to print values passed as an operand +// to a %#v format. +type GoStringer interface { + GoString() string +} + +// Use simple []byte instead of bytes.Buffer to avoid large dependency. +type buffer []byte + +func (b *buffer) write(p []byte) { + *b = append(*b, p...) +} + +func (b *buffer) writeString(s string) { + *b = append(*b, s...) +} + +func (b *buffer) writeByte(c byte) { + *b = append(*b, c) +} + +func (bp *buffer) writeRune(r rune) { + if r < utf8.RuneSelf { + *bp = append(*bp, byte(r)) + return + } + + b := *bp + n := len(b) + for n+utf8.UTFMax > cap(b) { + b = append(b, 0) + } + w := utf8.EncodeRune(b[n:n+utf8.UTFMax], r) + *bp = b[:n+w] +} + +// pp is used to store a printer's state and is reused with sync.Pool to avoid allocations. +type pp struct { + buf buffer + + // CUSTOM: hook fn for the redact package. + printArgSubstituteFn func(p *pp, a interface{}, verb rune) (newState int) + substituteState int + + // arg holds the current item, as an interface{}. + arg interface{} + + // value is used instead of arg for reflect values. + value reflect.Value + + // fmt is used to format basic items such as integers or strings. + fmt fmt + + // reordered records whether the format string used argument reordering. + reordered bool + // goodArgNum records whether the most recent reordering directive was valid. + goodArgNum bool + // panicking is set by catchPanic to avoid infinite panic, recover, panic, ... recursion. + panicking bool + // erroring is set when printing an error string to guard against calling handleMethods. + erroring bool + // wrapErrs is set when the format string may contain a %w verb. + wrapErrs bool + // wrappedErr records the target of the %w verb. + wrappedErr error +} + +var ppFree = sync.Pool{ + New: func() interface{} { return new(pp) }, +} + +// newPrinter allocates a new pp struct or grabs a cached one. +func newPrinter() *pp { + p := ppFree.Get().(*pp) + p.panicking = false + p.erroring = false + p.wrapErrs = false + p.fmt.init(&p.buf) + return p +} + +// free saves used pp structs in ppFree; avoids an allocation per invocation. +func (p *pp) free() { + // Proper usage of a sync.Pool requires each entry to have approximately + // the same memory cost. To obtain this property when the stored type + // contains a variably-sized buffer, we add a hard limit on the maximum buffer + // to place back in the pool. + // + // See https://golang.org/issue/23199 + if cap(p.buf) > 64<<10 { + return + } + + p.buf = p.buf[:0] + p.arg = nil + p.value = reflect.Value{} + p.wrappedErr = nil + ppFree.Put(p) +} + +func (p *pp) Width() (wid int, ok bool) { return p.fmt.wid, p.fmt.widPresent } + +func (p *pp) Precision() (prec int, ok bool) { return p.fmt.prec, p.fmt.precPresent } + +func (p *pp) Flag(b int) bool { + switch b { + case '-': + return p.fmt.minus + case '+': + return p.fmt.plus || p.fmt.plusV + case '#': + return p.fmt.sharp || p.fmt.sharpV + case ' ': + return p.fmt.space + case '0': + return p.fmt.zero + } + return false +} + +// Implement Write so we can call Fprintf on a pp (through State), for +// recursive use in custom verbs. +func (p *pp) Write(b []byte) (ret int, err error) { + p.buf.write(b) + return len(b), nil +} + +// Implement WriteString so that we can call io.WriteString +// on a pp (through state), for efficiency. +func (p *pp) WriteString(s string) (ret int, err error) { + p.buf.writeString(s) + return len(s), nil +} + +// These routines end in 'f' and take a format string. + +// Fprintf formats according to a format specifier and writes to w. +// It returns the number of bytes written and any write error encountered. +func Fprintf(w io.Writer, format string, a ...interface{}) (n int, err error) { + p := newPrinter() + p.doPrintf(format, a) + n, err = w.Write(p.buf) + p.free() + return +} + +// Printf formats according to a format specifier and writes to standard output. +// It returns the number of bytes written and any write error encountered. +func Printf(format string, a ...interface{}) (n int, err error) { + return Fprintf(os.Stdout, format, a...) +} + +// Sprintf formats according to a format specifier and returns the resulting string. +func Sprintf(format string, a ...interface{}) string { + p := newPrinter() + p.doPrintf(format, a) + s := string(p.buf) + p.free() + return s +} + +// These routines do not take a format string + +// Fprint formats using the default formats for its operands and writes to w. +// Spaces are added between operands when neither is a string. +// It returns the number of bytes written and any write error encountered. +func Fprint(w io.Writer, a ...interface{}) (n int, err error) { + p := newPrinter() + p.doPrint(a) + n, err = w.Write(p.buf) + p.free() + return +} + +// Print formats using the default formats for its operands and writes to standard output. +// Spaces are added between operands when neither is a string. +// It returns the number of bytes written and any write error encountered. +func Print(a ...interface{}) (n int, err error) { + return Fprint(os.Stdout, a...) +} + +// Sprint formats using the default formats for its operands and returns the resulting string. +// Spaces are added between operands when neither is a string. +func Sprint(a ...interface{}) string { + p := newPrinter() + p.doPrint(a) + s := string(p.buf) + p.free() + return s +} + +// These routines end in 'ln', do not take a format string, +// always add spaces between operands, and add a newline +// after the last operand. + +// Fprintln formats using the default formats for its operands and writes to w. +// Spaces are always added between operands and a newline is appended. +// It returns the number of bytes written and any write error encountered. +func Fprintln(w io.Writer, a ...interface{}) (n int, err error) { + p := newPrinter() + p.doPrintln(a) + n, err = w.Write(p.buf) + p.free() + return +} + +// Println formats using the default formats for its operands and writes to standard output. +// Spaces are always added between operands and a newline is appended. +// It returns the number of bytes written and any write error encountered. +func Println(a ...interface{}) (n int, err error) { + return Fprintln(os.Stdout, a...) +} + +// Sprintln formats using the default formats for its operands and returns the resulting string. +// Spaces are always added between operands and a newline is appended. +func Sprintln(a ...interface{}) string { + p := newPrinter() + p.doPrintln(a) + s := string(p.buf) + p.free() + return s +} + +// getField gets the i'th field of the struct value. +// If the field is itself is an interface, return a value for +// the thing inside the interface, not the interface itself. +func getField(v reflect.Value, i int) reflect.Value { + val := v.Field(i) + if val.Kind() == reflect.Interface && !val.IsNil() { + val = val.Elem() + } + return val +} + +// tooLarge reports whether the magnitude of the integer is +// too large to be used as a formatting width or precision. +func tooLarge(x int) bool { + const max int = 1e6 + return x > max || x < -max +} + +// parsenum converts ASCII to integer. num is 0 (and isnum is false) if no number present. +func parsenum(s string, start, end int) (num int, isnum bool, newi int) { + if start >= end { + return 0, false, end + } + for newi = start; newi < end && '0' <= s[newi] && s[newi] <= '9'; newi++ { + if tooLarge(num) { + return 0, false, end // Overflow; crazy long number most likely. + } + num = num*10 + int(s[newi]-'0') + isnum = true + } + return +} + +func (p *pp) unknownType(v reflect.Value) { + if !v.IsValid() { + p.buf.writeString(nilAngleString) + return + } + p.buf.writeByte('?') + p.buf.writeString(v.Type().String()) + p.buf.writeByte('?') +} + +func (p *pp) badVerb(verb rune) { + p.erroring = true + p.buf.writeString(percentBangString) + p.buf.writeRune(verb) + p.buf.writeByte('(') + switch { + case p.arg != nil: + p.buf.writeString(reflect.TypeOf(p.arg).String()) + p.buf.writeByte('=') + p.printArg(p.arg, 'v') + case p.value.IsValid(): + p.buf.writeString(p.value.Type().String()) + p.buf.writeByte('=') + p.printValue(p.value, 'v', 0) + default: + p.buf.writeString(nilAngleString) + } + p.buf.writeByte(')') + p.erroring = false +} + +func (p *pp) fmtBool(v bool, verb rune) { + switch verb { + case 't', 'v': + p.fmt.fmtBoolean(v) + default: + p.badVerb(verb) + } +} + +// fmt0x64 formats a uint64 in hexadecimal and prefixes it with 0x or +// not, as requested, by temporarily setting the sharp flag. +func (p *pp) fmt0x64(v uint64, leading0x bool) { + sharp := p.fmt.sharp + p.fmt.sharp = leading0x + p.fmt.fmtInteger(v, 16, unsigned, 'v', ldigits) + p.fmt.sharp = sharp +} + +// fmtInteger formats a signed or unsigned integer. +func (p *pp) fmtInteger(v uint64, isSigned bool, verb rune) { + switch verb { + case 'v': + if p.fmt.sharpV && !isSigned { + p.fmt0x64(v, true) + } else { + p.fmt.fmtInteger(v, 10, isSigned, verb, ldigits) + } + case 'd': + p.fmt.fmtInteger(v, 10, isSigned, verb, ldigits) + case 'b': + p.fmt.fmtInteger(v, 2, isSigned, verb, ldigits) + case 'o', 'O': + p.fmt.fmtInteger(v, 8, isSigned, verb, ldigits) + case 'x': + p.fmt.fmtInteger(v, 16, isSigned, verb, ldigits) + case 'X': + p.fmt.fmtInteger(v, 16, isSigned, verb, udigits) + case 'c': + p.fmt.fmtC(v) + case 'q': + if v <= utf8.MaxRune { + p.fmt.fmtQc(v) + } else { + p.badVerb(verb) + } + case 'U': + p.fmt.fmtUnicode(v) + default: + p.badVerb(verb) + } +} + +// fmtFloat formats a float. The default precision for each verb +// is specified as last argument in the call to fmt_float. +func (p *pp) fmtFloat(v float64, size int, verb rune) { + switch verb { + case 'v': + p.fmt.fmtFloat(v, size, 'g', -1) + case 'b', 'g', 'G', 'x', 'X': + p.fmt.fmtFloat(v, size, verb, -1) + case 'f', 'e', 'E': + p.fmt.fmtFloat(v, size, verb, 6) + case 'F': + p.fmt.fmtFloat(v, size, 'f', 6) + default: + p.badVerb(verb) + } +} + +// fmtComplex formats a complex number v with +// r = real(v) and j = imag(v) as (r+ji) using +// fmtFloat for r and j formatting. +func (p *pp) fmtComplex(v complex128, size int, verb rune) { + // Make sure any unsupported verbs are found before the + // calls to fmtFloat to not generate an incorrect error string. + switch verb { + case 'v', 'b', 'g', 'G', 'x', 'X', 'f', 'F', 'e', 'E': + oldPlus := p.fmt.plus + p.buf.writeByte('(') + p.fmtFloat(real(v), size/2, verb) + // Imaginary part always has a sign. + p.fmt.plus = true + p.fmtFloat(imag(v), size/2, verb) + p.buf.writeString("i)") + p.fmt.plus = oldPlus + default: + p.badVerb(verb) + } +} + +func (p *pp) fmtString(v string, verb rune) { + switch verb { + case 'v': + if p.fmt.sharpV { + p.fmt.fmtQ(v) + } else { + p.fmt.fmtS(v) + } + case 's': + p.fmt.fmtS(v) + case 'x': + p.fmt.fmtSx(v, ldigits) + case 'X': + p.fmt.fmtSx(v, udigits) + case 'q': + p.fmt.fmtQ(v) + default: + p.badVerb(verb) + } +} + +func (p *pp) fmtBytes(v []byte, verb rune, typeString string) { + switch verb { + case 'v', 'd': + if p.fmt.sharpV { + p.buf.writeString(typeString) + if v == nil { + p.buf.writeString(nilParenString) + return + } + p.buf.writeByte('{') + for i, c := range v { + if i > 0 { + p.buf.writeString(commaSpaceString) + } + p.fmt0x64(uint64(c), true) + } + p.buf.writeByte('}') + } else { + p.buf.writeByte('[') + for i, c := range v { + if i > 0 { + p.buf.writeByte(' ') + } + p.fmt.fmtInteger(uint64(c), 10, unsigned, verb, ldigits) + } + p.buf.writeByte(']') + } + case 's': + p.fmt.fmtBs(v) + case 'x': + p.fmt.fmtBx(v, ldigits) + case 'X': + p.fmt.fmtBx(v, udigits) + case 'q': + p.fmt.fmtQ(string(v)) + default: + p.printValue(reflect.ValueOf(v), verb, 0) + } +} + +func (p *pp) fmtPointer(value reflect.Value, verb rune) { + var u uintptr + switch value.Kind() { + case reflect.Chan, reflect.Func, reflect.Map, reflect.Ptr, reflect.Slice, reflect.UnsafePointer: + u = value.Pointer() + default: + p.badVerb(verb) + return + } + + switch verb { + case 'v': + if p.fmt.sharpV { + p.buf.writeByte('(') + p.buf.writeString(value.Type().String()) + p.buf.writeString(")(") + if u == 0 { + p.buf.writeString(nilString) + } else { + p.fmt0x64(uint64(u), true) + } + p.buf.writeByte(')') + } else { + if u == 0 { + p.fmt.padString(nilAngleString) + } else { + p.fmt0x64(uint64(u), !p.fmt.sharp) + } + } + case 'p': + p.fmt0x64(uint64(u), !p.fmt.sharp) + case 'b', 'o', 'd', 'x', 'X': + p.fmtInteger(uint64(u), unsigned, verb) + default: + p.badVerb(verb) + } +} + +func (p *pp) catchPanic(arg interface{}, verb rune, method string) { + if err := recover(); err != nil { + // If it's a nil pointer, just say "". The likeliest causes are a + // Stringer that fails to guard against nil or a nil pointer for a + // value receiver, and in either case, "" is a nice result. + if v := reflect.ValueOf(arg); v.Kind() == reflect.Ptr && v.IsNil() { + p.buf.writeString(nilAngleString) + return + } + // Otherwise print a concise panic message. Most of the time the panic + // value will print itself nicely. + if p.panicking { + // Nested panics; the recursion in printArg cannot succeed. + panic(err) + } + + oldFlags := p.fmt.fmtFlags + // For this output we want default behavior. + p.fmt.clearflags() + + p.buf.writeString(percentBangString) + p.buf.writeRune(verb) + p.buf.writeString(panicString) + p.buf.writeString(method) + p.buf.writeString(" method: ") + p.panicking = true + p.printArg(err, 'v') + p.panicking = false + p.buf.writeByte(')') + + p.fmt.fmtFlags = oldFlags + } +} + +func (p *pp) handleMethods(verb rune) (handled bool) { + if p.erroring { + return + } + if verb == 'w' { + // It is invalid to use %w other than with Errorf, more than once, + // or with a non-error arg. + err, ok := p.arg.(error) + if !ok || !p.wrapErrs || p.wrappedErr != nil { + p.wrappedErr = nil + p.wrapErrs = false + p.badVerb(verb) + return true + } + p.wrappedErr = err + // If the arg is a Formatter, pass 'v' as the verb to it. + verb = 'v' + } + + // Is it a Formatter? + if formatter, ok := p.arg.(Formatter); ok { + handled = true + defer p.catchPanic(p.arg, verb, "Format") + formatter.Format(p, verb) + return + } + + // If we're doing Go syntax and the argument knows how to supply it, take care of it now. + if p.fmt.sharpV { + if stringer, ok := p.arg.(GoStringer); ok { + handled = true + defer p.catchPanic(p.arg, verb, "GoString") + // Print the result of GoString unadorned. + p.fmt.fmtS(stringer.GoString()) + return + } + } else { + // If a string is acceptable according to the format, see if + // the value satisfies one of the string-valued interfaces. + // Println etc. set verb to %v, which is "stringable". + switch verb { + case 'v', 's', 'x', 'X', 'q': + // Is it an error or Stringer? + // The duplication in the bodies is necessary: + // setting handled and deferring catchPanic + // must happen before calling the method. + switch v := p.arg.(type) { + case error: + handled = true + defer p.catchPanic(p.arg, verb, "Error") + p.fmtString(v.Error(), verb) + return + + case Stringer: + handled = true + defer p.catchPanic(p.arg, verb, "String") + p.fmtString(v.String(), verb) + return + } + } + } + return false +} + +// CUSTOM: printArg() renamed to printArgOrig(). +func (p *pp) printArgOrig(arg interface{}, verb rune) { + p.arg = arg + p.value = reflect.Value{} + + if arg == nil { + switch verb { + case 'T', 'v': + p.fmt.padString(nilAngleString) + default: + p.badVerb(verb) + } + return + } + + // Special processing considerations. + // %T (the value's type) and %p (its address) are special; we always do them first. + switch verb { + case 'T': + p.fmt.fmtS(reflect.TypeOf(arg).String()) + return + case 'p': + p.fmtPointer(reflect.ValueOf(arg), 'p') + return + } + + // Some types can be done without reflection. + switch f := arg.(type) { + case bool: + p.fmtBool(f, verb) + case float32: + p.fmtFloat(float64(f), 32, verb) + case float64: + p.fmtFloat(f, 64, verb) + case complex64: + p.fmtComplex(complex128(f), 64, verb) + case complex128: + p.fmtComplex(f, 128, verb) + case int: + p.fmtInteger(uint64(f), signed, verb) + case int8: + p.fmtInteger(uint64(f), signed, verb) + case int16: + p.fmtInteger(uint64(f), signed, verb) + case int32: + p.fmtInteger(uint64(f), signed, verb) + case int64: + p.fmtInteger(uint64(f), signed, verb) + case uint: + p.fmtInteger(uint64(f), unsigned, verb) + case uint8: + p.fmtInteger(uint64(f), unsigned, verb) + case uint16: + p.fmtInteger(uint64(f), unsigned, verb) + case uint32: + p.fmtInteger(uint64(f), unsigned, verb) + case uint64: + p.fmtInteger(f, unsigned, verb) + case uintptr: + p.fmtInteger(uint64(f), unsigned, verb) + case string: + p.fmtString(f, verb) + case []byte: + p.fmtBytes(f, verb, "[]byte") + case reflect.Value: + // Handle extractable values with special methods + // since printValue does not handle them at depth 0. + if f.IsValid() && f.CanInterface() { + p.arg = f.Interface() + if p.handleMethods(verb) { + return + } + } + p.printValue(f, verb, 0) + default: + // If the type is not simple, it might have methods. + if !p.handleMethods(verb) { + // Need to use reflection, since the type had no + // interface methods that could be used for formatting. + p.printValue(reflect.ValueOf(f), verb, 0) + } + } +} + +// printValue is similar to printArg but starts with a reflect value, not an interface{} value. +// It does not handle 'p' and 'T' verbs because these should have been already handled by printArg. +func (p *pp) printValue(value reflect.Value, verb rune, depth int) { + // Handle values with special methods if not already handled by printArg (depth == 0). + if depth > 0 && value.IsValid() && value.CanInterface() { + p.arg = value.Interface() + if p.handleMethods(verb) { + return + } + } + p.arg = nil + p.value = value + + switch f := value; value.Kind() { + case reflect.Invalid: + if depth == 0 { + p.buf.writeString(invReflectString) + } else { + switch verb { + case 'v': + p.buf.writeString(nilAngleString) + default: + p.badVerb(verb) + } + } + case reflect.Bool: + p.fmtBool(f.Bool(), verb) + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + p.fmtInteger(uint64(f.Int()), signed, verb) + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + p.fmtInteger(f.Uint(), unsigned, verb) + case reflect.Float32: + p.fmtFloat(f.Float(), 32, verb) + case reflect.Float64: + p.fmtFloat(f.Float(), 64, verb) + case reflect.Complex64: + p.fmtComplex(f.Complex(), 64, verb) + case reflect.Complex128: + p.fmtComplex(f.Complex(), 128, verb) + case reflect.String: + p.fmtString(f.String(), verb) + case reflect.Map: + if p.fmt.sharpV { + p.buf.writeString(f.Type().String()) + if f.IsNil() { + p.buf.writeString(nilParenString) + return + } + p.buf.writeByte('{') + } else { + p.buf.writeString(mapString) + } + sorted := fmtsort.Sort(f) + for i, key := range sorted.Key { + if i > 0 { + if p.fmt.sharpV { + p.buf.writeString(commaSpaceString) + } else { + p.buf.writeByte(' ') + } + } + p.printValue(key, verb, depth+1) + p.buf.writeByte(':') + p.printValue(sorted.Value[i], verb, depth+1) + } + if p.fmt.sharpV { + p.buf.writeByte('}') + } else { + p.buf.writeByte(']') + } + case reflect.Struct: + if p.fmt.sharpV { + p.buf.writeString(f.Type().String()) + } + p.buf.writeByte('{') + for i := 0; i < f.NumField(); i++ { + if i > 0 { + if p.fmt.sharpV { + p.buf.writeString(commaSpaceString) + } else { + p.buf.writeByte(' ') + } + } + if p.fmt.plusV || p.fmt.sharpV { + if name := f.Type().Field(i).Name; name != "" { + p.buf.writeString(name) + p.buf.writeByte(':') + } + } + p.printValue(getField(f, i), verb, depth+1) + } + p.buf.writeByte('}') + case reflect.Interface: + value := f.Elem() + if !value.IsValid() { + if p.fmt.sharpV { + p.buf.writeString(f.Type().String()) + p.buf.writeString(nilParenString) + } else { + p.buf.writeString(nilAngleString) + } + } else { + p.printValue(value, verb, depth+1) + } + case reflect.Array, reflect.Slice: + switch verb { + case 's', 'q', 'x', 'X': + // Handle byte and uint8 slices and arrays special for the above verbs. + t := f.Type() + if t.Elem().Kind() == reflect.Uint8 { + var bytes []byte + if f.Kind() == reflect.Slice { + bytes = f.Bytes() + } else if f.CanAddr() { + bytes = f.Slice(0, f.Len()).Bytes() + } else { + // We have an array, but we cannot Slice() a non-addressable array, + // so we build a slice by hand. This is a rare case but it would be nice + // if reflection could help a little more. + bytes = make([]byte, f.Len()) + for i := range bytes { + bytes[i] = byte(f.Index(i).Uint()) + } + } + p.fmtBytes(bytes, verb, t.String()) + return + } + } + if p.fmt.sharpV { + p.buf.writeString(f.Type().String()) + if f.Kind() == reflect.Slice && f.IsNil() { + p.buf.writeString(nilParenString) + return + } + p.buf.writeByte('{') + for i := 0; i < f.Len(); i++ { + if i > 0 { + p.buf.writeString(commaSpaceString) + } + p.printValue(f.Index(i), verb, depth+1) + } + p.buf.writeByte('}') + } else { + p.buf.writeByte('[') + for i := 0; i < f.Len(); i++ { + if i > 0 { + p.buf.writeByte(' ') + } + p.printValue(f.Index(i), verb, depth+1) + } + p.buf.writeByte(']') + } + case reflect.Ptr: + // pointer to array or slice or struct? ok at top level + // but not embedded (avoid loops) + if depth == 0 && f.Pointer() != 0 { + switch a := f.Elem(); a.Kind() { + case reflect.Array, reflect.Slice, reflect.Struct, reflect.Map: + p.buf.writeByte('&') + p.printValue(a, verb, depth+1) + return + } + } + fallthrough + case reflect.Chan, reflect.Func, reflect.UnsafePointer: + p.fmtPointer(f, verb) + default: + p.unknownType(f) + } +} + +// intFromArg gets the argNumth element of a. On return, isInt reports whether the argument has integer type. +func intFromArg(a []interface{}, argNum int) (num int, isInt bool, newArgNum int) { + newArgNum = argNum + if argNum < len(a) { + num, isInt = a[argNum].(int) // Almost always OK. + if !isInt { + // Work harder. + switch v := reflect.ValueOf(a[argNum]); v.Kind() { + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + n := v.Int() + if int64(int(n)) == n { + num = int(n) + isInt = true + } + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: + n := v.Uint() + if int64(n) >= 0 && uint64(int(n)) == n { + num = int(n) + isInt = true + } + default: + // Already 0, false. + } + } + newArgNum = argNum + 1 + if tooLarge(num) { + num = 0 + isInt = false + } + } + return +} + +// parseArgNumber returns the value of the bracketed number, minus 1 +// (explicit argument numbers are one-indexed but we want zero-indexed). +// The opening bracket is known to be present at format[0]. +// The returned values are the index, the number of bytes to consume +// up to the closing paren, if present, and whether the number parsed +// ok. The bytes to consume will be 1 if no closing paren is present. +func parseArgNumber(format string) (index int, wid int, ok bool) { + // There must be at least 3 bytes: [n]. + if len(format) < 3 { + return 0, 1, false + } + + // Find closing bracket. + for i := 1; i < len(format); i++ { + if format[i] == ']' { + width, ok, newi := parsenum(format, 1, i) + if !ok || newi != i { + return 0, i + 1, false + } + return width - 1, i + 1, true // arg numbers are one-indexed and skip paren. + } + } + return 0, 1, false +} + +// argNumber returns the next argument to evaluate, which is either the value of the passed-in +// argNum or the value of the bracketed integer that begins format[i:]. It also returns +// the new value of i, that is, the index of the next byte of the format to process. +func (p *pp) argNumber(argNum int, format string, i int, numArgs int) (newArgNum, newi int, found bool) { + if len(format) <= i || format[i] != '[' { + return argNum, i, false + } + p.reordered = true + index, wid, ok := parseArgNumber(format[i:]) + if ok && 0 <= index && index < numArgs { + return index, i + wid, true + } + p.goodArgNum = false + return argNum, i + wid, ok +} + +func (p *pp) badArgNum(verb rune) { + p.buf.writeString(percentBangString) + p.buf.writeRune(verb) + p.buf.writeString(badIndexString) +} + +func (p *pp) missingArg(verb rune) { + p.buf.writeString(percentBangString) + p.buf.writeRune(verb) + p.buf.writeString(missingString) +} + +func (p *pp) doPrintf(format string, a []interface{}) { + end := len(format) + argNum := 0 // we process one argument per non-trivial format + afterIndex := false // previous item in format was an index like [3]. + p.reordered = false +formatLoop: + for i := 0; i < end; { + p.goodArgNum = true + lasti := i + for i < end && format[i] != '%' { + i++ + } + if i > lasti { + p.buf.writeString(format[lasti:i]) + } + if i >= end { + // done processing format string + break + } + + // Process one verb + i++ + + // Do we have flags? + p.fmt.clearflags() + simpleFormat: + for ; i < end; i++ { + c := format[i] + switch c { + case '#': + p.fmt.sharp = true + case '0': + p.fmt.zero = !p.fmt.minus // Only allow zero padding to the left. + case '+': + p.fmt.plus = true + case '-': + p.fmt.minus = true + p.fmt.zero = false // Do not pad with zeros to the right. + case ' ': + p.fmt.space = true + default: + // Fast path for common case of ascii lower case simple verbs + // without precision or width or argument indices. + if 'a' <= c && c <= 'z' && argNum < len(a) { + if c == 'v' { + // Go syntax + p.fmt.sharpV = p.fmt.sharp + p.fmt.sharp = false + // Struct-field syntax + p.fmt.plusV = p.fmt.plus + p.fmt.plus = false + } + p.printArg(a[argNum], rune(c)) + argNum++ + i++ + continue formatLoop + } + // Format is more complex than simple flags and a verb or is malformed. + break simpleFormat + } + } + + // Do we have an explicit argument index? + argNum, i, afterIndex = p.argNumber(argNum, format, i, len(a)) + + // Do we have width? + if i < end && format[i] == '*' { + i++ + p.fmt.wid, p.fmt.widPresent, argNum = intFromArg(a, argNum) + + if !p.fmt.widPresent { + p.buf.writeString(badWidthString) + } + + // We have a negative width, so take its value and ensure + // that the minus flag is set + if p.fmt.wid < 0 { + p.fmt.wid = -p.fmt.wid + p.fmt.minus = true + p.fmt.zero = false // Do not pad with zeros to the right. + } + afterIndex = false + } else { + p.fmt.wid, p.fmt.widPresent, i = parsenum(format, i, end) + if afterIndex && p.fmt.widPresent { // "%[3]2d" + p.goodArgNum = false + } + } + + // Do we have precision? + if i+1 < end && format[i] == '.' { + i++ + if afterIndex { // "%[3].2d" + p.goodArgNum = false + } + argNum, i, afterIndex = p.argNumber(argNum, format, i, len(a)) + if i < end && format[i] == '*' { + i++ + p.fmt.prec, p.fmt.precPresent, argNum = intFromArg(a, argNum) + // Negative precision arguments don't make sense + if p.fmt.prec < 0 { + p.fmt.prec = 0 + p.fmt.precPresent = false + } + if !p.fmt.precPresent { + p.buf.writeString(badPrecString) + } + afterIndex = false + } else { + p.fmt.prec, p.fmt.precPresent, i = parsenum(format, i, end) + if !p.fmt.precPresent { + p.fmt.prec = 0 + p.fmt.precPresent = true + } + } + } + + if !afterIndex { + argNum, i, afterIndex = p.argNumber(argNum, format, i, len(a)) + } + + if i >= end { + p.buf.writeString(noVerbString) + break + } + + verb, size := rune(format[i]), 1 + if verb >= utf8.RuneSelf { + verb, size = utf8.DecodeRuneInString(format[i:]) + } + i += size + + switch { + case verb == '%': // Percent does not absorb operands and ignores f.wid and f.prec. + p.buf.writeByte('%') + case !p.goodArgNum: + p.badArgNum(verb) + case argNum >= len(a): // No argument left over to print for the current verb. + p.missingArg(verb) + case verb == 'v': + // Go syntax + p.fmt.sharpV = p.fmt.sharp + p.fmt.sharp = false + // Struct-field syntax + p.fmt.plusV = p.fmt.plus + p.fmt.plus = false + fallthrough + default: + p.printArg(a[argNum], verb) + argNum++ + } + } + + // Check for extra arguments unless the call accessed the arguments + // out of order, in which case it's too expensive to detect if they've all + // been used and arguably OK if they're not. + if !p.reordered && argNum < len(a) { + p.fmt.clearflags() + p.buf.writeString(extraString) + for i, arg := range a[argNum:] { + if i > 0 { + p.buf.writeString(commaSpaceString) + } + if arg == nil { + p.buf.writeString(nilAngleString) + } else { + p.buf.writeString(reflect.TypeOf(arg).String()) + p.buf.writeByte('=') + p.printArg(arg, 'v') + } + } + p.buf.writeByte(')') + } +} + +func (p *pp) doPrint(a []interface{}) { + prevString := false + for argNum, arg := range a { + isString := arg != nil && reflect.TypeOf(arg).Kind() == reflect.String + // Add a space between two non-string arguments. + if argNum > 0 && !isString && !prevString { + p.buf.writeByte(' ') + } + p.printArg(arg, 'v') + prevString = isString + } +} + +// doPrintln is like doPrint but always adds a space between arguments +// and a newline after the last argument. +func (p *pp) doPrintln(a []interface{}) { + for argNum, arg := range a { + if argNum > 0 { + p.buf.writeByte(' ') + } + p.printArg(arg, 'v') + } + p.buf.writeByte('\n') +} diff --git a/github.com/cockroachdb/redact/internal/print.go.diff b/github.com/cockroachdb/redact/internal/print.go.diff new file mode 100644 index 0000000000..4451b4b104 --- /dev/null +++ b/github.com/cockroachdb/redact/internal/print.go.diff @@ -0,0 +1,58 @@ +--- print.go.orig 2020-08-26 14:49:30.396184000 +0200 ++++ print.go 2020-08-26 14:50:29.330829000 +0200 +@@ -1,3 +1,6 @@ ++// Code generated from print.go.orig. DO NOT EDIT ++// GENERATED FILE DO NOT EDIT ++// + // Copyright 2009 The Go Authors. All rights reserved. + // Use of this source code is governed by a BSD-style + // license that can be found in the LICENSE file. +@@ -5,12 +8,16 @@ + package fmt + + import ( +- "internal/fmtsort" ++ // CUSTOM: needed to avoid a type mismatch on Formatter. ++ origFmt "fmt" + "io" + "os" + "reflect" + "sync" + "unicode/utf8" ++ ++ // CUSTOM: our own import since we can't use internal from go stdlib. ++ "github.com/cockroachdb/redact/internal/fmtsort" + ) + + // Strings for use with buffer.WriteString. +@@ -51,7 +58,8 @@ + // The implementation of Format may call Sprint(f) or Fprint(f) etc. + // to generate its output. + type Formatter interface { +- Format(f State, c rune) ++ // CUSTOM: refer to the original type, not the one defined here. ++ Format(f origFmt.State, c rune) + } + + // Stringer is implemented by any value that has a String method, +@@ -105,6 +113,10 @@ + type pp struct { + buf buffer + ++ // CUSTOM: hook fn for the redact package. ++ printArgSubstituteFn func(p *pp, a interface{}, verb rune) (newState int) ++ substituteState int ++ + // arg holds the current item, as an interface{}. + arg interface{} + +@@ -635,7 +647,8 @@ + return false + } + +-func (p *pp) printArg(arg interface{}, verb rune) { ++// CUSTOM: printArg() renamed to printArgOrig(). ++func (p *pp) printArgOrig(arg interface{}, verb rune) { + p.arg = arg + p.value = reflect.Value{} + diff --git a/github.com/cockroachdb/redact/internal/refresh.sh b/github.com/cockroachdb/redact/internal/refresh.sh new file mode 100644 index 0000000000..1be06b5cb3 --- /dev/null +++ b/github.com/cockroachdb/redact/internal/refresh.sh @@ -0,0 +1,29 @@ +#!/usr/bin/env bash +# +# This file re-generates the sources in this directory from the Go +# standard library. +# +set -euxo pipefail + +( + echo "// Code generated from the Go standard library. DO NOT EDIT" + echo "// GENERATED FILE DO NOT EDIT" + cat $GOROOT/src/fmt/format.go +) >format.go + +cp $GOROOT/src/fmt/print.go print.go +patch -p0 fmtsort/"$n" +done + diff --git a/github.com/cockroachdb/redact/make_format.go b/github.com/cockroachdb/redact/make_format.go index 534300ba7d..e088b93799 100644 --- a/github.com/cockroachdb/redact/make_format.go +++ b/github.com/cockroachdb/redact/make_format.go @@ -31,9 +31,10 @@ func reproducePrintf(w io.Writer, s fmt.State, verb rune, arg interface{}) { } } -// MakeFormat reproduces the format currently active in fmt.State and -// verb. This is provided because Go's standard fmt.State does not -// make the original format string available to us. +// MakeFormat is a helper for use by implementations of the +// SafeFormatter interface. It reproduces the format currently active +// in fmt.State and verb. This is provided because Go's standard +// fmt.State does not make the original format string available to us. // // If the return value justV is true, then the current state // was found to be %v exactly; in that case the caller diff --git a/github.com/cockroachdb/redact/markers.go b/github.com/cockroachdb/redact/markers.go index 49f7f6c015..7b719183d7 100644 --- a/github.com/cockroachdb/redact/markers.go +++ b/github.com/cockroachdb/redact/markers.go @@ -14,6 +14,8 @@ package redact +import "bytes" + // RedactableString is a string that contains a mix of safe and unsafe // bits of data, but where it is known that unsafe bits are enclosed // by redaction markers ‹ and ›, and occurrences of the markers @@ -42,6 +44,13 @@ func (s RedactableString) ToBytes() RedactableBytes { return RedactableBytes([]byte(string(s))) } +// SafeFormat formats the redactable safely. +func (s RedactableString) SafeFormat(sp SafePrinter, _ rune) { + // As per annotateArgs() in markers_internal_print.go, + // we consider the redactable string not further formattable. + sp.Print(s) +} + // RedactableBytes is like RedactableString but is a byte slice. // // Instances of RedactableBytes should not be constructed directly; @@ -67,15 +76,21 @@ func (s RedactableBytes) ToString() RedactableString { return RedactableString(string([]byte(s))) } +// SafeFormat formats the redactable safely. +func (s RedactableBytes) SafeFormat(sp SafePrinter, _ rune) { + // As per annotateArgs() in markers_internal_print.go, + // we consider the redactable bytes not further formattable. + sp.Print(s) +} + // EscapeBytes escapes markers inside the given byte slice and encloses // the entire byte slice between redaction markers. func EscapeBytes(s []byte) RedactableBytes { - s = reStripMarkers.ReplaceAll(s, escapeBytes) - enclosed := make([]byte, 0, len(s)+len(startRedactableBytes)+len(endRedactableBytes)) - enclosed = append(enclosed, startRedactableBytes...) - enclosed = append(enclosed, s...) - enclosed = append(enclosed, endRedactableBytes...) - return RedactableBytes(enclosed) + var buf bytes.Buffer + buf.Grow(len(s) + len(startRedactableS) + len(endRedactableS)) + e := escapeWriter{w: &buf, enclose: true, strip: false} + _, _ = e.Write(s) + return RedactableBytes(buf.Bytes()) } // StartMarker returns the start delimiter for an unsafe string. diff --git a/github.com/cockroachdb/redact/markers_internal_escape.go b/github.com/cockroachdb/redact/markers_internal_escape.go index da555058a0..9d6989f407 100644 --- a/github.com/cockroachdb/redact/markers_internal_escape.go +++ b/github.com/cockroachdb/redact/markers_internal_escape.go @@ -74,10 +74,11 @@ func (p *escapeWriter) Write(b []byte) (int, error) { // entirely if there was nothing but empty space: // if len(b) == 0 { return 0, nil } - start := startRedactableBytes - ls := len(startRedactableS) - end := endRedactableBytes - le := len(endRedactableS) + // Note: we use len(...RedactableS) and not len(...RedactableBytes) + // because the ...S variant is a compile-time constant so this + // accelerates the loops below. + start, ls := startRedactableBytes, len(startRedactableS) + end, le := endRedactableBytes, len(endRedactableS) escape := escapeBytes if p.enclose { @@ -85,22 +86,50 @@ func (p *escapeWriter) Write(b []byte) (int, error) { } // Now write the string. + + // k is the index in b up to (and excluding) the byte which we've + // already copied into the output. k := 0 + for i := 0; i < len(b); i++ { - // Ensure that occurrences of the delimiter inside the string get - // escaped. - if i+ls <= len(b) && bytes.Equal(b[i:i+ls], start) { - st = p.doWrite(b[k:i], st, true) - st = p.doWrite(escape, st, false) - st.l += ls - k = i + ls - i += ls - 1 - } else if i+le <= len(b) && bytes.Equal(b[i:i+le], end) { + if b[i] == '\n' && p.enclose { + // Avoid enclosing newline characters inside redaction markers. + // This is important when redact is used to render errors, where + // sub-strings split at newline characters are rendered + // separately. st = p.doWrite(b[k:i], st, true) - st = p.doWrite(escape, st, false) - st.l += le - k = i + le - i += le - 1 + st = p.doWrite(end, st, false) + // Advance to the last newline character. We want to forward + // them all in a single call to doWrite, for performance. + lastNewLine := i + for b[lastNewLine] == '\n' && lastNewLine < len(b) { + lastNewLine++ + } + st = p.doWrite(b[i:lastNewLine], st, true) + st = p.doWrite(start, st, false) + // Advance the counters by the number of newline characters. + k = lastNewLine + i = lastNewLine - 1 /* -1 because we have i++ at the end of every iteration */ + } else { + // Ensure that occurrences of the delimiter inside the string get + // escaped. + // Reminder: ls and le are likely greater than 1, as we are scanning + // utf-8 encoded delimiters (the utf-8 encoding is multibyte). + if i+ls <= len(b) && bytes.Equal(b[i:i+ls], start) { + st = p.doWrite(b[k:i], st, true) + st = p.doWrite(escape, st, false) + // Advance the counters by the length (in bytes) of the delimiter. + st.l += ls + k = i + ls + i += ls - 1 /* -1 because we have i++ at the end of every iteration */ + } else if i+le <= len(b) && bytes.Equal(b[i:i+le], end) { + st = p.doWrite(b[k:i], st, true) + st = p.doWrite(escape, st, false) + // Advance the counters by the length (in bytes) of the delimiter. + st.l += le + k = i + le + i += le - 1 /* -1 because we have i++ at the end of every iteration */ + } } } st = p.doWrite(b[k:], st, true) @@ -128,3 +157,61 @@ func (p *escapeWriter) doWrite(b []byte, st escapeResult, count bool) escapeResu st.err = err return st } + +// internalEscapeBytes escapes redaction markers in the provided buf +// starting at the location startLoc. +// The bytes before startLoc are considered safe (already escaped). +func internalEscapeBytes(b []byte, startLoc int) (res []byte) { + // Note: we use len(...RedactableS) and not len(...RedactableBytes) + // because the ...S variant is a compile-time constant so this + // accelerates the loops below. + start, ls := startRedactableBytes, len(startRedactableS) + end, le := endRedactableBytes, len(endRedactableS) + escape := escapeBytes + + // res is the output slice. In the common case where there is + // nothing to escape, the input slice is returned directly + // and no allocation takes place. + res = b + // copied is true if and only if `res` is a copy of `b`. It only + // turns to true if the loop below finds something to escape. + copied := false + // k is the index in b up to (and excluding) the byte which we've + // already copied into res (if copied=true). + k := 0 + + for i := startLoc; i < len(b); i++ { + // Ensure that occurrences of the delimiter inside the string get + // escaped. + // Reminder: ls and le are likely greater than 1, as we are scanning + // utf-8 encoded delimiters (the utf-8 encoding is multibyte). + if i+ls <= len(b) && bytes.Equal(b[i:i+ls], start) { + if !copied { + // We only allocate an output slice when we know we definitely + // need it. + res = make([]byte, 0, len(b)+len(escape)) + copied = true + } + res = append(res, b[k:i]...) + res = append(res, escape...) + // Advance the counters by the length (in bytes) of the delimiter. + k = i + ls + i += ls - 1 /* -1 because we have i++ at the end of every iteration */ + } else if i+le <= len(b) && bytes.Equal(b[i:i+le], end) { + if !copied { + // See the comment above about res allocation. + res = make([]byte, 0, len(b)+len(escape)) + copied = true + } + res = append(res, b[k:i]...) + res = append(res, escape...) + // Advance the counters by the length (in bytes) of the delimiter. + k = i + le + i += le - 1 /* -1 because we have i++ at the end of every iteration */ + } + } + if copied { + res = append(res, b[k:]...) + } + return +} diff --git a/github.com/cockroachdb/redact/markers_internal_print.go b/github.com/cockroachdb/redact/markers_internal_print.go index 1f9b92857d..aea6a247f7 100644 --- a/github.com/cockroachdb/redact/markers_internal_print.go +++ b/github.com/cockroachdb/redact/markers_internal_print.go @@ -14,57 +14,173 @@ package redact -import "fmt" +import ( + "fmt" -// annotateArgs wraps the arguments to one of the print functions with -// an indirect formatter which either encloses the result of the -// display between redaction markers, or not. -func annotateArgs(args []interface{}) { - for i, arg := range args { + internalFmt "github.com/cockroachdb/redact/internal" +) + +// printArgFn is the hook injected into the standard fmt logic +// by the printer functions in markers_print.go. +func printArgFn(p *internalFmt.InternalPrinter, arg interface{}, verb rune) (newState int) { + redactLastWrites(p) + + switch verb { + case 'T': + // If the value was wrapped, reveal its original type. Anything else is not very useful. switch v := arg.(type) { - case RedactableString: - // Already formatted as redactable. Include as-is. - // We need an intermediate struct because we don't - // want the fmt machinery to re-format the string - // object by adding quotes, expanding the byte slice, etc. - args[i] = &passthrough{arg: []byte(v)} - case RedactableBytes: - args[i] = &passthrough{arg: v} - - case SafeFormatter: - // calls to Format() by fmt.Print will be redirected to - // v.SafeFormat(). This delegates the task of adding markers to - // the object itself. - args[i] = &redactFormatRedirect{v} - - case SafeValue: - // calls to Format() by fmt.Print will be redirected to a - // display of v without redaction markers. - // - // Note that we can't let the value be displayed as-is because - // we must prevent any marker inside the value from leaking into - // the result. (We want to avoid mismatched markers.) - args[i] = &escapeArg{arg: arg, enclose: false} - - case SafeMessager: - // Obsolete interface. - // TODO(knz): Remove this. - args[i] = &escapeArg{arg: v.SafeMessage(), enclose: false} - - default: + case safeWrapper: + arg = v.a + case unsafeWrap: + arg = v.a + } + + // Shortcut: %T is always safe to print as-is. + internalFmt.PrintArg(p, arg, verb) + return len(internalFmt.Buf(p)) + case 'p': + // Printing a pointer via %p is handled as special case in printf, + // so we need a special case here too. The other cases of + // printing a pointer via %v / %d %x etc is handled by the common path. + + switch v := arg.(type) { + case safeWrapper: + // If the value was meant to be safe, then print it as-is. + internalFmt.PrintArg(p, v.a, verb) + return len(internalFmt.Buf(p)) + + case unsafeWrap: + // If it's been wrapped, unwrap it. This helps preserve the + // original pointer value in the output. + arg = v.a + } + // Now perform an unsafe print: we are assuming that the pointer + // representation by the fmt.printf code does not contain + // redaction markers, and go a short route. If that assumption did + // not hold (or is invalidated by changes upstream after this + // comment is written), this code should be changed to use the + // escapeWriter instead. + internalFmt.Append(p, startRedactableBytes) + internalFmt.PrintArg(p, arg, verb) + internalFmt.Append(p, endRedactableBytes) + return len(internalFmt.Buf(p)) + } + + // nil arguments are printed as-is. Note: a nil argument under + // interface{} is not the same as a nil pointer passed via a pointer + // of a concrete type. The latter kind of nil goes through + // redaction as usual, because it may have its own custom Format() method. + if arg == nil { + internalFmt.PrintArg(p, arg, verb) + return len(internalFmt.Buf(p)) + } + + // RedactableBytes/RedactableString are already formatted as + // redactable. Include them as-is. + // + // NB: keep this logic synchronized with + // (RedactableString/Bytes).SafeFormat(). + switch v := arg.(type) { + case RedactableString: + internalFmt.Append(p, []byte(v)) + return len(internalFmt.Buf(p)) + case RedactableBytes: + internalFmt.Append(p, []byte(v)) + return len(internalFmt.Buf(p)) + } + + arg = annotateArg(arg, internalFmt.CollectingError(p)) + internalFmt.PrintArg(p, arg, verb) + return len(internalFmt.Buf(p)) +} + +// redactLastWrites escapes any markers that were added by the +// internals of the printf functions, for example +// if markers were present in the format string. +func redactLastWrites(p *internalFmt.InternalPrinter) { + state := internalFmt.GetState(p) + newBuf := internalEscapeBytes(internalFmt.Buf(p), state) + internalFmt.SetState(p, newBuf) +} + +// annotateArg wraps the arguments to one of the print functions with +// an indirect formatter which ensures that redaction markers inside +// the representation of the object are escaped, and optionally +// encloses the result of the display between redaction markers. +// +// collectingError is true iff we are in the context of +// HelperForErrorf, where we want the %w verb to work properly. This +// adds a little overhead to the processing, but this is OK because +// typically the error path is not perf-critical. +func annotateArg(arg interface{}, collectingError bool) interface{} { + var newArg fmt.Formatter + err, isError := arg.(error) + + switch v := arg.(type) { + case SafeFormatter: + // calls to Format() by fmt.Print will be redirected to + // v.SafeFormat(). This delegates the task of adding markers to + // the object itself. + newArg = &redactFormatRedirect{ + func(p SafePrinter, verb rune) { v.SafeFormat(p, verb) }, + } + + case SafeValue: + // calls to Format() by fmt.Print will be redirected to a + // display of v without redaction markers. + // + // Note that we can't let the value be displayed as-is because + // we must prevent any marker inside the value from leaking into + // the result. (We want to avoid mismatched markers.) + newArg = &escapeArg{arg: arg, enclose: false} + + case SafeMessager: + // Obsolete interface. + // TODO(knz): Remove this. + newArg = &escapeArg{arg: v.SafeMessage(), enclose: false} + + default: + if isError && redactErrorFn != nil { + // We place this case after the other cases above, in case + // the error object knows how to print itself safely already. + newArg = &redactFormatRedirect{ + func(p SafePrinter, verb rune) { redactErrorFn(err, p, verb) }, + } + } else { // calls to Format() by fmt.Print will be redirected to a // display of v within redaction markers if the type is // considered unsafe, without markers otherwise. In any case, // occurrences of delimiters within are escaped. - args[i] = &escapeArg{arg: v, enclose: !isSafeValue(v)} + newArg = &escapeArg{arg: v, enclose: !isSafeValue(v)} } } + + if isError && collectingError { + // Ensure the arg still implements the `error` interface for + // detection by the handling of %w, while also implementing + // fmt.Formatter to forward the implementation to the objects + // constructed above. + newArg = &makeError{err: err, arg: newArg} + } + + return newArg +} + +type makeError struct { + err error + arg fmt.Formatter } -// redactFormatRedirect wraps a SafeFormatter object and uses its -// SafeFormat method as-is to implement fmt.Formatter. +// Error implements error. +func (m *makeError) Error() string { return m.err.Error() } + +// Format implements fmt.Formatter. +func (m *makeError) Format(f fmt.State, verb rune) { m.arg.Format(f, verb) } + +// redactFormatRedirect wraps a safe print callback and uses it to +// implement fmt.Formatter. type redactFormatRedirect struct { - arg SafeFormatter + printFn func(p SafePrinter, verb rune) } // Format implements fmt.Formatter. @@ -77,7 +193,7 @@ func (r *redactFormatRedirect) Format(s fmt.State, verb rune) { }() p := &printer{} p.escapeState = makeEscapeState(s, &p.buf) - r.arg.SafeFormat(p, verb) + r.printFn(p, verb) _, _ = s.Write(p.buf.Bytes()) } @@ -135,3 +251,14 @@ type printerfn struct { func (p printerfn) SafeFormat(w SafePrinter, _ rune) { p.fn(w) } + +// redactErrorFn can be injected from an error library +// to render error objects safely. +var redactErrorFn func(err error, p SafePrinter, verb rune) + +// RegisterRedactErrorFn registers an error redaction function for use +// during automatic redaction by this package. +// Provided e.g. by cockroachdb/errors. +func RegisterRedactErrorFn(fn func(err error, p SafePrinter, verb rune)) { + redactErrorFn = fn +} diff --git a/github.com/cockroachdb/redact/markers_internal_printer.go b/github.com/cockroachdb/redact/markers_internal_printer.go index 17da9e36cb..7b0fd75a4e 100644 --- a/github.com/cockroachdb/redact/markers_internal_printer.go +++ b/github.com/cockroachdb/redact/markers_internal_printer.go @@ -17,6 +17,7 @@ package redact import ( "bytes" "fmt" + "unicode/utf8" ) // printer implements SafePrinter. @@ -70,7 +71,13 @@ func (b *printer) UnsafeRune(s rune) { // UnsafeByte is part of the SafeWriter interface. func (b *printer) UnsafeByte(s byte) { _, _ = b.buf.WriteRune(startRedactable) - _ = b.buf.WriteByte(s) + if s >= utf8.RuneSelf || + s == startRedactableBytes[0] || s == endRedactableBytes[0] { + // Unsafe byte. Escape it. + _, _ = b.buf.Write(escapeBytes) + } else { + _ = b.buf.WriteByte(s) + } _, _ = b.buf.WriteRune(endRedactable) } diff --git a/github.com/cockroachdb/redact/markers_print.go b/github.com/cockroachdb/redact/markers_print.go index 55ceade95b..50eea32b4b 100644 --- a/github.com/cockroachdb/redact/markers_print.go +++ b/github.com/cockroachdb/redact/markers_print.go @@ -15,8 +15,9 @@ package redact import ( - "fmt" "io" + + internalFmt "github.com/cockroachdb/redact/internal" ) // Sprint prints out the arguments and encloses unsafe bits @@ -26,8 +27,13 @@ import ( // If a RedactableString or RedactableBytes argument is passed, // it is reproduced as-is without escaping. func Sprint(args ...interface{}) RedactableString { - annotateArgs(args) - return RedactableString(fmt.Sprint(args...)) + p := internalFmt.NewInternalPrinter() + internalFmt.SetHook(p, printArgFn) + internalFmt.DoPrint(p, args) + redactLastWrites(p) + s := RedactableString(internalFmt.Buf(p)) + internalFmt.Free(p) + return s } // Sprintf formats the arguments and encloses unsafe bits @@ -38,8 +44,37 @@ func Sprint(args ...interface{}) RedactableString { // is responsible to ensure that the markers are not present // in the format string. func Sprintf(format string, args ...interface{}) RedactableString { - annotateArgs(args) - return RedactableString(fmt.Sprintf(format, args...)) + p := internalFmt.NewInternalPrinter() + internalFmt.SetHook(p, printArgFn) + internalFmt.DoPrintf(p, format, args) + redactLastWrites(p) + s := RedactableString(internalFmt.Buf(p)) + internalFmt.Free(p) + return s +} + +// HelperForErrorf is a helper to implement a redaction-aware +// fmt.Errorf-compatible function in a different package. It formats +// the string according to the given format and arguments in the same +// way as Sprintf, but in addition to this if the format contains %w +// and an error object in the proper argument position it also returns +// that error object. +// +// Note: This function only works if an error redaction function +// has been injected with RegisterRedactErrorFn(). +func HelperForErrorf(format string, args ...interface{}) (RedactableString, error) { + p := internalFmt.NewInternalPrinter() + internalFmt.SetCollectError(p) + internalFmt.SetHook(p, printArgFn) + internalFmt.DoPrintf(p, format, args) + redactLastWrites(p) + s := RedactableString(internalFmt.Buf(p)) + e := internalFmt.WrappedError(p) + internalFmt.Free(p) + if m, ok := e.(*makeError); ok { + e = m.err + } + return s, e } // Sprintfn produces a RedactableString using the provided @@ -64,14 +99,24 @@ func StringWithoutMarkers(f SafeFormatter) string { // Fprint is like Sprint but outputs the redactable // string to the provided Writer. -func Fprint(w io.Writer, args ...interface{}) (int, error) { - annotateArgs(args) - return fmt.Fprint(w, args...) +func Fprint(w io.Writer, args ...interface{}) (n int, err error) { + p := internalFmt.NewInternalPrinter() + internalFmt.SetHook(p, printArgFn) + internalFmt.DoPrint(p, args) + redactLastWrites(p) + n, err = w.Write(internalFmt.Buf(p)) + internalFmt.Free(p) + return } // Fprintf is like Sprintf but outputs the redactable string to the // provided Writer. -func Fprintf(w io.Writer, format string, args ...interface{}) (int, error) { - annotateArgs(args) - return fmt.Fprintf(w, format, args...) +func Fprintf(w io.Writer, format string, args ...interface{}) (n int, err error) { + p := internalFmt.NewInternalPrinter() + internalFmt.SetHook(p, printArgFn) + internalFmt.DoPrintf(p, format, args) + redactLastWrites(p) + n, err = w.Write(internalFmt.Buf(p)) + internalFmt.Free(p) + return } diff --git a/github.com/coreos/go-oidc/.gitignore b/github.com/coreos/go-oidc/.gitignore new file mode 100644 index 0000000000..c96f2f47bc --- /dev/null +++ b/github.com/coreos/go-oidc/.gitignore @@ -0,0 +1,2 @@ +/bin +/gopath diff --git a/github.com/coreos/go-oidc/.travis.yml b/github.com/coreos/go-oidc/.travis.yml new file mode 100644 index 0000000000..3fddaaac90 --- /dev/null +++ b/github.com/coreos/go-oidc/.travis.yml @@ -0,0 +1,16 @@ +language: go + +go: + - "1.12" + - "1.13" + +install: + - go get -v -t github.com/coreos/go-oidc/... + - go get golang.org/x/tools/cmd/cover + - go get golang.org/x/lint/golint + +script: + - ./test + +notifications: + email: false diff --git a/github.com/coreos/go-oidc/CONTRIBUTING.md b/github.com/coreos/go-oidc/CONTRIBUTING.md new file mode 100644 index 0000000000..6662073a84 --- /dev/null +++ b/github.com/coreos/go-oidc/CONTRIBUTING.md @@ -0,0 +1,71 @@ +# How to Contribute + +CoreOS projects are [Apache 2.0 licensed](LICENSE) and accept contributions via +GitHub pull requests. This document outlines some of the conventions on +development workflow, commit message formatting, contact points and other +resources to make it easier to get your contribution accepted. + +# Certificate of Origin + +By contributing to this project you agree to the Developer Certificate of +Origin (DCO). This document was created by the Linux Kernel community and is a +simple statement that you, as a contributor, have the legal right to make the +contribution. See the [DCO](DCO) file for details. + +# Email and Chat + +The project currently uses the general CoreOS email list and IRC channel: +- Email: [coreos-dev](https://groups.google.com/forum/#!forum/coreos-dev) +- IRC: #[coreos](irc://irc.freenode.org:6667/#coreos) IRC channel on freenode.org + +Please avoid emailing maintainers found in the MAINTAINERS file directly. They +are very busy and read the mailing lists. + +## Getting Started + +- Fork the repository on GitHub +- Read the [README](README.md) for build and test instructions +- Play with the project, submit bugs, submit patches! + +## Contribution Flow + +This is a rough outline of what a contributor's workflow looks like: + +- Create a topic branch from where you want to base your work (usually master). +- Make commits of logical units. +- Make sure your commit messages are in the proper format (see below). +- Push your changes to a topic branch in your fork of the repository. +- Make sure the tests pass, and add any new tests as appropriate. +- Submit a pull request to the original repository. + +Thanks for your contributions! + +### Format of the Commit Message + +We follow a rough convention for commit messages that is designed to answer two +questions: what changed and why. The subject line should feature the what and +the body of the commit should describe the why. + +``` +scripts: add the test-cluster command + +this uses tmux to setup a test cluster that you can easily kill and +start for debugging. + +Fixes #38 +``` + +The format can be described more formally as follows: + +``` +: + + + +