Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 16 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,40 +324,34 @@ the value correctly.
</details>

<details>
<summary>Using Secret Manager references (AWS/GCP/Vault)</summary>
<summary>Using Secret Manager references (AWS/GCP/Vault/Kubernetes)</summary>

If the database runs on AWS or Google Cloud, you might want to store the DSN in their Secret Manager services and allow
SQL Exporter to access it from there. This way you can avoid hardcoding credentials in the configuration file and
benefit from the security features of these services. In addition, Vault is also available as a secret manager option
for SQL Exporter.
SQL Exporter supports multiple secret management backends:

The secrets can be referenced in the configuration file as a value for `data_source_name` item using the following
syntax:
**Kubernetes Secrets** (for Kubernetes deployments):
```
k8ssecret://[namespace/]secret-name?key=field&template=dsn_template
```
Recommended for Kubernetes deployments. Requires RBAC permissions for the service account to read secrets. See [k8s-secret example](examples/k8s-secret/) for detailed setup instructions and the Helm chart automatically creates necessary RBAC resources.

**Cloud-based Secret Managers**:
If the database runs on AWS or Google Cloud, you might want to store the DSN in their Secret Manager services and allow SQL Exporter to access it from there. This way you can avoid hardcoding credentials in the configuration file and benefit from the security features of these services. In addition, Vault is also available as a secret manager option for SQL Exporter.

The secrets can be referenced in the configuration file as a value for `data_source_name` item using the following syntax:

```
awssecretsmanager://<SECRET_NAME>?region=<AWS_REGION>&key=<JSON_KEY>
gcpsecretsmanager://<SECRET_NAME>?project_id=<GCP_PROJECT_ID>&key=<JSON_KEY>
hashivault://<MOUNT>/<SECRET_PATH>?key=<JSON_KEY>
```

The secret value can be a simple string or a JSON object. If it's a JSON object, you need to specify the `key` query
parameter to indicate which value to use as the DSN. If the secret is a valid json but the key is not specified, SQL
Exporter will try to use the value of `data_source_name` key by default. If a simple string, then it will be used as
the DSN directly. Using JSON format gives more flexibility and allows to store additional information or multiple DSNs
in the same secret resource.
The secret value can be a simple string or a JSON object. If it's a JSON object, you need to specify the `key` query parameter to indicate which value to use as the DSN. If the secret is a valid json but the key is not specified, SQL Exporter will try to use the value of `data_source_name` key by default. If a simple string, then it will be used as the DSN directly. Using JSON format gives more flexibility and allows to store additional information or multiple DSNs in the same secret resource.

Secret references are supported for both single-target and jobs setups, so you can use them in both cases without any
issues. Just make sure to use the correct syntax and provide the necessary parameters for the secret manager you
choose. Also check the permissions and access policies for the secret manager to ensure that SQL Exporter has the
necessary access to read the secrets.
Secret references are supported for both single-target and jobs setups, so you can use them in both cases without any issues. Just make sure to use the correct syntax and provide the necessary parameters for the secret manager you choose. Also check the permissions and access policies for the secret manager to ensure that SQL Exporter has the necessary access to read the secrets.

Secrets are only resolved at startup, so if the secret value changes, you need to restart SQL Exporter to pick up the
new value. Or use the `reload` endpoint to trigger a configuration reload without restarting the process, but keep in
mind that this will also reload the entire configuration, not just the secrets.
Secrets are only resolved at startup, so if the secret value changes, you need to restart SQL Exporter to pick up the new value. Or use the `reload` endpoint to trigger a configuration reload without restarting the process, but keep in mind that this will also reload the entire configuration, not just the secrets.

For Vault, you also need to specify the `VAULT_ADDR` and `VAULT_TOKEN` environment variables to allow SQL Exporter to
authenticate. This is a regular practice and goes beyond the scope of this document, so please refer to Vault
documentation for more details on how to set up and use Vault for secrets management.
For Vault, you also need to specify the `VAULT_ADDR` and `VAULT_TOKEN` environment variables to allow SQL Exporter to authenticate. This is a regular practice and goes beyond the scope of this document, so please refer to Vault documentation for more details on how to set up and use Vault for secrets management.
</details>

<details>
Expand Down
139 changes: 139 additions & 0 deletions config/secret_k8s.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
package config

import (
"context"
"fmt"
"net/url"
"os"
"strings"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
)

type k8sSecretProvider struct {
clientset kubernetes.Interface
namespace string // cached current namespace
}

var k8sProviderInstance *k8sSecretProvider

// getCurrentNamespace retrieves the current pod's namespace from the downward API.
func getCurrentNamespace() (string, error) {
nsBytes, err := os.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/namespace")
if err != nil {
return "", fmt.Errorf("unable to read current namespace: %w", err)
}
return string(nsBytes), nil
}

// getK8sProvider returns a singleton instance of the k8s secret provider, lazily initializing the client.
func getK8sProvider() (*k8sSecretProvider, error) {
if k8sProviderInstance != nil {
return k8sProviderInstance, nil
}

// Use in-cluster configuration
config, err := rest.InClusterConfig()
if err != nil {
return nil, fmt.Errorf("unable to load in-cluster Kubernetes config: %w", err)
}

clientset, err := kubernetes.NewForConfig(config)
if err != nil {
return nil, fmt.Errorf("unable to create Kubernetes client: %w", err)
}

// Get current namespace
namespace, err := getCurrentNamespace()
if err != nil {
return nil, err
}

k8sProviderInstance = &k8sSecretProvider{
clientset: clientset,
namespace: namespace,
}
return k8sProviderInstance, nil
}

// getDSN fetches a plain string value from a Kubernetes secret.
// URL format: k8ssecret://[namespace/]secret-name?key=field&template=dsn_template
//
// Examples (with explicit namespace):
// - k8ssecret://default/my-db-secret
// - k8ssecret://monitoring/db-creds?key=password&template=postgres://user:DSN_VALUE@host:5432/db
//
// Examples (current namespace, auto-detected):
// - k8ssecret://my-db-secret
// - k8ssecret://db-creds?key=password
//
// Parameters:
// - namespace: Kubernetes namespace (optional, defaults to pod's current namespace)
// - secret-name: Name of the Kubernetes secret (required)
// - key: The key within the secret to extract (optional, defaults to "data_source_name")
// - template: Template string for building the DSN (optional, uses DSN_VALUE as placeholder for secret value)
//
// The secret value is returned as-is (plain string). If template is provided, it replaces
// all occurrences of DSN_VALUE with the actual secret value.
func (p k8sSecretProvider) getDSN(ctx context.Context, ref *url.URL) (string, error) {
provider, err := getK8sProvider()
if err != nil {
return "", err
}

namespace := ref.Host
secretName := ref.Path

// Remove leading slash from secret name if present
if secretName != "" && secretName[0] == '/' {
secretName = secretName[1:]
}

// If namespace is empty or looks like a secret name (no slashes), treat Host as secret name in current namespace
if namespace == "" || (ref.Path == "" && namespace != "") {
// Single-part URL: k8ssecret://secret-name
secretName = namespace
namespace = provider.namespace
}

if secretName == "" {
return "", fmt.Errorf("invalid k8ssecret URL format: expected k8ssecret://[namespace/]secret-name, got %s", ref.String())
}

// Extract the key from the secret data
key := ref.Query().Get("key")
if key == "" {
key = "data_source_name"
}

// Fetch the secret from Kubernetes API
secret, err := provider.clientset.CoreV1().Secrets(namespace).Get(ctx, secretName, metav1.GetOptions{})
if err != nil {
return "", fmt.Errorf("unable to fetch secret %q from namespace %q: %w", secretName, namespace, err)
}

// Extract the key from secret data - check both Data (binary) and StringData (string)
var secretValue string

// Check in Data field first (for binary/encoded data)
if data, ok := secret.Data[key]; ok {
secretValue = string(data)
} else if stringData, ok := secret.StringData[key]; ok {
// Check in StringData field (for direct string values)
secretValue = stringData
} else {
return "", fmt.Errorf("key %q not found in Kubernetes secret %s/%s", key, namespace, secretName)
}

// Apply template if provided
templateStr := ref.Query().Get("template")
if templateStr != "" {
// Simple string replacement - replace all occurrences of DSN_VALUE with the secret value
result := strings.ReplaceAll(templateStr, "DSN_VALUE", secretValue)
return result, nil
}

return secretValue, nil
}
1 change: 1 addition & 0 deletions config/secret_resolver.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ var secretProviders = map[string]secretProvider{
"awssecretsmanager": awsSecretsManagerProvider{},
"gcpsecretmanager": gcpSecretManagerProvider{},
"hashivault": vaultProvider{},
"k8ssecret": k8sSecretProvider{},
}

// secretCacheKey returns a cache key for the secret, excluding query params so that multiple DSNs referencing the same
Expand Down
94 changes: 94 additions & 0 deletions examples/k8s-secret/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Kubernetes Secret DSN Resolution

SQL Exporter can read database connection strings (DSN) directly from Kubernetes secrets.

## Requirements

The service account running the pod must have permission to read secrets:
- `automountServiceAccountToken: true` - Automatically mounts the service account token
- RBAC Role with `secrets: get` permission

The provided Helm chart (`helm/templates/serviceaccount.yaml`) automatically creates the necessary Role and RoleBinding when `serviceAccount.create: true`.

## URL Format

```
k8ssecret://[namespace/]secret-name?key=field_name&[template=template_string]
```

**Parameters:**
- `namespace` (optional): Kubernetes namespace. If omitted, uses the pod's current namespace
- `secret-name`: Name of the Kubernetes secret (required)
- `key` (optional): Key within the secret to extract (defaults to `data_source_name`)
- `template` (optional): Template string using `DSN_VALUE` as placeholder for the secret value. If omitted, the secret value is used as-is

## Examples

### Full DSN Stored in Secret

Store a complete DSN in the secret:

```bash
kubectl create secret generic postgres-db \
--from-literal=data_source_name='postgres://user:password@host:5432/mydb?sslmode=require'
```

Use it directly (no template needed):

```yaml
config:
target:
data_source_name: 'k8ssecret://postgres-db'
collectors:
- collector1
```

### Partial DSN with Template

Store only the credentials and connection details, build the full DSN with template:

```bash
kubectl create secret generic db-creds \
--from-literal=APP_DB_CONNECTION='user:password@host:5432/database'
```

Build the full DSN with `postgres://` prefix and query parameters:

```yaml
config:
target:
data_source_name: 'k8ssecret://db-creds?key=APP_DB_CONNECTION&template=postgres://DSN_VALUE?application_name=sql-exporter&sslmode=require'
collectors:
- collector1
```

The `DSN_VALUE` placeholder will be replaced with the secret's value:
- Secret value: `user:password@host:5432/database`
- Result DSN: `postgres://user:password@host:5432/database?application_name=sql-exporter&sslmode=require`

### Cross-Namespace Secret

```yaml
config:
target:
data_source_name: 'k8ssecret://monitoring/db-secret' # From 'monitoring' namespace
collectors:
- collector1
```

⚠️ **Note**: Accessing secrets from a different namespace is **not recommended** for production deployments. It requires additional RBAC ClusterRole with cross-namespace permissions that are **not provided** with this Helm chart. For cross-namespace access, you would need to manually create a ClusterRole with `secrets: get` permission across all namespaces. It's recommended to always store secrets in the same namespace as the SQL Exporter pod.

## Deployment

Use the provided values file for quick deployment:

```bash
helm install sql-exporter ./helm \
-f deployment/values-override-static-config.yaml \
-n your-namespace
```

This automatically:
1. Creates a service account with `automountServiceAccountToken: true`
2. Creates the necessary RBAC Role and RoleBinding for secret access
3. Configures the DSN to read from the Kubernetes secret
83 changes: 83 additions & 0 deletions examples/k8s-secret/secret-examples.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Kubernetes Secrets for SQL Exporter DSN Resolution
#
# This file demonstrates the complete setup needed for SQL Exporter to read
# database connection strings from Kubernetes secrets, including:
# 1. Service account configuration
# 2. RBAC Role and RoleBinding
# 3. Database credential secrets
# 4. Example values configuration

---
# Service Account - Required for accessing Kubernetes secrets
apiVersion: v1
kind: ServiceAccount
metadata:
name: sql-exporter
namespace: default
labels:
app.kubernetes.io/name: sql-exporter
automountServiceAccountToken: true

---
# RBAC Role - Grants permission to read secrets
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: sql-exporter-secret-reader
namespace: default
labels:
app.kubernetes.io/name: sql-exporter
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]

---
# RBAC RoleBinding - Binds the role to the service account
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: sql-exporter-secret-reader
namespace: default
labels:
app.kubernetes.io/name: sql-exporter
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: sql-exporter-secret-reader
subjects:
- kind: ServiceAccount
name: sql-exporter
namespace: default

---
# Example 1: Full DSN stored in secret (complete connection string)
# Usage: k8ssecret://postgres-db
apiVersion: v1
kind: Secret
metadata:
name: postgres-db
namespace: default
labels:
app: sql-exporter
type: full-dsn
type: Opaque
stringData:
data_source_name: "postgres://user:password@postgres.default.svc.cluster.local:5432/mydb?sslmode=require"

---
# Example 2: Partial DSN stored in secret (credentials + host info only)
# This will be combined with a template to build the complete DSN
# Usage: k8ssecret://db-creds?key=APP_DB_CONNECTION&template=postgres://DSN_VALUE?application_name=sql-exporter&sslmode=require
apiVersion: v1
kind: Secret
metadata:
name: db-creds
namespace: default
labels:
app: sql-exporter
type: partial-dsn
type: Opaque
stringData:
# Partial DSN: credentials@host:port/database (no protocol, no query params)
APP_DB_CONNECTION: "user:password@postgres.default.svc.cluster.local:5432/mydb"
Loading
Loading