Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 12 additions & 3 deletions architecture/gateway-security.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,15 +170,15 @@ sequenceDiagram
The gateway supports three transport modes:

1. **mTLS (default)** -- TLS is enabled and client certificates are required.
2. **Dual-auth TLS** -- TLS is enabled, but the handshake also accepts clients without certificates (`allow_unauthenticated=true`). This is used for Cloudflare Tunnel deployments where the edge authenticates the user and forwards a Cloudflare JWT to the gateway.
2. **Dual-auth TLS** -- TLS is enabled, but the handshake also accepts clients without certificates (`allow_unauthenticated=true`). This is used for Cloudflare Tunnel deployments where the edge authenticates the user and forwards a Cloudflare token to the gateway. In this mode, the gateway enforces application-layer auth by requiring `cf-authorization` or `authorization: Bearer ...` on gRPC and tunnel endpoints.
3. **Plaintext behind edge** -- TLS is disabled at the gateway and the service listens on HTTP behind a trusted reverse proxy or tunnel.

### Server Configuration

`TlsAcceptor::from_files()` (`crates/openshell-server/src/tls.rs:27`) constructs the `rustls::ServerConfig`:

1. **Server identity**: loads the server certificate and private key from PEM files (supports PKCS#1, PKCS#8, and SEC1 key formats).
2. **Client verification**: builds a `WebPkiClientVerifier` from the CA certificate. In the default mode it requires a valid client certificate; in dual-auth mode it also accepts no-certificate clients and defers authentication to the HTTP/gRPC layer.
2. **Client verification**: builds a `WebPkiClientVerifier` from the CA certificate. In the default mode it requires a valid client certificate; in dual-auth mode it also accepts no-certificate clients and shifts identity checks to HTTP/gRPC token validation.
3. **ALPN**: advertises `h2` and `http/1.1` for protocol negotiation.

### Connection Flow
Expand All @@ -194,14 +194,23 @@ TCP accept

All traffic shares a single port. When TLS is enabled, the TLS handshake occurs before any HTTP parsing. In plaintext mode, the gateway expects an upstream reverse proxy or tunnel to be the outer security boundary.

### Application-Layer Auth Guard in Dual-Auth Mode

When `allow_unauthenticated=true`, transport security no longer guarantees caller identity. The server applies explicit token checks:

- gRPC requests are intercepted and rejected with `UNAUTHENTICATED` unless metadata contains `cf-authorization` or bearer `authorization`.
- `/_ws_tunnel` upgrades are rejected with HTTP `401` unless an edge token is present in headers or `CF_Authorization` cookie.

Health endpoints remain unauthenticated for probes.

### Cloudflare-Specific HTTP Endpoints

Cloudflare-fronted gateways add two HTTP endpoints on the same multiplexed port:

- `/auth/connect` -- browser login relay that reads the `CF_Authorization` cookie server-side and POSTs the token back to the CLI's localhost callback server.
- `/_ws_tunnel` -- WebSocket upgrade endpoint used to carry gRPC and SSH bytes through Cloudflare Access.

The WebSocket tunnel bridges directly into the gateway's `MultiplexedService` over an in-memory duplex stream. It does not re-enter the public listener, so it behaves the same whether the public listener is plaintext or TLS-backed.
The WebSocket tunnel bridges directly into the gateway's `MultiplexedService` over an in-memory duplex stream. It does not re-enter the public listener, so it behaves the same whether the public listener is plaintext or TLS-backed. In dual-auth mode, the upgrade request is rejected with `401` unless it includes an edge token (`cf-authorization`, bearer `authorization`, or `CF_Authorization` cookie).

### What Gets Rejected

Expand Down
143 changes: 143 additions & 0 deletions crates/openshell-server/src/edge_auth.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

//! Edge authentication helpers for Cloudflare/reverse-proxy deployments.

use axum::http::HeaderMap;
use tonic::metadata::MetadataMap;

/// Return `true` when headers include a usable edge auth token.
#[must_use]
pub fn has_edge_auth_http(headers: &HeaderMap) -> bool {
header_or_cookie_token(headers)
}

/// Return `true` when gRPC metadata includes a usable edge auth token.
#[must_use]
pub fn has_edge_auth_grpc(metadata: &MetadataMap) -> bool {
metadata_token(metadata, "cf-authorization") || metadata_bearer(metadata, "authorization")
}

fn header_or_cookie_token(headers: &HeaderMap) -> bool {
header_token(headers, "cf-authorization")
|| header_bearer(headers, "authorization")
|| cookie_token(headers)
}

fn header_token(headers: &HeaderMap, name: &str) -> bool {
headers
.get(name)
.and_then(|v| v.to_str().ok())
.is_some_and(|v| !v.trim().is_empty())
}

fn header_bearer(headers: &HeaderMap, name: &str) -> bool {
headers
.get(name)
.and_then(|v| v.to_str().ok())
.is_some_and(has_bearer_prefix)
}

fn cookie_token(headers: &HeaderMap) -> bool {
headers
.get("cookie")
.and_then(|v| v.to_str().ok())
.and_then(|cookies| extract_cookie(cookies, "CF_Authorization"))
.is_some_and(|v| !v.trim().is_empty())
}

fn extract_cookie(cookies: &str, name: &str) -> Option<String> {
cookies.split(';').find_map(|c| {
let mut parts = c.trim().splitn(2, '=');
let key = parts.next()?.trim();
let val = parts.next()?.trim();
if key == name {
Some(val.to_string())
} else {
None
}
})
}

fn metadata_token(metadata: &MetadataMap, name: &str) -> bool {
metadata
.get(name)
.and_then(|v| v.to_str().ok())
.is_some_and(|v| !v.trim().is_empty())
}

fn metadata_bearer(metadata: &MetadataMap, name: &str) -> bool {
metadata
.get(name)
.and_then(|v| v.to_str().ok())
.is_some_and(has_bearer_prefix)
}

fn has_bearer_prefix(value: &str) -> bool {
value
.strip_prefix("Bearer ")
.is_some_and(|token| !token.trim().is_empty())
}

#[cfg(test)]
mod tests {
use super::*;
use axum::http::{HeaderMap, HeaderValue};
use tonic::metadata::{MetadataMap, MetadataValue};

#[test]
fn http_accepts_cf_authorization_header() {
let mut headers = HeaderMap::new();
headers.insert("cf-authorization", HeaderValue::from_static("jwt-token"));
assert!(has_edge_auth_http(&headers));
}

#[test]
fn http_accepts_bearer_authorization_header() {
let mut headers = HeaderMap::new();
headers.insert(
"authorization",
HeaderValue::from_static("Bearer jwt-token"),
);
assert!(has_edge_auth_http(&headers));
}

#[test]
fn http_accepts_cf_authorization_cookie() {
let mut headers = HeaderMap::new();
headers.insert(
"cookie",
HeaderValue::from_static("foo=bar; CF_Authorization=jwt-token"),
);
assert!(has_edge_auth_http(&headers));
}

#[test]
fn http_rejects_missing_token() {
let headers = HeaderMap::new();
assert!(!has_edge_auth_http(&headers));
}

#[test]
fn grpc_accepts_cf_authorization_metadata() {
let mut metadata = MetadataMap::new();
metadata.insert("cf-authorization", MetadataValue::from_static("jwt-token"));
assert!(has_edge_auth_grpc(&metadata));
}

#[test]
fn grpc_accepts_bearer_authorization_metadata() {
let mut metadata = MetadataMap::new();
metadata.insert(
"authorization",
MetadataValue::from_static("Bearer jwt-token"),
);
assert!(has_edge_auth_grpc(&metadata));
}

#[test]
fn grpc_rejects_missing_token() {
let metadata = MetadataMap::new();
assert!(!has_edge_auth_grpc(&metadata));
}
}
1 change: 1 addition & 0 deletions crates/openshell-server/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
//! - mTLS support

mod auth;
mod edge_auth;
mod grpc;
mod http;
mod inference;
Expand Down
39 changes: 34 additions & 5 deletions crates/openshell-server/src/multiplex.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,10 @@ use std::pin::Pin;
use std::sync::Arc;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, AsyncWrite};
use tonic::{Request as TonicRequest, Status};
use tower::ServiceExt;

use crate::{OpenShellService, ServerState, http_router, inference::InferenceService};
use crate::{OpenShellService, ServerState, edge_auth, http_router, inference::InferenceService};

/// Maximum inbound gRPC message size (1 MB).
///
Expand Down Expand Up @@ -53,10 +54,25 @@ impl MultiplexService {
where
S: AsyncRead + AsyncWrite + Unpin + Send + 'static,
{
let openshell = OpenShellServer::new(OpenShellService::new(self.state.clone()))
.max_decoding_message_size(MAX_GRPC_DECODE_SIZE);
let inference = InferenceServer::new(InferenceService::new(self.state.clone()))
.max_decoding_message_size(MAX_GRPC_DECODE_SIZE);
let enforce_edge_auth = self
.state
.config
.tls
.as_ref()
.is_some_and(|tls| tls.allow_unauthenticated);
let openshell_state = self.state.clone();
let openshell = OpenShellServer::with_interceptor(
OpenShellService::new(openshell_state),
move |request: TonicRequest<()>| grpc_edge_auth_interceptor(request, enforce_edge_auth),
)
.max_decoding_message_size(MAX_GRPC_DECODE_SIZE);

let inference_state = self.state.clone();
let inference = InferenceServer::with_interceptor(
InferenceService::new(inference_state),
move |request: TonicRequest<()>| grpc_edge_auth_interceptor(request, enforce_edge_auth),
)
.max_decoding_message_size(MAX_GRPC_DECODE_SIZE);
let grpc_service = GrpcRouter::new(openshell, inference);
let http_service = http_router(self.state.clone());

Expand All @@ -70,6 +86,19 @@ impl MultiplexService {
}
}

fn grpc_edge_auth_interceptor(
request: TonicRequest<()>,
enforce_edge_auth: bool,
) -> std::result::Result<TonicRequest<()>, Status> {
if !enforce_edge_auth || edge_auth::has_edge_auth_grpc(request.metadata()) {
return Ok(request);
}

Err(Status::unauthenticated(
"missing edge auth token: provide cf-authorization or authorization bearer token",
))
}

/// Combined gRPC service that routes between `OpenShell` and Inference services
/// based on the request path prefix.
#[derive(Clone)]
Expand Down
14 changes: 14 additions & 0 deletions crates/openshell-server/src/ws_tunnel.rs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
use axum::{
Router,
extract::{State, WebSocketUpgrade, ws::Message},
http::{HeaderMap, StatusCode},
response::IntoResponse,
routing::get,
};
Expand All @@ -41,15 +42,28 @@ pub fn router(state: Arc<ServerState>) -> Router {
/// Handle the WebSocket upgrade request.
async fn ws_tunnel_handler(
State(state): State<Arc<ServerState>>,
headers: HeaderMap,
ws: WebSocketUpgrade,
) -> impl IntoResponse {
if requires_edge_auth(&state) && !crate::edge_auth::has_edge_auth_http(&headers) {
return StatusCode::UNAUTHORIZED.into_response();
}

ws.on_upgrade(move |socket| async move {
if let Err(e) = handle_ws_tunnel(socket, state).await {
warn!(error = %e, "WebSocket tunnel connection failed");
}
})
.into_response()
}

fn requires_edge_auth(state: &ServerState) -> bool {
state
.config
.tls
.as_ref()
.is_some_and(|tls| tls.allow_unauthenticated)
}
/// Pipe bytes between the WebSocket and an in-memory `MultiplexService` stream.
async fn handle_ws_tunnel(
ws: axum::extract::ws::WebSocket,
Expand Down
36 changes: 13 additions & 23 deletions crates/openshell-server/tests/edge_tunnel_auth.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,11 @@
//! | false | none | — | rejected |
//! | true | valid | — | OK |
//! | true | none | present | OK (*) |
//! | true | none | absent | OK (**) |
//! | true | none | absent | rejected |
//!
//! (*) Simulates the edge tunnel path: no client cert but a JWT header.
//! (**) TLS handshake succeeds, but in production the auth middleware (not yet
//! implemented) would reject. This test proves the TLS layer alone does
//! not block unauthenticated connections when the flag is set.
//! (**) TLS handshake may succeed, but request auth is rejected without
//! a valid edge token in application-layer metadata.

use bytes::Bytes;
use http_body_util::Empty;
Expand Down Expand Up @@ -667,14 +666,10 @@ async fn dual_auth_mtls_still_accepted() {
server.abort();
}

/// With allow_unauthenticated=true, no-client-cert connections pass the TLS
/// handshake. This simulates Cloudflare Tunnel re-originating a connection.
///
/// The gRPC health check succeeds because there is no auth middleware yet —
/// this proves the TLS layer is no longer the gate. When auth middleware is
/// added, the test should be updated to expect 401 without a valid JWT.
/// With allow_unauthenticated=true, no-client-cert connections can pass the TLS
/// handshake, but are rejected at the application layer without edge auth.
#[tokio::test]
async fn tunnel_mode_no_cert_passes_tls_handshake() {
async fn tunnel_mode_no_cert_rejected_without_edge_token() {
install_rustls_provider();
let (temp, pki) = generate_pki();

Expand All @@ -688,28 +683,23 @@ async fn tunnel_mode_no_cert_passes_tls_handshake() {

let (addr, server) = start_test_server(tls_acceptor).await;

// gRPC without client cert — should pass TLS handshake
// gRPC without client cert and without edge token — rejected
let mut grpc = grpc_client_no_cert(addr, pki.ca_cert_pem.clone()).await;
let resp = grpc.health(HealthRequest {}).await.unwrap();
assert_eq!(
resp.get_ref().status,
ServiceStatus::Healthy as i32,
"gRPC health check should succeed without client cert in tunnel mode"
let result = grpc.health(HealthRequest {}).await;
assert!(
result.is_err(),
"gRPC call should fail without edge auth token"
);

// HTTP without client cert
// HTTP health remains available without edge token
let client = https_client_no_cert(&pki.ca_cert_pem);
let req = Request::builder()
.method("GET")
.uri(format!("https://localhost:{}/healthz", addr.port()))
.body(Empty::<Bytes>::new())
.unwrap();
let resp = client.request(req).await.unwrap();
assert_eq!(
resp.status(),
StatusCode::OK,
"HTTP health check should succeed without client cert in tunnel mode"
);
assert_eq!(resp.status(), StatusCode::OK);

server.abort();
}
Expand Down
Loading