diff --git a/docs/api-reference/index.mdx b/docs/api-reference/index.mdx index 2607aa9..e9cd6ff 100644 --- a/docs/api-reference/index.mdx +++ b/docs/api-reference/index.mdx @@ -1,7 +1,7 @@ --- -title: Client SDKs +title: SDKs and REST API Reference sidebarTitle: "SDKs" -description: "SDK & REST API reference for LanceDB" +description: "SDK and REST API reference for LanceDB Enterprise and OSS." --- @@ -15,20 +15,35 @@ If you're looking for conceptual and practical namespace guidance before diving ## Supported SDKs -Python, Typescript and Rust SDKs are officially supported by LanceDB. +Python, Typescript and Rust SDKs are officially supported by LanceDB. You can use these SDKs to interact with both LanceDB OSS and Enterprise deployments. -| SDK Reference | Description | +| Reference | Description | |:--------------|-------------------| -| [Python SDK](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas & numpy integration | -| [Typescript SDK](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` -| [Rust SDK](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent-storage and high performance | +| [Python](https://lancedb.github.io/lancedb/python/python/) | Full-featured Python client with pandas & numpy integration | +| [Typescript](https://lancedb.github.io/lancedb/js/) | A TypeScript wrapper around the Rust library, built with `napi-rs` +| [Rust](https://docs.rs/lancedb/latest/lancedb/index.html) | Native Rust library with persistent-storage and high performance | -## Examples in other languages +## REST API SDKs -Other language SDKs are available through examples or third-party contributions. +Enterprise -| SDK Examples | Description | +REST API-based SDKs provide a convenient way to interact with LanceDB Cloud and Enterprise deployments using the Lance REST Namespace API. + +| Reference | Description | +|:--------------|-------------------| +| [Java](https://lancedb.github.io/lancedb/java/java/)| REST API Enterprise SDK in Java | + +## Community-driven SDKs + +In addition to the officially supported SDKs, the LanceDB community may contribute SDKs in other languages. +These SDKs may not have the same level of support or feature parity as the official ones supported by LanceDB, but they can be an option +for users working in languages other than those listed above. + +| Reference | Description | |:--------------|-------------------| -| [Java API Quickstart]https://lancedb.github.io/lancedb/java/java/)| Streamline REST API interactions in Java| +| [Go](https://pkg.go.dev/github.com/lancedb/lancedb-go/pkg/lancedb) | Community-contributed Go SDK for LanceDB | +| [Ruby](https://github.com/scientist-labs/lancelot) | Community-contributed Ruby bindings for LanceDB | +| [Swift](https://github.com/RyanLisse/LanceDbSwiftKit) | Community-contributed Swift SDK for LanceDB | +| [R](https://github.com/CathalByrneGit/lancedb) | Community-contributed R package for LanceDB | +| [Flutter](https://github.com/Alexcn/flutter_lancedb) | Community-contributed Flutter bindings for LanceDB | -{/* TODO: Add Go bindings reference page here */} diff --git a/docs/namespaces.mdx b/docs/namespaces.mdx index 570c45b..e570a59 100644 --- a/docs/namespaces.mdx +++ b/docs/namespaces.mdx @@ -23,61 +23,81 @@ A namespace can contain a collection of tables, and it can also contain namespac ![](/static/assets/images/namespaces/lance-namespace.png) -## Root namespace and the familiar `data/` layout +Before diving into examples, it helps to keep two terms in mind: the **namespace client** is the abstraction that presents a consistent namespace API, while the **namespace implementation** is the concrete backend that resolves namespaces and table locations (for example, a local directory or an external catalog). +If you want to go deeper, see the Lance format [namespace documentation](https://lance.org/format/namespace/). + +## Directory namespaces The simplest namespace model in LanceDB is a single root namespace, often represented by one directory: ```bash -/data/ # root namespace -├─ users.lance # table ["users"] in root -└─ orders.lance # table ["orders"] in root +./local_lancedb/ # catalog root ({"root": "./local_lancedb"}) +└─ data/ # root namespace + ├─ users.lance # table ["users"] in root + └─ orders.lance # table ["orders"] in root ``` -As a user of LanceDB, you might never notice namespaces at first, because LanceDB exposes the single-level -hierarchy shown above, with the data stored in the `data/` directory, where the root namespace -is implicit. In alternative setups, you could have multiple namespaces that we won't cover here, -but you can learn more about them in the [namespace documentation](https://lance.org/format/namespace/) for the Lance format. +As a user of LanceDB OSS, you might never notice namespaces at first, because LanceDB exposes the single-level hierarchy shown above, with the data stored in the `data/` directory, where the root namespace is implicit. Connecting to this namespace is as simple as connecting to the catalog root: -## Best-practice guidance +```python Python icon="python" +import lancedb -- Use the default, single-level root namespace in LanceDB for locally stored, single-application, or early-stage projects. -- For remote storage locations, introduce explicit namespaces when multiple teams, environments, or domains share the same catalog. -- Treat namespace paths as stable identifiers (for example `"prod/search"`, `"staging/recs"`). -- Avoid hard-coding object-store table paths in application code -- instead, prefer catalog identifiers + namespaces. +# Connect to the directory namespace root +db = lancedb.connect("./local_lancedb") +``` -See the Python example below for how to use namespaces in practice. +You can also explicitly connect to this namespace in Python using `lancedb.connect_namespace(...)` with the directory namespace implementation: ```python Python icon="python" import lancedb # Local namespace-backed catalog root (DirectoryNamespace) -db = lancedb.connect_namespace("dir", {"root": "./namespace_test"}) - -# Business identifier + namespace path (stable app-level IDs) -namespace = ["prod", "recommendations"] -table_name = "user_profiles" - -# Create namespace hierarchy sequentially -for i in range(1, len(namespace) + 1): - db.create_namespace(namespace[:i], mode="exist_ok") - -# Good: resolve table through catalog + namespace -table = db.create_table( - table_name, - data=[{"id": 1, "vector": [0.1, 0.2], "name": "alice"}], - namespace=namespace, - mode="overwrite", +# See https://lance.org/format/namespace/dir/catalog-spec/ +db = lancedb.connect_namespace( + "dir", + { + "root": "./local_lancedb" + }, ) +``` + + +For simple search use cases in LanceDB OSS, you don't need to go too deep into namespaces. However, to integrate LanceDB with external catalogs and to use it as a true multimodal lakehouse, understanding the different namespace implementations and how to use them in your organization's setup is beneficial. + + +## Remote or external catalog namespaces -# Bad: Avoid hard-coded physical object-store table paths -# (it's bad for maintainability reasons) -# table = lancedb.connect( -# "s3://my-lakehouse/catalog/prod/recommendations/user_profiles.lance" -# ) +For remote object stores with central metadata/catalog services (either commercial or open source), use the REST namespace implementation. In LanceDB, this is the same namespace API surface as local directory namespaces, but backed by REST routes +(for example `POST /v1/namespace/{id}/create` and `GET /v1/namespace/{id}/list`) and server-provided table locations. + +For authentication, any property prefixed with `headers` is forwarded as an HTTP header +(for example `headers.Authorization` becomes `Authorization`, and `headers.X-API-Key` becomes `X-API-Key`). + +```python Python icon="python" +import os +import lancedb + +# Remote namespace-backed catalog root (RestNamespace) +# See https://lance.org/format/namespace/rest/catalog-spec/ +db = lancedb.connect_namespace( + "rest", + { + "uri": "https://.internal..com", + "headers.x-api-key": os.environ["API_KEY"], + # or: + # "headers.Authorization": f"Bearer {os.environ['REST_AUTH_TOKEN']}", + }, +) ``` +[LanceDB Enterprise](/enterprise) operates a REST namespace server on top of the Lance format, so any REST client that can speak the REST namespace API +contract can be used to interact with it. For authentication examples in LanceDB Enterprise, visit +the [Namespaces in SDKs](/tables/namespaces) page. -## SDK usage +## Best practices -1. For language-specific examples of `namespace` usage across Python, TypeScript, and Rust, see "[Using namespaces in SDKs](/tables/namespaces)". -2. For REST-level operations, see the [REST API Reference](/api-reference/rest). +Below, we list some best practices for working with namespaces: +- For simple use cases and single, stand-alone applications, the directory-based root namespace is sufficient and requires no special configuration. +- For remote storage locations, introduce explicit namespaces when multiple teams, environments, or domains share the same catalog. +- Treat namespace paths as stable identifiers (for example `"prod/search"`, `"staging/recs"`). +- For maintainability reasons, avoid hard-coding object-store table paths in application code -- instead, prefer catalog identifiers + namespaces. \ No newline at end of file diff --git a/docs/tables/namespaces.mdx b/docs/tables/namespaces.mdx index e2b96e4..774bd67 100644 --- a/docs/tables/namespaces.mdx +++ b/docs/tables/namespaces.mdx @@ -1,5 +1,5 @@ --- -title: "Using Namespaces in SDKs" +title: "Using Namespaces" sidebarTitle: "Namespaces" description: "Use LanceDB's namespace-aware table and catalog APIs in Python, TypeScript, and Rust." icon: "folder-tree" @@ -69,6 +69,42 @@ In TypeScript, namespace lifecycle management is not on `Connection`, so namespa created through another admin surface (for example REST/admin tooling) before use. +## Namespaces in LanceDB Enterprise + +In LanceDB Enterprise deployments, configure namespace-backed federated databases in a TOML file under your deployment's `config` directory. +LanceDB Enterprise supports both directory-based (`ns_impl = "dir"`) and REST-based (`ns_impl = "rest"`) namespace implementations. +The example below shows how to configure a directory-based namespace implementation in LanceDB Enterprise. +```toml +# Federated database configuration for DirectoryNamespace +# This example uses minio storage +[federated_dbs.federated_dir_test] +ns_impl = "dir" +root = "s3:///" +"storage.region" = "us-east-1" +"storage.endpoint" = "http://localhost:9000" +"storage.access_key_id" = "minioadmin" +"storage.secret_access_key" = "minioadmin" +"storage.allow_http" = "true" +# Far future expiration (year 2100) +"storage.expires_at_millis" = "4102444800000" +``` + +The example above uses MinIO, but the same approach applies to other cloud object storage platforms based on your deployment. + +For REST-based namespace servers, you can specify the namespace implementation as `"rest"` with forwarding prefixed headers +for authentication and context propagation. + +```toml +[federated_dbs.federated_rest_test] +ns_impl = "rest" +uri = "http://.internal.catalog.com" +forward_header_prefixes = ["X-forward"] +``` + +With `forward_header_prefixes = ["X-forward"]`, any incoming header starting with `X-forward` is forwarded to +`http://.internal.catalog.com`. This is useful for auth propagation, for example sending +`X-forward-authorization: Bearer xxxx`. + ## Related references - [Client SDK API references](/api-reference)