diff --git a/CodeGen/README.md b/CodeGen/README.md index 9aebba4472..3b97e2ed4b 100644 --- a/CodeGen/README.md +++ b/CodeGen/README.md @@ -106,18 +106,19 @@ flowchart LR This CodeGen example can be deployed manually on various hardware platforms using Docker Compose or Kubernetes. Select the appropriate guide based on your target environment: -| Hardware | Deployment Mode | Guide Link | -| :-------------- | :----------------------------------- | :--------------------------------------------------------------------------------------- | -| Intel Xeon CPU | Single Node (Docker) | [Xeon Docker Compose Guide](./docker_compose/intel/cpu/xeon/README.md) | -| Intel Xeon CPU | Single Node (Docker) with Monitoring | [Xeon Docker Compose with Monitoring Guide](./docker_compose/intel/cpu/xeon/README.md) | -| Intel Gaudi HPU | Single Node (Docker) | [Gaudi Docker Compose Guide](./docker_compose/intel/hpu/gaudi/README.md) | -| Intel Gaudi HPU | Single Node (Docker) with Monitoring | [Gaudi Docker Compose with Monitoring Guide](./docker_compose/intel/hpu/gaudi/README.md) | -| AMD EPYC CPU | Single Node (Docker) | [EPYC Docker Compose Guide](./docker_compose/amd/cpu/epyc/README.md) | -| AMD ROCm GPU | Single Node (Docker) | [ROCm Docker Compose Guide](./docker_compose/amd/gpu/rocm/README.md) | -| Intel Xeon CPU | Kubernetes (Helm) | [Kubernetes Helm Guide](./kubernetes/helm/README.md) | -| Intel Gaudi HPU | Kubernetes (Helm) | [Kubernetes Helm Guide](./kubernetes/helm/README.md) | -| Intel Xeon CPU | Kubernetes (GMC) | [Kubernetes GMC Guide](./kubernetes/gmc/README.md) | -| Intel Gaudi HPU | Kubernetes (GMC) | [Kubernetes GMC Guide](./kubernetes/gmc/README.md) | +| Hardware | Deployment Mode | Guide Link | +| :------------------ | :----------------------------------- | :--------------------------------------------------------------------------------------- | +| Intel Xeon CPU | Single Node (Docker) | [Xeon Docker Compose Guide](./docker_compose/intel/cpu/xeon/README.md) | +| Intel Xeon CPU | Single Node (Docker) with Monitoring | [Xeon Docker Compose with Monitoring Guide](./docker_compose/intel/cpu/xeon/README.md) | +| Intel Gaudi HPU | Single Node (Docker) | [Gaudi Docker Compose Guide](./docker_compose/intel/hpu/gaudi/README.md) | +| Intel Gaudi HPU | Single Node (Docker) with Monitoring | [Gaudi Docker Compose with Monitoring Guide](./docker_compose/intel/hpu/gaudi/README.md) | +| Intel Arc GPU (XPU) | Single Node (Docker) | [Arc XPU Docker Compose Guide](./docker_compose/intel/xpu/arc/README.md) | +| AMD EPYC CPU | Single Node (Docker) | [EPYC Docker Compose Guide](./docker_compose/amd/cpu/epyc/README.md) | +| AMD ROCm GPU | Single Node (Docker) | [ROCm Docker Compose Guide](./docker_compose/amd/gpu/rocm/README.md) | +| Intel Xeon CPU | Kubernetes (Helm) | [Kubernetes Helm Guide](./kubernetes/helm/README.md) | +| Intel Gaudi HPU | Kubernetes (Helm) | [Kubernetes Helm Guide](./kubernetes/helm/README.md) | +| Intel Xeon CPU | Kubernetes (GMC) | [Kubernetes GMC Guide](./kubernetes/gmc/README.md) | +| Intel Gaudi HPU | Kubernetes (GMC) | [Kubernetes GMC Guide](./kubernetes/gmc/README.md) | _Note: Building custom microservice images can be done using the resources in [GenAIComps](https://github.com/opea-project/GenAIComps)._ @@ -180,15 +181,16 @@ Intelยฎ Optimized Cloud Modules for Terraform provide an automated way to deploy ## Validated Configurations -| **Deploy Method** | **LLM Engine** | **LLM Model** | **Hardware** | -| ----------------- | -------------- | ------------------------------ | ------------ | -| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Gaudi | -| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Xeon | -| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD EPYC | -| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm | -| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Gaudi | -| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Xeon | -| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm | +| **Deploy Method** | **LLM Engine** | **LLM Model** | **Hardware** | +| ----------------- | -------------- | ------------------------------ | --------------- | +| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Gaudi | +| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Xeon | +| Docker Compose | vLLM | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Arc (XPU) | +| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD EPYC | +| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm | +| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Gaudi | +| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Xeon | +| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm | ## Contribution diff --git a/CodeGen/docker_compose/intel/xpu/arc/.gitignore b/CodeGen/docker_compose/intel/xpu/arc/.gitignore new file mode 100644 index 0000000000..8fce603003 --- /dev/null +++ b/CodeGen/docker_compose/intel/xpu/arc/.gitignore @@ -0,0 +1 @@ +data/ diff --git a/CodeGen/docker_compose/intel/xpu/arc/QUICK_START.md b/CodeGen/docker_compose/intel/xpu/arc/QUICK_START.md new file mode 100644 index 0000000000..adf6bbbdcb --- /dev/null +++ b/CodeGen/docker_compose/intel/xpu/arc/QUICK_START.md @@ -0,0 +1,196 @@ +# CodeGen on Intel Arc XPU - Quick Start Guide + +## ๐Ÿš€ Quick Deployment (3 Steps) + +### Step 1: Setup Environment (1 minute) + +```bash +cd GenAIExamples/CodeGen/docker_compose/intel/xpu/arc + +# Set your host IP and HuggingFace token +export HOST_IP=$(hostname -I | awk '{print $1}') +export HF_TOKEN="your_huggingface_token" + +# Optional: Configure proxy if needed +export no_proxy="localhost,127.0.0.1,${HOST_IP}" +export NO_PROXY="localhost,127.0.0.1,${HOST_IP}" + +source ./set_env.sh +``` + +### Step 2: Deploy Services (5-10 minutes) + +```bash +docker compose up -d +``` + +### Step 3: Wait for Model to Load (3-5 minutes) + +```bash +docker compose logs -f codegen-vllm-service +``` + +Wait for: `Application startup complete` + +--- + +## ๐Ÿงช Quick Test + +### Test 1: Health Check + +```bash +curl http://${HOST_IP}:8028/health +``` + +Expected: `{"status":"ok"}` + +### Test 2: Code Generation + +```bash +curl http://${HOST_IP}:8028/v1/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen2.5-Coder-7B-Instruct", + "prompt": "def fibonacci(n):", + "max_tokens": 100, + "temperature": 0.7 + }' +``` + +### Test 3: Access UI + +Open browser: http://${HOST_IP}:5173 + +--- + +## ๐Ÿ“Š Service Ports + +| Service | Port | URL | +| ----------- | ---- | ---------------------- | +| vLLM | 8028 | http://${HOST_IP}:8028 | +| LLM Service | 9000 | http://${HOST_IP}:9000 | +| Backend | 7778 | http://${HOST_IP}:7778 | +| UI | 5173 | http://${HOST_IP}:5173 | + +--- + +## ๐Ÿ› ๏ธ Useful Commands + +### View Logs + +```bash +# All services +docker compose logs -f + +# Specific service +docker compose logs -f codegen-vllm-service +``` + +### Check Status + +```bash +docker compose ps +``` + +### Stop Services + +```bash +docker compose down +``` + +### Restart Services + +```bash +docker compose restart +``` + +### Remove Everything + +```bash +docker compose down -v +``` + +--- + +## ๐Ÿ”ง Troubleshooting + +### GPU Not Detected? + +```bash +ls -la /dev/dri/ +sudo usermod -aG video,render $USER +# Logout and login +``` + +### Service Won't Start? + +```bash +docker compose logs codegen-vllm-service +docker compose ps +``` + +### Out of Memory? + +Edit `compose.yaml`: + +```yaml +shm_size: 16g # Increase from 10g +``` + +--- + +## ๐Ÿ“ Configuration + +### Change Model + +Edit `set_env.sh`: + +```bash +export CODEGEN_LLM_MODEL_ID="your-model-id" +``` + +### Change Ports + +Edit `set_env.sh`: + +```bash +export CODEGEN_VLLM_SERVICE_PORT=8029 +export CODEGEN_UI_SERVICE_PORT=5174 +``` + +--- + +## โœ… Validation Checklist + +- [ ] Docker daemon running +- [ ] Intel GPU detected at `/dev/dri` +- [ ] Environment variables set +- [ ] Services deployed +- [ ] Health endpoint responds +- [ ] Code generation works +- [ ] UI accessible + +--- + +## ๐Ÿ“š More Information + +- Full documentation: [README.md](./README.md) +- Main CodeGen docs: [../../README.md](../../../README.md) + +--- + +## ๐ŸŽฏ Expected Timeline + +| Phase | Duration | Status | +| ----------------- | ------------- | ------ | +| Environment setup | 1 min | โœ… | +| Pull images | 10-15 min | โณ | +| Start services | 2 min | โณ | +| Model loading | 3-5 min | โณ | +| **Total** | **15-20 min** | | + +--- + +**Hardware**: Intel Arc Pro B-series GPU +**Model**: Qwen/Qwen2.5-Coder-7B-Instruct +**Backend**: Intel vLLM 0.14.1-xpu diff --git a/CodeGen/docker_compose/intel/xpu/arc/README.md b/CodeGen/docker_compose/intel/xpu/arc/README.md new file mode 100644 index 0000000000..5d625ae33c --- /dev/null +++ b/CodeGen/docker_compose/intel/xpu/arc/README.md @@ -0,0 +1,306 @@ +# Deploy CodeGen Application on Intel Arc GPU (XPU) with Docker Compose + +This README provides instructions for deploying the CodeGen application using Docker Compose on a system equipped with Intel Arc Pro B-series GPUs, detailing the steps to configure, run, and validate the services. This guide uses the **vLLM** backend optimized for Intel XPU for LLM serving. + +## Table of Contents + +- [Overview](#overview) +- [Prerequisites](#prerequisites) +- [Quick Start](#quick-start) +- [Configuration Parameters](#configuration-parameters) + - [Environment Variables](#environment-variables) +- [Deploy the Services Using Docker Compose](#deploy-the-services-using-docker-compose) +- [Validate Services](#validate-services) + - [Check Container Status](#check-container-status) + - [Test the Pipeline](#test-the-pipeline) +- [Accessing the User Interface (UI)](#accessing-the-user-interface-ui) +- [Troubleshooting](#troubleshooting) +- [Stopping the Application](#stopping-the-application) +- [Next Steps](#next-steps) + +## Overview + +This guide focuses on running the pre-configured CodeGen service using Docker Compose on Intel Arc Pro B-series GPU platform. It leverages containers optimized for Intel XPU architecture for LLM serving using vLLM, along with the CodeGen gateway and UI components. + +## Prerequisites + +- Docker and Docker Compose installed +- Intel Arc Pro B-series GPU (or compatible Intel discrete GPU) +- Intel GPU drivers installed and properly configured +- Git installed (for cloning repository) +- Hugging Face Hub API Token (for downloading models) +- Access to the internet (or a private model cache) +- Clone the `GenAIExamples` repository: + +```bash +git clone https://github.com/opea-project/GenAIExamples.git +cd GenAIExamples/CodeGen/docker_compose/intel/xpu/arc/ +``` + +Checkout a released version, such as v1.3: + +```bash +git checkout v1.3 +``` + +## Quick Start + +### 1. Generate a HuggingFace Access Token + +Some HuggingFace resources, such as some models, are only accessible if you have an access token. If you do not already have a HuggingFace access token, you can create one by first creating an account by following the steps provided at [HuggingFace](https://huggingface.co/) and then generating a [user access token](https://huggingface.co/docs/transformers.js/en/guides/private#step-1-generating-a-user-access-token). + +### 2. Configure the Deployment Environment + +Set up environment variables for deploying CodeGen services: + +```bash +# Replace with your host's external IP address (do not use localhost or 127.0.0.1) +export HOST_IP=$(hostname -I | awk '{print $1}') +# Replace with your Hugging Face Hub API token +export HF_TOKEN="your_huggingface_token" + +# Optional: Configure proxy if needed +# export http_proxy="your_http_proxy" +# export https_proxy="your_https_proxy" +# export no_proxy="localhost,127.0.0.1,${HOST_IP}" +source ./set_env.sh +``` + +### 3. Deploy the Services Using Docker Compose + +```bash +docker compose up -d +``` + +This will start the following services: + +- **codegen-vllm-service**: vLLM service optimized for Intel XPU +- **codegen-llm-server**: LLM microservice that interfaces with vLLM +- **codegen-backend-server**: CodeGen backend (MegaService) +- **codegen-ui-server**: Web UI for CodeGen + +### 4. Check the Deployment Status + +Monitor the logs to ensure all services start successfully: + +```bash +docker compose logs -f +``` + +Check container status: + +```bash +docker ps +``` + +All containers should show as healthy or running. + +## Configuration Parameters + +### Environment Variables + +Key parameters are configured via environment variables set in `set_env.sh`: + +| Environment Variable | Description | Default Value | +| :----------------------------- | :---------------------------------------------------------------------------------------------------- | :------------------------------- | +| `HOST_IP` | External IP address of the host machine. **Required.** Must be exported before sourcing `set_env.sh`. | _None_ | +| `HF_TOKEN` | Your Hugging Face Hub token for model access. **Required.** | `${HF_TOKEN}` | +| `CODEGEN_LLM_MODEL_ID` | Hugging Face model ID for the CodeGen LLM | `Qwen/Qwen2.5-Coder-7B-Instruct` | +| `CODEGEN_VLLM_SERVICE_PORT` | Port for vLLM service | `8028` | +| `CODEGEN_LLM_SERVICE_PORT` | Port for LLM microservice | `9000` | +| `CODEGEN_BACKEND_SERVICE_PORT` | Port for CodeGen backend service | `7778` | +| `CODEGEN_UI_SERVICE_PORT` | Port for CodeGen UI | `5173` | +| `MODEL_CACHE` | Directory for model cache | `./data` | +| `REGISTRY` | Docker registry for OPEA images | `opea` | +| `TAG` | Docker image tag | `latest` | +| `http_proxy` / `https_proxy` | Network proxy settings (if required) | `""` | +| `no_proxy` | No proxy list | Includes localhost and `HOST_IP` | + +### Intel XPU Specific Environment Variables + +The following environment variables are set in the vLLM service for Intel XPU optimization: + +- `VLLM_TARGET_DEVICE: "xpu"` - Targets Intel XPU devices +- `VLLM_LOGGING_LEVEL: "DEBUG"` - Sets logging level for debugging +- `ZE_FLAT_DEVICE_HIERARCHY: "FLAT"` - Level Zero driver configuration +- `ONEAPI_DEVICE_SELECTOR: "level_zero:gpu;opencl:gpu"` - Device selector for oneAPI + +## Deploy the Services Using Docker Compose + +```bash +cd GenAIExamples/CodeGen/docker_compose/intel/xpu/arc/ +docker compose up -d +``` + +### Wait for Services to Be Ready + +The vLLM service may take several minutes to download the model and initialize. Monitor progress: + +```bash +docker compose logs -f codegen-vllm-service +``` + +Wait for a message indicating the server is ready to accept requests. + +## Validate Services + +### Check Container Status + +```bash +docker ps +``` + +Expected output should show all four containers running: + +- `codegen-vllm-service` (healthy) +- `codegen-llm-server` (running) +- `codegen-backend-server` (running) +- `codegen-ui-server` (running) + +### Test the vLLM Service + +```bash +curl http://${HOST_IP}:8028/v1/completions \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen2.5-Coder-7B-Instruct", + "prompt": "def fibonacci(n):", + "max_tokens": 100, + "temperature": 0.7 + }' +``` + +### Test the LLM Microservice + +```bash +curl http://${HOST_IP}:9000/v1/chat/completions \ + -X POST \ + -H "Content-Type: application/json" \ + -d '{ + "query": "Write a Python function to calculate factorial" + }' +``` + +### Test the CodeGen Backend Service + +```bash +curl http://${HOST_IP}:7778/v1/codegen \ + -X POST \ + -H "Content-Type: application/json" \ + -d '{ + "messages": "Write a function to sort an array in Python" + }' +``` + +## Accessing the User Interface (UI) + +Once all services are running and validated, access the CodeGen UI: + +```bash +http://${HOST_IP}:5173 +``` + +Open this URL in your web browser. You should see the CodeGen interface where you can: + +- Enter natural language prompts for code generation +- View generated code +- Interact with the CodeGen assistant + +## Troubleshooting + +### GPU Not Detected + +If vLLM cannot detect the Intel GPU: + +1. Verify GPU drivers are installed: + + ```bash + clinfo + ``` + +2. Check device permissions: + + ```bash + ls -la /dev/dri + ``` + +3. Verify the container has access to `/dev/dri`: + ```bash + docker compose exec codegen-vllm-service ls -la /dev/dri + ``` + +### vLLM Service Fails to Start + +1. Check logs for errors: + + ```bash + docker compose logs codegen-vllm-service + ``` + +2. Common issues: + - Model download failed: Check HF_TOKEN and network connectivity + - Out of memory: Reduce model size or adjust `shm_size` in compose.yaml + - Driver issues: Update Intel GPU drivers + +### Service Cannot Connect + +1. Check network connectivity between containers: + + ```bash + docker compose exec codegen-llm-server ping codegen-vllm-service + ``` + +2. Verify environment variables are set correctly: + ```bash + docker compose config + ``` + +### Performance Issues + +1. Monitor GPU utilization: + + ```bash + intel_gpu_top + ``` + +2. Check container resource usage: + ```bash + docker stats + ``` + +## Stopping the Application + +To stop all services: + +```bash +docker compose down +``` + +To also remove volumes (model cache): + +```bash +docker compose down -v +``` + +## Next Steps + +- **Customize the Model**: Change `CODEGEN_LLM_MODEL_ID` in `set_env.sh` to use a different model +- **Adjust Resources**: Modify `shm_size` and resource limits in `compose.yaml` +- **Enable Monitoring**: Add Prometheus and Grafana for monitoring (see main README) +- **Scale Services**: Deploy multiple vLLM instances for load balancing +- **Integrate with IDE**: Use the CodeGen API endpoint with your IDE or code editor + +## Additional Resources + +- [OPEA Project Documentation](https://opea-project.github.io/) +- [vLLM Documentation](https://docs.vllm.ai/) +- [Intel GPU Drivers](https://dgpu-docs.intel.com/) +- [GenAIComps Repository](https://github.com/opea-project/GenAIComps) + +## Support + +For issues and questions: + +- Open an issue in the [GenAIExamples repository](https://github.com/opea-project/GenAIExamples/issues) +- Check existing documentation and examples +- Join the OPEA community discussions diff --git a/CodeGen/docker_compose/intel/xpu/arc/compose.yaml b/CodeGen/docker_compose/intel/xpu/arc/compose.yaml new file mode 100644 index 0000000000..2853697a72 --- /dev/null +++ b/CodeGen/docker_compose/intel/xpu/arc/compose.yaml @@ -0,0 +1,90 @@ +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +services: + codegen-vllm-service: + image: intel/vllm:0.14.1-xpu + container_name: codegen-vllm-service + ports: + - "${CODEGEN_VLLM_SERVICE_PORT:-8028}:80" + volumes: + - "${MODEL_CACHE:-./data}:/root/.cache/huggingface/hub" + shm_size: 10g + devices: + - /dev/dri:/dev/dri + # Grant least-privilege access to the Intel GPU by joining the host's + # video/render groups instead of running the container as privileged. + group_add: + - ${VIDEOGROUPID:-44} + - ${RENDERGROUPID:-109} + environment: + no_proxy: ${no_proxy} + http_proxy: ${http_proxy} + https_proxy: ${https_proxy} + HF_TOKEN: ${CODEGEN_HUGGINGFACEHUB_API_TOKEN} + host_ip: ${HOST_IP} + VLLM_TARGET_DEVICE: "xpu" + VLLM_LOGGING_LEVEL: "DEBUG" + ZE_FLAT_DEVICE_HIERARCHY: "FLAT" + ONEAPI_DEVICE_SELECTOR: "level_zero:gpu;opencl:gpu" + healthcheck: + # Use python3 (always present in the vLLM image) rather than curl, which + # is not guaranteed to be installed and would leave the container unhealthy. + test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:80/health')\" || exit 1"] + interval: 10s + timeout: 10s + retries: 100 + command: --model ${CODEGEN_LLM_MODEL_ID} --host 0.0.0.0 --port 80 + codegen-llm-server: + image: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest} + container_name: codegen-llm-server + depends_on: + codegen-vllm-service: + condition: service_healthy + ports: + - "${CODEGEN_LLM_SERVICE_PORT:-9000}:9000" + ipc: host + environment: + no_proxy: ${no_proxy} + http_proxy: ${http_proxy} + https_proxy: ${https_proxy} + LLM_ENDPOINT: ${CODEGEN_VLLM_ENDPOINT} + LLM_MODEL_ID: ${CODEGEN_LLM_MODEL_ID} + HF_TOKEN: ${CODEGEN_HUGGINGFACEHUB_API_TOKEN} + LLM_COMPONENT_NAME: "OpeaTextGenService" + restart: unless-stopped + codegen-backend-server: + image: ${REGISTRY:-opea}/codegen:${TAG:-latest} + container_name: codegen-backend-server + depends_on: + - codegen-llm-server + ports: + - "${CODEGEN_BACKEND_SERVICE_PORT:-7778}:7778" + environment: + no_proxy: ${no_proxy} + https_proxy: ${https_proxy} + http_proxy: ${http_proxy} + MEGA_SERVICE_HOST_IP: ${CODEGEN_MEGA_SERVICE_HOST_IP} + LLM_SERVICE_HOST_IP: ${HOST_IP} + LLM_SERVICE_PORT: ${CODEGEN_LLM_SERVICE_PORT} + ipc: host + restart: always + codegen-ui-server: + image: ${REGISTRY:-opea}/codegen-ui:${TAG:-latest} + container_name: codegen-ui-server + depends_on: + - codegen-backend-server + ports: + - "${CODEGEN_UI_SERVICE_PORT:-5173}:5173" + environment: + no_proxy: ${no_proxy} + https_proxy: ${https_proxy} + http_proxy: ${http_proxy} + BASIC_URL: ${CODEGEN_BACKEND_SERVICE_URL} + BACKEND_SERVICE_ENDPOINT: ${CODEGEN_BACKEND_SERVICE_URL} + ipc: host + restart: always + +networks: + default: + driver: bridge diff --git a/CodeGen/docker_compose/intel/xpu/arc/set_env.sh b/CodeGen/docker_compose/intel/xpu/arc/set_env.sh new file mode 100644 index 0000000000..5ce402cc24 --- /dev/null +++ b/CodeGen/docker_compose/intel/xpu/arc/set_env.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash + +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +### The IP address or domain name of the server on which the application is running +export HOST_IP=${HOST_IP} +export EXTERNAL_HOST_IP=${HOST_IP} + +### The port of the vLLM service. On this port, the vLLM service will accept connections +export CODEGEN_VLLM_SERVICE_PORT=8028 +export CODEGEN_VLLM_ENDPOINT="http://${HOST_IP}:${CODEGEN_VLLM_SERVICE_PORT}" + +### A token for accessing repositories with models +export CODEGEN_HUGGINGFACEHUB_API_TOKEN=${HF_TOKEN} + +### Model ID +export CODEGEN_LLM_MODEL_ID="Qwen/Qwen2.5-Coder-7B-Instruct" + +### Model cache directory +export MODEL_CACHE=${MODEL_CACHE:-"./data"} + +### The port of the LLM service. On this port, the LLM service will accept connections +export CODEGEN_LLM_SERVICE_PORT=9000 + +### The IP address or domain name of the server for CodeGen MegaService +export CODEGEN_MEGA_SERVICE_HOST_IP=${HOST_IP} + +### The port for CodeGen backend service +export CODEGEN_BACKEND_SERVICE_PORT=7778 + +### The URL of CodeGen backend service, used by the frontend service +export CODEGEN_BACKEND_SERVICE_URL="http://${EXTERNAL_HOST_IP}:${CODEGEN_BACKEND_SERVICE_PORT}/v1/codegen" + +### The endpoint of the LLM service to which requests to this service will be sent +export CODEGEN_LLM_SERVICE_HOST_IP=${HOST_IP} + +### The CodeGen service UI port +export CODEGEN_UI_SERVICE_PORT=5173 + +### Docker registry and tag +export REGISTRY=${REGISTRY:-opea} +export TAG=${TAG:-latest} + +### Intel GPU device group IDs, used to grant the vLLM container least-privilege +### access to /dev/dri without running in privileged mode. Auto-detected from the +### host; falls back to common defaults if the groups are not present. +export VIDEOGROUPID=$(getent group video | awk -F: '{print $3}') +export RENDERGROUPID=$(getent group render | awk -F: '{print $3}') +export VIDEOGROUPID=${VIDEOGROUPID:-44} +export RENDERGROUPID=${RENDERGROUPID:-109} diff --git a/CodeGen/docker_compose/intel/xpu/arc/test_deployment.sh b/CodeGen/docker_compose/intel/xpu/arc/test_deployment.sh new file mode 100755 index 0000000000..5c020b0280 --- /dev/null +++ b/CodeGen/docker_compose/intel/xpu/arc/test_deployment.sh @@ -0,0 +1,93 @@ +#!/bin/bash +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +set -e + +echo "========================================" +echo "CodeGen XPU Deployment Test" +echo "========================================" +echo "" + +# Check prerequisites +echo "1. Checking prerequisites..." +echo " - Docker version:" +docker --version + +echo " - Intel GPU devices:" +if [ -d "/dev/dri" ] && ls -la /dev/dri/ | grep -qE "card|render"; then + ls -la /dev/dri/ | grep -E "card|render" + echo " โœ“ Intel GPU devices found" +else + echo " โœ— /dev/dri not found - Intel GPU may not be available" + exit 1 +fi + +echo "" +echo "2. Checking environment variables..." +if [ -z "$HOST_IP" ]; then + echo " โœ— HOST_IP not set" + exit 1 +else + echo " โœ“ HOST_IP: $HOST_IP" +fi + +if [ -z "$HF_TOKEN" ]; then + echo " โœ— HF_TOKEN not set" + exit 1 +else + echo " โœ“ HF_TOKEN is set (${#HF_TOKEN} characters)" +fi + +if [ -z "$CODEGEN_LLM_MODEL_ID" ]; then + echo " โœ— CODEGEN_LLM_MODEL_ID not set" + exit 1 +else + echo " โœ“ Model: $CODEGEN_LLM_MODEL_ID" +fi + +echo "" +echo "3. Validating Docker Compose configuration..." +if command -v docker-compose &> /dev/null; then + COMPOSE_CMD="docker-compose" +elif docker compose version &> /dev/null; then + COMPOSE_CMD="docker compose" +else + echo " โœ— Neither 'docker-compose' nor 'docker compose' found" + exit 1 +fi + +echo " Using: $COMPOSE_CMD" +if $COMPOSE_CMD config > /dev/null 2>&1; then + echo " โœ“ Docker Compose configuration is valid" +else + echo " โœ— Docker Compose configuration has errors" + exit 1 +fi + +echo "" +echo "4. Checking Docker Compose services..." +$COMPOSE_CMD config --services +echo "" + +echo "5. Summary of configuration:" +echo " - vLLM Service Port: $CODEGEN_VLLM_SERVICE_PORT" +echo " - LLM Service Port: $CODEGEN_LLM_SERVICE_PORT" +echo " - Backend Service Port: $CODEGEN_BACKEND_SERVICE_PORT" +echo " - UI Service Port: $CODEGEN_UI_SERVICE_PORT" +echo " - Model Cache: $MODEL_CACHE" +echo "" + +echo "========================================" +echo "Deployment configuration is valid!" +echo "========================================" +echo "" +echo "To deploy, run:" +echo " $COMPOSE_CMD up -d" +echo "" +echo "To monitor logs:" +echo " $COMPOSE_CMD logs -f" +echo "" +echo "To test vLLM service after deployment:" +echo " curl http://\${HOST_IP}:8028/health" +echo "" diff --git a/CodeGen/docker_compose/intel/xpu/arc/validate_config.sh b/CodeGen/docker_compose/intel/xpu/arc/validate_config.sh new file mode 100755 index 0000000000..c41715fde0 --- /dev/null +++ b/CodeGen/docker_compose/intel/xpu/arc/validate_config.sh @@ -0,0 +1,129 @@ +#!/bin/bash +# Copyright (C) 2024 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +set -e + +echo "========================================" +echo "CodeGen XPU Configuration Validation" +echo "========================================" +echo "" + +# Check prerequisites +echo "1. Checking prerequisites..." +echo " - Docker version:" +docker --version || { echo " โœ— Docker not installed"; exit 1; } + +echo " - Intel GPU devices:" +if [ -d "/dev/dri" ] && ls -la /dev/dri/ | grep -qE "card|render"; then + ls -la /dev/dri/ | grep -E "card|render" + echo " โœ“ Intel GPU devices found" +else + echo " โœ— /dev/dri not found - Intel GPU may not be available" + exit 1 +fi + +echo "" +echo "2. Checking environment variables..." +if [ -z "$HOST_IP" ]; then + echo " โœ— HOST_IP not set" + exit 1 +else + echo " โœ“ HOST_IP: $HOST_IP" +fi + +if [ -z "$HF_TOKEN" ]; then + echo " โœ— HF_TOKEN not set" + exit 1 +else + echo " โœ“ HF_TOKEN is set (${#HF_TOKEN} characters)" +fi + +if [ -z "$CODEGEN_LLM_MODEL_ID" ]; then + echo " โœ— CODEGEN_LLM_MODEL_ID not set" + exit 1 +else + echo " โœ“ Model: $CODEGEN_LLM_MODEL_ID" +fi + +echo "" +echo "3. Validating compose.yaml syntax..." +if command -v python3 &> /dev/null; then + if python3 -c "import yaml; yaml.safe_load(open('compose.yaml'))" 2>&1; then + echo " โœ“ compose.yaml syntax is valid" + else + echo " โœ— compose.yaml has syntax errors" + exit 1 + fi +else + echo " โš  Python3 not available, skipping YAML validation" +fi + +echo "" +echo "4. Configuration summary:" +echo " Services defined in compose.yaml:" +if command -v python3 &> /dev/null; then + python3 -c " +import yaml +with open('compose.yaml') as f: + config = yaml.safe_load(f) + for service in config.get('services', {}).keys(): + print(f' - {service}') +" +fi + +echo "" +echo " Port mappings:" +echo " - vLLM Service: $CODEGEN_VLLM_SERVICE_PORT -> 80" +echo " - LLM Service: $CODEGEN_LLM_SERVICE_PORT -> 9000" +echo " - Backend Service: $CODEGEN_BACKEND_SERVICE_PORT -> 7778" +echo " - UI Service: $CODEGEN_UI_SERVICE_PORT -> 5173" + +echo "" +echo " Environment endpoints:" +echo " - vLLM Endpoint: $CODEGEN_VLLM_ENDPOINT" +echo " - Backend URL: $CODEGEN_BACKEND_SERVICE_URL" + +echo "" +echo " Docker images to be used:" +echo " - vLLM: intel/vllm:0.14.1-xpu" +echo " - LLM Server: ${REGISTRY:-opea}/llm-textgen:${TAG:-latest}" +echo " - Backend: ${REGISTRY:-opea}/codegen:${TAG:-latest}" +echo " - UI: ${REGISTRY:-opea}/codegen-ui:${TAG:-latest}" + +echo "" +echo " Model configuration:" +echo " - Model ID: $CODEGEN_LLM_MODEL_ID" +echo " - Model Cache: $MODEL_CACHE" + +echo "" +echo "5. XPU-specific settings:" +echo " - VLLM_TARGET_DEVICE: xpu" +echo " - ZE_FLAT_DEVICE_HIERARCHY: FLAT" +echo " - ONEAPI_DEVICE_SELECTOR: level_zero:gpu;opencl:gpu" +echo " - Device mount: /dev/dri:/dev/dri" +echo " - GPU access: group_add (video: ${VIDEOGROUPID:-44}, render: ${RENDERGROUPID:-109})" +echo " - Shared memory: 10g" + +echo "" +echo "========================================" +echo "โœ“ Configuration validation passed!" +echo "========================================" +echo "" +echo "Next steps:" +echo "1. Install Docker Compose if not already installed:" +echo " sudo apt-get update && sudo apt-get install docker-compose-plugin" +echo "" +echo "2. Ensure you have access to Intel GPU:" +echo " sudo usermod -aG video,render \$USER" +echo " (logout and login again)" +echo "" +echo "3. Deploy the services:" +echo " docker compose up -d" +echo "" +echo "4. Monitor deployment:" +echo " docker compose logs -f" +echo "" +echo "5. Test the deployment:" +echo " curl http://\${HOST_IP}:8028/health" +echo ""