Skip to content

Enable CodeGen deployment for Intel Arc Pro B-series GPU (XPU) #2467

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tintisimone wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Enable CodeGen deployment for Intel Arc Pro B-series GPU (XPU) #2467

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 23 additions & 21 deletions CodeGen/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,18 +106,19 @@ flowchart LR

This CodeGen example can be deployed manually on various hardware platforms using Docker Compose or Kubernetes. Select the appropriate guide based on your target environment:

| Hardware | Deployment Mode | Guide Link |
| :-------------- | :----------------------------------- | :--------------------------------------------------------------------------------------- |
| Intel Xeon CPU | Single Node (Docker) | [Xeon Docker Compose Guide](./docker_compose/intel/cpu/xeon/README.md) |
| Intel Xeon CPU | Single Node (Docker) with Monitoring | [Xeon Docker Compose with Monitoring Guide](./docker_compose/intel/cpu/xeon/README.md) |
| Intel Gaudi HPU | Single Node (Docker) | [Gaudi Docker Compose Guide](./docker_compose/intel/hpu/gaudi/README.md) |
| Intel Gaudi HPU | Single Node (Docker) with Monitoring | [Gaudi Docker Compose with Monitoring Guide](./docker_compose/intel/hpu/gaudi/README.md) |
| AMD EPYC CPU | Single Node (Docker) | [EPYC Docker Compose Guide](./docker_compose/amd/cpu/epyc/README.md) |
| AMD ROCm GPU | Single Node (Docker) | [ROCm Docker Compose Guide](./docker_compose/amd/gpu/rocm/README.md) |
| Intel Xeon CPU | Kubernetes (Helm) | [Kubernetes Helm Guide](./kubernetes/helm/README.md) |
| Intel Gaudi HPU | Kubernetes (Helm) | [Kubernetes Helm Guide](./kubernetes/helm/README.md) |
| Intel Xeon CPU | Kubernetes (GMC) | [Kubernetes GMC Guide](./kubernetes/gmc/README.md) |
| Intel Gaudi HPU | Kubernetes (GMC) | [Kubernetes GMC Guide](./kubernetes/gmc/README.md) |
| Hardware | Deployment Mode | Guide Link |
| :------------------ | :----------------------------------- | :--------------------------------------------------------------------------------------- |
| Intel Xeon CPU | Single Node (Docker) | [Xeon Docker Compose Guide](./docker_compose/intel/cpu/xeon/README.md) |
| Intel Xeon CPU | Single Node (Docker) with Monitoring | [Xeon Docker Compose with Monitoring Guide](./docker_compose/intel/cpu/xeon/README.md) |
| Intel Gaudi HPU | Single Node (Docker) | [Gaudi Docker Compose Guide](./docker_compose/intel/hpu/gaudi/README.md) |
| Intel Gaudi HPU | Single Node (Docker) with Monitoring | [Gaudi Docker Compose with Monitoring Guide](./docker_compose/intel/hpu/gaudi/README.md) |
| Intel Arc GPU (XPU) | Single Node (Docker) | [Arc XPU Docker Compose Guide](./docker_compose/intel/xpu/arc/README.md) |
| AMD EPYC CPU | Single Node (Docker) | [EPYC Docker Compose Guide](./docker_compose/amd/cpu/epyc/README.md) |
| AMD ROCm GPU | Single Node (Docker) | [ROCm Docker Compose Guide](./docker_compose/amd/gpu/rocm/README.md) |
| Intel Xeon CPU | Kubernetes (Helm) | [Kubernetes Helm Guide](./kubernetes/helm/README.md) |
| Intel Gaudi HPU | Kubernetes (Helm) | [Kubernetes Helm Guide](./kubernetes/helm/README.md) |
| Intel Xeon CPU | Kubernetes (GMC) | [Kubernetes GMC Guide](./kubernetes/gmc/README.md) |
| Intel Gaudi HPU | Kubernetes (GMC) | [Kubernetes GMC Guide](./kubernetes/gmc/README.md) |

_Note: Building custom microservice images can be done using the resources in [GenAIComps](https://github.com/opea-project/GenAIComps)._

Expand Down Expand Up @@ -180,15 +181,16 @@ Intel® Optimized Cloud Modules for Terraform provide an automated way to deploy

## Validated Configurations

| **Deploy Method** | **LLM Engine** | **LLM Model** | **Hardware** |
| ----------------- | -------------- | ------------------------------ | ------------ |
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Gaudi |
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Xeon |
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD EPYC |
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm |
| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Gaudi |
| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Xeon |
| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm |
| **Deploy Method** | **LLM Engine** | **LLM Model** | **Hardware** |
| ----------------- | -------------- | ------------------------------ | --------------- |
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Gaudi |
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Xeon |
| Docker Compose | vLLM | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Arc (XPU) |
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD EPYC |
| Docker Compose | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm |
| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Gaudi |
| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | Intel Xeon |
| Helm Charts | vLLM, TGI | Qwen/Qwen2.5-Coder-7B-Instruct | AMD ROCm |

## Contribution

Expand Down
1 change: 1 addition & 0 deletions CodeGen/docker_compose/intel/xpu/arc/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
data/
196 changes: 196 additions & 0 deletions CodeGen/docker_compose/intel/xpu/arc/QUICK_START.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
# CodeGen on Intel Arc XPU - Quick Start Guide

## 🚀 Quick Deployment (3 Steps)

### Step 1: Setup Environment (1 minute)

```bash
cd GenAIExamples/CodeGen/docker_compose/intel/xpu/arc

# Set your host IP and HuggingFace token
export HOST_IP=$(hostname -I | awk '{print $1}')
export HF_TOKEN="your_huggingface_token"

# Optional: Configure proxy if needed
export no_proxy="localhost,127.0.0.1,${HOST_IP}"
export NO_PROXY="localhost,127.0.0.1,${HOST_IP}"

source ./set_env.sh
```

### Step 2: Deploy Services (5-10 minutes)

```bash
docker compose up -d
```

### Step 3: Wait for Model to Load (3-5 minutes)

```bash
docker compose logs -f codegen-vllm-service
```

Wait for: `Application startup complete`

---

## 🧪 Quick Test

### Test 1: Health Check

```bash
curl http://${HOST_IP}:8028/health
```

Expected: `{"status":"ok"}`

### Test 2: Code Generation

```bash
curl http://${HOST_IP}:8028/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen2.5-Coder-7B-Instruct",
"prompt": "def fibonacci(n):",
"max_tokens": 100,
"temperature": 0.7
}'
```

### Test 3: Access UI

Open browser: http://${HOST_IP}:5173

---

## 📊 Service Ports

| Service | Port | URL |
| ----------- | ---- | ---------------------- |
| vLLM | 8028 | http://${HOST_IP}:8028 |
| LLM Service | 9000 | http://${HOST_IP}:9000 |
| Backend | 7778 | http://${HOST_IP}:7778 |
| UI | 5173 | http://${HOST_IP}:5173 |

---

## 🛠️ Useful Commands

### View Logs

```bash
# All services
docker compose logs -f

# Specific service
docker compose logs -f codegen-vllm-service
```

### Check Status

```bash
docker compose ps
```

### Stop Services

```bash
docker compose down
```

### Restart Services

```bash
docker compose restart
```

### Remove Everything

```bash
docker compose down -v
```

---

## 🔧 Troubleshooting

### GPU Not Detected?

```bash
ls -la /dev/dri/
sudo usermod -aG video,render $USER
# Logout and login
```

### Service Won't Start?

```bash
docker compose logs codegen-vllm-service
docker compose ps
```

### Out of Memory?

Edit `compose.yaml`:

```yaml
shm_size: 16g # Increase from 10g
```

---

## 📝 Configuration

### Change Model

Edit `set_env.sh`:

```bash
export CODEGEN_LLM_MODEL_ID="your-model-id"
```

### Change Ports

Edit `set_env.sh`:

```bash
export CODEGEN_VLLM_SERVICE_PORT=8029
export CODEGEN_UI_SERVICE_PORT=5174
```

---

## ✅ Validation Checklist

- [ ] Docker daemon running
- [ ] Intel GPU detected at `/dev/dri`
- [ ] Environment variables set
- [ ] Services deployed
- [ ] Health endpoint responds
- [ ] Code generation works
- [ ] UI accessible

---

## 📚 More Information

- Full documentation: [README.md](./README.md)
- Main CodeGen docs: [../../README.md](../../../README.md)

---

## 🎯 Expected Timeline

| Phase | Duration | Status |
| ----------------- | ------------- | ------ |
| Environment setup | 1 min | ✅ |
| Pull images | 10-15 min | ⏳ |
| Start services | 2 min | ⏳ |
| Model loading | 3-5 min | ⏳ |
| **Total** | **15-20 min** | |

---

**Hardware**: Intel Arc Pro B-series GPU
**Model**: Qwen/Qwen2.5-Coder-7B-Instruct
**Backend**: Intel vLLM 0.14.1-xpu
Loading
Loading