Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
145 commits
Select commit Hold shift + click to select a range
8ba3dc1
refactor: etl handlers
badalprasadsingh Mar 2, 2026
9b33fab
initial commit
badalprasadsingh Mar 2, 2026
2283d86
lint fix
badalprasadsingh Mar 2, 2026
2a8b712
major backend APIs for fusion
badalprasadsingh Mar 5, 2026
e1d9715
refactor: domain-driver level
badalprasadsingh Mar 5, 2026
83b0118
chore: move config management away from beego
vishalm0509 Mar 6, 2026
1dce1de
chore: setup gin
vishalm0509 Mar 6, 2026
62c8916
chore: migrate server and handlers to gin
vishalm0509 Mar 6, 2026
e359dff
major refactoring
badalprasadsingh Mar 9, 2026
90fac51
feat: migrate beego-orm to gorm
vishalm0509 Mar 9, 2026
ddbdf10
fix: proper error code
vishalm0509 Mar 9, 2026
a6a2342
chore: version upgrade
vishalm0509 Mar 9, 2026
978a175
chore: beego cleanup
vishalm0509 Mar 9, 2026
176cf6b
chore: services and add cancel api
badalprasadsingh Mar 9, 2026
b86178e
chore: update CI to read go-version from mod
vishalm0509 Mar 9, 2026
3027ebc
fix: install golangci-lint using goinstall instead of binary
vishalm0509 Mar 9, 2026
4c2ba45
chore: final cleanup
vishalm0509 Mar 9, 2026
bfa257d
chore: fix lint
vishalm0509 Mar 9, 2026
95e2eea
fix: cron
badalprasadsingh Mar 10, 2026
8dcc0d5
add go.sum
badalprasadsingh Mar 10, 2026
cad2a9d
chore: revert compose changes
vishalm0509 Mar 10, 2026
933a2e9
minor lint fixes
badalprasadsingh Mar 10, 2026
29c8085
fix: compaction.go
badalprasadsingh Mar 12, 2026
8eb739b
minor lint
badalprasadsingh Mar 12, 2026
9dbdece
minor refactoring fix
badalprasadsingh Mar 12, 2026
da1a672
minor lint
badalprasadsingh Mar 12, 2026
0b6ca5d
Merge branch 'refactor/etl-handlers' into feat/fusion
badalprasadsingh Mar 12, 2026
7155561
chore: resolve comments
vishalm0509 Mar 12, 2026
244badd
chore: resolve comments
vishalm0509 Mar 12, 2026
c4e09ff
fix: update swagger annotations to match new response
vishalm0509 Mar 12, 2026
799f6e4
chore: migrate from app.conf to app.yaml
vishalm0509 Mar 13, 2026
49d9241
chore: update database type from text -> varchar(255)
vishalm0509 Mar 13, 2026
436399b
Merge branch 'staging' of https://github.com/datazip-inc/olake-ui int…
vishalm0509 Mar 16, 2026
0ed646c
chore: remove app.conf
vishalm0509 Mar 16, 2026
0709736
chore: update app.yaml
vishalm0509 Mar 16, 2026
a8fc6f3
add compaction enable
badalprasadsingh Mar 20, 2026
cbcea8c
fix: minor
badalprasadsingh Mar 20, 2026
f1e3b24
sort databases and catalogs
badalprasadsingh Mar 20, 2026
7fe1481
minor
badalprasadsingh Mar 21, 2026
8002be4
fix: minor
badalprasadsingh Mar 21, 2026
007f2c0
removed unceessary dependencies
badalprasadsingh Mar 21, 2026
a5c82d0
minor
badalprasadsingh Mar 21, 2026
11838f0
minor
badalprasadsingh Mar 21, 2026
22f95bf
fix: minor
badalprasadsingh Mar 21, 2026
5bee882
major refactoring
badalprasadsingh Mar 24, 2026
51434d0
fix: remove OptimizerGroup from constants
badalprasadsingh Mar 24, 2026
5c404e0
minor
badalprasadsingh Mar 25, 2026
8269fa8
Merge branch 'staging' into refactor/etl-handlers
badalprasadsingh Mar 25, 2026
d4e20a7
Merge branch 'refactor/etl-handlers' into feat/fusion
badalprasadsingh Mar 25, 2026
5a0b3c3
minor
badalprasadsingh Mar 25, 2026
09c9aaa
minor
badalprasadsingh Mar 25, 2026
9cd0290
minor
badalprasadsingh Mar 25, 2026
ce774c8
minor
badalprasadsingh Mar 25, 2026
0f2950e
optimisation -> optimization
badalprasadsingh Mar 26, 2026
2cdb9af
optimisation -> optimization
badalprasadsingh Mar 26, 2026
9d6a2a4
minor
badalprasadsingh Mar 27, 2026
1da9a6a
refresh token logic
badalprasadsingh Mar 27, 2026
e46ed36
update piggy backing to add sep download logic
badalprasadsingh Mar 27, 2026
a755b77
review changes
badalprasadsingh Mar 29, 2026
9e7b0b3
minor change
badalprasadsingh Mar 29, 2026
d46e7e1
minor change
badalprasadsingh Mar 29, 2026
9a185c9
minor change
badalprasadsingh Mar 29, 2026
c5120db
minor change
badalprasadsingh Mar 30, 2026
74cc892
minor change
badalprasadsingh Mar 30, 2026
0515b7b
fix: merge conflict changes
vishalm0509 Mar 30, 2026
0fe82a7
minor change
badalprasadsingh Mar 30, 2026
a2bfb45
chore: cleanup appconfig
vishalm0509 Mar 30, 2026
8c4260b
Merge branch 'staging' of https://github.com/datazip-inc/olake-ui int…
vishalm0509 Mar 30, 2026
f194ca4
minor todi
badalprasadsingh Mar 30, 2026
bcc3b91
chore: tried to resolve conflicts
vishalm0509 Mar 30, 2026
e923ab5
chore: cleanup
vishalm0509 Mar 30, 2026
2b7369c
chore: cleanup and generate swagger docs
vishalm0509 Mar 31, 2026
d5e64a6
Merge branch 'feat/fusion' of https://github.com/datazip-inc/olake-ui…
vishalm0509 Mar 31, 2026
0aa1cfe
adding todo for camel case
badalprasadsingh Mar 31, 2026
ca4c50e
adding todo for camel or snake case
badalprasadsingh Mar 31, 2026
deb3d6d
fix: minor
badalprasadsingh Mar 31, 2026
b3cd859
chore: add httpx and fix required-variables check
vishalm0509 Mar 31, 2026
3a4f9d5
chore: minor cleanup
vishalm0509 Mar 31, 2026
5503069
fix: check auth session reading
vishalm0509 Mar 31, 2026
57635f9
chore: cleanup sessionStore
vishalm0509 Mar 31, 2026
79d7a26
chore: modify appconfig
vishalm0509 Mar 31, 2026
8826660
apply insert filter on all apis
badalprasadsingh Mar 31, 2026
96f1b20
chore: remove viper constants
vishalm0509 Mar 31, 2026
ad90d9d
Merge branch 'feat/fusion' of https://github.com/datazip-inc/olake-ui…
vishalm0509 Mar 31, 2026
ed020d5
chore: revert golang-ci changes
vishalm0509 Mar 31, 2026
f76d266
chore: cleanup
vishalm0509 Apr 1, 2026
d4c1e7d
import feature
badalprasadsingh Apr 1, 2026
2047146
minor
badalprasadsingh Apr 1, 2026
f09a5e5
feat(optimization): import destination as catalog in opt (#343)
badalprasadsingh Apr 1, 2026
702255c
review comments
badalprasadsingh Apr 1, 2026
81ea927
fix: minor
badalprasadsingh Apr 1, 2026
ba4d57c
Merge branch 'staging' into refactor/etl-handlers
badalprasadsingh Apr 1, 2026
5c38c02
fix: lint
badalprasadsingh Apr 1, 2026
954ccfc
fix: lint
badalprasadsingh Apr 1, 2026
549f662
Merge branch 'feat/fusion' into feat/fusion-dev
badalprasadsingh Apr 1, 2026
5326f6b
Merge branch 'staging' into feat/fusion-dev
badalprasadsingh Apr 1, 2026
45e0217
fix: minor lint issues
badalprasadsingh Apr 1, 2026
32c9611
Merge branch 'refactor/etl-handlers' into feat/fusion
badalprasadsingh Apr 1, 2026
526f488
fix: minor
badalprasadsingh Apr 1, 2026
2c6816c
fix: minor
badalprasadsingh Apr 1, 2026
8168e49
fix: minor lint issues
badalprasadsingh Apr 2, 2026
87cba6f
fix: minor
badalprasadsingh Apr 2, 2026
5bba60f
merge dev branch
badalprasadsingh Apr 2, 2026
000a7a3
minor
badalprasadsingh Apr 2, 2026
cd3e854
Merge branch 'feat/fusion-dev' into feat/fusion
badalprasadsingh Apr 2, 2026
37dc85d
Merge branch 'feat/fusion' of https://github.com/datazip-inc/olake-ui…
vishalm0509 Apr 2, 2026
4b6d2b7
fix: catalogrequest type property name - OptimizeTableFormatList
vishalm0509 Apr 2, 2026
ac2c407
chore: move utils from /server/utils to /server/internal/utils
vishalm0509 Apr 2, 2026
5b82e30
chore: move from httpx to utils
vishalm0509 Apr 2, 2026
c07baa5
review changes + lock
badalprasadsingh Apr 6, 2026
986755e
Merge branch 'staging' into feat/fusion-dev
badalprasadsingh Apr 6, 2026
269356c
fix: lint
badalprasadsingh Apr 6, 2026
0e56be5
Merge branch 'feat/fusion-dev' into feat/fusion
badalprasadsingh Apr 6, 2026
82d389d
Merge branch 'staging' into refactor/etl-handlers
badalprasadsingh Apr 6, 2026
03f9755
Merge branch 'refactor/etl-handlers' into feat/fusion
badalprasadsingh Apr 6, 2026
d6bdb9e
fix: minor
badalprasadsingh Apr 6, 2026
e0a8def
Merge branch 'feat/fusion' of https://github.com/datazip-inc/olake-ui…
vishalm0509 Apr 6, 2026
2263ec1
fix: utils import error
vishalm0509 Apr 6, 2026
c45f3b5
feat: fusion (#333)
badalprasadsingh Apr 6, 2026
42d27b1
Merge branch 'refactor/etl-handlers' of https://github.com/datazip-in…
vishalm0509 Apr 7, 2026
6cb8244
fix: merge errors
vishalm0509 Apr 7, 2026
ea45630
chore: minor changes
vishalm0509 Apr 7, 2026
1be6085
fix: use no-group to fix route conflicts in router.go
vishalm0509 Apr 8, 2026
6cd63f1
Merge branch 'staging' of https://github.com/datazip-inc/olake-ui int…
vishalm0509 Apr 8, 2026
49f5837
chore: handle no-route handler
vishalm0509 Apr 8, 2026
bd75bf6
chore: self review changes
vishalm0509 Apr 8, 2026
64355d0
Merge branch 'staging' into feat/gin
vishalm0509 Apr 8, 2026
234e44f
fix: ci lint error
vishalm0509 Apr 8, 2026
54db03f
fix: ci errors
vishalm0509 Apr 9, 2026
ae95d1a
Merge branch 'staging' of https://github.com/datazip-inc/olake-ui int…
vishalm0509 Apr 9, 2026
a90b1eb
fix: optimization status endpoint
vishalm0509 Apr 9, 2026
6f8c311
Merge branch 'staging' into feat/gin
vishalm0509 Apr 10, 2026
068ac7b
chore: remove copyrequestbody
vishalm0509 Apr 13, 2026
8119ead
chore: resolve comments
vishalm0509 Apr 13, 2026
216f227
chore: remove dead code
vishalm0509 Apr 13, 2026
4918861
Merge branch 'staging' of https://github.com/datazip-inc/olake-ui int…
vishalm0509 Apr 14, 2026
c2707c3
chore: remove unused method
vishalm0509 Apr 14, 2026
2b79fdf
chore: add middleware to root api group
vishalm0509 Apr 14, 2026
4d9d1ba
chore: add session cleanup
vishalm0509 Apr 14, 2026
648da1a
chore: cleanup NewHandler
vishalm0509 Apr 14, 2026
bb2ba7f
chore: refactor no_route_handler and update functions to methods (ses…
vishalm0509 Apr 14, 2026
7968dd6
chore: update zero values as well
vishalm0509 Apr 15, 2026
6aa0db1
chore: add TODO for session_on
vishalm0509 Apr 16, 2026
a777928
fix: release CI to support dev-image release
vishalm0509 Apr 16, 2026
3bbbe06
chore: update config path from conf -> config in the docker bundle
vishalm0509 Apr 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 37 additions & 2 deletions .github/workflows/build-and-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ on:
required: true
default: ""
type: string
ref:
description: "Git ref (branch/tag/SHA) to build from. If empty, derived from environment mapping."
required: false
default: ""
type: string
version:
description: "Version to release"
required: true
Expand All @@ -23,6 +28,11 @@ on:
- master
- staging
- dev
ref:
description: "Git ref (branch/tag/SHA) to build from (optional). If empty, derived from environment mapping."
required: false
default: ""
type: string
version:
description: "Version to release"
required: true
Expand All @@ -43,10 +53,35 @@ jobs:
VERSION: ${{ inputs.version }}

steps:
- name: Checkout code
- name: Checkout code from explicit ref
if: ${{ inputs.ref != '' }}
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref }}

- name: Checkout code for master
if: ${{ inputs.ref == '' && inputs.environment == 'master' }}
uses: actions/checkout@v4
with:
ref: master

- name: Checkout code for staging
if: ${{ inputs.ref == '' && inputs.environment == 'staging' }}
uses: actions/checkout@v4
with:
ref: staging

- name: Checkout code for dev
if: ${{ inputs.ref == '' && inputs.environment == 'dev' }}
uses: actions/checkout@v4
with:
ref: ci/workerReleaseIssues

- name: Checkout code for other environments
if: ${{ inputs.ref == '' && inputs.environment != 'master' && inputs.environment != 'staging' && inputs.environment != 'dev' }}
uses: actions/checkout@v4
with:
ref: ${{ inputs.environment == 'master' && 'master' || (inputs.environment == 'staging' && 'staging' || inputs.environment == 'dev' && 'ci/workerReleaseIssues' || 'develop') }}
ref: staging

- name: Set up Node.js
uses: actions/setup-node@v4
Expand Down
4 changes: 0 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,10 +91,6 @@ You can also change the local PostgreSQL connection or any other settings based
- **Go** ≥ 1.20
- **Node.js** and **pnpm**
- **Docker** and **Docker Compose**
- **Beego CLI**:
```bash
go install github.com/beego/bee/v2@latest
```

### Start Services
1. **Temporal Services**
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -53,11 +53,11 @@ RUN apk update && apk add --no-cache docker-cli
WORKDIR /app/olake-ui

# Create directories for applications
RUN mkdir -p conf /opt/frontend/dist
RUN mkdir -p conf config /opt/frontend/dist

# Copy built artifacts from builder stages
COPY --from=go-builder /app/olake-server ./olake-server
COPY server/conf/app.conf ./conf/app.conf
COPY server/conf/app.yaml ./config/app.yaml
COPY --from=node-builder /app/ui/dist /opt/frontend/dist

# Expose the Go backend port (which serves both API and frontend)
Expand Down
3 changes: 1 addition & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,6 @@ BACKEND_ENV_VARS = \
APP_NAME=olake-server \
HTTP_PORT=8000 \
RUN_MODE=localdev \
COPY_REQUEST_BODY=true \
OLAKE_POSTGRES_USER=temporal \
OLAKE_POSTGRES_PASSWORD=temporal \
OLAKE_POSTGRES_HOST=localhost \
Expand All @@ -75,7 +74,7 @@ start-frontend:
# Start backend server with env vars
start-backend:
cd $(SERVER_DIR) && \
$(BACKEND_ENV_VARS) bee run
$(BACKEND_ENV_VARS) go run .


# Start Temporal services using Docker Compose
Expand Down
7 changes: 3 additions & 4 deletions server/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Olake Server

Olake Server is a RESTful API service built with the Beego framework that manages data sources, destinations, jobs, and users for data ingestion workflows.
Olake Server is a RESTful API service built with Gin that manages data sources, destinations, jobs, and users for data ingestion workflows.

## Prerequisites

Expand All @@ -24,7 +24,6 @@ Review and update the configuration in `conf/app.conf` as needed:
appname = olake-server
httpport = 8000
runmode = dev
copyrequestbody = true
postgresdb = postgres://postgres:testing@testing-postgres.postgres.database.azure.com:5432/olakedb

# Session configuration
Expand All @@ -45,7 +44,7 @@ CREATE TABLE session (

```bash
go mod tidy
bee run
go run .
# or
make run
```
Expand Down Expand Up @@ -119,5 +118,5 @@ For production deployment:

1. Set `runmode = prod` in `conf/app.conf`
2. Configure a secure database connection
3. Set up proper CORS settings in `routers/router.go`
3. Set up proper CORS settings in `internal/httpserver/server.go`
4. Enable authentication middleware
21 changes: 0 additions & 21 deletions server/conf/app.conf

This file was deleted.

30 changes: 30 additions & 0 deletions server/conf/app.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
APP_NAME: olake-server
HTTP_PORT: "8000"
RUN_MODE: dev

MAX_MEMORY: 67108864
MAX_UPLOAD_SIZE: 67108864

# Either provide a full DSN here...
POSTGRES_DB: ""
# ...or configure via individual parts below.
OLAKE_POSTGRES_USER: temporal
OLAKE_POSTGRES_PASSWORD: temporal
OLAKE_POSTGRES_HOST: postgresql
OLAKE_POSTGRES_PORT: "5432"
OLAKE_POSTGRES_DBNAME: postgres
OLAKE_POSTGRES_SSLMODE: disable

LOGS_DIR: ./logger/logs
SESSION_ON: true

TEMPORAL_ADDRESS: temporal:7233
CONTAINER_REGISTRY_BASE: registry-1.docker.io

# Prefer setting this via environment.
OLAKE_SECRET_KEY: ""

# Optimization module configuration
ENABLE_OPTIMIZATION: false
OPTIMIZATION_BASE_URL: http://127.0.0.1:1630
OPTIMIZATION_GROUP: spark-container
Loading
Loading