feat: migrate backend from beego to gin#335
Conversation
Signed-off-by: badalprasadsingh <badal@datazip.io>
Signed-off-by: badalprasadsingh <badal@datazip.io>
Signed-off-by: badalprasadsingh <badal@datazip.io>
Signed-off-by: badalprasadsingh <badal@datazip.io>
Signed-off-by: badalprasadsingh <badal@datazip.io>
Signed-off-by: badalprasadsingh <badal@datazip.io>
Signed-off-by: badalprasadsingh <badal@datazip.io>
Signed-off-by: badalprasadsingh <badal@datazip.io>
| if err != nil || value == "" { | ||
| panic("Required config variable not found: ," + v) | ||
| func checkForRequiredVariables() { | ||
| cfg := appconfig.Load() |
There was a problem hiding this comment.
Init() calls appconfig.Load() at line 117 and checkForRequiredVariables() also calls here, pass cfg in the function
There was a problem hiding this comment.
Config is loaded once at startup and then every time we call appconfig.Load() it returns the already loaded config. Therefore, no re-computation of all the envs.
// this loads first on startup
var cfg = loadConfig()
// this serves the already loaded cfg
func Load() Config {
return cfg
}| gormlogger "gorm.io/gorm/logger" | ||
|
|
||
| "github.com/datazip-inc/olake-ui/server/internal/constants" | ||
| "github.com/datazip-inc/olake-ui/server/internal/appconfig" |
There was a problem hiding this comment.
how can we avoid import of appconfig in multiple packages
There was a problem hiding this comment.
Config is loaded once at startup and then every time we call appconfig.Load() it returns the already loaded config. Therefore, no re-computation of all the envs.
// this loads first on startup
var cfg = loadConfig()
// this serves the already loaded cfg
func Load() Config {
return cfg
}It is similar to that of web.AppConfig.String(constants.ConfOLakePostgresUser) we import web everywhere. appconfig is similar to that
There was a problem hiding this comment.
can we think of any alternate to avoid this now
There was a problem hiding this comment.
We have decided to use this now
| ) | ||
|
|
||
| // ValidateStruct validates any struct that has `validate` tags. | ||
| func Validate(s interface{}) error { |
There was a problem hiding this comment.
don't we require this anymore?
There was a problem hiding this comment.
No, here c.ShouldBindJSON() handles bindings.
func bindAndValidate(c *gin.Context, target interface{}) error {
return c.ShouldBindJSON(target)
}We have replaced Username string json:"username" validate:"required" example:"admin" (validate) with this (binding) which does the same work.
| SessionData string `json:"session_data" orm:"column(session_data);type(text)"` | ||
| SessionExpiry time.Time `json:"session_expiry" orm:"column(session_expiry)"` | ||
| SessionKey string `json:"session_key" gorm:"column:session_key;primaryKey;size:64"` | ||
| SessionData []byte `json:"session_data" gorm:"column:session_data;type:bytea"` |
There was a problem hiding this comment.
why? Session.SessionData TEXT -> BYTEA
There was a problem hiding this comment.
In the earlier code beego also wrote data to postgres in []byte
// Add session table if sessions are enabled
if web.BConfig.WebConfig.Session.SessionOn {
_, err = orm.NewOrm().Raw(`CREATE TABLE IF NOT EXISTS session (
session_key VARCHAR(64) PRIMARY KEY,
session_data BYTEA,
session_expiry TIMESTAMP WITH TIME ZONE
);`).Exec()
| BaseModel | ||
| ID int `json:"id" gorm:"column:id;primaryKey;autoIncrement"` | ||
| Name string `json:"name" gorm:"column:name;size:100"` | ||
| SourceID int `json:"source_id" gorm:"column:source_id"` |
There was a problem hiding this comment.
won't it cause any issue changing *Source -> int
There was a problem hiding this comment.
This is FK reference ID, we still can access the Source object with preLoad
Source *Source `json:"source,omitempty" gorm:"foreignKey:SourceID;references:ID"`
Destination *Destination `json:"destination,omitempty" gorm:"foreignKey:DestID;references:ID"`
CreatedBy *User `json:"created_by,omitempty" gorm:"foreignKey:CreatedByID;references:ID"`
UpdatedBy *User `json:"updated_by,omitempty" gorm:"foreignKey:UpdatedByID;references:ID"`| // @Router /signup [post] | ||
| func (h *Handler) Signup() { | ||
| func (h *Handler) Signup(c *gin.Context) { | ||
| var req models.User |
There was a problem hiding this comment.
can we create dto's for these as well?
| return | ||
| } | ||
| utils.SuccessResponse(&h.Controller, "Destinations listed successfully", items) | ||
| successResponse(c, "Destinations listed successfully", items) |
There was a problem hiding this comment.
| successResponse(c, "Destinations listed successfully", items) | |
| successResponse(c, "destinations listed successfully", items) |
| return | ||
| } | ||
|
|
||
| if req.Source.ID == nil { | ||
| if err := dto.ValidateSourceType(req.Source.Type); err != nil { |
There was a problem hiding this comment.
don't we require source type validation anymore?
| utils.ErrorResponse(&h.Controller, http.StatusBadRequest, fmt.Sprintf("failed to validate request: %s", err), err) | ||
| return | ||
| } | ||
| if req.Source.Name == "" || req.Source.Version == "" || req.Source.Config == "" { |
There was a problem hiding this comment.
don't we require this check anymore?
| h.Ctx.Output.Header("Access-Control-Expose-Headers", "Content-Disposition") | ||
|
|
||
| if err := h.etl.StreamLogArchive(id, filePath, h.Ctx.ResponseWriter); err != nil { | ||
| logger.Errorf("failed to stream log archive job_id[%d]: %s", id, err) |
Signed-off-by: badalprasadsingh <badal@datazip.io>
Signed-off-by: badalprasadsingh <badal@datazip.io>
Signed-off-by: badalprasadsingh <badal@datazip.io>
Signed-off-by: badalprasadsingh <badal@datazip.io>
* initial commit Signed-off-by: badalprasadsingh <badal@datazip.io> * lint fix Signed-off-by: badalprasadsingh <badal@datazip.io> * major backend APIs for fusion Signed-off-by: badalprasadsingh <badal@datazip.io> * refactor: domain-driver level Signed-off-by: badalprasadsingh <badal@datazip.io> * major refactoring Signed-off-by: badalprasadsingh <badal@datazip.io> * chore: services and add cancel api Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: cron Signed-off-by: badalprasadsingh <badal@datazip.io> * add go.sum Signed-off-by: badalprasadsingh <badal@datazip.io> * minor lint fixes Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: compaction.go Signed-off-by: badalprasadsingh <badal@datazip.io> * minor lint Signed-off-by: badalprasadsingh <badal@datazip.io> * add compaction enable Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor Signed-off-by: badalprasadsingh <badal@datazip.io> * sort databases and catalogs Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor Signed-off-by: badalprasadsingh <badal@datazip.io> * removed unceessary dependencies Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor Signed-off-by: badalprasadsingh <badal@datazip.io> * major refactoring Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: remove OptimizerGroup from constants Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * optimisation -> optimization Signed-off-by: badalprasadsingh <badal@datazip.io> * optimisation -> optimization Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * refresh token logic Signed-off-by: badalprasadsingh <badal@datazip.io> * update piggy backing to add sep download logic Signed-off-by: badalprasadsingh <badal@datazip.io> * review changes Signed-off-by: badalprasadsingh <badal@datazip.io> * minor change Signed-off-by: badalprasadsingh <badal@datazip.io> * minor change Signed-off-by: badalprasadsingh <badal@datazip.io> * minor change Signed-off-by: badalprasadsingh <badal@datazip.io> * minor change Signed-off-by: badalprasadsingh <badal@datazip.io> * minor change Signed-off-by: badalprasadsingh <badal@datazip.io> * minor change Signed-off-by: badalprasadsingh <badal@datazip.io> * minor todi Signed-off-by: badalprasadsingh <badal@datazip.io> * adding todo for camel case Signed-off-by: badalprasadsingh <badal@datazip.io> * adding todo for camel or snake case Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor Signed-off-by: badalprasadsingh <badal@datazip.io> * apply insert filter on all apis Signed-off-by: badalprasadsingh <badal@datazip.io> * chore: cleanup * import feature Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * feat(optimization): import destination as catalog in opt (#343) * import feature Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> --------- Signed-off-by: badalprasadsingh <badal@datazip.io> Co-authored-by: Ankit Sharma <111491139+hash-data@users.noreply.github.com> * review comments Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: lint Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: lint Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor lint issues * import feature Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * Merge pull request #346 from datazip-inc/fix/trivy_docker fix: suppress CVE-2026-34040 in Trivy until testcontainers-go upgrade * review comments Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: lint Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: lint Signed-off-by: badalprasadsingh <badal@datazip.io> --------- Signed-off-by: badalprasadsingh <badal@datazip.io> Co-authored-by: vishal-datazip <vishal@datazip.io> Co-authored-by: Ankit Sharma <111491139+hash-data@users.noreply.github.com> * fix: minor Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor lint issues Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor Signed-off-by: badalprasadsingh <badal@datazip.io> * minor Signed-off-by: badalprasadsingh <badal@datazip.io> * review changes + lock Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: lint Signed-off-by: badalprasadsingh <badal@datazip.io> * fix: minor Signed-off-by: badalprasadsingh <badal@datazip.io> --------- Signed-off-by: badalprasadsingh <badal@datazip.io> Co-authored-by: vishalm0509 <vishal@datazip.io> Co-authored-by: Ankit Sharma <111491139+hash-data@users.noreply.github.com>
| // Project settings routes | ||
| web.Router("/api/v1/project/:projectid/settings", h, "put:UpsertProjectSettings") | ||
| web.Router("/api/v1/project/:projectid/settings", h, "get:GetProjectSettings") | ||
|
|
||
| // validation routes | ||
| web.Router("/api/v1/project/:projectid/check-unique", h, "post:CheckUniqueName") | ||
|
|
||
| // platform routes |
| CopyRequestBody bool | ||
| MaxMemory int64 | ||
| MaxUploadSize int64 |
There was a problem hiding this comment.
can we remove CopyRequestBody
| return projectID, nil | ||
| } | ||
|
|
||
| func getIDParam(c *gin.Context, key string) (int, error) { |
There was a problem hiding this comment.
key always will be "id" here, can we avoid passing in function
There was a problem hiding this comment.
Right, but for other modules will it still be id always ? since we are using this as a global util for all the modules
There was a problem hiding this comment.
I have made changes anyway since the name is explicitly GetIDParam but the above is something we should think about
| } | ||
|
|
||
| // GetAll retrieves all jobs | ||
| func (db *Database) ListJobs() ([]*models.Job, error) { |
| var users []*models.User | ||
| _, err := db.ormer.QueryTable(constants.TableNameMap[constants.UserTable]).All(&users) | ||
| err := db.conn.Find(&users).Error | ||
| return users, err |
There was a problem hiding this comment.
fmt.Errorf("failed to find user: %s", err)
There was a problem hiding this comment.
here, err can be nil as well. This is the design we are following in the database layer.
We are adding a proper error message in the service layer to which this is appended
| c.Status(http.StatusOK) | ||
| if err := h.etl.StreamLogArchive(id, filePath, c.Writer); err != nil { |
There was a problem hiding this comment.
what if StreamLogArchive returns an error without writing any bytes, it would return 200 OK with an empty response
There was a problem hiding this comment.
Yes, made changes
| } | ||
| } | ||
|
|
||
| func (s *Server) Engine() *gin.Engine { |
There was a problem hiding this comment.
removed, was using it earlier
| // User-related methods on AppService | ||
|
|
||
| func (s Service) CreateUser(_ context.Context, req *models.User) error { | ||
| func (s *Service) CreateUser(_ context.Context, req *models.User) error { |
| // platform routes | ||
| web.Router("/api/v1/platform/releases", etlHandler, "get:GetReleaseUpdates") | ||
| web.Router("/api/v1/platform/opt/status", h, "get:GetoptimizationStatus") | ||
| etl := engine.Group("/api/v1") |
There was a problem hiding this comment.
in beego we were applying auth for /api/* path
There was a problem hiding this comment.
// Apply auth middleware to protected routes
web.InsertFilter("/api/*", web.BeforeRouter, middleware.AuthMiddleware)Auth middlware only applied to routes with prefix /api
There was a problem hiding this comment.
yes
now it is applying to -> etl := engine.Group("/api/v1"),etl.Use(h.AuthMiddleware())
There was a problem hiding this comment.
Yes, made changes
| source.Config = eConfig | ||
| _, err = db.ormer.Update(source) | ||
| return err | ||
| return db.conn.Updates(source).Error |
There was a problem hiding this comment.
gorm.Updates() Silently Skips Zero-Values on Structs
please check following doc
https://gorm.io/docs/update.html#Updates-multiple-columns
There was a problem hiding this comment.
still we should use correct function
| v := viper.New() | ||
|
|
||
| // Note: config priority: env variables -> file (app.yaml) | ||
| v.SetConfigFile("./conf/app.yaml") |
There was a problem hiding this comment.
can it cause any issue in k8s where all configs come from env vars
There was a problem hiding this comment.
If env is set then we prioritize env over file
| return fmt.Errorf("failed to marshal session payload: %w", err) | ||
| } | ||
|
|
||
| err = s.db.UpsertSession(sessionID, payload, expiresAt) |
There was a problem hiding this comment.
Create makes sense here. Made changes
| // platform routes | ||
| web.Router("/api/v1/platform/releases", etlHandler, "get:GetReleaseUpdates") | ||
| web.Router("/api/v1/platform/opt/status", h, "get:GetoptimizationStatus") | ||
| etl := engine.Group("/api/v1") |
There was a problem hiding this comment.
yes
now it is applying to -> etl := engine.Group("/api/v1"),etl.Use(h.AuthMiddleware())
| func configureMode(cfg *appconfig.Config) { | ||
| switch cfg.RunMode { | ||
| case "localdev": | ||
| gin.SetMode(gin.DebugMode) |
There was a problem hiding this comment.
gin.SetMode() is a global setter that must be called before any gin.New() or gin.Default() call. Calling it after the engine is already created and middleware is already registered changes the global mode but has no effect on the already-constructed engine.
There was a problem hiding this comment.
It will not affect the code. But it's mentioned as good practice. Moved it to the top
| configureMode(cfg) | ||
| configureBaseRoutes(engine) | ||
|
|
||
| if cfg.RunMode == "localdev" { |
There was a problem hiding this comment.
we are checking runmode redundantly can we avoid this and check run mode at one place for all ops
There was a problem hiding this comment.
We are just reading the value from the already loaded config. All these are separate operations and cannot be grouped together.
| if h.Optimization != nil { | ||
| optHandler := h.Optimization | ||
| opt := engine.Group("/api/opt/v1") | ||
| opt.Use(h.AuthMiddleware()) |
There was a problem hiding this comment.
h.AuthMiddleware() is called twice
There was a problem hiding this comment.
It was called twice because we had 2 separate route groups for ETL and OPT. Now we are using a root API group where I have applied the middleware. Solved
|
|
||
| // piggy backing | ||
| web.Router("/api/opt/v1/*", optHandler, "get:PiggyBacking;post:PiggyBacking;put:PiggyBacking;delete:PiggyBacking") | ||
| type ModuleNoRouteHandler struct { |
There was a problem hiding this comment.
can we move the struct and function in httpserver folder
There was a problem hiding this comment.
Yes we can move it. DOne
There was a problem hiding this comment.
The noModuleConfiguration is better if it's in router.go as it's related to that. Move the actual implementation to httpserver package.
| // Register optimization as a module fallback for unmatched /api/opt/v1/*. | ||
| // This avoids route tree conflicts from wildcard catch-all registration. | ||
| moduleHandlers = append(moduleHandlers, routes.ModuleNoRouteHandler{ | ||
| PathPrefix: "/api/opt/v1/", |
There was a problem hiding this comment.
we should place all the routes together in router.go
|
|
||
| const frontendDistPath = "/opt/frontend/dist" | ||
|
|
||
| type Server struct { |
There was a problem hiding this comment.
it would be better if we create all the functions with server as receiver

Description
Fixes # (issue)
Type of change
How Has This Been Tested?
Screenshots or Recordings
Related PR's (If Any):