[Inference] Inference platform

[DanThrane]

Expand the UCloud inference platform with a new model configuration structure. Model configurations are stored in the UCloud/IM database and can be created in one of the following ways:

• Automatically discovered from the backing inference platform (LocalAI/Dynamo)
• Manually updated via the CLI
• Manually updated via the inference UI

The model configuration is defined as follows:

type InferenceModel struct {
	Name 						string
	Title 					string
	Capabilities 		[]InferenceCapability
	PriceMultiplier InferencePricing
	Endpoint 				InferenceEndpoint
	Availability		InferenceAvailability
}

type InferencePricing struct {
	CachedInput 	int
	Input 				int
	Output 				int
}

type InferenceCapability string

const (
	InferenceTextGeneration InferenceCapability = "TextGeneration"
	InferenceTextToImage 		InferenceCapability = "TextToImage"
	InferenceSpeechToText 	InferenceCapability = "SpeechToText"
)

type InferenceEndpoint struct {
	BasePath string // e.g. http://some-model-frontend.some-namespace.svc.cluster.local:8000/v1/
	BackendModelName string
}

type InferenceAvailability struct {
	Public 			bool
	AvailableTo []string // project IDs
}

Pricing

Pricing of models is being updated with this issue. This will introduce differentiated pricing based on the type of tokens along with adding a multiplier to the type of token.

The multiplier is stored in PriceMultiplier. Each multiplier is a fixed-point decimal number. A value of 1000 is equal to 1x (100%), a value of 1 is equal to 0.001x (0.1%). Token consumption will be extracted as the code already does it with the addition of differentiating between cached input tokens and non-cached input tokens. The token consumption of a request will use the following formula, using only integer math:

usageNonNormalized = 
  (cachedTokens * PriceMultiplier.CachedInput) +
  (inputTokens * PriceMultiplier.Input) +
  (outputTokens * PriceMultiplier.Output)

usage = usageNonNormalized / 1000
if usage == 0 && usageNonNormalized > 0 {
  usage = 1
}

Where:

• inputTokens = usage.prompt_tokens - usage.prompt_tokens_details.cached_tokens.
• cachedTokens = usage.prompt_tokens_details.cached_tokens
• outputTokens = usage.completion_tokens

If usage.prompt_tokens_details.cached_tokens is unavailable, then this value gets a default of 0.

Discovery

Discovery is the process of automatically finding available models in the current environment. If a new model is discovered, then the model is automatically inserted into the model catalog with the following values:

InferenceModel{
	Name: discoveredName,
	Title: discoveredName,
	Capabilities: []InferenceCapability{InferenceTextGeneration},
	PriceMultiplier: InferencePricing{
		CachedInput: 	1000,
		Input: 				1000,
		Output: 			1000,
	},
	Endpoint: discoveredEndpoint,
	Availability: InferenceAvailability{
		Public: 			false,
		AvailableTo: 	configuredDefaultInferenceTesters,
	}
}

Development mode

Extend the current development mode code to automatically bootstrap discovery of the existing LocalAI models. This will also include the existing code for automatically downloading and installing models.

Note that the capability estimation code is replaced with the capabilities listed in the model configuration.

The development mode discovery will automatically run at startup under the same circumstances as the existing LocalAI bootstrap code does.

Dynamo

Dynamo uses a configured Dynamo namespace which is used for discovering new models. The Dynamo discoverer will automatically look for all services in this namespace having the -frontend suffix. The discoverer will assume an endpoint using the following format:

http://<svcName>.<svcNamespace>.svc.cluster.local:<svcPort>/v1/

The model name is automatically extracted by querying the /v1/models endpoint and extract entries of type "model". This may result in the creation of multiple InferenceModels. The BackendModelName and discovered Name will initially match the name returned by the endpoint. The Name can later be changed by an administrator through the UI/CLI. This is a destructive action which will remove it from the global map and potentially disrupt users. It will be inserted back into the map under the new name. This action can only be done when the model has public set to false. The model.Name is the one which will be exposed and used by the model router. When delegating the response to the upstream inference provider all references to this model name will be transformed to the BackendModelName. The Dynamo discoverer will automatically determine if the BackendModelName has already been registered, that is, it will match on the BackendModelName and not the Name.

The Dynamo discoverer will automatically run at startup and periodically every 60 seconds.

Model catalog

The model catalog will primarily be stored as an in-memory map protected by a sync.RWMutex. This will be accessible as:

var modelGlobals struct {
	Mu sync.RWMutex
	Models map[string]InferenceModel
}

The modelGlobals are initially loaded from a database table using the
following schema:

create table inference_model(
    name text primary key,
    title text not null,
    capabilities jsonb not null, -- capabilities jsonified
    price_cached_input int not null,
    price_input int not null,
    price_output int not null,
    inference_endpoint_path text not null,
    inference_endpoint_model text not null,
    public bool not null,
    available_to jsonb not null -- available_to jsonified
)

Whenever a new model is discovered then both the modelGlobals and the database are updated. Models are not automatically removed by any discoverer, this must be done manually through the CLI/UI.

CLI

The CLI will following existing CLI conventions established in the integration module. Communication via IPC, as done by the rest of UCloud/IM, will be done to make changes to the system. Below is a rough outline of the operations available via the CLI:

$ ucloud inference models ls
$ ucloud inference models update <name>
  [--name "NewName"]
  [--capabilities "TextGeneration,TextToImage"]
  [--price-cached 500]
  [--price-input 1000]
  [--price-output 5000]
  [--public true]
  [--available-to "project-id-a,project-id-b"]
  [--base-path "http://..."]
  [--backend-model-name "foobar"]
  [--title "Foobar"]
$ ucloud inference models rm <name>

UI

Extend the inference playground UI to contain a "models" page which lists the pricing of each model. This page will also be used as the foundation for administrators to edit the model catalog.

The model page will be shown when the route is /models. UCX should be extended such that it is possible to declare that a given page will remain static until a route change. When non SP admins visit this page, the UCX app should simply yield (close the WS connection without disrupting the UI). SP admins are defined from the configuration and matches against the resource owner passed to the app.

The default UI for the playground should not be shown on the models page and should be considered completely separate. In the code, this should no longer be referred to as a playground.

The models page will show a table like this:

Model	Name	Cached	Input	Output	Capabilities
Foobar	foobar	0.5x	1x	2x	...
Foobar	foobar	0.5x	1x	2x	...
Foobar	foobar	0.5x	1x	2x	...
Foobar	foobar	0.5x	1x	2x	...
Foobar	foobar	0.5x	1x	2x	...

When the price multiplier is 0, show N/A. This can be used for models for which input tokens, for example, aren't billed.

Administrators will get a button in a new column to the right which allows for editing.

Configuration format

Below is an illustrative example of how the configuration format in UCloud/IM is likely to look:

inference:
  enabled: true
  provider: development|dynamo
  development: # only active when provider is development
    provider: localai
    server: http://localai:8080/v1
  dynamo: # only active when provider is dynamo
    namespace: ucloud-inference
  access:
    administrators: [] # array of project ids
    testers: [] # array of project ids (implied to also contain administrators)

Usage metrics

Expand the internal inference API to track relevant usage statistics for the different models. This should, at the very least, include the following:

• Cached tokens per model
• Non-cached input tokens per model
• Output tokens per model
• Requests per model

UCloud/Core product support

Extend UCloud/Core with a new product type called INFERENCE. This type will be used on the inference product instead of the current hack which sets it to license. The UI, and UCloud/Core internal code and database all need to be updated for this change.

In the accounting system of UCloud/Core, insert a temporary hack which sets all products of type INFERENCE to be non-capacity based even though they are not time-based.

Changes to the existing inference service

The existing inference service will make a number of changes to primarily use this model catalog instead of the current solution. In particular this will change:

• /v1/models will be populated from the model catalog. Only models available to the entity requesting it will be available. This endpoint will contain the updated capabilities and pricing information.
• The previous BackendServer variable will be replaced with the endpoint extracted from the model catalog based on the requested model.
• All operations that look up in the model catalog will have to authorize access to the model according to the availability policy
• Billing is changed according to the changes in "Pricing"