inference

Index

• v1
  • Inference

Schemas

Inference

Inference is a module schema consisting of model, framework and so on

Attributes

name	type	description	default value
framework required	"Ollama" | "KubeRay"	The framework or environment in which the model operates.
model required	str	The model name to be used for inference.
num_ctx	int	The size of the context window used to generate the next token.	2048
num_predict	int	Maximum number of tokens to predict when generating text.	128
system	str	The system message, which will be set in the template.	""
temperature	float	A parameter determines whether the model's output is more random and creative or more predictable.	0.8
template	str	The full prompt template, which will be sent to the model.	""
top_k	int	A higher value (e.g. 100) will give more diverse answers, while a lower value (e.g. 10) will be more conservative.	40
top_p	float	A higher value (e.g. 0.9) will give more diverse answers, while a lower value (e.g. 0.5) will be more conservative.	0.9

Examples

import inference.v1.infer

accessories: {
    "inference@v0.1.0": infer.Inference {
        model: "llama3"
        framework: "Ollama"

        system: "You are Mario from super mario bros, acting as an assistant."
        template: "{{ if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}{{ if .Prompt }}<|im_start|>user {{ .Prompt }}<|im_end|> {{ end }}<|im_start|>assistant"

        top_k: 40
        top_p: 0.9
        temperature: 0.8

        num_predict: 128
        num_ctx: 2048
    }
}