Can I serve a LoRA adapter I trained?

Yes. Merge it into its base, import the weights into Amazon Bedrock, and register the ARN with While. The endpoint calls the import on your behalf through a role in your AWS account, so While never holds a key of yours. Any OpenAI-compatible server you run works the same way, registered by URL.

Does the served model score the same as the trained one?

On the text-to-SQL adapter we checked, yes. Served through Bedrock it solved 35 of 100 held-out tasks; served on vLLM it solved 35 of 100. The paired difference over the same 140 tasks is half a point, with an interval that covers zero.

Can I use my own domain?

You get a subdomain of your own on models.withwhile.com by claiming it once, and only your keys work there. For your own domain in front of it, write to us.

Hosting

Serve your own model from one endpoint.

You trained it. Put it behind an OpenAI-compatible URL on While., with a subdomain of your own, and point any client at it.

One URL for every model

Register a model once and it answers at models.withwhile.com. Your key picks your account and the model name picks the row.

Runs where it already runs

A Bedrock import in your AWS account or in ours, or any OpenAI-compatible server you host. Nothing moves and no key of yours is stored.

A subdomain of your own

Claim acme.models.withwhile.com in one call. Only your keys work there. Streaming works everywhere.

For a model in your own AWS account, add a role named WhileModelsInvoke that trusts While. and can invoke the model. The endpoint assumes it per call.

curl -X POST https://models.withwhile.com/models \
  -H "Authorization: Bearer $WHILEAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"name": "nemotron-8b-t2s-r1",
       "arn": "arn:aws:bedrock:us-east-1:<account>:imported-model/<id>",
       "roleArn": "arn:aws:iam::<account>:role/WhileModelsInvoke"}'

Call it

The OpenAI SDK, curl, or wai.Endpoint. The reply is an ordinary chat completion.

from openai import OpenAI

client = OpenAI(base_url="https://models.withwhile.com/v1", api_key="zp_...")
reply = client.chat.completions.create(
    model="nemotron-8b-t2s-r1",
    messages=[{"role": "user", "content": "Which customers never ordered?"}],
    stream=True,
)

Your subdomain

curl -X PUT https://models.withwhile.com/domain \
  -H "Authorization: Bearer $WHILEAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"subdomain": "acme"}'
# https://acme.models.withwhile.com/v1 now answers for your account only

The served weights are the trained weights

We imported a text-to-SQL adapter into Bedrock and scored it on the same 140 held-out tasks as the vLLM run.

Served through the endpoint: 35 of 100 tasks
Served on vLLM: 35 of 100 tasks
The base model: 26 of 100 tasks

The difference between the two servings is half a point and its interval covers zero. The gain over the base is 8 points and its interval excludes zero, the same gain the training run reported. The recipe that produced it.

What we keep

Never a prompt, a completion, or a request log. Nothing you send is stored or read.
Per model per day, three numbers: calls, errors, and tokens. That is how you see it is alive and how it will be billed.
Each account gets 600 requests a minute across its keys, so a leaked key cannot run up a bill. Every call is over TLS, behind a firewall with a per-IP rate limit.

What it costs

Your model’s compute is billed where it runs. On Bedrock, an 8B import costs about $7 an hour while it answers and about $4 a month when idle, billed to the AWS account that holds it. An idle import takes about two minutes to wake; the endpoint says so with a retry hint that the OpenAI SDKs follow.

Questions

Can I serve a LoRA adapter I trained?: Yes. Merge it into its base, import the weights into Amazon Bedrock, and register the ARN with While.. The endpoint calls the import on your behalf through a role in your AWS account, so While. never holds a key of yours. Any OpenAI-compatible server you run works the same way, registered by URL.
Does the served model score the same as the trained one?: On the text-to-SQL adapter we checked, yes. Served through Bedrock it solved 35 of 100 held-out tasks; served on vLLM it solved 35 of 100. The paired difference over the same 140 tasks is half a point, with an interval that covers zero.
Can I use my own domain?: You get a subdomain of your own on models.withwhile.com by claiming it once, and only your keys work there. For your own domain in front of it, write to us.

Read the docs Ask about your own domain Home