LLM Inference API Service¶

The LLM Inference API service provides OpenAI/Anthropic-compatible inference endpoints running selected open-weight LLM models such as Apertus and other vetted models. CSCS takes care of deploying, patching, scaling, and operating the underlying serving stack.

In order to maximize utilization and reduce costs, a reduced set of models is available. Private model deployment is not supported. If you are interested to deploy a model that is not available in this service, we encourage using the sml tool developed by the Swiss AI community.

Privacy and confidentiality are essential to us. CSCS does not record user prompts or model responses, and your data does not leave the infrastructure we control. Usage follows the CSCS user regulations.

Service at a glance¶

Quick start¶

Before using the API, obtain a key by following the access section. Include this API key in every API request. The base URL for the inference API is https://api.inference.cscs.ch/v1.

Query available models using the /v1/models endpoint:

curl -X GET "https://api.inference.cscs.ch/v1/models" \
  -H "Authorization: Bearer $CSCS_INFERENCE_API_KEY" \
  -H "Content-Type: application/json"

$ curl -s -X GET "https://api.inference.cscs.ch/v1/models" -H "Authorization: Bearer $CSCS_INFERENCE_API_KEY" -H "Content-Type: application/json" | jq
{
  "data": [
    {
      "id": "swiss-ai/Apertus-8B-Instruct-2509",
      "created": 1782315799,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "swiss-ai/Apertus-70B-Instruct-2509",
      "created": 1782315799,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "swiss-ai/Apertus-v1.5-8B",
      "created": 1782315799,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "swiss-ai/Apertus-v1.5-8B-thinking",
      "created": 1782315799,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "swiss-ai/Apertus-v1.5-70B",
      "created": 1782315799,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "swiss-ai/Apertus-v1.5-70B-thinking",
      "created": 1782315799,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "google/gemma-4-31B-it",
      "created": 1782315799,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "moonshotai/Kimi-K2.7-Code",
      "created": 1782315799,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16",
      "created": 1782315799,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    },
    {
      "id": "zai-org/GLM-5.2",
      "created": 1782315799,
      "object": "model",
      "owned_by": "Envoy AI Gateway"
    }
  ],
  "object": "list"
}

Get a response using the Apertus 70B model using the /v1/chat/completions endpoint:

curl -X POST "https://api.inference.cscs.ch/v1/chat/completions" \
    -H "Authorization: Bearer $CSCS_INFERENCE_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"model": "swiss-ai/Apertus-v1.5-70B", "messages": [{"role": "user", "content": "Explain gradient descent in one paragraph."}], "temperature": 0.2}'

$ curl -s -X POST "https://api.inference.cscs.ch/v1/chat/completions" -H "Authorization: Bearer $CSCS_INFERENCE_API_KEY" -H "Content-Type: application/json" -d '{"model": "swiss-ai/Apertus-v1.5-70B", "messages": [{"role": "user", "content": "Explain gradient descent in one paragraph."}], "temperature": 0.2}' | jq
{
  "id": "chatcmpl-426afafa-2bfb-4412-a1cb-859fdc3ada0c",
  "object": "chat.completion",
  "created": 1782485315,
  "model": "swiss-ai/Apertus-v1.5-70B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Gradient descent is a fundamental optimization algorithm used in machine learning to minimize the cost or loss function of a model. It works by iteratively adjusting the model's parameters in the direction of steepest descent of the cost function, which is determined by the negative of the gradient of the cost function with respect to the parameters. The gradient points in the direction of the greatest increase of the function, so by moving in the opposite direction (negative gradient), the algorithm reduces the cost. The step size, or learning rate, determines how much to adjust the parameters in each iteration. If the learning rate is too small, the algorithm may take too long to converge; if it's too large, the algorithm may overshoot the minimum and fail to converge. Gradient descent is widely used in training neural networks and other machine learning models.",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [],
        "reasoning": null
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null,
      "token_ids": null,
      "routed_experts": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.23.0-tp4-712aba24",
  "usage": {
    "prompt_tokens": 69,
    "total_tokens": 233,
    "completion_tokens": 164,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "prompt_text": null,
  "kv_transfer_params": null
}

Access¶

Access to the inference service is granted at the project level. In order to use the service users must have an active project granted via an open call for proposals, a partnership, or cscs2go.

Available models and pricing¶

Available models, along with pricing information, are listed on the Inference API UI pricing page. The available models can also be listed for a given API key using the models endpoint or on the Inference API UI when creating a new key.

The available models, their maximum context length, and their supported input modalities are also listed in the table below. Most coding agents benefit from being configured with the given context sizes so that they can do context compaction before hitting the context limit.

Create an inference resource¶

An inference resource must be created for your project before any project member can create API keys. Which procedure applies depends on the organization your project belongs to: non-SwissAI projects must request a resource through the CSCS Service Desk, while projects in the SwissAI organization can create the resource directly in the project management portal (self-service).

Create an API key¶

Once an inference resource has been created for your project, any project member can create API keys through the inference API UI.

1. Navigate to the Inference API UI and authenticate.
2. Expand the inference resource created by the PI and press “Add Key”.
   • Enter a key alias for the key. Choose a memorable name that you can distinguish among other keys in your project and resource.
   • Optionally set a token budget, reset period, or restrict the available models. Please note that the global resource limits apply as well.
3. Click “Create Key” and copy the generated key and store securely, for example in a password manager. The key will be displayed once.
4. Test that the key works by following the quick start guide.

API¶

The base URL for the inference API is https://api.inference.cscs.ch/v1. The following OpenAI- and Anthropic-compatible endpoints are available.

When using the endpoints for example through agents, the framework will handle API requests for you. For information on how to use the endpoints directly, see the OpenAI and Anthropic documentation.

Setting up coding agents to use the inference service¶

Below are instructions for setting up Claude Code, OpenCode and VS Code Copilot to use the inference service. For more information on using coding agents on Alps, see the coding agents guide.

See the available models table for context sizes. Most agents benefit from having the maximum context length configured explicitly so that they can do context compaction before hitting the context limit.

Claude Code¶

Set the following environment variables before starting a claude session.

export ANTHROPIC_AUTH_TOKEN=$CSCS_INFERENCE_API_KEY

export ANTHROPIC_BASE_URL=https://api.inference.cscs.ch
export ANTHROPIC_MODEL=moonshotai/Kimi-K2.7-Code
claude

OpenCode¶

Add a custom provider to your OpenCode config file (typically ~/.config/opencode/opencode.jsonc).

{
    "$schema": "https://opencode.ai/config.json",
    // Set Kimi as default OpenCode model
    "model": "cscs/moonshotai/Kimi-K2.7-Code",
    "provider": {
        "cscs": {
            "npm": "@ai-sdk/anthropic",
            "name": "CSCS Inference",
            "options": {
                "baseURL": "https://api.inference.cscs.ch/v1",
                "systemMessageMode": "system",
            },
            "models": {
                "moonshotai/Kimi-K2.7-Code": {
                    "name": "Kimi K2.7-Code",
                    "modalities": {
                        "input": ["text", "image"],
                        "output": ["text"]
                    },
                    "limit": {
                        "context": 262144,
                        "output": 16384
                    }
                }
            }
        }
    }
}

{
    "$schema": "https://opencode.ai/config.json",
    // Set Kimi as default OpenCode model
    "model": "cscs/moonshotai/Kimi-K2.7-Code",
    "provider": {
        "cscs": {
            "npm": "@ai-sdk/anthropic",
            "name": "CSCS Inference",
            "options": {
                "baseURL": "https://api.inference.cscs.ch/v1",
                "systemMessageMode": "system",
                "apiKey": "{env:CSCS_INFERENCE_API_KEY}" 
            },
            "models": {
                "moonshotai/Kimi-K2.7-Code": {
                    "name": "Kimi K2.7-Code",
                    "modalities": {
                        "input": ["text", "image"],
                        "output": ["text"]
                    },
                    "limit": {
                        "context": 262144,
                        "output": 16384
                    }
                }
            }
        }
    }
}

Once configured, you can choose models configured in the config with /models or Ctrl-X+M.

Model support for different input modalities are described in the available models table.

VS Code Copilot¶

GitHub Copilot Chat in VS Code can use models served by the inference API through its custom endpoint provider. You need to be signed in to GitHub in VS Code. See the VS Code language models documentation for the full reference.

To add the inference API as a model provider:

• Open the command palette (Cmd+Shift+P on macOS, Ctrl+Shift+P on Linux and Windows) and run “Chat: Manage Language Models”.
• Click “Add Models” in the top right corner and select “Custom Endpoint”.
• Enter a name for the group of models, for example CSCS Inference.
• Enter your API key when prompted for the apiKey value. VS Code stores the key in its secret storage and only references it from the configuration file.
• Select “Chat Completions” as the API type.

VS Code then opens the chatLanguageModels.json configuration file with an entry for the new provider. Add the models that you want to use to the models list, leaving the generated apiKey reference untouched:

[
    {
        "name": "CSCS Inference",
        "vendor": "customendpoint",
        "apiKey": "${input:chat.lm.secret.xxxxxxxx}",
        "apiType": "chat-completions",
        "models": [
            {
                "id": "moonshotai/Kimi-K2.7-Code",
                "name": "Kimi K2.7-Code",
                "url": "https://api.inference.cscs.ch/v1",
                "toolCalling": true,
                "vision": true,
                "maxInputTokens": 245760,
                "maxOutputTokens": 16384
            }
        ]
    }
]

The fields of each model entry are:

• id is the model identifier as returned by the /v1/models endpoint,
• name is the display name shown in the model picker,
• url is the base URL of the inference API, to which VS Code appends /chat/completions,
• toolCalling enables tool use, which is required for agent mode,
• vision enables image input and should only be set for models that support image input (see the available models table),
• maxOutputTokens is the maximum number of tokens in a single response, for which 16384 is a good default for coding tasks, and
• maxInputTokens is the maximum context length of the model from the available models table minus maxOutputTokens.

After filling in the file, save and close it. Then open the Chat view, click the model picker at the bottom of the chat input, and select the model under “Other models”.

Setting up OpenWebUI to use the inference service¶

OpenWebUI is an open source interface for accessing AI providers. It can be hosted locally for a single user and can be set up to access the CSCS inference endpoints. Please see the OpenWebUI documentation for help setting up an instance for yourself.

In order to configure OpenWebUI to connect to the CSCS inference endpoint you must be admin on an instance. Then:

• Go to “Admin Panel” under your profile menu.
• Go to “Settings” and “Connections”.
• Under “OpenAI API” press the + symbol to add a new provider.
• Add the CSCS inference base URL and your API key under “Auth”.

The models will be automatically detected and available for use.

Announcements¶

Planned maintenance, incidents, and changes to the available models are published on the service status page. Everyone with access to a project that has an inference resource is subscribed automatically and can opt out at any time.

The same announcements are also posted in the #inference-service channel of the CSCS User Slack; see get in touch to join.

Known issues and limitations¶

• Detailed self-service telemetry is limited today. Users interested in hourly/daily usage should record it from the client side.
• Documentation and model-specific configuration transparency are work in progress.
• The service is currently offered from a single infrastructure. Interruptions of the service should be expected due to incidents and/or planned maintenances.
• We currently do not distinguish between cached and uncached input tokens; please beware the costs when performing typical agentic usage.