Model Deployment Introduction

After completing model training, you can deploy the model to a server to provide real-time model inference services. "Model Deployment (Serving)" is the server-side model inference functionality provided by HyperAI.

Deployment Modes

HyperAI model deployment supports two deployment modes:

1. Custom Deployment (Recommended): Fully customize the service startup method by writing a start.sh startup script
2. Traditional predictor.py approach: Use the predefined framework provided by HyperAI

API Key Authentication

HyperAI model deployment supports secure authentication using API Keys. Compared to JWT Token authentication, API Keys offer:

• More fine-grained access control
• Support for independent key management and tracking
• Alignment with industry standard practices (such as OpenAI, HuggingFace, etc.)

You can create and manage API Keys in the model deployment settings page. For detailed information, please refer to API Key Management.

Custom Deployment Method (Recommended)

This is the simplest and most flexible deployment method. You only need to:

1. Prepare model files
2. Write a start.sh script to start your service

Custom Deployment Requirements

1. Required files:

   • start.sh - Startup script, must ensure:
     • Listen on port 80
     • Handle HTTP requests
   • Model files and other dependency files

2. Optional files:

   • requirements.txt - For installing Python dependencies
   • conda-packages.txt - For installing Conda dependencies
   • dependencies.sh - For installing system dependencies
   • .env - For setting environment variables

Example

You can use any framework (such as FastAPI, Flask, Gradio, etc.) to provide services. Here is a simple example using FastAPI:

# app.py
from fastapi import FastAPI
import uvicorn

app = FastAPI()

@app.get("/")
def predict():
    return {"message": "Hello World"}

# start.sh
#!/bin/bash
pip install fastapi uvicorn
uvicorn app:app --host 0.0.0.0 --port 80

Data Binding

When creating a model deployment, you can bind one or more data directories. The data binding method is basically the same as data binding for model training, and you can choose from the following sources:

• Public datasets or models
• Personal private datasets or models
• Working directory of computing containers
• Data repositories uploaded via file upload

Data Binding Characteristics

Model deployment data binding differs from model training in the following ways:

1. Read-only Binding: All data bindings are in read-only mode, with no write or modification operations allowed
2. Multiple Directory Binding: Multiple data directories can be bound simultaneously to different mount points:
   • /hyperai/input/input0
   • /hyperai/input/input1
   • /hyperai/input/input2
   • /hyperai/input/input3
   • /hyperai/input/input4
3. Working Directory Characteristics:
   • Contents in the working directory (/hyperai/home) are copied from the binding source when the deployment starts
   • Important note: Since the contents of the working directory will be lost after restart, it is recommended to place all necessary model files and dependencies in the bound data directories

Selecting Binding Directories

The selection method for binding is similar to compute containers. For detailed operations, please refer to Computing Container Data Binding.

Version Management

Model deployment supports version management:

• Versions are independent of each other and can support different runtime environments, resource types, and deployment contents
• When a new version is deployed, the old version will be automatically taken offline
• Version numbers increment as numeric sequences

Detailed operations are introduced in Managing Model Deployments.

Startup & Readiness

After a deployment starts, the platform does not route traffic to it immediately. Instead, it waits for your service to listen on 0.0.0.0:80 inside the container; only once that port is reachable will the version be marked ready and begin receiving requests.

Loading large models (downloading weights, loading into VRAM) typically takes several minutes. During this time the container being "not ready" is normal and does not indicate a deployment failure. So you can follow the progress, the platform outputs startup-stage hints in the deployment logs:

===== Waiting for the service to listen on port 80; requests will only be accepted once listening succeeds =====
===== The service is not yet listening on port 80; waited N minutes. Please ensure the service listens on 0.0.0.0:80 =====
===== Port 80 detected, the service is ready =====

If the service listens on the wrong port (not 80), the log will instead show the following hint, followed by a restart; after repeated failures the deployment is stopped (the "service ready" message above will not appear):

===== Detected the service is listening on port 8000, but the platform requires 0.0.0.0:80; the platform is about to restart the service. After multiple failures the deployment will be stopped automatically =====

• Seeing "service ready" means the port is ready and the version has started receiving requests.
• If you repeatedly see "the service is not yet listening on port 80; waited N minutes", troubleshoot in this order:
  • Bind address: Check the host argument start.sh passes to the framework; binding to 0.0.0.0 is recommended. The platform probes the port from inside the container (loopback) — binding to a specific container IP cannot be probed and will be judged as failed; binding to 127.0.0.1 / localhost can still be detected as ready, but using 0.0.0.0 is recommended to cover all access paths.
  • Listening port: Confirm the framework is actually listening on port 80.
  • Startup failure or still loading: Check the output from the framework itself (vLLM, FastAPI, etc.) in the deployment logs to confirm whether there are errors or it is still downloading / loading the model.
• The platform judges definitive startup failures as soon as possible without waiting for the full timeout:
  • Crashes immediately on startup: The process exits shortly after starting (the script finishes and ends, or crashes). The platform judges it failed within minutes and reclaims resources (releasing the GPU), with a hint in the log. Fix: ensure the service runs in the foreground and stays alive; do not let start.sh exit at the end.
  • Listening on the wrong port or address: The process stays alive but is not listening on 0.0.0.0:80 inside the container (for example, mistakenly listening on 8000, or bound to a specific container IP instead of 0.0.0.0). Once detected, the platform will tell you the actual port the service is listening on in the log, and judge it failed within minutes, reclaiming resources. Fix: make the service listen on 0.0.0.0:80.
  • Still loading (genuinely slow): The process is alive but not listening on any port yet (for example, a large model is being downloaded / loaded into VRAM). The platform grants a longer startup grace period (up to about 1 hour) for it to become ready; only if port 80 is still not listened on after that is it judged as failed. Please ensure time-consuming operations can finish within this time; if necessary, pre-place the model in the bound dataset to avoid re-downloading on every cold start.

Traditional Deployment Method (predictor.py)

If you wish to use the predefined framework provided by HyperAI, you can choose this method.

1. Required files:

   • predictor.py - Model deployment script containing the Predictor class
   • Model files

2. Optional files:

   • requirements.txt, conda-packages.txt - For installing dependencies
   • dependencies.sh - For installing system dependencies
   • .env - For setting environment variables

Detailed writing rules are introduced in Writing Serving Services, and writing examples are available for reference in the openbayes-serving-examples repository.