Engines

All three run in the same sandbox, with the same readiness check and limits. sglang and vllm differ only in the framework, its source and its flags; other runs from your folder, and its launch.sh starts the server itself.

engine.json

engine.json is a JSON object with three keys: "format", which is 1, "engine", and "model". For other it has a fourth key, "env", the path of your environment folder (Your own engine). The site refuses "env" for sglang and vllm.

{
  "format": 1,
  "engine": "sglang",
  "model": "nvidia/Qwen3.6-35B-A3B-NVFP4@1355db6a052410cfd62085d94b58866fd0f2c3c5"
}

The model is the course's model, the catalog entry nvidia/Qwen3.6-35B-A3B-NVFP4@1355db6a052410cfd62085d94b58866fd0f2c3c5, which both basic setups serve. Every grading in Assignment 4 serves this model from the course's copy of its weights, so "model" is always this entry, and the site refuses any other value, a folder of your own included.

Your server serves the model under the name in $A4_SERVED_MODEL_NAME, qwen3.6-35b-a3b-fp4, which is the name your agent calls.

launch.sh

The grading runs engine/launch.sh with bash on a fresh RTX PRO 4500 node, in place of serve.sh or your own batch script, so #SBATCH lines in it have no effect. It must have Unix line endings.

What launch.sh runs in

It runs inside the course's sandbox, which has no network and hides /shared, /scratch and every home folder. Nothing under $A4 or ~/a4 is there, so a launch.sh that runs $A4/bin/serve.sh or reads ~/a4 fails. Your engine/, the framework's source and the model are read-only; write caches and temporary files under HOME or /tmp. The sandbox gives launch.sh these variables:

What your server must do

1. Listen on 127.0.0.1:$A4_PORT.
2. Require the key in $A4_KEY_FILE (Authorization: Bearer <key>) on every request, and answer 401 without it. Only /health and /metrics may answer without the key, as they do in SGLang.
3. Serve exactly one model, named $A4_SERVED_MODEL_NAME, with OpenAI's chat completions API.
4. Pass the readiness check within 45 minutes of the start of the grading job.

The readiness check runs every 5 seconds. It passes when GET /v1/models answers 401 without the key and with a wrong key, answers 200 with the key and lists exactly $A4_SERVED_MODEL_NAME, a chat completion with max_tokens 1 answers 200, and POST /invocations without the key is refused (any answer but a 2xx). The 45 minutes include the course's setup of the node, which runs before launch.sh and took 8-10 minutes (490-571 s) on fresh nodes in October 2026. A server that exits before it is ready ends the grading as Engine failed to start, and one that is not ready in time as Engine not ready in 45 minutes. In both cases the Past submissions page shows the end of its log.

While your server runs, the grading stops it as a failure of your engine when the node has less than 3 GB of memory available, when the node's local disk has less than 5 GB free, or when HOME holds more than 20 GB.

The simplest launch.sh

#!/bin/bash
set -euo pipefail
exec a4-serve --flags-file "$A4_ENGINE_DIR/flags.txt"

a4-serve starts the framework A4_ENGINE names (python -m sglang.launch_server, or vLLM's vllm serve) with the flags in the file, then any flags given after it on its command line, and last the arguments the course sets from the variables above: the model (--model-path for SGLang, the model argument for vLLM), --served-model-name, --host, --port and --api-key. vLLM's --api-key guards only the routes under /v1, /v2, /inference and /cohere, so for vLLM a4-serve also adds --middleware a4_auth.RequireKey, the course's middleware that requires the key on every other route too. A missing flags file means no flags. The starter code's engine/launch.sh is this file with comments. A launch.sh may also start the server itself, as long as the server does the four things above. For other it must, since a4-serve starts only SGLang and vLLM (Your own engine).

flags.txt is split into words as Python's shlex splits them (quotes, backslashes and # comments), and no variable in it is expanded. A flag that names a file in engine/ therefore goes on the a4-serve line, where bash expands it:

exec a4-serve --flags-file "$A4_ENGINE_DIR/flags.txt" --chat-template "$A4_ENGINE_DIR/files/chat_template.jinja"

flags.txt

Every flag of the framework is yours to set except the flags that set what the course sets: the model path, the served model name, the address (host and port), the key, and the number of GPUs (tensor, data, pipeline and expert parallel sizes). --config is refused too, since a config file could set any of them. A flag that converts the model's weights as they load serves another model, which is part of Assignment 5 (engine.json).

The site refuses these flags at upload. The grading checks flags.txt again with the framework's own parser before your server starts: a flag the parser does not know, an abbreviated flag, a value it cannot read, or for vLLM a word that belongs to no flag (a model name, for instance) ends the grading as Engine check failed. check-submission.sh runs the same check on the cluster (Test before you submit).

Patches

If you change the framework's source, engine/ holds your changes as one patch: sglang.patch for SGLang or vllm.patch for vLLM, the output of git diff against the release the grading uses. The grading applies it with git apply to a fresh copy of that release on the node. The patch holds your clone as it is, committed or not, against the release, which is also what serve.sh runs. git add every file you create first, since git diff leaves out files git does not track.

git -C ~/a4/sglang add -A python/sglang && git -C ~/a4/sglang diff --no-renames v0.5.20 -- python/sglang > ~/a4/engine/sglang.patch
git -C ~/a4/vllm add -A vllm && git -C ~/a4/vllm diff --no-renames v0.31.0 -- vllm > ~/a4/engine/vllm.patch   # Using vLLM, below

If you changed nothing, leave the patch out: the site refuses an empty one.

• A patch changes only the framework's Python source: files under python/sglang/ for SGLang, except python/sglang/kernels/aot/, and files under vllm/ for vLLM. Kernels compiled ahead of time (python/sglang/kernels/aot/ and SGLang's sgl-kernel, vLLM's C++ and CUDA extensions) cannot be rebuilt on the nodes; kernels compiled at run time may change.
• It is plain git diff output: no renames (hence --no-renames), binary changes, symbolic links or submodules, and no sitecustomize.py, usercustomize.py or .pth file. At most 1 MB.
• It must apply to the release. The site checks the rules above at upload; the grading and check-submission.sh also check that it applies.

engine/files/ may hold up to 32 text files, 256 KB in all and no folders, for files your launch.sh or your flags read, such as a chat template. Nothing else goes in engine/: engine.json, launch.sh, flags.txt, one patch and files/, all UTF-8 text and at most 2 MB in all. For other there is no patch, since your environment folder holds your engine's code.

Using vLLM

setup.sh --vllm builds vLLM's environment on the node next to the course environment. On a fresh node it took about 2 minutes more than setup.sh and left 8-13 GB of the node's local disk free (two runs, October 2026). A grading of a vllm engine runs the same step. To serve vLLM on the cluster:

git clone --depth 1 --branch v0.31.0 https://github.com/vllm-project/vllm.git ~/a4/vllm   # once, on the login node
cp $A4/basic-vllm-flags.txt ~/a4/engine/flags.txt          # and "engine": "vllm" in ~/a4/engine/engine.json
bash $A4/bin/setup.sh --vllm                               # in every job on a GPU node
bash $A4/bin/serve.sh > ~/a4/server.log 2>&1 & tail -f ~/a4/server.log

serve.sh starts the engine that ~/a4/engine/engine.json names. For vLLM it applies your clone's changes against v0.31.0 (git diff v0.31.0 -- vllm, committed or not; git add new files) to a copy of the installed package, as a grading applies vllm.patch, and it adds the course's key middleware, as a4-serve does. vLLM is ready when its log prints Application startup complete. Three differences from SGLang matter to an agent or a flag:

• vLLM returns the model's reasoning in the message's reasoning field, where SGLang uses reasoning_content.
• vLLM's metrics on /metrics are named vllm:..., where SGLang's are named sglang:....
• The basic vLLM flags set --max-num-seqs 32. With vLLM's default of 256, the server refuses to start on this card, since the model's linear-attention state for that many sequences does not fit.

Your own engine

With "engine": "other", a grading runs a serving framework that you install yourself, in an environment folder that you build on the cluster. engine.json names the folder in "env", and your launch.sh starts the server from it. The course does not fix problems inside your own engine. If your framework does not install, does not start, or fails during the run, you debug it, and a grading that fails this way counts toward your daily gradings, as with any engine.

{
  "format": 1,
  "engine": "other",
  "model": "nvidia/Qwen3.6-35B-A3B-NVFP4@1355db6a052410cfd62085d94b58866fd0f2c3c5",
  "env": "/shared/courseSharedFolders/176914outer/176914/a4-models/<your cluster username>/vllm-env"
}

What the folder must be

• Under your own folder, /shared/courseSharedFolders/176914outer/176914/a4-models/<your cluster username>/<folder>. The site checks this at upload. The folder holds your engine, not the model: your server loads the course's model at $A4_MODEL_PATH, as every engine does. If a4-models/<your cluster username> already exists and you did not create it, ask the course staff to move it, since a grading accepts only a folder that you own.
• Self-contained. It holds everything your server runs, including Python itself. The sandbox hides /shared except this folder, and it hides all of /scratch and every home folder. A program or library that your folder takes from elsewhere (a conda installation, ~/.local, the course environment) is therefore missing during a grading. So is the source of an editable install (pip install -e of a clone outside the folder), so install your changed framework into the folder instead. The node's own system is visible read-only: /usr with gcc 11.5, /usr/local/cuda-13.0, and the GPU driver's libraries (October 2026). The sandbox has no network, so nothing can be downloaded.
• Writable by you alone. Every file and folder in it, and your own folder above it, belongs to you and is writable only by you. Build it with umask 022, or run chmod -R go-w on it afterwards.
• Readable by everyone. A grading reads the folder as another account, so every file must be readable and every folder searchable by other accounts (chmod -R a+rX). Everyone in the course can therefore read your folder, including any change you make to your framework's code. To keep your changes private, install the framework unchanged in the folder and put your changes in engine/files/, which is part of your zip and not readable by other students. For example, a files/sitecustomize.py can replace the functions you changed (the rule against that file name covers only patches). Python loads it when it starts, in the server and in every worker process, because launch.sh sets PYTHONPATH=$A4_ENGINE_DIR/files, as the example below does.
• Files, folders and symbolic links only. Each link's target, read from the link's own folder, must stay inside the folder. A venv's links from bin/python to a Python inside the folder, and from lib64 to lib, pass; a link to /usr/bin/python3 does not.
• Within the limits. At most 32 GiB of files, and at most 400,000 files, folders and links in all. No link target may be longer than 1,024 characters, and resolving all the links may take at most 200,000 steps, where a step is one part of a target, counted through every link it leads to. The example below has 157,350 entries, and its links take 1,727 steps.

When a grading starts, it copies the folder to the GPU node's local disk and checks the rules as it copies. The copy must leave 8 GB of the disk free. For other, the course's setup copies only the course model's weights (setup.sh --weights-only), which leaves about 44 GB free on a fresh node (October 2026), so a folder of up to 32 GiB fits. A folder that breaks a rule ends the grading as Engine check failed before your server starts, with a message that names the file, for example venv/bin/python is a symlink to /usr/bin/python3, which leaves the folder. Up to three of your gradings a day that end this way, or on the course's side, do not count toward your daily gradings; any more do. The copy counts toward the 45 minutes, and a copy that has not finished by then ends the grading as Engine not ready in 45 minutes.

The grading checks the finished copy again and mounts it read-only at the folder's own path. The absolute paths that a venv records (in pyvenv.cfg and in the first line of its scripts) therefore work, and a change you make to the folder after the copy does not reach that grading. Build the folder at the path where it will stay, since a venv that is moved or renamed no longer runs.

launch.sh for your own engine

launch.sh gets the variables of What launch.sh runs in, with A4_ENV_DIR and without A4_SRC and PYTHONPATH. It runs from the node's own shell, not from the course environment. It must start the server itself, under the same contract as every engine (What your server must do): listen on $A4_HOST:$A4_PORT, serve the model at $A4_MODEL_PATH under the name $A4_SERVED_MODEL_NAME, and require the key in $A4_KEY_FILE on every route but /health and /metrics. $A4_MODEL_PATH is nvidia/Qwen3.6-35B-A3B-NVFP4, which a framework that reads Hugging Face's cache finds in HF_HOME. Its files are in $HF_HOME/hub/models--nvidia--Qwen3.6-35B-A3B-NVFP4/snapshots/1355db6a052410cfd62085d94b58866fd0f2c3c5. launch.sh passes the model, the served name, the address and the key to your server from these variables. flags.txt is allowed and not checked, and launch.sh may read it. engine/files/ holds whatever else launch.sh reads.

The key must close every route. If your framework's own key option guards every route, as SGLang's --api-key does, use it. vLLM's --api-key guards only /v1, /v2, /inference and /cohere, so the example below also passes --middleware a4_auth.RequireKey, the course's middleware that a4-serve adds for vllm. It answers 401 to any request without the key except /health and /metrics. It is a plain ASGI middleware ($A4/bin/a4_auth.py, standard library only), so a framework whose server is an ASGI app, such as one built on FastAPI or Starlette, can wrap its app in it. The readiness check refuses a server whose /v1/models or POST /invocations answers without the key.

Example: vLLM from PyPI in a uv environment

This example builds vLLM 0.31.0, the release that vllm uses, into a uv environment whose Python is installed inside the folder. The course tested this example; to use another framework, replace vLLM with it. First install uv once, on the login node. The installer puts uv in ~/.local/bin, and UV_NO_MODIFY_PATH=1 stops it from editing your shell's startup files:

curl -LsSf https://astral.sh/uv/0.12.17/install.sh | env UV_NO_MODIFY_PATH=1 sh

Then save this script as build-env.sh and run sbatch build-env.sh, since the install needs more CPU time than the login node allows:

#!/bin/bash
#SBATCH -p general
#SBATCH -c 8
#SBATCH -t 2:00:00
#SBATCH -o build-env-%j.out
set -euo pipefail
umask 022                                  # files writable by you alone
export PATH=$HOME/.local/bin:$PATH         # uv
D=/shared/courseSharedFolders/176914outer/176914/a4-models/$USER/vllm-env
mkdir -p "$D"
UV_CACHE_DIR=$(mktemp -d /tmp/$USER-uv-cache.XXXXXX)   # uv's cache: a new folder that only you can read
trap 'rm -rf "$UV_CACHE_DIR"' EXIT                      # deleted when the script ends, also after an error
export UV_CACHE_DIR UV_PYTHON_INSTALL_DIR=$D/python UV_LINK_MODE=copy
uv python install --no-bin 3.12            # Python itself, inside the folder
[ -e "$D/venv" ] || uv venv --python 3.12 --python-preference only-managed "$D/venv"
uv pip install --python "$D/venv/bin/python" -c /shared/courseSharedFolders/176914outer/176914/a4/vllm-env-constraints.txt \
  vllm==0.31.0 flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache-sm120f==0.7.0.post1 \
  --extra-index-url https://flashinfer.ai/whl/cu130 --extra-index-url https://flashinfer.ai/whl --index-strategy unsafe-best-match
uv pip install --python "$D/venv/bin/python" --no-deps flashinfer-jit-cache==0.7.0.post1 --index-url https://flashinfer.ai/whl/cu130
find "$D" -perm /022 ! -type l -exec chmod go-w {} +   # uv leaves its .lock files writable by everyone
"$D/venv/bin/python" -c "import vllm, transformers; print(vllm.__version__, transformers.__version__)"

The constraints file, $A4/vllm-env-constraints.txt, pins every package to the version the course tested. The FlashInfer packages hold FlashInfer's kernels prebuilt for this card, as the course's vLLM environment does, so the server does not compile them when it starts. The result has 148,881 files and takes 14.3 GB. The course folder is on a network file system, so the install is slow. In October 2026 one install ran for 40 minutes and then stopped with failed to rename file ... No such file or directory on a file of transformers. Building on a node's local disk and copying the result into the folder took longer still (the copy had not finished after 52 minutes). If the install stops with this error, add --reinstall-package <name> to the first uv pip install command, where <name> is the PyPI package that installed the file (transformers for a file under site-packages/transformers/), and run sbatch build-env.sh again. It keeps what is already installed. The last line, the import check, fails when a package is only partly installed. If you created a4-models/$USER without umask 022, also run chmod go-w on it.

Then make the engine in ~/a4/engine: engine.json as above with your cluster username, flags.txt a copy of $A4/basic-vllm-flags.txt, the course's middleware in files/, and the launch.sh below.

cp $A4/basic-vllm-flags.txt ~/a4/engine/flags.txt
mkdir -p ~/a4/engine/files && cp $A4/bin/a4_auth.py ~/a4/engine/files/

#!/bin/bash
# Engine other: vLLM 0.31.0 from the environment folder that engine.json's env names, which a grading mounts
# read-only at its own path, $A4_ENV_DIR. files/a4_auth.py, the course's middleware, requires the key on every route
# but /health and /metrics.
set -euo pipefail
FLAGS=()
if [ -f "$A4_ENGINE_DIR/flags.txt" ]; then   # split as a4-serve splits it: quotes, backslashes, # comments
  out=$("$A4_ENV_DIR/venv/bin/python" -c 'import shlex, sys; print("\n".join(shlex.split(open(sys.argv[1]).read(), comments=True)))' "$A4_ENGINE_DIR/flags.txt")
  [ -z "$out" ] || mapfile -t FLAGS <<< "$out"
fi
export PYTHONPATH=$A4_ENGINE_DIR/files
exec "$A4_ENV_DIR/venv/bin/vllm" serve "$A4_MODEL_PATH" "${FLAGS[@]}" \
  --served-model-name "$A4_SERVED_MODEL_NAME" --host "$A4_HOST" --port "$A4_PORT" \
  --api-key "$(cat "$A4_KEY_FILE")" --middleware a4_auth.RequireKey

If flags.txt does not split (a quote that is not closed, for example), launch.sh stops with Python's error before the server starts.

Run it while you develop

serve.sh starts only sglang and vllm, so it refuses an engine.json that names other. After setup.sh, ENGINE=sglang bash $A4/bin/serve.sh still starts the course's SGLang. To start your own engine in a job on a GPU node, after setup.sh --weights-only, run your launch.sh with the variables a grading sets:

mkdir -p /var/tmp/$USER-a4/home
A4_ENGINE=other A4_ENGINE_DIR=$HOME/a4/engine A4_ENV_DIR=/shared/courseSharedFolders/176914outer/176914/a4-models/$USER/vllm-env \
  A4_MODEL_PATH=nvidia/Qwen3.6-35B-A3B-NVFP4 A4_SERVED_MODEL_NAME=qwen3.6-35b-a3b-fp4 A4_HOST=0.0.0.0 A4_PORT=30000 \
  A4_KEY_FILE=$HOME/a4/serve.key HF_HOME=/var/tmp/$USER-a4/hf HF_HUB_OFFLINE=1 HOME=/var/tmp/$USER-a4/home \
  bash ~/a4/engine/launch.sh > ~/a4/server.log 2>&1 & tail -f ~/a4/server.log

This runs your server outside the sandbox, with the key in ~/a4/serve.key and on 0.0.0.0, as serve.sh does, so your agent reaches it through the tunnel to <node>:30000. setup.sh --weights-only puts the weights in /var/tmp/$USER-a4/hf, and a HOME on the node's disk keeps the framework's caches out of your home folder. Stop the server with kill %1. Outside the sandbox your server can read files that a grading hides, so test it as a grading runs it before you submit.

Test it as a grading runs it

In a job on a GPU node, after cleanup.sh if the node holds the course's environment from an earlier setup.sh:

bash $A4/bin/setup.sh --weights-only        # the course model's weights only: your folder brings the engine
bash $A4/bin/check-submission.sh            # the rules, your folder included (a few minutes: it checks every file in it)
bash $A4/bin/check-submission.sh --launch   # copies your folder and runs launch.sh as a grading does

On nodes in October 2026, setup.sh --weights-only took 142-163 s on a fresh node, the check of this folder 138-296 s, and --launch 574-710 s to PASS, of which the folder's copy took 167-254 s. A grading of this example on a fresh node was ready 911 s (15 minutes) after its setup started.

Test before you submit

In a job on a GPU node, after setup.sh (setup.sh --vllm for a vllm engine, setup.sh --weights-only for other), with your engine/ in ~/a4/engine:

bash $A4/bin/check-submission.sh            # the rules on this page: seconds, or a few minutes for other, whose folder it checks file by file
bash $A4/bin/check-submission.sh --launch   # then runs launch.sh as a grading does, and prints PASS or FAIL

Stop your own server first, since --launch needs the GPU to itself. It uses the leaderboard's sandbox and helpers, a fresh HOME, the same readiness check and the same 45 minutes, and on FAIL it prints the end of the log as the Past submissions page would show it. The whole log is in ~/a4/launch-check.log. check-submission.sh [--launch] DIR tests DIR/engine instead.

Before you zip your submission, copy ~/a4/engine back into your repository's engine/, so that the leaderboard runs what you tested.