Getting Started
This guide runs one EdgeBench task end to end on the default Docker backend: install SForge, get the task images, run an agent, and look at the result.
Scale
The Docker backend is meant for a small number of tasks. Running many tasks concurrently on one host (roughly 20+) will exhaust even a high-end server. Use the Kubernetes backend for full-suite runs, with the official leaderboard settings.
Cost
Frontier-model runs are expensive: one task over the official 12-hour budget can cost hundreds to over a thousand USD in API usage. Start with a single task and a short --timeout (as in this guide) to gauge your burn rate.
Prerequisites
Linux (x86_64). SForge currently targets Linux hosts.
WARNING
The Docker backend needs direct access to the host Docker daemon; running SForge inside a container introduces Docker-in-Docker issues.
Docker Engine, running. Check with:
bashdocker run hello-worldPython 3.11–3.13.
Install SForge
The tasks live in the repository, so install from a clone:
git clone https://github.com/ByteDance-Seed/EdgeBench.git
cd EdgeBench
pip install -e . # add the e2b extra for the E2B backend: pip install -e ".[e2b]"Run the commands below from the repository root: SForge reads EdgeBench from edgebench/ (another benchmark: -b <dir> or SFORGE_BENCHMARK). List its tasks with:
sforge listRun Ad Placement Optimization
Ad Placement Optimization is a C++ optimization task, used here as a small end-to-end example.
1. Get the images
Pull the pre-built work and judge images:
sforge pull --task ad_placement_optimizationImages are tagged by a hash of the task directory, for example edgebench.work.ad_placement_optimization:7b2f533ff7d4, so the pulled images always match your checkout. If you changed the task, or prefer not to download, build them instead:
sforge build --task ad_placement_optimizationSee Pre-built Images for where the images come from.
2. Run an agent
SFORGE_AGENT_API_KEY="sk-ant-xxxx" \
sforge run --task ad_placement_optimization --agent claude-code \
--model claude-opus-4-8 \
--timeout 7200 \
--run-id ad-placement-001sforge run starts its own judge server in the same process. The agent works inside the work container, submits code with sforge-submit, gets a score and feedback, and keeps improving until the timeout. Its model requests go through the judge server, which holds the real API key; the key never enters the container.
View Results
Start the run viewer:
sforge viewerThen open http://127.0.0.1:8000/ for score curves, submissions and token usage. To read the agent's own transcript in its UI:
sforge replay --run-id ad-placement-001 --task ad_placement_optimizationRun outputs are files under logs/runs/<run-id>/<task-id>/:
tail -f logs/runs/ad-placement-001/ad_placement_optimization/agent_output.txt
cat logs/runs/ad-placement-001/ad_placement_optimization/final_result.jsonCommon Next Steps
- A different agent or model: Supported Agents.
- API keys, proxies, resource limits: Environment Variables and Network Setup.
- Every command and flag: CLI Commands.
- Fixing or adding a task: Task Format.