# Evaluate coding agents on private SWE-bench tasks with ForwardLane ForwardLane's SWE-bench Task Instances API sells SWE-bench-format evaluation tasks minted from private financial-services repositories. Those repositories were never public, so the tasks are not in frontier models' training data the way public SWE-bench is. The task catalog and the model evaluation summaries cost $0, and one full task costs $10. Send `GET https://agents.forwardlane.com/zcj/vy06zd2bcewp/api/benchmark/tasks` to pick a task, then `GET https://agents.forwardlane.com/zcj/vy06zd2bcewp/api/benchmark/tasks/{instance_id}` and pay the 402, the price quote you settle and retry. ## At a glance - **Browse tasks:** `GET https://agents.forwardlane.com/zcj/vy06zd2bcewp/api/benchmark/tasks`, filtered by `repo`, `domain`, `difficulty` and `limit`, at $0 - **Model results:** `GET https://agents.forwardlane.com/zcj/vy06zd2bcewp/api/benchmark/eval-results`, filtered by `model`, `repo` and `limit`, at $0 - **Buy one task:** `GET https://agents.forwardlane.com/zcj/vy06zd2bcewp/api/benchmark/tasks/{instance_id}`, $10 - **Payment:** a 402 paid over x402 or MPP (wallet rails settling in USDC) and retried. A call of $0.50 or more can also be paid by card. No account or API key. - **Full task:** the catalog metadata plus `problem_statement`, `base_commit`, `environment_setup_commit`, the gold `patch` and the `FAIL_TO_PASS` and `PASS_TO_PASS` test lists - **Limits:** catalog `limit` 1 to 500 (default 20), results `limit` 1 to 100 (default 10), one task per paid request - **Full purchase flow:** [the buyer skill](https://agents.forwardlane.com/zcj/vy06zd2bcewp/SKILL.md) ## Browse the task catalog 1. List the newest tasks: ```bash curl -s "https://agents.forwardlane.com/zcj/vy06zd2bcewp/api/benchmark/tasks?limit=3" ``` Items come newest first. `total` counts every task that matches your filters before `limit` applies. A request with `limit=3` returned three items and a `total` of 49, the first of them shown here. ```json {"items": [{"instance_id": "", "repo": "forwardlane-backend", "difficulty": "easy", "domain": "tasks", "fail_to_pass_count": 1, "pass_to_pass_count": 0, "created_at": "2026-03-25T08:48:25.515671+00:00", "version": null, "solve_rate": 0}, "… 2 more"], "total": 49} ``` Each item also carries `complexity` when the task has been scored. The catalog does not include the problem statement, the patch or the test lists. 2. Narrow the list to the tasks you want to run: | To get | Add | | --- | --- | | Tasks from one repository | `repo=` | | Tasks in one domain | `domain=` | | Tasks of one difficulty | `difficulty=easy`, `medium` or `hard` | | More or fewer tasks | `limit=1` to `limit=500` | ```bash curl -s "https://agents.forwardlane.com/zcj/vy06zd2bcewp/api/benchmark/tasks?difficulty=hard&limit=50" ``` Note the `instance_id` of each task you choose. 3. Read the model evaluation summaries: ```bash curl -s "https://agents.forwardlane.com/zcj/vy06zd2bcewp/api/benchmark/eval-results?limit=5" ``` The response is a JSON object with `items`, a list of model evaluation result summaries, and `total`. Filter with `model=` or `repo=`. A request with `limit=5` returned an empty `items` list and a `total` of 0. ```json {"items": [], "total": 0} ``` ## Buy a full task 1. Request the task by its id: ```bash curl -si "https://agents.forwardlane.com/zcj/vy06zd2bcewp/api/benchmark/tasks/" ``` The storefront answers with a 402 quoting $10. Nothing is charged until you pay it, so you can stop at this point. 2. Pay the 402 and retry the same request. [The buyer skill](https://agents.forwardlane.com/zcj/vy06zd2bcewp/SKILL.md) has the full payment flow. 3. Read the task from the response. It is a JSON object with the catalog fields plus these: | Field | Contents | | --- | --- | | `problem_statement` | The issue the model must resolve | | `base_commit` | The commit the task starts from | | `environment_setup_commit` | The commit for setting up the test environment | | `patch` | The gold fix, as a unified diff | | `FAIL_TO_PASS` | Test list in SWE-bench format | | `PASS_TO_PASS` | Test list in SWE-bench format | To pay for many tasks from one prepaid balance, buy Credits; [the buyer skill](https://agents.forwardlane.com/zcj/vy06zd2bcewp/SKILL.md) covers buying them. ## Limits - One task per request: a run of 10 tasks is 10 requests, each with its own 402, and costs $100. - Every endpoint is a synchronous read that returns JSON: task data and evaluation summaries. - The data comes from the repositories ForwardLane analysed, not from your own code. ## Errors | Response | Meaning | Action | | --- | --- | --- | | `402` | The price quote. Nothing is charged yet. | Pay and retry the same request, or stop | | `404` | The `instance_id` is unknown | Take an id from the catalog | | `422` | A parameter failed validation, such as `limit` out of range | Fix the query and resend | | `settlement_failed` | The payment did not settle | Fix the wallet and send a fresh request for a new 402 | | `401 bearer_required` | A Credits purchase was sent without an agent identity | Register at [auth.md](https://agents.forwardlane.com/zcj/vy06zd2bcewp/auth.md), or pay per task | ## Frequently asked questions ### Why are these SWE-bench tasks uncontaminated? Public SWE-bench is in frontier models' training data. These tasks come from private financial-services repositories that have never been public, so frontier models cannot have trained on them. ### Is browsing the task catalog charged? No. The catalog and the model evaluation summaries cost $0. Only the full task, with its problem statement, patch and test lists, is charged at $10 per instance. ### Can I download several tasks in one call? No. `GET /api/benchmark/tasks/{instance_id}` returns one task. Fetch each chosen task with its own request, or buy Credits to pay for a run from one balance. ### What format are the tasks in? SWE-bench format: a problem statement, a base commit, an environment setup commit, a gold patch as a unified diff, and `FAIL_TO_PASS` and `PASS_TO_PASS` test lists. The catalog adds difficulty, domain, test counts, version and solve rate. ### What do the model evaluation results contain? `GET /api/benchmark/eval-results` returns model evaluation result summaries in `items`, with `total` counting them. Filter by `model` or `repo`, and set `limit` from 1 to 100 (default 10). The endpoint costs $0. ## Keep reading - [Buyer skill](https://agents.forwardlane.com/zcj/vy06zd2bcewp/SKILL.md): the full purchase flow - [How to pay](https://agents.forwardlane.com/zcj/vy06zd2bcewp/payment.md): x402, MPP, card and Credits - [Error reference](https://agents.forwardlane.com/zcj/vy06zd2bcewp/errors.md): every error code - [SWE-bench Task Instances API reference](https://agents.forwardlane.com/zcj/vy06zd2bcewp/services/swebench-tasks/openapi.json): parameters and responses --- Updated 2026-10-26. All ForwardLane guides: https://agents.forwardlane.com/guides.md. Storefront: https://agents.forwardlane.com/zcj/vy06zd2bcewp/llms.txt.