Prompting LabAI Lab for Lawyers

Behind the scenes

For instructors

How the Lab grades, what it does and does not store, and how to change the model or the exercises.

How grading works

When a student presses Grade my prompt, the server makes two calls to a language model through OpenRouter, in parallel:

  1. Run. The student's prompt is sent to the model as written, so the student sees what their instructions actually produce. If the prompt refers to the materials by placeholder (for example [paste lease extract] or [CLAUSE]), or does not include them at all, the Lab appends the exercise's run_materials (or, if that field is absent, the materials) in a delimited block. Step 1 sets run_materials to null: what the student pastes is what runs, so the unredacted email never reaches the model through the back door. Step 4 supplies clause 12.2 so [CLAUSE] has something to work on. The run output is shown to the student but is not what is graded.
  2. Grade. A second call acts as an LLM judge. It receives the exercise brief, the materials, the rubric with weights and “what good looks like”, the exercise's hidden grader notes (point schedules and traps), a hidden reference prompt for calibration, a <measurements> block computed by the server (see below), and the student's prompt inside a tag whose name carries a random per-request suffix, flagged as data rather than instructions. Any literal boundary tag the student types is neutralised before interpolation, so a prompt cannot “close” the data block and smuggle notes to the grader. The judge is told to score the prompt, never an imagined output; to ignore length and markdown; to write a rationale before each score; and to answer in strict JSON.

The server then parses the JSON defensively: code fences are stripped, the shape is validated, each criterion score is clamped to 0 to its weight, missing criteria score 0, and the total is recomputed from the rubric weights rather than trusted from the model. The student sees per-criterion scores with a one-line rationale, the total out of 100, two or three concrete suggestions, and an improved version of their prompt.

What the grader is good and bad at

  • It is reliable on observable, binary criteria: is an audience named, is a length ceiling given, are the documents delimited, does the prompt still contain the divorce gossip. The rubrics are written that way on purpose.
  • It is less reliable at counting and at fine distinctions, so the deterministic parts are measured server-side and injected as a <measurements> block the grader is told to use instead of its own count: the instruction word count (tagged document blocks and [bracketed] placeholders stripped first), and, for Step 1, a case-insensitive hit list of the names and gossip terms the exercise expects the student to remove (leak_terms in the JSON). Expect a few points of noise between runs on the judgement criteria. Scores are indicative, not authoritative, and the footer says so.
  • Students can and will try to argue with the grader inside the prompt. The judge is told to treat such text as a defect. It is not bulletproof; a human sanity check of top scores is sensible in a live class.
  • The pass mark of 70 per exercise and the 6,000-character limit on prompts are constants in lib/progress.ts and lib/grading.ts.

Changing the model

The model is read from the OPENROUTER_MODEL environment variable at request time; the default is anthropic/claude-sonnet-5. Any chat-completion model on OpenRouter works, but the grader relies on the model returning well-formed JSON and following a long rubric, so mid-tier or better models are recommended. Set OPENROUTER_REASONING_EFFORT (low to max) to control reasoning depth on the grading call for models that support it. Both the run and the grade use the same model.

Data and privacy

  • Progress, best scores and unsent drafts live in the student's browser localStorage. There is no account and no database.
  • Prompts are sent to OpenRouter and on to the model provider under OpenRouter's and the provider's terms. The exercises use fictional parties; tell students not to paste real client material.
  • The OpenRouter key is read only inside the server route and never shipped to the client. The hidden model_prompt and grader_notes fields of each exercise are stripped from everything sent to the browser.
  • A best-effort rate limit is kept in server memory with three buckets: per browser (a random UUID the page stores in localStorage and sends as X-Client-Id; default 12 grades per 10 minutes), per trusted IP as a backstop (default 120 per 10 minutes; the IP is taken from x-vercel-forwarded-for or cf-connecting-ip, or from the last entry of x-forwarded-for only when TRUSTED_PROXY=1), and global per server instance (default 240 per 10 minutes) so total spend is capped whatever headers a caller sends. On Vercel every bucket is per function instance, so treat all of this as a speed bump, not a budget control; set a credit limit on the OpenRouter key at openrouter.ai/keys.
  • Optional course code: set LAB_ACCESS_CODE and the grading route refuses requests without a matching X-Lab-Code header. The page asks the student for the code once and keeps it in localStorage. Share it in the room, not on the public site.

Classroom settings

A class of thirty behind one office or venue network shares one IP but not one browser, so the per-browser bucket is the limit students feel. The defaults suit a session with a retry or two per exercise. For a hands-on workshop where students iterate quickly, raise them in the environment, for example GRADE_RATE_LIMIT=30/10m, GRADE_IP_RATE_LIMIT=600/10m and GRADE_GLOBAL_RATE_LIMIT=900/10m. The format is count/window with s, m or h. Budget roughly two model calls per grade. The 62 minutes on the home page is writing time; allow about 90 minutes with a second attempt at each exercise.

Editing the exercises

All exercises live in content/exercises.json and are imported at build time. Each entry needs an id (lower-case slug), title, minutes, technique, a short technique_label for badges, learning_objective, scenario, task, materials, hints, a rubric whose weights sum to 100, a model_prompt and grader_notes. Optional: run_materials (a string to append at run time when the prompt uses a placeholder, or null to never append) and leak_terms (words whose presence the server reports to the grader). Grader notes should express every cap per criterion; the total is always recomputed from the per-criterion scores. Rebuild and redeploy after editing.