rulelab run
onel 2026-11-03 · rulelab 0.1.0 · Darwin arm64 (Apple M4, 16 GB RAM).
Nothing below is simulated.Nada de lo que sigue es simulado.
Which rules in¿Qué reglas de AGENTS.md earn their tokens?se ganan sus tokens?
Each rule was removed on its own and the agent re-ran the same real tasks. A rule helps if success drops without it; it costs tokens on every model call either way.Se quitó cada regla por separado y el agente repitió las mismas tareas reales. Una regla ayuda si el acierto baja sin ella; en cualquier caso cuesta tokens en cada llamada al modelo.
RulesReglas
Effect on success = success with the full file minus success without the rule, in percentage points, with a 95% bootstrap interval. The shaded band is ±Efecto en el acierto = acierto con el fichero completo menos acierto sin la regla, en puntos porcentuales, con intervalo bootstrap del 95 %. La banda sombreada es ±10 pp: an interval entirely inside it counts as no effect.un intervalo que cae entero dentro cuenta como sin efecto.
Tokens are approximate (characters / 4). "Input tok/run" is the measured difference in input tokens per run with and without the rule; it also moves when the agent takes a different path. "Per month" is the rule's tokens × model calls per run ×Los tokens son aproximados (caracteres / 4). «Tok entrada/ejec.» es la diferencia medida de tokens de entrada por ejecución con y sin la regla; también cambia cuando el agente toma otro camino. «Al mes» es tokens de la regla × llamadas al modelo por ejecución × 1,0001.000 sessions atsesiones a 0.20,2 USD / 1M tokUS$ / 1M tok (mistral/ministral-14b-latest input priceprecio de entrada de mistral/ministral-14b-latest), without prompt caching., sin caché de prompt. The whole file:El fichero entero: 0.86 USD0,86 US$.
Annotated fileFichero anotado
Every rule as it appears in the file, with its verdict. Grey lines are kept in every configuration.Cada regla tal como aparece en el fichero, con su veredicto. Las líneas en gris se mantienen en todas las configuraciones.
# AGENTS.md
## Repository overview sqlparse is a non-validating SQL parser for Python. The package lives in `sqlparse/`: `lexer.py` and `keywords.py` turn text into tokens, `engine/` groups tokens into statements, `sql.py` defines the token tree, `filters/` and `formatter.py` implement formatting, and `cli.py` is the `sqlformat` command. Tests live in `tests/`.
## Running tests Run the tests that cover your change with `python -m pytest -q tests/<file>.py`, then the whole suite with `python -m pytest -q`. Do not finish while a test is failing.
## How to fix a bug Read the failing test first and reproduce the failure before editing anything. Fix the root cause with the smallest change that makes the test pass. Never edit or delete existing tests to make them pass.
## Lexer and keywords Token rules are regular expressions in `sqlparse/keywords.py` and they are tried in order: the first match wins. When adding or changing a pattern, put more specific patterns before more general ones and check that neighbouring patterns still match.
## Code style Follow PEP 8, keep lines under 88 characters, and do not add new dependencies. Keep public function signatures unchanged.
## Communication Before finishing, write a short summary of what you changed and why, listing every file you touched.
ConfigurationsConfiguraciones
Raw success per configuration, pooled over tasks.Acierto bruto por configuración, sumando todas las tareas.
| ConfigurationConfiguración | SuccessAcierto | RunsEjec. | Input tok/runTok entrada/ejec. | USD/runUS$/ejec. |
|---|---|---|---|---|
| No fileSin fichero | 12%12 % (1/8) | 8/8 | 59,05359.053 | 0.0120 USD0,0120 US$ |
| Full fileFichero completo | 25%25 % (2/8) | 8/8 | 59,82959.829 | 0.0122 USD0,0122 US$ |
| WithoutSin R1 | 12%12 % (1/8) | 8/8 | 61,78361.783 | 0.0126 USD0,0126 US$ |
| WithoutSin R2 | 25%25 % (2/8) | 8/8 | 58,34358.343 | 0.0119 USD0,0119 US$ |
| WithoutSin R3 | 12%12 % (1/8) | 8/8 | 53,28953.289 | 0.0109 USD0,0109 US$ |
| WithoutSin R4 | 25%25 % (2/8) | 8/8 | 55,57455.574 | 0.0113 USD0,0113 US$ |
| WithoutSin R5 | 25%25 % (2/8) | 8/8 | 51,49551.495 | 0.0105 USD0,0105 US$ |
| WithoutSin R6 | 25%25 % (2/8) | 8/8 | 53,03253.032 | 0.0108 USD0,0108 US$ |
MethodMétodo
Leave-one-out.Quitar de una en una. Configurations: no file, the full file, and the full file minus each rule. Every configuration runs the same tasks the same number of times, in an interleaved order so a partial run stays balanced.Configuraciones: sin fichero, fichero completo y fichero completo sin cada regla. Todas corren las mismas tareas el mismo número de veces, en orden intercalado para que una ejecución parcial siga equilibrada.
Real tasks.Tareas reales. Tasks come from commits of the repository and are scored by its own tests, run inside a Docker sandbox with no network. The model is called from the host.Las tareas salen de commits del repositorio y se puntúan con sus propios tests, dentro de un sandbox Docker sin red. El modelo se llama desde el host.
Interval.Intervalo. Two-level bootstrap: tasks are resampled with replacement and each configuration's per-task success rate is drawn from a Beta(s+½, f+½) posterior, so 3/3 vs 3/3 does not read as certainty.Bootstrap en dos niveles: se remuestrean tareas con reemplazo y el acierto por tarea de cada configuración se saca de una posterior Beta(s+½, f+½), para que 3/3 frente a 3/3 no parezca una certeza.
Verdict.Veredicto. Helps if the whole interval is above 0, hurts if below 0, no effect if it sits inside ±Ayuda si todo el intervalo está por encima de 0, empeora si está por debajo, sin efecto si cabe dentro de ±10 pp, inconclusive otherwise.y no concluyente en otro caso.
Why measure.Por qué medir. Gloaguen et al. (ETH Zurich, LogicStar.ai) found that context files do not generally improve task success while raising inference cost by over 20% on average.Gloaguen et al. (ETH Zúrich, LogicStar.ai) vieron que los ficheros de contexto no mejoran en general el acierto y suben el coste de inferencia más de un 20 % de media. arXiv 2602.11988