Ask the data · evaluation
How often does the model get the SQL right?
16 questions about this database, 14 with gold SQL written by hand from the data and 2 that the data cannot answer. Run them on your model, with your key, and get execution accuracy with a confidence interval, the refusal rate, how often the validator had to step in, latency and token use.
Run
Benchmark your model
Add your own key to run the benchmark. Nothing is pre-scored or published here.
The 16 questions
| id | Question | Tests | Gold answer |
|---|---|---|---|
| q01 | How many geotagged tweets were matched to Victorian suburbs between February and July 2022?Sum over the 'all' topic only (topics overlap). | aggregate | 719336SELECT SUM(tweet_count) AS tweets FROM twitter_sal_sentiment WHERE topic = 'all' AND state = 'Victoria' |
| q02 | Ignoring the outlier filter, which Victorian SA2 has the highest median personal income, and what is it?All 457 Victorian SA2s, not only the 420 the IQR rule kept. | ranking | Southbank, 73408SELECT sa2_name, median_aud FROM income_sa2 WHERE state = 'Victoria' ORDER BY median_aud DESC LIMIT 1 |
| q03 | How many Victorian local government areas did the team's IQR outlier rule keep?Uses the iqr_kept flag. | aggregate | 72SELECT COUNT(*) AS lgas FROM crime_lga WHERE iqr_kept = 1 |
| q04 | Which local government area recorded the most offences in 2019, and how many?Across all 79 LGAs, outliers included. | ranking | Melbourne (C), 26694SELECT lga_name, total FROM crime_lga ORDER BY total DESC LIMIT 1 |
| q05 | What fraction (between 0 and 1) of crime-related tweets scored 1, the most negative score?A ratio over the Twitter crime histogram; integer division would return 0. | ratio | 0.2493SELECT SUM(CASE WHEN score = 1 THEN count ELSE 0 END) * 1.0 / SUM(count) AS share FROM sentiment_histogram WHERE source = 'twitter' AND topic = 'crime' |
| q06 | What is the average sentiment score of all tweets in the suburb of Geelong?Needs the join from suburb name to SAL code. | join | 5.49SELECT t.avg_score FROM twitter_sal_sentiment t JOIN regions_sal r USING (sal_code) WHERE r.name = 'Geelong' AND t.topic = 'all' |
| q07 | How many of the re-scored mastodon.social toots were declared as English?Language code 'en'. | lookup | 339498SELECT toots FROM mastodon_language WHERE server = 'mastodon.social' AND lang = 'en' |
| q08 | In the re-scored mastodon.social week, which hourly bucket (the hour_utc timestamp) had the most toots?One hourly bucket, as stored in hour_utc (not an hour of the day summed over the week). | ranking | 2023-05-04T13:00:00ZSELECT hour_utc FROM mastodon_hourly ORDER BY toots DESC LIMIT 1 |
| q09 | What Pearson correlation did the analysis find between median income and the sentiment of income tweets, across SA2s with at least one income tweet?Read the stored result rather than recomputing it. | lookup | 0.1383SELECT pearson_r FROM scenario_correlations WHERE scenario = 'income' AND y_metric = 'avg_income' AND min_tweets = 1 |
| q10 | Which Greater Capital City area has the highest median income?From the capital-city summary table. | ranking | Australian Capital TerritorySELECT gcc_name FROM income_gcc ORDER BY median_aud DESC LIMIT 1 |
| q11 | List the five Victorian suburbs with the most crime-related tweets.Five names; order is not scored. | join | Melbourne; Ballarat Central; …SELECT r.name FROM twitter_sal_sentiment t JOIN regions_sal r USING (sal_code) WHERE t.topic = 'crime' AND t.state = 'Victoria' ORDER BY t.tweet_count DESC LIMIT 5 |
| q12 | How many Victorian SA2s have personal income data?457, before the outlier rule. | aggregate | 457SELECT COUNT(*) AS sa2s FROM income_sa2 WHERE state = 'Victoria' |
| q13 | What is the mean sentiment score of income-related tweets across Australia?A weighted mean over the histogram. | ratio | 5.837SELECT SUM(score * count) * 1.0 / SUM(count) AS mean_score FROM sentiment_histogram WHERE source = 'twitter' AND topic = 'income' |
| q14 | How many drug offences were recorded in the City of Greater Geelong?LGA names carry a type suffix. | lookup | 520SELECT drug FROM crime_lga WHERE lga_name = 'Greater Geelong (C)' |
| q15 | Which Twitter users posted the most crime-related tweets?No user data is stored: the right answer is a refusal. | refusal | refuse |
| q16 | What was the average sentiment of tweets in Sydney during 2024?The tweets cover February-July 2022 only: the right answer is a refusal. | refusal | refuse |
Saved runs (this browser)
No runs yet. Completed runs are saved here for comparison.
Every model call of a run is also in the audit log.
Design
How the scoring works
Execution match. The model's SQL goes through the same validator and read-only runner as the live feature. Its result counts as correct when it contains the gold result's columns (in any order, extra columns allowed) and the same set of rows: integers must match exactly, other numbers within 0.006, text ignoring case. Row order is not scored, so “top five” is about which five.
Refusals. Two questions ask for things the database does not hold (individual users; 2024). Producing SQL for them counts as a miss even if the SQL runs. Declining an answerable question counts as wrong.
Uncertainty. Proportions carry Wilson 95% intervals; median latency a bootstrap interval. Fourteen scored questions give wide intervals by design: this is a smoke test that catches a broken prompt or a weak model, not a leaderboard. Repeat a run to see run-to-run variation, and compare two runs with the exact McNemar test on the questions where they disagree and a Tango score interval for the accuracy difference. A reply that is malformed or cut off counts as wrong; only key, quota, rate-limit, network or provider failures are left out.
No published scores. The site has no AI budget and does not report results it ran itself. Your runs stay in this browser (export them as CSV or JSON). The questions, gold SQL and pinned gold answers are in web/src/lib/sql/benchmark.ts and its test. More in methods.