COMP90024 · The University of Melbourne · 2023
In 2023 a five-person team built a cloud system that read 37,823,414 tweets and harvested 1,659,690 toots, scored their sentiment, pinned them to Victorian suburbs and set them against official income and crime statistics. This is that project, rebuilt so anyone can explore it.
Short on time? Take the guided tour: three captioned walkthroughs
1,085 Victorian suburbs · 719,336 geotagged tweets, Feb-Jul 2022
dot size = tweets more negative neutral more positive
The brief
Assignment 2 of COMP90024 asked each team to use the Melbourne Research Cloud to harvest social media, store it in a distributed database, analyse it at scale and present scenarios that compare online talk with official data from the Spatial Urban Data Observatory (SUDO), all deployed automatically.
Team 57 chose two questions: do people in higher- and lower-income areas talk about money differently, and does the tone of crime talk follow where crime is recorded?
How it worked
Scroll to follow a tweet through the original system. The cloud no longer exists; every number it produced has been recovered and checked.
01 · Harvest
MPI ranks split the 57 GB Twitter file by byte ranges and parse it line by line, while three Mastodon harvesters ask mastodon.social, mastodon.au and tictoc.social for the next 40 toots every ten minutes.
37.8M tweets read · 1.66M toots
02 · Score and store
Each post is cleaned, tokenised, lemmatised and scored by VADER on a 1-9 scale, matched to a suburb, then bulk-loaded 1,000 at a time into a CouchDB cluster replicated across three nodes.
2.42M geotagged tweets
03 · Summarise with MapReduce
CouchDB views emit every score keyed by suburb, once for all tweets and again for posts mentioning income or crime keywords, and a _stats reduce keeps count, sum, min and max.
3 views · 6,799 suburb summaries
04 · Serve and compare
A Flask API joins the view results with SUDO income and crime tables and ABS boundaries, renders Plotly figures, gzips them, and a React dashboard draws them side by side.
37-46 MB of gzipped Plotly JSON per map
05 · Automate
Ansible playbooks create the instances, volumes and security groups on the Melbourne Research Cloud, form the CouchDB cluster and deploy the services onto Docker Swarm, where they can be scaled with one script.
8 vCPUs · 500 GB · 6 instances
What the data says
Each card links to an interactive page with the full data and method.
48% of geotagged tweets score a neutral 5, and the positive side outweighs the negative. Suburb averages show no geographic pattern.
| Score | Count | Share |
|---|---|---|
| 1 (Extremely Negative) | 39726 | 1.6% |
| 2 (Very Strongly Negative) | 80238 | 3.3% |
| 3 (Strongly Negative) | 132745 | 5.5% |
| 4 (Negative) | 120487 | 5.0% |
| 5 (Neutral) | 1158765 | 47.9% |
| 6 (Positive) | 187854 | 7.8% |
| 7 (Strongly Positive) | 321908 | 13.3% |
| 8 (Very Strongly Positive) | 241710 | 10.0% |
| 9 (Extremely Positive) | 135184 | 5.6% |
Scenario 1 · Income
Across 167 SA2 areas, median income and the tone of income tweets barely move together (r = 0.14, explaining 1.9% of the variation). Income talk is concentrated in Melbourne's CBD.
Median income (x) vs average tone of income tweets (y), one dot per SA2 · dashed line: least squares
Scenario 2 · Crime
The most common score for crime tweets is 1 (extremely negative). But suburbs with more crime talk are not measurably gloomier (r = −0.05).
| Score | Count | Share |
|---|---|---|
| 1 (Extremely Negative) | 4436 | 24.9% |
| 2 (Very Strongly Negative) | 3205 | 18.0% |
| 3 (Strongly Negative) | 2382 | 13.4% |
| 4 (Negative) | 1457 | 8.2% |
| 5 (Neutral) | 3039 | 17.1% |
| 6 (Positive) | 910 | 5.1% |
| 7 (Strongly Positive) | 1058 | 5.9% |
| 8 (Very Strongly Positive) | 822 | 4.6% |
| 9 (Extremely Positive) | 487 | 2.7% |
Mastodon
mastodon.social looked almost uniformly neutral in 2023 while mastodon.au resembled Twitter. Re-scoring a surviving week shows why: non-English toots fall to neutral, and German 'die' reads as negative.
mastodon.social
| Score | Count | Share |
|---|---|---|
| 1 (Extremely Negative) | 7731 | 1.4% |
| 2 (Very Strongly Negative) | 8289 | 1.5% |
| 3 (Strongly Negative) | 5038 | 0.9% |
| 4 (Negative) | 15756 | 2.9% |
| 5 (Neutral) | 429414 | 78.5% |
| 6 (Positive) | 20428 | 3.7% |
| 7 (Strongly Positive) | 24447 | 4.5% |
| 8 (Very Strongly Positive) | 19356 | 3.5% |
| 9 (Extremely Positive) | 16460 | 3.0% |
mastodon.au
| Score | Count | Share |
|---|---|---|
| 1 (Extremely Negative) | 11331 | 2.1% |
| 2 (Very Strongly Negative) | 27363 | 5.0% |
| 3 (Strongly Negative) | 16426 | 3.0% |
| 4 (Negative) | 51299 | 9.4% |
| 5 (Neutral) | 232983 | 42.6% |
| 6 (Positive) | 33381 | 6.1% |
| 7 (Strongly Positive) | 75399 | 13.8% |
| 8 (Very Strongly Positive) | 46061 | 8.4% |
| 9 (Extremely Positive) | 52277 | 9.6% |
2026 upgrade
The revival added the questions a statistician would ask of the 2023 results, and a way to query the data in plain English that shows its working.
Spatial statistics, with intervals
Moran's I and a LISA cluster map with permutation tests, small-area suppression, bootstrap and robust intervals on both scenarios. The apparent clustering turns out to be noise from areas with a handful of tweets.
OpenAsk the data, with your own key
Your model writes SQL; the site validates it and runs it read-only; the answer cites the rows. Labelled, logged in your browser, and measurable with a 16-question benchmark.
OpenHands on
The original Python scoring code (NLTK's tokenisers, WordNet and VADER) has been ported to TypeScript and verified against the original on 44,156 real toots. Type a post and watch it become a number.
About this project
| 2023 original | 2026 revival | |
|---|---|---|
| Compute | Melbourne Research Cloud, 6 instances | Static pages + serverless functions (Vercel) |
| Data store | CouchDB 3.2 cluster, 3 nodes, ~400 GB | SQLite analytics.db, 1.7 MB, read-only |
| Processing | mpi4py, MapReduce views | uv scripts re-running the original code |
| Backend | Flask + Gunicorn, gzipped Plotly JSON | Next.js Server Components |
| Frontend | React 18, MUI, Plotly, Tailwind | Next.js 16, Tailwind v4, MapLibre, SVG |
| Maps | Plotly mapbox, 37-46 MB per map | OpenFreeMap tiles + 1.2 MB of TopoJSON |
| Sentiment | NLTK 3.8.1 in Python | Same algorithm, ported to TypeScript |
| Deployment | Ansible + Docker Swarm | git push |
Academic integrity. The original submission is preserved unchanged in the repository's coursework/ folder for reference. The revival rebuilds the presentation and re-runs the team's own code; it does not alter the methods or the conclusions. The assignment specification and course-provided raw data are not published here.