Engage Columbus Scrape-to-RSS
A GitHub Action that runs daily, scrapes the Engage Columbus sitemap, discovers new pages, and publishes an RSS feed of newly found links.
How It Works
This project runs a daily scraper which:
- Fetches
https://engage.columbus.gov/sitemapeach day. - Extracts links from the page
- Filters URLs to just columbus.gov URLs, removes URL parameters.
- Adds new URLs to a CSV.
- Generates an RSS feed of items
- If anything changed, the workflow commits and pushes the updated CSV and feed back to the repository.
Feed
The RSS feed is available at:
- https://raw.githubusercontent.com/benlk/engage-columbus-scrape-to-rss/refs/heads/trunk/feed.xml
- https://benlk.github.io/engage-columbus-scrape-to-rss/feed.xml
RSS item fields:
| Field | Source |
|---|---|
<title> |
Link text from sitemap at time of first discovery |
<link> / <guid> |
Normalized URL |
<pubDate> |
UTC timestamp of first discovery |
The feed is capped at the most-recent 500 items; this isn't likely to be hit any time soon.
Data
data/links.csv is the source of truth for all discovered URLs. Columns:
| Column | Description |
|---|---|
url |
Normalized URL (primary key) |
title |
Anchor text at time of first discovery |
first_seen_utc |
ISO 8601 UTC timestamp |
source |
Always sitemap, for now. |
run_id |
Timestamp of the run that discovered the URL |
Schedule
The workflow runs daily at 06:00 UTC. You can also trigger it manually:
- Go to Actions → Scrape Engage Columbus Sitemap.
- Click Run workflow.
Only one run executes at a time (concurrency lock). A run that finds no new URLs produces no commit.
To roll back a bad run: use git log -- data/links.csv to find the commit and git revert or git checkout <sha> -- data/links.csv feed.xml to restore a previous state.
AI usage statement
This was almost entirely vibe-coded, using Github Copilot. The decision to use Copilot was made in order to see what's possible with a from-scratch project.
Lessons learned:
- High-level project descriptions can be converted to functional requirements documents pretty easily.
- Copilot defaults to Python for building Github Action scrapers.
- Copilot defaults to PSR-12 when writing PHP.
- Copilot can be convinced to obey your
phpcs.xml, in time. - First-pass generated code often contains a number of archaicisms and bugs, which are only picked up after test-running the code and viewing the logs.
- First-pass generated
composer.jsonand GitHub Actions frequently refer to out-of-date versions of packages, and must be manually verified for current-ness. This is no doubt because of outdated training data.
License
Because of the AI usage, I've chosen to license this in the public domain, under The Unlicense.
Local Development
Run the Scraper
# Optional: validate PHP requirement metadata
php -v
# Run the scraper once
php main.php
# Diff, then commit changes to the scraped URLs.
Requires PHP 8.5
Build the site
composer installcomposer run buildphp -S localhost:8000 -t _site/to start a local server in that folder
GitHub Pages is deployed from the trunk branch by GitHub Actions. The Pages workflow also runs after successful scraper runs so feed updates committed by GitHub Actions are published.