Engage Columbus Scrape-to-RSS

-> subscribe to feed <-

A GitHub Action that runs daily, scrapes the Engage Columbus sitemap, discovers new pages, and publishes an RSS feed of newly found links.

How It Works

This project runs a daily scraper which:

  1. Fetches https://engage.columbus.gov/sitemap each day.
  2. Extracts links from the page
  3. Filters URLs to just columbus.gov URLs, removes URL parameters.
  4. Adds new URLs to a CSV.
  5. Generates an RSS feed of items
  6. If anything changed, the workflow commits and pushes the updated CSV and feed back to the repository.

Feed

The RSS feed is available at:

RSS item fields:

Field Source
<title> Link text from sitemap at time of first discovery
<link> / <guid> Normalized URL
<pubDate> UTC timestamp of first discovery

The feed is capped at the most-recent 500 items; this isn't likely to be hit any time soon.

Data

data/links.csv is the source of truth for all discovered URLs. Columns:

Column Description
url Normalized URL (primary key)
title Anchor text at time of first discovery
first_seen_utc ISO 8601 UTC timestamp
source Always sitemap, for now.
run_id Timestamp of the run that discovered the URL

Schedule

The workflow runs daily at 06:00 UTC. You can also trigger it manually:

  1. Go to Actions → Scrape Engage Columbus Sitemap.
  2. Click Run workflow.

Only one run executes at a time (concurrency lock). A run that finds no new URLs produces no commit.

To roll back a bad run: use git log -- data/links.csv to find the commit and git revert or git checkout <sha> -- data/links.csv feed.xml to restore a previous state.

AI usage statement

This was almost entirely vibe-coded, using Github Copilot. The decision to use Copilot was made in order to see what's possible with a from-scratch project.

Lessons learned:

License

Because of the AI usage, I've chosen to license this in the public domain, under The Unlicense.

Local Development

Run the Scraper

# Optional: validate PHP requirement metadata
php -v

# Run the scraper once
php main.php
# Diff, then commit changes to the scraped URLs.

Requires PHP 8.5

Build the site

  1. composer install
  2. composer run build
  3. php -S localhost:8000 -t _site/ to start a local server in that folder

GitHub Pages is deployed from the trunk branch by GitHub Actions. The Pages workflow also runs after successful scraper runs so feed updates committed by GitHub Actions are published.