All projects

AI Engineering

Vision-LLM Event Extraction Pipeline

A scheduled pipeline that reads tour operators' Instagram carousels with a vision model and publishes a searchable outings site

Small tour operators around Medellín publish their trips only as Instagram carousels: dates, prices and itineraries baked into images. None of it is searchable. This pipeline reads those posts with a vision model, turns them into structured events, and publishes them to a public site, a Notion database and Google Calendar, twice a week.

ARCHITECTURE

  1. Fetch. Recent posts per operator are fetched in batches, under a charge cap; ticket sites have their own adapters.
  2. Extract. A vision model (through OpenRouter) reads each carousel into activities under a strict JSON schema. Post-validation clamps date ranges and rejects dates that fall before the post itself.
  3. Store. SQLite is the source of truth. Writes to Notion and Google Calendar are idempotent upserts keyed by deterministic event IDs, so a rerun never duplicates an event.
  4. Place. Each event is resolved to a municipality (aliases, the national municipality list, cached geocoding) with an estimated drive time from Medellín.
  5. Publish. Events and WebP thumbnails export to Cloudflare R2; an Astro site on Cloudflare Pages renders them, with a feedback form behind Turnstile.
  6. Schedule. A cron job runs the pipeline twice a week and sends a summary or failure alert to my phone.

MODEL CHOICE BY BENCHMARK

  • Test set: 4 real carousels, 58 slides.
  • Every candidate model found all 33 activities with correct names and prices; they differed only in how they wrote date ranges.
  • I picked the cheapest model that was perfect on the set: about $0.001 per post.

WHAT’S HARD

  • Dates. Models confuse an availability window (“June to August”) with a departure date. The schema and the post-validation rules handle it.
  • Transient failures. A second pass retries failed posts; a post that fails three times is marked and skipped instead of blocking the run.
  • Safe publishing. The export refuses to publish if the event list would shrink below half of what’s live, and orphaned images get a 7-day grace period before cleanup.

RESULTS

  • Live public site, refreshed twice a week without manual work.
  • About $0.001 per post in extraction cost.
  • 65 automated tests, none of which touch the network.