Community Media Archive & Archive Corps

Community Media Archive & Archive Corps — Project Notes

The strategic shift

What "Move focus from Community Media Archive operations to CMA Stewardship" means:

• Move off operations, onto the underdeveloped/unexplored aspects
• Bring the community in to understand the resource they have and what it could become
• Drop the "that video's not going to archive itself!" pressure

Why Archive Corps matters (the value proposition to participants):

• Discover a passion or interest without a scary commitment — no career change, no months of full-time retraining, no degree that might end in "I don't like this field"
• Earn peer recognition for real contributions
• Contribute at scale to something that preserves this community's work and history, and gives researchers a resource for studying community and local civic media

The integration question: how does Archive Corps map onto operating / feeding / improving / extending the beast?

Curriculum: IA interface fundamentals first

Taught with real examples, three levels:

1. GUI
2. Search → advanced search
3. CLI

Rationale: you can't understand how metadata decisions affect discovery, findability, searchability, and usability until you've worked the retrieval side. Only then does the link between what you put in and how people find it click.

Outcomes: participants can archive their own media — born-digital, whether or not it's on YouTube/Vimeo. Also a jumping-off point for physical digitization projects.

Then: operating the CMA. Worked example — default TelVue archiving module vs. exporting metadata alongside the videos.

CMA work areas

Operating the beast

• Onboarding new YouTube channels
• Reporting to operators, fleet managers, channel owners

Feeding / expanding the fleet

• New worker process for Linux/Mac/Windows that doesn't require deep Linux tooling knowledge
• Migrate existing fleet to the new Linux process

Incorporating Agendas & Minutes

• Existing corpus: ~800,000 CivicPlus documents, ~1,500 communities, run 2022–2023 covering 2018–2022, via BigLocalNews civic-scraper
• CivicPlus UI has changed; the 2022 scraper code no longer works
• Need extractors matching current UIs for the common agenda vendors in civic.db
• Need a way to track vendor changes over time

Improving the beast — metadata grooming bots

• Language attribution: mark Spanish based on description text, not naive title matching
• Clean \r and \" in descriptions
• Automated transcript local dictionaries

Infrastructure

• Captioning machines: Faster-Whisper-XXL on cards that need newer Nvidia software than 3060 generation can support

Extending the beast — volunteer systems

• Microtask distribution, volunteer direction, recognition tracking

The biggest unfinished areas

1. Volunteer infrastructure — microtasks, recognition, coordination. Nothing else in Archive Corps works without it.
2. The onboarding curriculum — IA fundamentals → CMA operation. Currently a sketch.
3. Agendas/minutes rebuild — highest-value corpus, currently broken.
4. Cross-platform worker process — the gate on fleet expansion beyond Linux-fluent people.

↑ back to contents00.0-stepping-back-from-CMA-operations/cma-notes.md

Civic Data Pipeline (Cities)

A pipeline that, for every incorporated city/town in a U.S. state, discovers its official website, its YouTube channel, and its Agendas & Minutes page (with any copyright statement). All data lives in a single SQLite database (civic.db by default) with one row per place across every state. CSV files are used only for review and hand-editing, via export and import.

Data source: the U.S. Census Bureau 2024 Places Gazetteer.

Files

All files must live in the same directory (civic.py imports the others).

Requirements

• Python 3.8+ (SQLite is in the standard library)
• pip install requests beautifulsoup4

Quick start

# Stage 1 — discover official sites for WA (downloads/caches the gazetteer,
# caches every homepage it fetches). Writes into civic.db.
python3 civic.py scrape-sites --state WA \
    --gazetteer ~/civic/gazetteer.zip --cache-dir ~/civic/html_cache

# Stage 2 — YouTube channels (reuses cached homepages, no re-fetch).
python3 civic.py scrape-youtube --state WA --cache-dir ~/civic/html_cache

# Stage 3 — Agendas & Minutes + copyright.
python3 civic.py scrape-agendas --state WA --cache-dir ~/civic/html_cache

# See coverage across everything in the database.
python3 civic.py stats

# Get a CSV of the finished data.
python3 civic.py export --state WA --out wa_final.csv

Use --db path/to/other.db on any command to target a different database (default is civic.db in the current directory). Process more states by re-running scrape-sites --state XX; they all accumulate in the same DB.

The --cache-dir and --gazetteer flags above are optional — both default to ~/.civic_cache/ (the homepage cache lives in ~/.civic_cache/html/), so you can omit them entirely and caching still works across runs. Pass them only to use a custom location.

Output schema

These columns live in the places table (and in every exported CSV). scrape-sites fills the first group; scrape-youtube and scrape-agendas fill theirs. The database also keeps an updated_at timestamp per row (not exported).

A blank official_site (confidence unverified) means no site cleared the verification bar; the youtube/agenda stages skip those rows.

How detection works (brief)

• Stage 1 guesses common municipal URL patterns (cityofX.gov, Xwa.gov, X-wa.gov, ci.X.wa.us, townofX.com, …), type-aware (towns try townof… first). A fast DNS pre-check skips hosts that don't resolve. Each responding candidate is scored against the target city and state (name in title/page, state in domain/page, .gov host, wrong-state penalties) so a same-named city in another state or a domain squatter is rejected. Hard rejects (no signal can rescue them): real-estate portals (nhrealestate.com, *realty*, Zillow/Redfin/Coldwell Banker…), domain marketplaces (afternic, hugedomains, expireddomains, */buy-domain/*, */domain/*), and parked/for-sale pages — these score well on name+state signals (a listing for "Nashua, NH" has the city, state, and an address), so they are refused outright rather than penalized. A .gov host is never hard-rejected.
• A name match alone is not a match. A generic site that merely carries the city's word in its title (liberty.org for Liberty TX, commerce.gov for Commerce TX, scotland.org for Scotland SD) is rejected, not stored at low confidence. This matters: a bad match occupies the site field, so needs=site skips the row forever and the real domain is never found — a blank is strictly better. Corroboration is required: state evidence, or a governmental signal that isn't just a bare <name>.gov (federal departments live at bare <name>.gov, so that alone proves nothing). Wrong-state defenses are strict: a domain or <title> announcing a different state (adjuntas-ny.gov, "Springfield, MO" when seeking IL) is penalized heavily; street addresses are ZIP-verified against the target state (a city-hall address whose ZIP places it in another state is a strong rejection, while a ZIP-confirmed in-state address is strong acceptance evidence); and a candidate with no positive evidence of the right state can't reach "confident" on name + .gov host alone — it's capped to "low" and flagged rather than accepted. A bare ", ST" token (e.g. a sister-city link) no longer counts as state evidence on its own.
• Stage 2 scans the homepage for YouTube links — including social-icon bars, data-* attributes, entity-encoded and protocol-relative URLs. If the homepage has none, it follows the most promising internal subpages ("Watch Meetings", "Live Stream", "Connect With Us", …) and scans those (youtube_source='subpage'). Every found channel is then fetched once to (a) canonicalize it — @handle / /user/ / /c/ forms resolve to the stable /channel/UC… id the wider ecosystem joins on — and (b) verify it belongs to this place: a channel page with zero overlap with the place name and no civic vocabulary (e.g. a linked news station) is dropped rather than stored; an unreachable channel page keeps the URL as found (benefit of the doubt). Stage 2 also — and ranks real channels above shared videos. An optional search fallback (off by default; see Extending) is verified before use.
• Stage 3 finds the link whose text/href best matches "Agendas & Minutes" / "Minutes & Agendas" (combined phrases rank highest), fetches it, extracts a copyright statement, and identifies the agenda portal vendor (Granicus, Legistar, CivicPlus, CivicClerk, PrimeGov, Municode, …) from the URL — the routing key for downstream extraction.

Which places count: governance models and data sources

The hard part of "list every local government" isn't scraping — it's that the United States has no single, uniform tier of local government. Most states have incorporated cities and towns, but several do not, and the Census Bureau splits these different structures across different Gazetteer files. A naïve tool that only reads the "Places" file silently returns almost nothing for those states.

This pipeline handles the common structural cases with a per-state source profile, choosing the right Gazetteer file(s) automatically:

Each row records which file it came from in the source column (place / cousub / county), and Stage 1 uses source-appropriate URL patterns (e.g. Xcounty.gov and co.X.st.us for counties, townof… first for New England towns).

Concretely, this is the difference between scrape-sites --state MA returning ~350 town + city governments instead of just the ~58 cities, and --state HI returning Hawaii's 5 counties — or --state PR returning Puerto Rico's 78 municipios — instead of nothing usable.

Known limitation: township states are not yet special-cased

States with civil townships — e.g. NY, NJ, PA, OH, IN, IL, MI, WI, MN, MO, KS, ND, SD, NE — are currently treated as DEFAULT (Places file only). In some of these, townships exercise real governing authority (and have their own websites and meeting agendas); in others they are weak or nominal. Because that authority varies so much, they are intentionally left for a later refinement rather than swept in wholesale. For now, a run of one of these states covers its incorporated cities/villages but not its townships. If you need township coverage for a specific state, that state's governments live in the County Subdivisions file and could be added to the NEW_ENGLAND-style merge — ask, or edit STATE_SOURCE_PROFILE and the COUSUB_GOVERNING_SUFFIXES map in civic_common.py.

Territories

Puerto Rico (PR) and Hawaii use the COUNTY profile (see the table above). The other U.S. territories — Guam (GU), the U.S. Virgin Islands (VI), American Samoa (AS), and the Northern Mariana Islands (MP) — each have their own governance structure and are not yet profiled; running them falls through to DEFAULT and may return statistical areas rather than governments, exactly as Puerto Rico did before it was added. They can be profiled the same way once their Gazetteer structure is verified.

The county layer (opt-in)

By default the pipeline is municipal-focused: a DEFAULT state returns its incorporated cities/towns, and counties are pulled only for COUNTY- profile states (HI, PR) where the county is the primary government. Counties everywhere else are not included unless you ask for them.

To add the full county layer, use the opt-in scrape-counties command:

python3 civic.py scrape-counties --state TX      # one state's counties
python3 civic.py scrape-counties --all-states    # every state's counties

County rows carry source='county' and coexist with the municipal rows (different GEOIDs — a "Los Angeles city" place row and a "Los Angeles County" county row both live in places, distinguishable by source). All county equivalents are handled: Louisiana parishes, Alaska boroughs / census areas / municipalities, and Virginia independent cities. Because the source column separates them, you can export just cities, just counties, or both.

Two honest caveats. First, adding counties roughly doubles the row count and changes what the database represents (municipal → municipal + county). Second, county site-discovery is weaker than municipal — county domains are highly varied, so expect a higher "no site" rate than for cities even with the county-specific URL patterns. County rows with no site can be corrected via the same export → edit → import round-trip.

More broadly: "local government" is genuinely fuzzy. Special districts (school, water, fire), tribal governments, and independent authorities are out of scope entirely. This pipeline models the general-purpose municipal and county tier, which covers the vast majority of what people mean by "city/county websites," but is not a complete census of every governing body in the U.S.

Re-running just the failures

Re-running is automatic and built into the scrape commands: each one only processes rows that still need it. scrape-sites does cities with no official_site; scrape-youtube does sites with no youtube_channel; scrape-agendas does sites with no agenda_page. Run the same command again and it picks up only the gaps — already-filled rows are skipped, and manual rows are never touched.

python3 civic.py scrape-sites   --state WA   # fills only missing sites
python3 civic.py scrape-youtube --state WA   # fills only missing channels
python3 civic.py scrape-agendas --state WA   # fills only missing agendas

Each scrape writes results row-by-row inside transactions, so an interrupted run leaves the database consistent and the next run resumes from the gaps. Omit --state on scrape-youtube/scrape-agendas to process every state in the database at once.

Manual corrections — the export → edit → import round-trip

Automated discovery will miss or mis-identify some sites. You correct them by editing a CSV exported from the database, then importing it back. Manual rows are authoritative and are never overwritten by future scrapes.

Workflow:

1. Export the rows you want to review. You can target exactly the gaps:

   python3 civic.py export --state WA --needs site --out wa_fix.csv

   (Drop --needs to export everything; drop --state to export all states.)

2. Open wa_fix.csv in a spreadsheet and fix the bad rows: set official_site to the correct URL and set match_confidence to manual. The score is normalized to 9 for you. Works for both "no site" rows and wrong-URL rows. Don't change the geoid — it's the match key.

3. Import the edits back into the database:

   python3 civic.py import --in wa_fix.csv --cache-dir ~/civic/html_cache

   Each manual URL is fetched once to confirm it's reachable and to populate the HTML cache (so the YouTube/agenda stages work without re-fetching). A URL that doesn't resolve (or a manual row with no URL) is flagged manual-unreachable and listed in a warning.

Why your edits survive: manual rows live in the same database as everything else, distinguished only by match_confidence = manual. The gazetteer upsert in scrape-sites explicitly skips manual rows, and the scrape stages only process rows still missing data — so a manual correction is never re-scraped or overwritten. There is no sidecar to manage; the database is the durable store.

Correcting YouTube channels and agenda pages

The same round-trip fixes a wrong or missing YouTube channel or agenda page, locked independently of the site (and of each other):

1. export the rows (e.g. --needs youtube or --needs agenda, or just the whole state).
2. In the CSV, set the correct value and mark its source column manual:
   • YouTube: set youtube_channel and youtube_source = manual.
   • Agenda: set agenda_page and agenda_source = manual.
3. import the CSV. These values are trusted as entered — not fetched or validated (unlike manual site URLs, which drive both later stages and so are still validated + cached).

A manual source locks that field: scrape-youtube / scrape-agendas skip it on every future run. This also lets you assert a negative — set youtube_channel to blank with youtube_source = manual to record "this city has no channel; stop looking." Each field is independent, so you can lock a channel while leaving the agenda open for the scraper, or vice versa.

Audit your corrections at any time (read-only):

python3 civic.py list-manual            # all states
python3 civic.py list-manual --state WA # one state

This prints every manual / manual-unreachable site row. The stats command shows per-field manual counts (mSITE, mYT, mAG) so you can see how many channels and agendas you've locked.

Zeroing out a contaminated row

If a row's official_site is flat wrong (e.g. a same-named city in another state), the YouTube and agenda data derived from it are wrong too. clear wipes the site and everything derived from it in one step, and (with --cache-dir) drops the now-wrong cached homepage:

# Blank site + youtube + agenda, leave the row OPEN for re-scraping:
python3 civic.py clear --geoid 7250123 --cache-dir ~/civic/cache

# Blank everything and LOCK it so the scraper won't re-fill it
# (use when you'll fix it by hand, or there is no correct site):
python3 civic.py clear --geoid 7250123 --lock

Without --lock the next scrape re-attempts the row. With --lock the site/youtube/agenda fields are set to manual so all three stages skip it; set the correct values later via the export → edit → import round-trip, or run clear again without --lock to reopen it.

Why was a verified URL flagged manual-unreachable? The importer fetches each manual site URL once; it only passes if the response is HTTP 200 with more than ~500 characters of HTML. A URL that works in your browser can still fail this if the site blocks non-browser User-Agents (a 403 from a WAF), is JavaScript-rendered (the raw HTML is a near-empty shell), or needs a scheme /www. you didn't include. The flag does not discard your URL — it's kept and still flows to later stages; only the one-time homepage caching was skipped. To see the exact reason per URL:

python3 civic.py diagnose                      # every manual-unreachable row
python3 civic.py diagnose --geoid 5377105      # one place by geoid
python3 civic.py diagnose --url https://x.gov  # any URL directly

It reports each step (scheme, DNS, HTTP status, body size), gives a verdict, and retries with a browser-like User-Agent to tell you whether that alone would fix it.

Automatic recovery from User-Agent blocks: the fetcher identifies itself honestly (Civic-Research/1.0) by default, but if a request is refused with a 401/403/429 it automatically retries that one request with a browser-like User-Agent. So sites behind a WAF that blocks non-browser agents are handled transparently — you don't need to do anything. If some sites were flagged manual-unreachable before this behavior existed, just re-import the same CSV: the rows are re-validated and flip to manual once they fetch.

Mislabeled status codes: some municipal servers and bot-mitigation layers return a full, real page under an odd non-200 status (e.g. a 415 with 400 KB of HTML — observed on Puerto Rico municipio sites). The fetcher accepts any response carrying a substantial body, so these pages are used rather than discarded on a status technicality, while genuine error pages (small bodies) stay rejected.

Non-English pages: site verification recognizes Spanish-language civic terms ("municipio de", "gobierno municipal", "alcaldía", "asamblea municipal"), so a legitimate Puerto Rico municipio page (often a bare <name>.com or <name>.pr.gov) clears the confidence bar — while an unrelated same-named .com with no civic content is still rejected.

Caching

The pipeline caches two things to avoid redundant network work.

Gazetteer (--gazetteer PATH)

The pipeline may use up to three national Gazetteer files (Places, County Subdivisions, Counties — see Governance models above). Each is downloaded at most once and cached on disk; subsequent runs (any state) load from disk.

--gazetteer accepts either a directory (the three files are stored inside it under their standard Census names) or a specific .zip (used as the Places file, back-compat; the other two are derived as siblings in the same directory).

• Default location: ~/.civic_cache/ (e.g. ~/.civic_cache/2024_Gaz_place_national.zip, …cousubs…, …counties…).
• Override with --gazetteer on scrape-sites, or set GAZETTEER_PATH.

Only the file(s) a given state needs are downloaded — a DEFAULT state fetches just Places; a New England state also fetches County Subdivisions; Hawaii and Puerto Rico fetch Counties. Only scrape-sites reads the gazetteer.

Homepage HTML (--cache-dir DIR)

scrape-sites saves each official homepage it fetched (keyed by geoid, as DIR/<geoid>.html); scrape-youtube and scrape-agendas read homepages from the same DIR instead of re-fetching — turning three homepage fetches per city into one. import also caches the homepage of each validated manual URL.

This cache is on by default: if you don't pass --cache-dir, it defaults to ~/.civic_cache/html/ (a flat html/ subdirectory alongside the gazetteer cache), so caching "just works" across runs. Files are stored flat — GEOIDs are globally unique, so no per-state subdirectories are needed. Override the location with --cache-dir, or set the CIVIC_CACHE_DIR environment variable.

Cache control flags on Stages 2 and 3:

Note: only homepages are cached. scrape-agendas still fetches each discovered agendas subpage live, since that URL isn't known until runtime.

All command-line options

Global: --db PATH (database to use, default civic.db) goes before the subcommand: python3 civic.py --db wa.db stats.

The three scrape-* commands share: --cache-dir DIR, --workers N (default 8), and a selection option for partial runs:

• --limit N — the first N items (deterministic).
• --sample N — N items chosen at random (sorted by geoid). Mutually exclusive with --limit.
• --seed S — fix the random seed for --sample (reproducible subset).

scrape-sites

--state STATE     USPS code or full name. Required.
--gazetteer PATH  Gazetteer zip location (load if present, else download+save).
--cache-dir DIR   Save fetched homepages here for the youtube/agenda stages.

export (with population filters)

--state STATE     One state (default: all).
--needs FIELD     Only rows missing site / youtube / agenda.
--min-pop N       Only rows with population >= N   (applied live, no reload).
--max-pop N       Only rows with population <= N.
--tier XL,L,...   Only these size buckets (XL/L/M/S/XS).
--sort-pop        Order biggest-first (unknown population last).
--format FMT      Field delimiter: csv (default), psv (vertical bar '|'),
                  or tsv (tab). For psv/tsv the delimiter is also stripped
                  from values so every row splits cleanly for naive tools.
--psv             Shorthand for --format psv.
--out FILE        Destination file. Required.

Population thresholds are computed at query time from the stored number, so a different cutoff is just a different --min-pop — never a reload. Volunteer triage — biggest governments still missing an agenda, largest first:

python3 civic.py export --needs agenda --min-pop 25000 --sort-pop --out gaps.csv

Pipe-delimited (PSV) output for tools that prefer it — the | character is stripped from any field value first, so a scraped description containing a pipe can't shift columns:

python3 civic.py export --psv --out all.psv          # or --format psv

CSV output is plain, greppable text

Scraped fields (agenda_description, copyright_statement, …) come from arbitrary HTML and can carry control characters — CR (^M), shift-out/in (^N/^O), NUL. Left alone these make grep report "binary file matches" and break normal text tooling. Every CSV this pipeline writes is therefore sanitized: embedded CR/LF collapse to a space (so one row is always one physical line), other control bytes are removed, and lines end with a plain Unix \n rather than the CSV-spec \r\n. Values are otherwise preserved.

Data scraped before this was added is still dirty in the DB. Scrub it in place (no re-scrape needed):

python3 check_rescrape.py --sanitize-text         # dry run: what's affected
python3 check_rescrape.py --sanitize-text --yes   # scrub

adjudicate-dupes (who does this domain actually belong to?)

--state STATE    Limit to one state.
--report FILE    Full per-claimant scoring to CSV.
--clear-losers   Blank the non-owning rows (needs --yes).
--yes            Confirm the clear.

dupe-sites shows which rows share a domain; this answers who owns it. Each duplicated domain is fetched once and scored with the current verifier against every claiming place:

• WINNER — one place verifies; the others are cleared and reopened for re-scrape.
• NO-OWNER — nobody verifies (a generic site like liberty.org, or a real-estate/domain-sale host). All claimants are cleared.
• TIE — several verify equally; left for a human.

python3 civic.py adjudicate-dupes --report adj.csv          # dry run
python3 civic.py adjudicate-dupes --clear-losers --yes      # apply
python3 civic.py scrape-sites --state TX                    # re-find the real ones

dupe-sites (find shared-domain false positives)

--state STATE  Limit to one state (default: all).
--top N        How many duplicated domains to print (default 40).
--report FILE  Write EVERY individual row behind each duplicate to CSV.

A domain used by more than one place is nearly always a false positive. The report assigns each a verdict:

• REJECT-ALL — a known non-government host (real-estate portal, domain marketplace) or a domain-sale URL. Every row using it should be cleared.
• REVIEW — a plausible municipal domain shared by several places (the "same city name, wrong state" case). One row probably owns it; the rest need clearing. The CSV includes population / pop_tier so a volunteer can see which place most plausibly owns the domain.

python3 civic.py dupe-sites                            # summary
python3 civic.py dupe-sites --report dupes.csv         # + every row, for review
# then clear the wrong ones:
python3 civic.py clear --geoid <geoid>                 # reopen for re-scrape
python3 civic.py clear --geoid <geoid> --lock          # keep empty

reverify (re-score existing matches)

--state STATE     Limit to one state (default: all non-manual matched rows).
--yes             Write updated confidence/scores (default: dry-run report).
--clear-failed    With --yes: clear rows now below the 'low' floor —
                  site + derived youtube/agenda fields blanked, cache entry
                  removed, row reopened for re-scrape. Manual rows are
                  never touched.
--refresh / --max-age DAYS   Homepage cache staleness controls.

Verification logic improves over time, but scrape-sites only visits rows with a blank site — old matches are never re-examined. reverify re-scores every already-matched site with the current verifier, reading homepages from the cache (live fetch only on a cache miss), and reports upgrades, downgrades, and outright failures (e.g. wrong-state false positives caught by newer checks). Typical flow after a verifier improvement:

python3 civic.py reverify --state WA              # dry run: see what changes
python3 civic.py reverify --state WA --yes --clear-failed
python3 civic.py scrape-sites --state WA          # re-discover cleared rows

load-backflush-towns (surgical township add)

--in FILE     CSV with state + city_name columns — typically the
              <report>_unmatched.csv from import-backflush.
--gazetteer P Gazetteer cache location (dir or .zip).
--yes         Add the rows (default: dry-run report).

For the township-belt states (NY/PA/MI/NJ/WI/…), a back-flush "Town of X" row often has no match because only the places layer was loaded, not the County Subdivisions (cousub) layer where townships live. Loading a whole state's townships would add thousands of mostly-unwanted rows, so this command does it surgically: it treats the unmatched list as a whitelist and adds only those specific towns, looked up in the cousub gazetteer. Townships parse as place_type='town', source='cousub'. Rows not found in the cousub file (CSV spelling variants, non-cousub places) are reported and skipped. Typical A-pass flow after an import-backflush dry run:

python3 civic.py load-backflush-towns --in bf_review_unmatched.csv         # preview
python3 civic.py load-backflush-towns --in bf_review_unmatched.csv --yes   # add towns
python3 civic.py import-backflush --in civic_backflush_2026-05-26.csv --yes # attach channels

import-backflush (CMA channel back-flush)

--in FILE   CMA back-flush CSV (dircode, state, city_name, org_name,
            video_channel, channel_platform, ia_collection_url, in_cma, …).
--yes       Apply (default: dry-run report).
--report F  Write the FULL unmatched + ambiguous + resolved lists to
            F_unmatched.csv / F_ambiguous.csv / F_resolved.csv. Unmatched rows
            carry a fuzzy 'did_you_mean' nearest-city suggestion.
--apply-disambiguated
            Also apply rows the org-name rule resolves ('City of X' picks the
            city row over a same-named town/village/cousub; place-source is the
            tiebreak). Needs --yes. Rows the rule can't resolve stay ambiguous.

Imports archive-verified video channels so onboarding volunteers don't re-derive them. Filters (case-insensitive): rejects any org_name containing Library / School / University / College; requires a municipal signal in org_name — city or town, or a consolidated-government word (consolidated / metro / metropolitan / unified / municipal) so consolidated city-counties (Augusta, Nashville, Louisville, Athens…) match. Plain counties/parishes/boroughs — not loaded outside HI — are still excluded. Matching normalizes names, so a backflush "Augusta" reaches the places-file row "Augusta-Richmond County consolidated government (balance)". Matches by (state, city_name); ambiguous city+state (more than one row) is skipped and reported. Stores video_channel + channel_platform + channel_source='cma_backflush', keeps youtube_channel in sync for YouTube rows, and records in_cma / cma_collection_url. Imported channels are locked (the crawler skips youtube_source='cma_backflush') and never clobber an existing or manually-locked channel (fill-only-where- empty). Dry-run first to see the filter/match breakdown:

python3 civic.py import-backflush --in civic_backflush_2026-05-26.csv        # preview
python3 civic.py import-backflush --in civic_backflush_2026-05-26.csv --yes  # apply

load-population (opt-in, run once)

FILES...          One or more 2020 Decennial population files (place / county
                  P1 tables; CSV or pipe/tab-delimited; prefixed GEOIDs OK).
--threshold N     Population at/above which pop_over_threshold='yes'
                  (default 10000).
--yes             Write the values (default: dry-run report).

Joins population onto existing rows by GEOID — it never adds places, so run it after your places exist. It matches any geography whose GEOID is in your DB, so to cover incorporated places and New England cousub towns and counties, load all three 2020 Decennial P1 files (Place, County Subdivision, County); the loader keys purely on GEOID, so each row is filled from whichever file contains it.

Only the raw population number and its pop_tier bucket are stored. Any population threshold is applied at retrieval time, not baked in at load — so you never reload to change a cutoff (see export --min-pop below). Get the files from data.census.gov (2020 Decennial table P1, "Total Population", for geographies Place and County), or the Census API. The loader auto-detects the GEOID and P1_001N columns and strips the 1600000US… / 0500000US… prefixes. Dry-run shows the tier distribution and how many rows clear the threshold before writing.

Volunteer-triage example — export the biggest cities still missing an agenda:

python3 civic.py load-population place_P1.csv county_P1.csv --yes
python3 civic.py export --needs agenda --out gaps.csv
#   then sort/filter gaps.csv by pop_tier / pop_over_threshold

scrape-counties (opt-in county layer)

--state STATE     One state's counties (mutually exclusive with --all-states).
--all-states      Load counties for every state.
--gazetteer PATH  Gazetteer cache location.
--cache-dir DIR   Save fetched homepages here.
(also supports --limit / --sample / --seed / --workers)

scrape-youtube / scrape-agendas

--state STATE     Limit to one state (default: every state in the DB).
--cache-dir DIR   Read homepages from cache instead of fetching.
--refresh         Re-fetch homepages live, ignoring the cache.
--max-age DAYS    Re-fetch cached homepages older than DAYS.

export

--state STATE     Limit to one state (default: all).
--needs FIELD     Only rows missing this field: site | youtube | agenda.
--out PATH        Output CSV path. Required.

import

--in PATH         CSV to import (must have a geoid column). Required.
--cache-dir DIR   Cache homepages of validated manual URLs here.
--workers N       Concurrency for manual-URL validation.

list-manual / stats

list-manual [--state STATE]   Print manual / manual-unreachable rows.
stats                         Per-state coverage summary.

Typical workflows

Process several states into one database, gazetteer downloaded once:

for st in WA OR CA; do
  python3 civic.py scrape-sites --state $st \
      --gazetteer ~/civic/gaz.zip --cache-dir ~/civic/cache
done
python3 civic.py scrape-youtube --cache-dir ~/civic/cache   # all states
python3 civic.py scrape-agendas --cache-dir ~/civic/cache   # all states
python3 civic.py stats

Refresh a months-old dataset cheaply (only re-fetch homepages older than 30 days; everything fresh is reused):

python3 civic.py scrape-youtube --cache-dir ~/civic/cache --max-age 30
python3 civic.py scrape-agendas --cache-dir ~/civic/cache --max-age 30

Test on a handful of cities first:

# First 10 (deterministic):
python3 civic.py scrape-sites --state WA --limit 10

# A random 10, reproducible via the seed (unbiased spot-check across the
# whole alphabet rather than just A-named cities):
python3 civic.py scrape-sites --state WA --sample 10 --seed 42

Politeness & good citizenship

• Requests are rate-limited per host and run under a worker pool, so unrelated city sites are hit concurrently but no single host is hammered.
• Set a real contact address in the USER_AGENT string in civic_common.py before large runs.
• Consider checking each site's robots.txt. These are public .gov sites, but courtesy still applies.

Extending

civic_common.py contains two stubs you can implement to raise hit rates:

• search_fallback(query) — return a best-guess official-site URL from a search API (used by Stage 1 when URL guessing fails).
• youtube_search_fallback(place_name, usps) — return a channel URL from a search API (used by Stage 2). Results are passed through verify_youtube_candidate() before being trusted.

Both return None by default, so the pipeline runs without API keys.

Known limitations

• URL-pattern discovery can't find sites on domains that follow no convention; implement search_fallback for the long tail.
• JavaScript-rendered homepages may hide links from a plain HTTP fetch; those need a headless browser (not included).
• The HTML cache never expires on its own unless you pass --max-age (or --refresh); a redesigned site keeps serving the old cached copy.
• Township states are not yet special-cased — NY/NJ/PA/MI/WI/etc. return their incorporated cities/villages but not their civil townships. See Which places count above. New England towns and Hawaii's counties are handled via per-state source profiles.
• Scope is the general-purpose municipal + county tier. Special districts (school/water/fire), tribal governments, and independent authorities are out of scope.

↑ back to contents10.0-civic-data-pipeline(22)/civic-data-pipeline/README.md

IA Metadata XML → SQLite / Datasette

Ingest Internet Archive _meta.xml files (~3.2M items across ~2,200 collections) into a SQLite database for analysis with Datasette. Handles a weekly incremental sync of new/modified items and new collections, plus out-of-band detection of items that Internet Archive has "made dark."

Bundle layout

This bundle is organized so the documents are easy to send out for review and the code stays separate:

README.md                 ← you are here (entry point)
docs/                     markdown sources of every document
  docs/pdf/               the same documents as PDF   (for reviewers / posting)
  docs/docx/              the same documents as Word   (for academics & comment)
code/                     all runnable code, config, and fixtures
  code/conf/              skip-list, directory→collection map, secrets example
  code/fixtures/          sample caption files used by the caption design

Every document under docs/ is provided in all three formats — edit the markdown; the docx and PDF are regenerated from it by code/make_bundle.sh. (OVERVIEW is the one exception: its Word/PDF are built by a small dedicated script for a tight two-page layout.) File names below are given without their code/ or docs/ prefix for brevity.

Contents

Quick start

pip install lxml datasette

# 1. Build the directory->collection map
python3 build_map.py --root /mnt/md0/.../archive_stats \
    --out conf/directory_collection_map.psv --jobs 16 --include-identity

# 2. Initial bulk load (once)
python3 bootstrap.py --root /mnt/md0/.../archive_stats \
    --db /mnt/md0/scratch/archive.build.db --shard-dir /mnt/md0/scratch/shards \
    --map conf/directory_collection_map.psv --skip conf/skip.conf --jobs 36

# 3. Weekly incremental (cron)
python3 sync.py --root /mnt/md0/.../archive_stats --db /mnt/md0/datasette/archive.db \
    --map conf/directory_collection_map.psv --skip conf/skip.conf --auto --jobs 36

# 4. Dark reconciliation (separate cadence; needs IA keys)
set -a && . conf/ia_secrets.env && set +a
python3 dark_reconcile.py --db /mnt/md0/datasette/archive.db \
    --map conf/directory_collection_map.psv --skip conf/skip.conf --fraction 1.0

# 5. Serve
datasette -i /mnt/md0/datasette/archive.db --metadata datasette.yaml

Full procedures — including the build-then-swap publish flow and recovery scenarios — are in RUNBOOK.md. Read DESIGN.md first for the why.

Requirements

Python 3.10+, lxml. Everything else is stdlib (sqlite3, hashlib, urllib, concurrent.futures). Datasette for serving.

Security note

The dark job authenticates to Internet Archive with S3-style keys. Keep them in conf/ia_secrets.env (chmod 600), never in code or version control. If a key is ever exposed, rotate it at https://archive.org/account/s3.php.

↑ back to contents10.1-ia_metadata_xml_to_sql(26)/ia_metadata_xml_to_sqlite/README.md

Distributed Archive Worker Dashboard

A monitoring system for ~10 distributed Linux "worker" machines that download YouTube/Vimeo channels via yt-dlp (orchestrated by GNU parallel), process them through bash scripts, and upload to the Community Media collections on the Internet Archive (IA). The system tracks, per channel, how far each has progressed through the pipeline (discovered → downloaded → processed → uploaded → verified-in-archive), tracks machine health, and surfaces — in plain language for non-technical viewers — the few things that need a human.

How it fits together

  workers (agent.py)  ── POST status every 2 min ──▶  collector.py
   read-only observers                                 (always-on service)
   of the existing pipeline                            • ingest (fast write)
                                                        • rollup + alerting
  reconcile.py  ── hourly, queries IA ──▶ collector DB  • serves dashboard
   (separate process, holds IA creds)                  • SQLite (only durable state)
                                                              │
                                                              ▼
                                                        dashboard (in browser)
                                                        "is it working, and
                                                         what needs a human?"

Core principles: workers are dumb read-only counters that push; the collector owns all reconciliation, alerting, and rollup; reconciliation runs as a separate process so a slow IA query never blocks ingest; the channel (its canonical UC…/Vimeo id) is the stable join key; and everything is stateless-restartable with one SQLite file as the only durable truth.

Three numbers, three different sources (don't conflate them)

Per channel, the system tracks three counts that look similar but come from different places and answer different questions. Keeping them straight is the key to reading the dashboard correctly:

The division of labor follows the source: the agent (on each worker) talks to YouTube (remote_total) and the local filesystem (downloaded); the reconcile.py job (on the collector) talks to IA (verified). Reconciliation checks IA only — never YouTube and never the worker's disk.

Archival progress on the dashboard is verified / remote_total — an IA number over a YouTube number. It deliberately does not use downloaded, because a worker's local inventory reflects only what's on that machine now (especially on migrated or multi-generation workers), not what's actually preserved in the archive. verified is the authoritative, worker-independent measure of "is the content safe"; downloaded is a local operational detail shown as a secondary stat. (See the design doc's "Three numbers easy to conflate" for the full treatment.)

What's in this package

sql/

Project A aggregate rollups over the IA metadata DB (archive.db): per-collection / per-year / fleet trend tables, a refresh script, and example research queries. Tested against the real schema. See sql/README.md.

code/ — the programs, split by where they run

The code is organized by deployment target so it's clear what goes where (each subdirectory has its own README):

• code/worker/ → deploy to /opt/archive-agent/ on each worker. The stateless agent (agent.py + agent_platform.py), config/claims tools, and the enrich_missing.py diagnostic. None touch a database.
• code/collector/ → deploy to /opt/archive-collector/ on the one always-on collector. The service (collector.py), IA verification (reconcile.py), the DB-writing operator tools, and the SQL exports/cleanups. This side owns the only durable state (collector.db).
• code/shared/ → role-aware tools that run on either side: rebuild_missing.sh (export on collector, redownload on worker) and migrate_env.sh (OS-upgrade recovery on whichever machine you upgraded). Kept single-copy to avoid skew. | File | Role | |---|---| | agent.py | Runs on each worker from cron. Reads the existing pipeline's files (download-archive, joblogs, .info.json, markers) across the metadata/download/TEMP volumes, computes the funnel counts, and POSTs a status blob. Read-only; never touches the pipeline. | | agent_platform.py | Cross-platform helpers (time, file mtime, process counting, single-instance lock, volume/disk resolution) so agent.py runs unmodified on Linux, macOS, and WSL2. No shell-outs to date/stat/pgrep/flock/df. | | collector.py | The always-on service: ingest, rollup, alerting, the dashboard, DB migrations. Stdlib only (http.server + sqlite3). | | reconcile.py | Verifies claimed uploads against Internet Archive only (never YouTube, never local disk), matching on the youtube-id/vimeo-id metadata every existing item already carries. Path A (query by collection) with a Path B (match by video id) fallback for multi-channel collections. Produces the verified count. | | generate_config.py | Builds a worker's config.toml by parsing the existing per-directory get_new_video_metadata_backlog.sh scripts. Supports --verify (stat the paths), --append (merge into an existing config, with auto-backup and dedup), and --channel-id-map (use pre-resolved channel ids to skip network calls). | | yt_channel_names.sh | One-time pre-resolver: reads a dirs list, resolves each YouTube channel's UC… id via yt-dlp (politely, resumable), and writes a dirname → channel_id TSV map. Feed it to generate_config.py --channel-id-map so config generation is fast and network-free. Vimeo is skipped (resolved URL-side by the generator). | | set_collections.py | Populates each channel's IA collection in the collector DB from a dirname → ia_collection mapping file (pipe/tab/whitespace delimited). This is the prerequisite that lets reconciliation verify a channel against IA — without it, channels stay "upload not yet verified". Run once per worker on the collector. | | set_channel_names.py | Populates human-readable channel names in the collector DB (from the yt_channel_names.sh TSV or a channel_id\|name map), so the dashboard shows "Anchorage School District" instead of a raw UC… id. The collector also falls back to the dirname before the id. Run once per worker. | | scan_claims.py | Worker-side batch job: scans each channel's download_dir (read from config.toml) for .info.json files, extracts the video ids, and POSTs them to the collector's /claims endpoint. This populates the claims table so reconciliation can compute missing (claimed-but-not-in-archive). Heavier than the agent's per-cycle work, so run on a slow timer (e.g. daily or post-pipeline), not every cycle. Has --dry-run. | | seed_one_channel.py | Test helper: seeds one channel + collection + a few claimed video ids into the collector DB, for a live single-channel reconciliation check (Level 3 test). Not a production tool. | | rebuild_missing.sh | Re-fetch a worker's missing videos per channel (dirname). Exports each channel's missing ids to {dirname}_missing_ids.txt (from the v3 missing_video table), then either (--mode redownload, recommended) re-downloads only those ids into the channel's configured download_dir with --no-download-archive (shared archive untouched, zero blast radius), or (--mode archive) backs up the shared archive.txt timestamped, removes just those ids from it, and re-runs the dirname's backlog script. --dry-run, --only DIRNAME, --export-only/--skip-export for split collector/worker execution. Throttle-safe pacing mirrors the agent's model: yt-dlp per-video sleeps (--sleep-min/-max, default 3-8s) so a channel's downloads never burst the shared IP, plus optional channel-level GNU --parallel (--jobs, --delay stagger) — parallelism at the channel granularity, never by splitting one channel's id list. | | enrich_missing.py | Worker-side diagnostic for the missing-video failure analysis (Archive Corps). Takes a channel's missing video ids (from the missing_videos_by_worker.sql export) and enriches each from the cheapest source: local .info.json (full metadata — title, duration, tags, etc.), then optionally a yt-dlp --flat-playlist probe (one call/channel) for ids with no local file, then optionally a full per-video fetch. Tags each record with its source tier. JSONL or TSV output. The source itself is diagnostic — no .info.json means the video failed earlier in the pipeline than one that downloaded but never uploaded. | | migrate_env.sh | OS-upgrade recovery/repair tool. After a worker (or collector) is upgraded across the PEP 668 boundary (Ubuntu 22.04 → 24.04), pip --user deps become invisible to the new Python and the agent silently dies. This script idempotently re-homes the machine onto a venv, fixes yt-dlp, repoints cron at the venv interpreter, and proves the agent runs. Safe to run on any machine in any state (--dry-run, --role, --venv). | | test_reconcile.py | Offline test of the verification logic — substitutes a fake IA client and asserts verified/missing/orphan outcomes across both reconciliation paths. Run with python3 test_reconcile.py; no IA account or network needed. |

docs/

• archive-dashboard-design.md — the full design document (architecture, data model, status-blob schema, the collector, reconciliation, schema-version rollout, deployment, SQLite backup/migration, the dashboard, and design principles). The reference for why everything works the way it does.
• collector-schema.md — the database reference: every table and column, a data-access map (which program reads/writes which table), the blob-vs-DB schema version distinction, and Mermaid entity-relationship diagrams for the v1 and v2 schema. Generated from collector.py and validated against a real migrated DB.
• collector-sql-reference.md — a teaching inventory of every SQL statement the code actually runs, verbatim, grouped by pattern (upserts, INSERT OR IGNORE vs OR REPLACE, delete-then-insert, safe optional filtering) and annotated with why each is written that way. A model to learn from when writing new SQL.
• how-it-works-now.md — the pre-existing worker fleet & pipeline this monitoring system observes (scale, hardware, schedule, networking, output). Current-state context, not the dashboard design.
• metadata-enhancements.md — forward-looking research roadmap: what a researcher would ask of the ~3.2M-item corpus across all collections, and a sequenced series of metadata-enhancement projects (aggregates → transcription → entity/topic extraction → research API) to answer those questions.
• volunteer-contribution.md — a "how can I help?" guide for volunteer contributors (laypersons, community-media pros, editors, MARA/MLIS students, locals with deep community knowledge): roles mapped to the enhancement tasks, a contribution ladder, a FAQ, and the management/supervisory infrastructure to coordinate and quality-control their work.
• civic-data-integration.md — federating the civic-data-pipeline (Census-GEOID discovery of every US local government's official site, YouTube channel, and agenda/minutes portal) as the discovery source: confidence-gated channel onboarding, the civic-coverage denominator for Democracy's Library, and a sibling agendas-&-minutes extraction project (with the video↔︎agenda linking decision).
• metadata-db-integration.md — how to federate this project's collector.db ("what we sent") with the separate IA metadata database ("what's actually on IA", incl. post-upload deletions): join keys, the DARKENED failure class, reconcile-against-local-truth, and where each report draws its data.
• report-mockups.md — worked mock-ups of all three reports (operator/manager/owner) on sample data: exactly what each shows, what issues it surfaces, and the questions each answers.
• reporting-and-expansion.md — analysis + recommendations: audience-scoped reporting (operator/manager/channel-owner) with suggested cadences, and an automated capacity-first channel-expansion process that fills spare worker capacity before provisioning new machines.

runbooks/

Suggested reading / build order

1. docs/archive-dashboard-design.md — understand the target system (the why).
2. docs/collector-schema.md — skim the data model: the tables, the channel_id join key, and the three-numbers distinction in concrete columns. Useful context before you stand anything up.
3. runbooks/collector-standup-runbook.md — stand up the one always-on collector first, so there's a live endpoint for workers to post to. Note its Step 4 (host allowlist): every worker's host must be added there (and the env file regenerated + service restarted) or its posts are rejected 400 unknown host.
4. runbooks/migration-runbook.md — convert the first existing worker. This is the main per-worker procedure; follow its steps in order:
   • install (Step 1 auto-handles the 22.04 pip --user vs 24.04 venv / PEP 668 split and sets $AGENT_PY; verify yt-dlp is on the agent's PATH),
   • generate config from the on-disk scripts (Steps 2–3),
   • trial-run --no-remote and confirm the worker appears (Step 4),
   • schedule cron (Step 5), map collections (5b), set names (5c), optionally populate claims (5d), then seed remote totals so the bars appear (5e). Validate the dashboard against the old monitoring before trusting it.
5. Repeat step 4 for the remaining workers. Each is independent: add its host to the allowlist, run its config/collections/names/seed. The per-worker checklist at the end of the migration runbook is the quick reference.
6. As needed: add fresh or cross-platform machines via linux-/windows-/macos-worker-runbook.md; after any worker's OS upgrade across the 22.04→24.04 boundary, run code/migrate_env.sh (see the migration runbook's "Maintenance: OS upgrade" section).

Status of the code

All programs were tested during development (unit tests of the parsing and reconciliation logic, and end-to-end runs of the agent → collector → dashboard path against fixtures), and the system has since been deployed and debugged across a live multi-worker fleet. Two environment-specific wiring points to confirm per deployment:

• reconcile.py's IAClient needs live IA credentials (configured on the collector per the standup runbook). The real query layer is isolated in one injectable class.
• agent.py's remote-total fetch resolves yt-dlp by absolute path (env YTDLP_PATH → PATH → ~/.local/bin → /usr/local/bin → /usr/bin) and fails loudly if it can't find it, rather than silently returning null totals. Confirm sudo -u access yt-dlp --version works (symlink into /usr/local/bin if it lives in ~/.local/bin); see the migration runbook's Step 1 and the "remote_total stays null" troubleshooting.

Nothing requires changes to the existing upload pipeline before reconciliation works — it matches on the youtube-id/vimeo-id metadata the corpus already has.

Dependencies

• Workers (agent.py): Python 3.8+, psutil (live job count; optional — degrades gracefully), tomli on Python <3.11 (3.11+ has tomllib built in). The pipeline already provides yt-dlp, jq, ffmpeg, GNU parallel. On Ubuntu 22.04 install deps with pip --user; on 24.04 / Debian 12+ that's blocked by PEP 668, so use a venv — the migration runbook's Step 1 auto-detects which and sets $AGENT_PY accordingly, and code/migrate_env.sh re-homes a worker onto a venv after an OS upgrade.
• Collector (collector.py): Python 3.8+ stdlib only.
• Reconciliation (reconcile.py): the internetarchive package.
• All machines: Tailscale (already in use).

↑ back to contents10.2-archive-dashboard-project(71)/archive-dashboard-project/README.md

Discover Agenda Systems via Wayback Machine

Tools and documentation for identifying which agenda-management vendor each US local government uses, by enumerating vendor tenant subdomains from the Internet Archive's Wayback CDX index.

Purpose: expand the set of known jurisdictions using one of ten agenda systems, so existing scrapers can be pointed at more targets and new extractor work can be prioritized by real client counts.

Status at packaging (2026-08-03): discovery complete — 12,362 tenants enumerated across ten vendors, up from a prior all-vendor working list of 2,414 — a 5.1x expansion. Next step is classification.

/code

All are stdlib-only except cc_columnar_hosts.py, which needs duckdb.

Quick start:

python3 code/vendor_host_harvest.py --sources wayback --sleep 1 --resume \
    --out-dir ./vendor_hosts

Read the runbook before a first run — particularly the four run modes (--resume / --append / --force / none), which govern whether existing results are protected.

/docs

Each document is provided as .md, .docx, and .pdf.

Not included

Harvested data (hosts_*.csv, raw_*.csv, all_vendor_hosts.csv, summary.txt) and scan state files are not packaged here — they are outputs, not assets, and are regenerated by running the tools.

Version History

↑ back to contents20.0-discover_agenda_systems_via_wayback_machine/README.md

Civic meeting scrapers — review bundle

2026-08-03

Start here

docs/EXECUTIVE-SUMMARY_2026-07-26 — scope, effort, what went wrong, and the decisions now needed. Everything else is detail behind it.

Each document is provided three ways: .pdf (read anywhere), .docx (comment and track changes), .md (plain text, version-controllable). The content is identical.

What is in here

docs/    seven documents, each as .pdf / .docx / .md
code/    six tools and three test suites

Documents, in reading order

Code

Three things a reviewer should know

The numbers answer different questions. The corpus counts places (cities, towns, villages). The Wayback CDX search counts vendor client sites, which include school districts, counties and special districts. They should never be added together — the executive summary explains why this matters for scope.

"Self-hosted" does not mean bespoke. It means no vendor name appeared in the web address. Councils commonly run a branded page on top of a commercial agenda system, with the join hidden in JavaScript. An earlier version of this summary mistook that label for a finding and wrote off ~4,800 sites; the correction is recorded in the summary and the tier-4 document.

Nothing here is finished collecting. The tools work and have been run against real sites, but collection is early. The summary's open-items list is the current state.

↑ back to contents20.1-civic-scrapers-bundle_2026-08-03/README.md

Municipal Agenda Scraper — Bundle Manifest

Bundle date: 2026-07-21 Purpose: Self-contained snapshot of the municipal agenda-scraper project — working code plus all project documentation — for review and for importing into other working sessions.

What this project is (30 seconds)

A toolkit for collecting agendas and minutes from municipal government websites at scale. A working scraper handles CivicPlus AgendaCenter sites today. The project has mapped the full ~2,400-site landscape across ~11 meeting-management platforms and has a decided build order for extending coverage.

If you read one file: docs/civicplus-scraper-exec-summary_2026-07-21.* (one page, non-technical).

Bundle layout

/
├── MANIFEST.md          ← you are here
├── code/
│   └── civicplus_agendas.py   (the scraper, ~1,600 lines, Python 3)
├── data/                (small representative sample — see data/README.md)
│   ├── sample_input_sites.csv        (what --preflight-csv consumes)
│   ├── sample_rejected_sites.csv     (what --summarize reads)
│   ├── sample_recovered_sites.csv    (a passing-output example)
│   └── vendor_distribution_summary.txt  (full 2,414-site platform tally)
└── docs/                (4 documents, each in 3 formats: .md .docx .pdf)
    ├── civicplus-scraper-exec-summary_2026-07-21.*
    ├── civicplus-scraper-project_2026-07-21.*
    ├── civicplus-scraper-runbook_2026-07-21.*
    └── civicplus-agenda-scraper_2026-07-20.*

Formats: every document is provided as Markdown (.md, source), Word (.docx), and PDF. Non-technical reviewers should use the .pdf or .docx; the .md is the editable source.

The documents — what each is for, and who should read it

Read order for a new person: exec-summary → project → runbook. The 07-20 doc is background.

The code

code/civicplus_agendas.py — single-file Python 3 scraper.

• Dependencies: pip install requests beautifulsoup4 lxml
• Handles: CivicPlus AgendaCenter (categories, years, agendas, minutes), PDF download into a State/City/Category/Year/Type/ tree, legacy vendor subdomain resolution, endpoint auto-detection with manual overrides.
• Batch tooling: --preflight-csv (test many sites for compatibility), --summarize (triage the rejects by platform/vendor), vendor detection.
• Quick start: python3 code/civicplus_agendas.py --help

Full operating instructions are in the runbook.

Current status at a glance

• CivicPlus AgendaCenter scraper: working (~1,300+ sites addressable).
• Full platform map: complete (2,414 sites classified).
• Build order: decided — CivicClerk first (public API, largest untapped block), then the Granicus/Legistar family. Details and rationale in the project doc.
• Not yet built: refactor into a per-platform plugin architecture; any second extractor.

Important caveats (please read before acting on the numbers)

• Vendor counts come from an imperfect upstream classifier (a separate effort). Treat site counts as approximate (±10–20%). Confirm before committing resources.
• Platform feasibility is unconfirmed for platforms beyond CivicPlus. Each new extractor should begin with a quick check that its API/site structure is accessible with plain HTTP (some may need a real browser).
• The docs candidly record several mid-project course corrections. That transparency is intentional — the lessons are part of the deliverable.

Sample data included

A small representative slice of the working data is in data/ (see data/README.md) — enough to understand the file formats and try the tooling. The full working data (complete input site lists, full preflight output, debug HTML dumps) is not bundled: it is large, environment-specific, and regenerable by the code. The docs reference these fuller files by name where relevant.

↑ back to contents30.0-agenda-scraper-bundle_2026-07-21/MANIFEST.md

CivicPlus Archive — Forensic Assessment Bundle
==============================================

  docs/   Start here.  (All findings current as of the 2026-08 full run.)
            EXECUTIVE_SUMMARY  — 3 pages, non-technical: what this collection is,
                                 what survived, and what to do next.
            PROGRESS_SUMMARY   — what was investigated and established.
            RUNBOOK            — how to re-run the tools, what every category
                                 means, and what to investigate first.
            SQL_REFERENCE      — the database schema, how the tables join, and
                                 ready-made queries for your own questions.
            LESSONS_LEARNED    — what the collection taught, and what the
                                 assessment's own mistakes taught.
          Each is provided as .md (source), .docx (Word) and .pdf.

  code/   civicplus_forensics.py    builds the index and classifies everything
          civicplus_investigate.py  named diagnostic checks, no SQL needed
          civicplus_refetch.py      probe whether source sites still answer,
                                    fingerprint who serves them now, and
                                    repair what they will serve
          civicplus_migration_test.py  for an address that has gone dark: did the
                                    jurisdiction leave, or just move?
          civicplus_blank_scan.py   find "blank" PDFs -- structurally valid
                                    (%PDF, %%EOF, one page) but no text, fonts,
                                    or images, so they render empty. Passes every
                                    other check yet carries nothing. Modes:
                                    strata (size histogram), sample (blank rate),
                                    full (exact count), breakdown (--resolve the
                                    www bucket + --rebuild-url a re-fetch
                                    worklist), fetch (re-download and verify not
                                    blank), ledger (roll the fetch log into a
                                    per-outcome/per-tenant census), check (spot a
                                    file).
          Python 3.8+, standard library only. The corpus is read-only.
          (blank_scan's optional --render uses poppler's pdftoppm if present.)

  data/   civicplus_host_lookup.csv       web address -> state/place (2,063 rows)
          civicplus_slug_lookup.csv       registry, normalized (1,550 rows)
          civicplus_script_crosswalk.csv  each batch script and its target
          sample_*                        shape of each input and output file

Quick start
-----------
  python3 code/civicplus_forensics.py \
      --root ~/civic-data-project \
      --sites civicplus_sites.csv \
      --host-lookup data/civicplus_host_lookup.csv \
      --db civicplus_index.db --out reports --stage all --verify-magic

  python3 code/civicplus_investigate.py --db civicplus_index.db --check orphans

See docs/RUNBOOK for everything else.

↑ back to contents30.1-civicplus_forensics_bundle(13)/README.txt

Municipal Meeting-Document Scrapers — Bundle

Bundle date: 2026-07-26 Purpose: complete, current snapshot of the CivicClerk/CivicWeb scraper effort — code + docs in one place, so work can resume in a fresh conversation with no context loss.

Layout

/code   — the tools (current versions only; superseded drafts excluded)
/docs   — runbook, master plan, project summary

/code — tools (run order per docs/runbook.md)

NOT YET BUILT: civicclerk_reconcile.py (Stage F audit — joins runs/ + disk tree + snapshots). Design is in runbook §7; records + snapshots are being captured now so it can run retroactively.

Primary scraper is at feature level: records emission + Ctrl-C-safe atomic records + archive-then-write eviction + progress heartbeat. (Built up over versions v1→v3; only the current consolidated file is included here.)

/docs

Key facts that took real work to learn (don't re-derive)

• civic-scraper library is BROKEN for modern CivicClerk (parses a __VIEWSTATE field React portals lack). All tools here are from-scratch against the real API.
• CivicClerk wildcards DNS — every hostname resolves. Existence must be checked via the API (/v1/Events → JSON vs empty body), never DNS.
• Pagination is mandatory (@odata.nextLink, ~15 rows/page). Files come from the single-event Events({id}) record, not the collection.
• Preflight samples events across the WHOLE date range (recent events are often empty placeholders — sampling only recent = false negatives).
• Two recurring "honest" failures: phantom tenants (@events: no rows → drop) and video-only CivicClerk (@files fail → docs are on CivicPlus, route there).
• Data root /mnt0/civicclerk_output/ is self-describing: {ST}/{Place}/{Board}/{Year}/*.pdf
  • runs/{siteid}.csv (records) + discovery/ (snapshots) + logs/.

Environment

• Ubuntu; Python 3, standard library only (no third-party deps for the API tools; civicclerk_html_archive.py uses BeautifulSoup if present, else stdlib).
• Politeness: keep --delay ≥ 0.3–0.5, modest --jobs.

↑ back to contents31.0-civicclerk_scrapers_bundle/civicclerk-scrapers-bundle_2026-07-26/MANIFEST.md

CivicClerk — Texas run: diagnostics, findings and code changes

Handoff note. This work happened inside a Legistar-focused conversation and belongs with the CivicClerk effort. Everything needed to pick it up is here.

Date: 2026-08-05/06. Scraper went from v3 (2026-07-26) to v5 (2026-08-05).

1. The run

119 clients | 74,177 downloaded | 5,785 cached | 1 failed | 32h24m | ~41/min

Command as issued:

python3 civicclerk_api_scrape-2026-07-26-v3.py \
    --portals TX_civicclerk_remaining.csv --jobs 8 --delay 1 \
    --start-date 2000-01-01 --end-date 2026-12-31 --download \
    --download-dir /mnt0/civicclerk_output

Output tree /mnt0/civicclerk_output/{ST}/{Place}/{Board}/{Year}/, records CSVs in runs/<client>.csv.

2. Findings

2.1 Zero-file meetings — 11,553, and nothing recorded them

The log carries lines like 2022-09-15 | City Council (#26) - 0 file(s). A meeting with no documents was not a match, not a failure, and not written anywhere — it existed only as a line in a multiplexed stdout log. The run summary (1518 file(s) matched; downloaded 1518, cached 0, failed 0) cannot express it.

Counted from the log:

By year, past only: 2016 302 · 2017 550 · 2018 406 · 2019 708 · 2020 845 · 2021 885 · 2022 997 · 2023 1,261 · 2024 1,378 · 2025 1,378 · 2026 665, with a thin tail back to 2006.

Three distinct causes are mixed together, and only one is benign:

1. Scheduled placeholders. Cities create a year of calendar entries in advance. Watauga's monthly City Council pairs run empty to 2026-05-25 while 2026-06-22 has agendas. The 1,464 future-dated ones are all of this kind — and they were in scope only because --end-date 2026-12-31 was five months past the run date.
2. Genuinely never published. Wimberley's third-Thursday council series (#24, #25, #26) goes empty while the first-Thursday series keeps publishing. A real property of that body, not a scrape failure.
3. The one that matters: documents the API did not report but the portal shows. This is unverified. It is exactly the failure that cost 60,781 meetings on the Legistar side, where a status field's silence was read as absence.

--section archived is a Legistar flag; CivicClerk's scraper has no equivalent. Bounding --end-date to the run date is what excludes placeholders.

2.2 The log could not be demultiplexed

With --jobs 8, eight workers write one stdout. Lines naming a file are attributable by their path; the 0 file(s) lines named no client at all. Wimberley file paths appear inside Watauga's section in the same log. So 31 hours of work produced 74,177 documents and no way to attribute any gap to a city — not by cleverness at analysis time, because the information was never written.

2.3 Duplicate place directories

Baycity/Bay_City and Bigspring/Big_Spring — an earlier run was killed, place names corrected, and the state re-run. Verified before deleting:

TX/Baycity:  762 files   TX/Bay_City:  1,123    in OLD not NEW: 0
TX/Bigspring: 308 files  TX/Big_Spring:  343    in OLD not NEW: 0

New runs were strict supersets, so the old trees were removable. Worth noting the general practice: a killed run can leave a file the re-run never reached, so the set-difference is what proves deletion is safe.

2.4 GUID-named documents — a false alarm

Bay City logs MEET-Agenda-f68bb121… where other cities log human titles. The GUID is the meeting's, not the file's — an Agenda and Minutes for one meeting share it. Checked on disk: all %PDF, sizes scattering 30 KB–410 KB, which is the signature of uploaded documents rather than generated placeholders. The name never reaches the tree, since on-disk filenames are built from date/board/type/file-id. Nothing to fix.

2.5 "General" board — 6,613 files, 106 tenants, the real defect

General was the most common board directory in Texas (93 places). It is not a body: CivicClerk defaults an uncategorized meeting's categoryName to "General", and 106 tenants leave it there.

Friendswood shows the damage — four bodies in one directory:

Friendswood | 2018-01-08 | Agenda  | City Council 01-08-2018 Regular Agenda
Friendswood | 2018-01-08 | Agenda  | P&Z 01-08-2018 Agenda
Friendswood | 2018-01-09 | Agenda  | FDEDC 01-09-2018 Agenda
Friendswood | 2018-01-09 | Agenda  | Senior Citizen 01-09-2018 Agenda

Diagnosis, in order:

• board == category for all 6,613 rows — the records CSV offers no alternative field.
• Filename-prefix extraction matches only 1,292 of 6,613 (20%), and conventions differ per city (Royse City puts the date first). A regex approach would mis-attribute more than it fixed.
• The API probe settled it. A raw event dump showed eventName carrying the body — "City Council Meeting", "Investment Committee Meeting" — while categoryName is the literal "General".

Root cause, one line:

board = cat or ev_name or "Board"      # "General" is truthy, so ev_name never ran

A generic value is not a value. Test for meaning, not for presence.

2.6 event_id was never written

_record() passed event_id="" on every row. The key tying a document to its meeting is blank in every CivicClerk records CSV produced to date, including the completed Texas run.

2.7 Flagged, not resolved

Notice_of_Possible_Quorum appears as a board directory. It is a Texas posting category, not a deliberative body — the same class as Legistar's Miscellaneous Agendas and Public Notice buckets. Decide before item identifiers are minted: keep with a note, or map to the posting body.

3. Code changes

v4 — attribution and pending records

• PendingWriter (subclasses RecordsWriter, same atomic partial-then-promote): writes runs/<client>_pending.csv, one row per zero-file meeting, with a classified reason — scheduled for future dates, no_files_reported for past ones. This is the CivicClerk equivalent of Legistar's _pending.csv, and without it the gap is uncountable.
• ClientLog — every per-meeting line prefixed [<client>], so a combined log demultiplexes with grep.
• --client-log-dir DIR — optional per-client log files alongside the combined stream.

v5 — the body name

• body_from_event(category, event_name) → (board, source). A category is used only if it names something; GENERIC_CATEGORIES covers general, uncategorized, default, other, misc, n/a and blank, so this is not a Texas-specific patch. Falls back to eventName.
• Trailing "Meeting" trimmed so City Council Meeting and City Council — both present in one tenant — are one body. _MEETING_KIND protects Special Meeting, Regular Meeting and similar from becoming Special.
• board_source column (category | event_name) in the records CSV, so a derived body name is visible and reversible rather than silently overwriting.
• event_id now actually written.

Files: civicclerk_api_scrape-2026-08-05-v5.py, civicclerk_v3_to_v5.diff (270 lines against v3).

civicclerk_regeneral.py — remediating the 6,613 already on disk

Three phases, each stopping for review; deletion is never automatic.

python3 civicclerk_regeneral.py --output-dir /mnt0/civicclerk_output plan
python3 civicclerk_regeneral.py --output-dir /mnt0/civicclerk_output rerun \
    --scraper ~/Downloads/civicclerk_api_scrape-2026-08-05-v5.py \
    --portals TX_civicclerk_remaining.csv --jobs 8 --delay 1 --apply
python3 civicclerk_regeneral.py --output-dir /mnt0/civicclerk_output verify

plan finds affected clients from the records CSVs. rerun slices the portals CSV to just those clients and re-runs them — the destination path changes (General → City_Council), so skip-if-exists cannot match the old copy and the files are re-fetched into the right place. verify set-differences each General/ tree against the rest of that city's tree and lists a directory as safe only when every filename appears elsewhere; it writes the delete list but never deletes.

Expect some directories to remain unsafe: meetings where eventName was empty too have nowhere else to go and legitimately stay General.

4. Open items

1. Regenerate the Texas inventory. A metadata-only pass (no --download) with v5 writes the pending CSVs for all 119 clients in minutes rather than 31 hours, and bounds the window to the run date:

   python3 civicclerk_api_scrape-2026-08-05-v5.py \
       --portals TX_civicclerk_remaining.csv --jobs 8 --delay 1 \
       --start-date 2000-01-01 --end-date $(date +%F) \
       --records-dir /mnt0/civicclerk_output/runs \
       --client-log-dir /mnt0/civicclerk_output/logs/clients

2. Answer the index-or-source question. Sample ten no_files_reported rows across different cities and open the portal pages by hand. Documents present means CivicClerk's API is an index like Legistar's and 10,089 meetings need a recovery path. This decides whether the Texas corpus is complete or 12% short.

3. Run the General remediation (§3) — 6,613 files across 106 tenants.

4. Decide on Notice_of_Possible_Quorum and any similar posting categories.

5. Investigate the single failed 1. One failure across 32 hours is findable now and impossible later.

6. Note that existing records CSVs lack event_id. Any join built on the Texas run's current CSVs has no meeting key until they are regenerated.

5. Carried over from the Legistar effort

Three patterns that recurred and are worth applying to CivicClerk directly:

• A field being populated is not the same as its contents being usable. On Legistar this appeared four separate times (bare filenames that broke the downloader, the pending path, a CSV validator, and provenance selection). On CivicClerk it is "General" — populated, truthy, and meaningless.
• "We did not look" must never be recorded as "it is not there." Every guard that stops trying needs its own reason code, or the gap silently becomes a coverage statistic.
• Count the thing itself. downloaded/kept conflates fetches with skips; grep -c '^\[saved\]' does not. Order-of-magnitude wrong conclusions came from the former.

The Legistar runbook's §13 is a symptom-to-cause diagnostic playbook built from these; most of it transfers with the vendor name changed.

↑ back to contents31.1-civicclerk_TX_diagnostics_and_code_changes/CIVICCLERK_TX_HANDOFF.md

LEGISTAR SCRAPER -- BUILD #1 SNAPSHOT
=====================================
Packaged: 2026-08-05

READ THIS FIRST
---------------
This archive is a point-in-time snapshot of the first working build of the
Legistar civic-meeting scraper, packaged for review. Development continued
after this was cut: a later working session may have productionalized or
superseded what is here. If you have received more than one archive, or have
access to the project's later conversations, CHECK FOR A NEWER VERSION before
building on this one. The findings log (docs/legistar-notes.md) is the
authoritative record of what was known at packaging time.

WHAT THIS PROJECT IS
--------------------
The third scraper in a family (Granicus, CivicClerk, Legistar) that collects
municipal meeting agendas and minutes into a common archive layout
(State/Place/Board/Year/DocType) with a shared record schema. This build
covers the 215 US municipalities running Legistar, of which 170 are live,
scrapeable clients. It scrapes the Legistar web API for enumeration and
falls back to the human-facing InSite meeting pages for the roughly one
tenth of documents the API links to but does not report.

CONTENTS
--------
MANIFEST.txt                          this file

code/
  legistar_api_scrape.py              the scraper (~4,000 lines). All modes in
                                      one file: scrape, download, preflight,
                                      field census, InSite survey, InSite
                                      document recovery, pending-document
                                      audit. Run with --help for usage; the
                                      design doc explains the architecture.
  test_legistar_offline.py            339 offline tests (no network needed):
                                      python3 -m unittest test_legistar_offline
  legistar_recovery_audit.py          joins a run's pending CSVs against its
                                      records CSVs to report how many
                                      API-missing documents were recovered
                                      from InSite, and what is still gone.
  legistar_action_summary_clients.py  filters an --insite-survey result down
                                      to the clients whose boards publish
                                      "Action Summary" documents, for targeted
                                      collection.

docs/  (each document in three identical-content formats: .md source,
        .docx for commenting in Word, .pdf for reading)
  legistar-executive-summary.*        ONE PAGE. Start here: status, scope,
                                      planned-vs-actual effort, the five
                                      unexpected issues, lessons, and the
                                      decisions still open.
  legistar-scraper-design.*           the original design/plan written before
                                      implementation. Kept as-written; the
                                      notes record where reality disagreed.
  legistar-notes.*                    the findings log, v1-v26: every live
                                      run, every anomaly, every bug (mine and
                                      Legistar's), in chronological order.
                                      The project's institutional memory.

SUGGESTED READING ORDER (non-technical reviewers)
-------------------------------------------------
1. legistar-executive-summary.pdf    (3 pages)
2. legistar-scraper-design.pdf       (9 pages, skim)
3. legistar-notes.pdf                (35 pages, reference -- skim the section
                                      headers, which are dated and titled by
                                      finding)

THE ONE-SENTENCE FINDING
------------------------
The Legistar API is a reliable index and an unreliable inventory: it
eliminated the structural guesswork of HTML scraping, but silently omits
documents that exist, so this build uses the API to decide what to fetch and
the human-facing meeting pages to actually fetch it.

NOT INCLUDED
------------
No scraped documents, no run outputs, no target lists with per-client tokens.
The scraper produces those; this archive is the tool and its paper trail.

↑ back to contents32.0-legistar-scraper-build1/MANIFEST.txt

legistar_client_runs

Tooling and documentation for collecting US local-government meeting agendas and minutes from Legistar, one of ten agenda-management vendors.

Packaged 2026-08-02. Scraper version 2026-08-02.6, pipeline driver 2026-07-30.1.

What this produced

418,871 documents from 170 jurisdictions across 33 states, spanning 2000–2026. Every file traces to a source URL, event id, publishing body, meeting date and document type — reconciliation is clean in both directions, with zero unexplained files and 1,918 records (0.48%) whose document could not be retrieved, each classified rather than reported as generic loss.

The documents themselves are not in this archive. This is the method: the scrapers, the verification tooling, and the record of why each decision was made.

Start here

If you have been handed a corpus to verify or upload and did not do the collection, start at runbook §12. It is self-contained: it explains the artifacts before asking you to act on them, and every check states what a good result looks like.

docs/

Each document is provided in three formats — .md (source), .docx (Word, for review and comment) and .pdf (for reading and printing). The .md files are authoritative; the other two are generated from them, so edit the markdown and regenerate rather than editing a Word file that will be overwritten.

cd docs
for f in *.md; do pandoc "$f" -o "${f%.md}.docx" --from=gfm --toc --standalone; done
soffice --headless --convert-to pdf --outdir . *.docx

• EXECUTIVE_SUMMARY.md — scope, effort estimated versus actual, the six categories of problem we did not anticipate, and the lessons that generalise to other vendors.
• IA_DECISION_BRIEF.md — eight decisions pending with the Internet Archive: item granularity, identifier scheme, delivery model, document scope, and who owns ongoing refresh. Each section ends in a question.
• LEGISTAR_RUNBOOK.md — operator procedure. Collection stages, the per-state pipeline, document format policy, the weekly refresh job, and (§12) reconciliation, remediation and upload preparation.
• legistar-notes.md — the engineering log. Every non-obvious decision with the evidence that forced it, in version order. The most transferable artifact here: it records which failure modes are real rather than theoretical.

code/

Collection

• legistar_api_scrape.py — the scraper. Preflight, survey, download, InSite recovery, per-tenant host repair, content sniffing. --version reports what it supports; check it before any long run.
• legistar_run_state.sh — pipeline driver. Walks one state through slice → preflight → survey → plan → download → audit, with pinned filenames, gates and a status ledger. -n for a dry run.
• legistar_weekly.sh — cron-schedulable refresh across every state, with audit gates and a count of what arrived.
• legistar_recovery_clients.py — splits a state's clients into the batch that needs InSite recovery and the batch that does not.

Verification

• legistar_reconcile.py — the tree against the records CSVs, both directions. Files with no provenance, and records with no file, with a per-client host census that classifies why.
• legistar_recovery_audit.py — recovery yield per pending reason, each against its own denominator. --min-pct for the weekly gate.
• legistar_minutes_gap_count.py — sizes the missing-minutes gap by body, separating bodies that publish minutes sometimes from those that never do.
• legistar_packet_scan.py — finds agenda packets misfiled as agendas, by size relative to each tenant's own norm, confirmed by page count.

Repair

• legistar_refetch.py — re-fetches documents listed in missing.csv using the scraper's host-repair ladder, for records the API no longer returns.
• legistar_unfile.py — moves documents written under a placeholder state back to the client tree they belong to.

Upload

• legistar_ia_manifest.py — builds an ia CLI upload spreadsheet from the download tree joined to the records CSVs. Configurable granularity and identifier prefix; emits neutral column names for late field mapping.

legistar_api_scrape_v34.diff is the cumulative diff of the scraper against the version that began this round of work, if you want to see what changed rather than what it does.

The one thing worth knowing before you read the code

The Legistar API is an index, not a source. It reports what it stored when a meeting was published; the meeting's own page is a superset — more document classes, current links, and the truth about what exists today. The architecture is API-to-decide-which-pages, page-to-fetch-content, and most of the complexity in this codebase exists because that was learned late rather than assumed early.

For the next vendor, the ten-minute test: fetch one record from the API and its human-facing page, and diff the document sets. If they differ, design for an index from day one.

↑ back to contents32.1-legistar_client_runs(2)/legistar_client_runs/README.md

Volunteers & Contribution — "How can I help?"

The metadata-enhancement roadmap (metadata-enhancements.md) has a large amount of work that is better done by people than by machines — and specifically by people with local knowledge, subject expertise, or archival training. This document maps volunteer contributors to that work: who can help, what they'd do, how they're supported and supervised, and answers to the questions a prospective volunteer asks.

The core insight: enrichment projects like transcription-correction, place mapping, speaker identification, and topic tagging are exactly the kind of work that scales through motivated volunteers — if the coordination and quality-control infrastructure exists to make their contributions trustworthy. This document is about building that infrastructure, not just recruiting.

1. Who can help, and what they're uniquely good at

Different contributors bring different strengths. The work is deliberately sliced so each profile has a natural, valuable lane.

The last row deserves emphasis: local knowledge is the one input that cannot be automated or outsourced. A retired city clerk who can identify every council member by voice across fifteen years of meetings is providing data that no ASR or NER pipeline can produce. Designing roles that capture that knowledge is the highest-value part of a volunteer program for this corpus.

2. What volunteers actually do — tasks mapped to the roadmap

Each task ties to a project in metadata-enhancements.md. Tasks are sized so a newcomer can do a useful unit in one sitting.

Caption & transcript correction (Project C)

The harvest tiers (on-disk YouTube captions, pre-2019 IA captions) produce usable but imperfect text; ASR on the gaps produces more. Volunteers review and correct segments — fixing misheard words, speaker turns, and punctuation. This is the largest pool of volunteer work and the most accessible: anyone literate can improve a transcript against the audio.

• Layperson / editor / senior. Editors raise consistency; seniors catch local names ("that's Commissioner Yoakum, not 'commissioner yokum'") that others can't.

Place & jurisdiction mapping (Project B)

The dirname→place map (lvnwks → Leavenworth, KS) is mostly mechanical but has exceptions and ambiguities a human resolves quickly. Volunteers confirm/complete the locality, county, and coordinates per collection.

• Community-media pro / senior / student. Locals disambiguate same-named towns; students apply geographic authority control.

Speaker & entity identification (Project E)

After NER proposes people/orgs, volunteers confirm, correct, and link them — especially recurring speakers across meetings. This is where local knowledge is decisive.

• Senior / community-media pro. The "name that voice / face" work is uniquely human and uniquely valuable.

Topic & subject tagging (Projects A/E)

Apply a controlled vocabulary of civic topics (zoning, budget, public safety, …) to items or transcript segments, so topical search and prevalence analysis work.

• Editor / student / layperson. Students design/steward the vocabulary; others apply it.

Coverage-gap flagging (Sub-project C-gap; Project G)

Volunteers spot and report gaps: a collection missing recent meetings, an item whose caption is garbage, a video that's gone dark. This mirrors the operational missing-video work, done by humans who watch the content.

• Anyone. A great first task — low skill floor, immediately useful.

Description & context writing (Project A)

Write or improve item/collection descriptions — what a meeting covered, why a collection matters. Turns thin auto-metadata into human-useful context.

• Editor / community-media pro / senior.

Metadata standards & authority control (cross-cutting)

Design the controlled vocabularies, name-authority files, and quality rubrics the other tasks depend on. This is the librarian layer that makes volunteer output consistent and trustworthy.

• MARA / MLIS students and professionals. This is their discipline; it's also the supervisory backbone (see §4).

3. Contribution levels (a ladder, not a wall)

Volunteers should be able to start trivially and grow into responsibility. A tiered ladder also is the quality-control structure (higher tiers review lower tiers):

1. Contributor — do discrete micro-tasks (correct a transcript, confirm a place, tag a topic). No commitment, no training beyond a short guide. Work is queued and reviewed before it lands.
2. Reviewer — trusted contributors who check others' work and approve it. Earned by a track record of accepted contributions. This tier is what lets the program scale without every edit hitting a paid supervisor.
3. Domain steward — owns a slice (a region, a collection group, the topic vocabulary, the speaker-authority file). Coordinates contributors, resolves disputes, maintains standards in their area. Often MARA/MLIS students/pros or deeply-engaged locals.
4. Program coordinator — the paid/lead role(s); see §4.

Progression is earned through accepted work, not tenure — which keeps quality tied to demonstrated reliability.

4. Management & supervisory infrastructure

Volunteer enrichment fails without coordination and quality control. The insight: the same "claimed vs. verified" discipline the archiving pipeline uses for videos applies to human contributions. A volunteer edit is a claim; a reviewer's approval is verification; unreviewed edits don't land in the authoritative dataset. That mental model — already proven in this project — is the backbone.

The people

• Program coordinator(s) — paid or lead volunteers who recruit, onboard, set priorities, and handle escalations. Realistically 1 coordinator per ~30–50 active contributors. This is the role that most needs to be resourced; everything else can be volunteer-run if a coordinator holds it together.
• Domain stewards (tier 3 above) — the distributed supervisory layer; each owns a region/collection-group/vocabulary and reviews or delegates review.
• A metadata/archival advisor — a MARA/MLIS professional (or faculty partner) who owns standards, vocabularies, and authority control. Part-time but essential: they keep volunteer output aligned with real archival practice, and they make the program a credential-worthy experience for students (see §5, practicum angle).

The systems (what has to exist)

• A task queue / work-assignment system — hands out micro-tasks, tracks who did what, prevents duplicate work. This is the single most important piece of infrastructure; without it, volunteers don't know what to do and effort collides.
• A review/approval workflow — every contribution enters as pending, gets reviewed (by a tier-2+ reviewer), and only then merges into the authoritative metadata. Mirrors the collector's claimed→verified model exactly.
• Provenance on every edit — who changed what, when, from which source, reviewed by whom. The metadata DB's soft-delete/history discipline extends naturally to human edits; nothing is ever silently overwritten.
• Contributor guides & rubrics — short, task-specific "how to correct a transcript," "how to tag a topic," maintained by the stewards/advisor.
• Recognition — visible credit, contribution stats, and (for students) a documentable record for their program. Recognition is the volunteer program's currency; budget for it.

Quality control model

• Redundancy for high-stakes fields — e.g. two independent confirmations before a speaker identification is authoritative; consensus resolves disagreement.
• Sampling — stewards spot-check accepted work to keep reviewers calibrated.
• Reputation — contributors whose work is consistently accepted earn lighter review; new contributors get more. This concentrates supervisory attention where it's needed and rewards reliability.
• Nothing lands unreviewed — the authoritative dataset only ever ingests verified contributions; raw edits live in a staging layer. (Same architecture as keeping archive.db authoritative and collector.db claims pending until reconciled.)

5. FAQ

Q: I'm not technical at all. Can I still help? Yes — most of the work is not technical. Correcting a transcript against the audio, confirming which town a collection belongs to, identifying a speaker you recognize, or tagging what a meeting was about needs care and knowledge, not coding. The task queue hands you a small, clear unit of work with a short guide.

Q: How much time do I need to commit? None ongoing. The work is designed as micro-tasks — do one transcript segment or one tagging task in a few minutes, or spend an afternoon. Contribute once or become a steward; both are welcome.

Q: I have deep knowledge of my town's history/government. Where's that most useful? Speaker and place identification, and historical context — the things no algorithm can know. Recognizing officials by voice across years of meetings, disambiguating local places and events, and explaining "why this meeting mattered" is the single most valuable contribution for this corpus, and it's irreplaceable.

Q: I'm a MARA/MLIS student — is there work at my level, and does it count? Yes, and this is designed as a practicum-quality experience: controlled-vocabulary design, name-authority control, provenance modeling, and supervising contributor quality are core archival practice on a real 3.2M-item corpus. With a faculty/advisor partnership it can be structured as documentable, credential-relevant work.

Q: How do I know my corrections actually matter / won't be lost? Every edit is attributed, reviewed, and versioned — nothing is silently overwritten, and you can see your accepted contributions. Reviewed work merges into the authoritative archive that researchers and the public use.

Q: What if I make a mistake? That's what the review layer is for — contributions are checked before they land, so mistakes are caught, not shipped. New contributors get more review; there's no penalty for honest errors.

Q: Can my community-media station / historical society / class participate as a group? Yes — a group can adopt a collection or region as domain stewards. Group participation with a local coordinator is one of the most effective patterns, because it concentrates local knowledge where it's most relevant.

Q: Who decides the standards (vocabularies, how to tag, name spellings)? The metadata advisor and domain stewards, following established archival practice. Standards are documented in the contributor guides so everyone works consistently; proposed changes go through the stewards.

Q: Is my work going to train AI / be sold? [A policy question for the program to answer explicitly and honestly up front — volunteers deserve a clear, truthful statement of how their contributions are used and licensed. State it plainly in the volunteer agreement.]

6. Why this fits this project specifically

• The work is already sliced. The enhancement roadmap's tasks (caption correction, place mapping, speaker ID, tagging, gap-flagging) are naturally human-sized and queueable — the volunteer program is mostly coordinating work the roadmap already defines.
• The quality model already exists. "Claimed vs. verified," soft-delete history, provenance-on-everything, coverage-gap worklists — the disciplines this project built for videos transfer directly to human contributions. The supervisory infrastructure is the same architecture applied to people.
• Local knowledge is the corpus's missing input. Civic media is inherently local; the people who understand it are distributed across exactly the communities the archive covers. A volunteer program isn't just cheaper labor — it's the only source of the local-knowledge metadata that makes the corpus truly navigable.
• It creates a virtuous loop with the reporting work. Channel-owner reports (report-mockups.md) put the archive in front of the very communities whose members are the ideal volunteers — the report that shows a town "here's your preserved public record" is also the natural on-ramp to "…and here's how you can help make it better."

This is a planning document for a future volunteer program, not an operational guide. It assumes the metadata-enhancement roadmap is underway and focuses on the human-coordination infrastructure that enrichment at scale requires.

↑ back to contents10.2-archive-dashboard-project(71)/archive-dashboard-project/docs/volunteer-contribution.md

Volunteer Contribution Platform — Design Notes

Status: design/scoping, not yet built. This document works through the infrastructure the volunteer program (volunteer-contribution.md) needs: how volunteers see and edit data, how edits round-trip safely into the canonical databases, whether SQLite still suffices, and how tasks, review, reputation, and gamification fit together. It builds on the three-database ecosystem (metadata-db-integration.md) and reuses its "claimed → verified" discipline.

The core realization: one new multi-writer store, not four converted ones

The instinct to consider Postgres comes from a real worry — many volunteers writing at once will make SQLite's single-writer lock a bottleneck. But that worry applies to the wrong databases. The three canonical stores (collector.db, archive.db, civic.db) are single-writer by design, each written by one pipeline on a cadence, and that is a feature, not a limitation: it is what makes them safe to rebuild, promote, and federate read-only.

Volunteers must never write the canonical databases directly. The moment they did, we'd lose the single-writer guarantee, the atomic build-then-promote, and the clean federation. So the multi-writer requirement doesn't belong to the canonical DBs at all — it belongs to a new, separate store that holds volunteer activity:

   VOLUNTEERS (many, concurrent)
        │  submit edits, claim tasks
        ▼
  +-------------------------------------------------+
  |  contrib store   (the ONLY multi-writer store)  |
  |  tasks · submissions · reviews · reputation     |
  +-------------------------------------------------+
        │ reads canonical DBs read-only (to build tasks + show context)
        │ writes back ONLY accepted edits, via a reviewed merge job
        ▼
  +------------+   +------------+   +------------+
  | collector  |   |  archive   |   |   civic    |   ← still single-writer,
  |    .db     |   |    .db     |   |    .db     |     still federated read-only
  +------------+   +------------+   +------------+

This keeps everything the project already relies on and localizes the concurrency problem to one component we can choose the right technology for.

Does SQLite still work? Mostly yes — with one caveat

• Canonical DBs: keep SQLite. Nothing changes. Single writer, WAL for concurrent readers, Datasette for serving. The volume and access pattern that made SQLite the right call still hold.
• Contrib store: SQLite in WAL is fine to start; Postgres if/when write volume demands it. SQLite in WAL mode allows many concurrent readers and one writer at a time; writes are serialized but each volunteer edit is tiny (milliseconds), so a single writer can absorb a surprising rate — realistically hundreds of submissions/minute, far beyond early program scale. The honest threshold: if sustained concurrent writes ever cause lock timeouts, that is the signal to move the contrib store (and only that store) to Postgres. The schema is designed to port cleanly, so this is a swap, not a redesign.

The decision rule, stated plainly: don't migrate to Postgres preemptively. Build the contrib store on SQLite/WAL, measure, and move just that one store to Postgres only if real write contention appears. The canonical three never move.

How edits round-trip (the "claimed → verified → merge" reuse)

The project already has the exact pattern this needs, proven on videos:

1. A volunteer edit is a claim. It's written to the contrib store as a submission (status submitted), never to the canonical DB. It records the target (which DB, table, row, field), the proposed value, the volunteer, and a timestamp.
2. A reviewer verifies it. A trusted volunteer (see the ladder in volunteer-contribution.md §3) approves or rejects; status becomes accepted or rejected. High-stakes fields can require two independent approvals.
3. A merge job applies accepted edits. On a cadence, a job reads all accepted submissions for a canonical DB and applies them — exactly like the existing sync/promote flow: build a new copy, apply the accepted edits with full provenance, integrity-check, atomically promote. The canonical DB thus still has exactly one writer (the merge job), preserving every guarantee.

Provenance travels with every applied edit (who proposed, who reviewed, when, from which submission), so nothing is silently overwritten and any edit can be traced or rolled back — the same soft-delete/history discipline already in the schema.

How volunteers access the data — three options, phased

Option A — CSV round-trip (today's approach). Export a slice ("give me 200 unconfirmed governments in Nebraska"), volunteers edit in a spreadsheet, re-import as submissions. Pros: zero new services, familiar tools. Cons: manual, no live validation, no concurrency awareness, easy to fumble a re-import. Verdict: fine for a handful of expert volunteers right now; it does not scale to an open program, but it's a valid Phase 0 that needs only small export/import helpers.

Option B — Mutable Datasette over the contrib store. Datasette already serves these databases read-only; a writable Datasette instance (canned write queries / the write API) pointed at the contrib store gives volunteers a web UI with no local database access, per-row editing, and built-in query views. Pros: reuses tech already in the stack, quick to stand up, good for the civic catalog's tabular editing. Cons: Datasette is read-optimized; it's a thin editing surface, not a task-queue/review system, so it covers access but not workflow. Verdict: a strong Phase 1 for the civic-catalog track specifically — tabular, low-volume, benefits most from a simple grid UI.

Option C — A purpose-built contribution web app + API. A lightweight web application whose API owns all writes to the contrib store, presents micro-tasks, captures submissions, runs the review queue, and tracks reputation. Pros: the real answer — it's where task distribution, review, reputation, and gamification live; it can present the right micro-task UI per track (a transcript editor, a "confirm this government" card, a "is this the right channel?" yes/no). Cons: the most build effort. Verdict: the Phase 2 destination; Options A and B are stepping stones that don't block it.

Phasing: A now (expert volunteers, CSV), B for the civic catalog (writable Datasette, quick win), C as the program opens up (the full platform).

The task-distribution, review, reputation & gamification system

All of this lives in the contrib store and (in Phase 2) the contribution app.

Task queue

• Task generation. A job reads the canonical DBs read-only and materializes tasks — e.g. "confirm government X" (civic rows with unconfirmed), "find the agendas page for Y" (governments with no portal URL), "correct segment Z" (items flagged needs_review), "is channel C official for station S?" A task names its track, its target row, and what "done" means.
• Leasing, not assigning. A volunteer requests a task; the queue hands out the next appropriate one and leases it (a short hold with a timeout) so two people don't do the same unit. If the lease expires unfinished, the task returns to the pool. This prevents duplicate work without locking anyone out.
• Right-sized & routed. Tasks are filtered by track and by the volunteer's tier (newcomers get well-defined, low-stakes tasks; trusted volunteers can lease review tasks). Locality-aware routing — offering a volunteer governments/channels in their own region — makes the local-knowledge work land where it's strongest.

Review workflow (states)

  submitted ──▶ in_review ──▶ accepted ──▶ (merged into canonical DB)
       │
       └──────────▶ rejected (with a reason; volunteer can learn/redo)

• Every submission enters submitted. A tier-2+ reviewer moves it to accepted or rejected (with a short reason). Only accepted submissions are eligible for the merge job. High-stakes fields (e.g. a speaker identification, a government's canonical name) can require two independent accepted votes; consensus resolves disagreement.
• Sampling. Stewards spot-check a fraction of already-accepted work to keep reviewers calibrated — quality control on the quality-controllers.

Reputation (climbing the ladder)

• Each volunteer has a reputation score driven by accepted contributions (not raw submissions — quality, not volume). Rejections don't punish beyond not earning; honest mistakes carry no penalty.
• Reputation gates the ladder from volunteer-contribution.md §3: enough accepted work in a track promotes a Contributor to Reviewer (can approve others' work), and sustained stewardship to Domain steward. This is how the program scales its own supervision — trusted volunteers review newcomers, so not every edit hits a paid coordinator.
• Trust reduces friction. Higher reputation earns lighter review (e.g. sampled rather than full review), concentrating scarce reviewer attention on newcomers and high-stakes fields.

Gamification (motivation, tied to quality)

Recognition is the volunteer program's currency; a few mechanics, all keyed to accepted work so they reward real contribution, not gaming:

• Leaderboards — per track and per region ("top contributors in Kansas this month"), plus all-time. Regional boards tap local pride and keep them human-scaled rather than one intimidating global list.
• Streaks & milestones — consecutive active weeks; badges at 10/100/1000 accepted contributions; a "first accepted edit" welcome badge.
• Track-specific badges — "Portal Historian" (traced N vendor migrations in Track 1), "Channel Scout" (onboarded N channels in Track 2), "Transcript Editor" (N corrected segments in Track 3), "Local Historian" (N speaker/place identifications). Badges map to the tracks so people see progress in the work they chose.
• Visible impact — "your confirmed governments are now searchable by 1,200 researchers," "the meeting you transcribed has been viewed 340 times." Impact beats points for sustained motivation on civic work.

A caution: gamify accepted, reviewed work only. Rewarding raw submission volume invites low-quality spam; rewarding accepted work aligns the incentive with the mission.

A sketch of the contrib store schema (illustrative)

Not final — just to show the shape. One store, multi-writer, portable to Postgres.

• task — id, track, target_db, target_table, target_key, kind, prompt, state (open/leased/done), lease_owner, lease_expires.
• submission — id, task_id, volunteer_id, target (db/table/key/field), proposed value, status (submitted/in_review/accepted/rejected), created_at.
• review — id, submission_id, reviewer_id, decision, reason, created_at.
• volunteer — id, display_name, region, tier, reputation, joined_at.
• badge / volunteer_badge — the gamification layer.
• merge_log — which accepted submissions were applied to which canonical DB when (provenance + rollback).

The canonical DBs are attached read-only for task generation and context; they are written only by their own merge job.

Open questions

1. Contrib store engine: start on SQLite/WAL (recommended) and port to Postgres only if measured write contention demands — or start on Postgres now to avoid a later migration? (Trade-off: operational simplicity now vs. a possible swap later. The schema is written to port cleanly either way.)
2. Identity & auth: how do volunteers sign in (email magic-link, OAuth via a civic/education identity, etc.), and what's the minimum PII we store?
3. Datasette-write vs. custom app for Phase 1 civic editing: is writable Datasette enough for the civic catalog to start, or do we jump to the custom app sooner?
4. Merge cadence: how often does the merge job fold accepted edits into each canonical DB — nightly, weekly (aligned to the existing sync), on a threshold?
5. Conflict handling: two accepted edits touching the same field — last-write, steward-resolves, or task-level locking to prevent it upstream?
6. The "use of contributions" policy (volunteer-contribution.md FAQ) — the licensing/AI-training answer should be settled before the platform opens.

↑ back to contents10.1-ia_metadata_xml_to_sql(26)/ia_metadata_xml_to_sqlite/docs/VOLUNTEER_PLATFORM_DESIGN.md

archive.org 30-day search challenge

A volunteer engagement challenge for the Archive Corps effort — a 30-day community-media search activity with its schedule and participant tracking.

Contents

↑ back to contents40.2-archive.org-30-day-search-challenge/README.md

Patch — fix ASCII diagram collapsing in OVERVIEW.docx

Symptom: the ASCII federation diagram renders with correct monospace spacing in the Markdown and PDF, but collapses to proportional spacing in the .docx, destroying the box alignment.

Root cause: the builder specified Consolas, which is a Windows-only font. When the reader's Word/LibreOffice cannot find it and no monospace fallback is declared, it substitutes the proportional document default (Calibri) — and every character width changes, so the boxes collapse.

This is environment-dependent, which is why it did not show up in the Linux LibreOffice conversion used to build the PDF: there, Consolas silently falls back to DejaVu Sans Mono, which is monospace, so the PDF looked fine while the .docx broke on the reader's machine.

Fix: name a font that exists everywhere, and declare it in two places so a renderer that drops one still honours the other.

Change 1 — the font constant

// BEFORE
const MONO = "Consolas";

// AFTER
const MONO = "Courier New";   // present on Windows, macOS, and every Office
                              // install; Linux/LibreOffice maps it to
                              // Liberation Mono. Consolas is Windows-only and
                              // silently falls back to a PROPORTIONAL font
                              // elsewhere, which collapses the diagram.

Change 2 — add a paragraph style carrying the mono font

Add this constant near the other helpers:

// Paragraph style for ASCII diagrams. Belt-and-braces: if a renderer ignores
// the run-level font property, the paragraph style still supplies monospace.
const MONO_STYLE = {
  id: "DiagramMono",
  name: "Diagram Mono",
  basedOn: "Normal",
  quickFormat: false,
  run: { font: MONO, size: 15 },
  paragraph: { spacing: { after: 0, line: 216 } },
};

Register it in the Document styles block:

const doc = new Document({
  creator: "Community Media Preservation",
  styles: {
    default: { document: { run: { font: FONT, size: 20 } } },
    paragraphStyles: [MONO_STYLE],          // <-- ADD THIS LINE
  },
  sections: [{ /* ... unchanged ... */ }],
});

Change 3 — apply the style in diagram()

// BEFORE
function diagram(lines) {
  return lines.map((ln, i) =>
    new Paragraph({
      spacing: { after: 0, line: 216, before: i === 0 ? 40 : 0 },
      children: [new TextRun({ text: ln || " ", font: MONO, size: 15 })],
    })
  );
}

// AFTER
function diagram(lines) {
  return lines.map((ln, i) =>
    new Paragraph({
      style: "DiagramMono",                          // <-- ADD THIS LINE
      spacing: { after: 0, line: 216, before: i === 0 ? 40 : 0 },
      children: [new TextRun({ text: ln || " ", font: MONO, size: 15 })],
    })
  );
}

Verification

After patching, rebuild and confirm the font appears in both XML parts:

node build_overview_docx.js /tmp/ov.docx
mkdir -p /tmp/ovx && cd /tmp/ovx && unzip -o -q /tmp/ov.docx

grep -o 'w:styleId="DiagramMono"' word/styles.xml     # paragraph style present
grep -o 'Courier New' word/styles.xml                 # font in the style
grep -o 'w:ascii="Courier New"' word/document.xml     # font on the runs

All three must return matches. Then open the .docx in Word (not just the converted PDF) and confirm the box corners line up — the PDF alone will not reveal this class of bug.

Note for other documents

The other bundle documents render their diagrams inside Markdown fenced code blocks, which pandoc maps to its own code style — those were not affected. This fix applies only to build_overview_docx.js, the one purpose-built generator.

If a future document needs an ASCII diagram in a hand-built docx, reuse the DiagramMono style rather than setting a run font alone.

↑ back to contents90.0-diagram-docx-font-style-fix/PATCH-overview-docx-mono-font.md