vjrj
959fa2551c
Grow the bundled species catalog from 14 hand-curated entries to ~1200 edible/cultivated species, internationalized in 13 languages (es, en, fr, de, it, pt, ca, gl, eu, ar, zh, ja, ru — Latin + Arabic RTL + CJK + Cyrillic). - Add a reproducible generator (tool/gen_species_catalog.dart) that queries Wikidata (CC0, no attribution burden) in two phases, filters out non-vernacular noise (author citations, ranks, initials) and applies a relevance floor, then merges hand-curated, authoritative core-crop data (tool/curated_overrides.json: names, family, viability_years). GBIF is used only as an identifier. The generated species.json (v3) is committed. - Carry wikidata_qid and gbif_key through the parse/seed pipeline; the columns already existed, so no DB migration. - Rewrite seedBundled to one read + one batch (was a SELECT per species on every startup — a real cost at ~1200 rows) and keep it idempotent with backfill of the new reference fields. - Move species search filtering to SQL (LIKE) so a large catalog is not pulled into memory on every keystroke. - Cover the generator transform, the generated asset, and the new fields with tests.
2.9 KiB
2.9 KiB
Species catalog generator
Regenerates assets/catalog/species.json — the
bundled, offline, multilingual catalog of edible / cultivated plant species the
app seeds into the encrypted database at first run.
cd apps/app_seeds
dart run tool/gen_species_catalog.dart # writes species.json (version 3)
dart run tool/gen_species_catalog.dart --dry # prints stats, writes nothing
The generated JSON is committed to the repo — the app never queries the network for it (local-first, offline).
Sources & licensing
- Names come only from Wikidata labels and aliases. Wikidata is CC0 (public domain), so the bundled data carries no attribution burden and is clean for redistribution under the app's AGPL-3.0 license.
gbif_keylinks the GBIF taxonomic backbone. It is used only as an identifier (a fact, not copyrightable text) — no GBIF vernacular text is bundled, so no CC-BY attribution is triggered.- Curated core crops (names, family and
viability_years) live incurated_overrides.json, hand-maintained. For a listed species the curated common names win (placed first, never dropped) and its family/viability override Wikidata; Wikidata still adds more languages. A curated species is always kept even if it would miss the relevance floor. viability_years(seed longevity) is not in Wikidata — it comes only from the curated overrides (public-domain agricultural-extension figures). Species without an entry simply get no age-based expiry warning.
What it does
- Phase 1 — species list (SPARQL): taxa with a binomial name that are a
crop (
P279* Q235352), the source/ingredient of a food (P1672/P366/reverseP1582→P279* Q2095), with scientific name (P225) and GBIF key (P846). - Phase 2 — names + family:
rdfs:label(primary) +skos:altLabel(aliases), batched over the species and filtered to the bundled languages; the family is walked upP171*to the family-rank ancestor. - Transform (
buildCatalog, pure & unit-tested): dedup by scientific name; drop non-vernacular noise (the binomial itself, author citations,var./×ranks, abbreviated initials); merge the curated overrides (authoritative, first); require a real vernacular in ≥ 4 languages as a relevance floor so the long tail of obscure single-label taxa is dropped; sort; serialize.
The current run yields ~1200 species (assets/catalog/species.json ≈ 1.5 MB).
Bundled languages
es, en, fr, de, it, pt, ca, gl, eu, ar, zh, ja, ru — a multilingual seed,
not a ceiling (Latin + Arabic RTL + CJK + Cyrillic). Edit kCatalogLanguages in
gen_species_catalog.dart to widen coverage, then
regenerate. A species with no name in any bundled language is dropped (add its
language rather than assuming es/en).