feat(species): expand catalog to ~1200 edible species from Wikidata

Grow the bundled species catalog from 14 hand-curated entries to ~1200
edible/cultivated species, internationalized in 13 languages (es, en, fr,
de, it, pt, ca, gl, eu, ar, zh, ja, ru — Latin + Arabic RTL + CJK + Cyrillic).

- Add a reproducible generator (tool/gen_species_catalog.dart) that queries
  Wikidata (CC0, no attribution burden) in two phases, filters out
  non-vernacular noise (author citations, ranks, initials) and applies a
  relevance floor, then merges hand-curated, authoritative core-crop data
  (tool/curated_overrides.json: names, family, viability_years). GBIF is used
  only as an identifier. The generated species.json (v3) is committed.
- Carry wikidata_qid and gbif_key through the parse/seed pipeline; the columns
  already existed, so no DB migration.
- Rewrite seedBundled to one read + one batch (was a SELECT per species on
  every startup — a real cost at ~1200 rows) and keep it idempotent with
  backfill of the new reference fields.
- Move species search filtering to SQL (LIKE) so a large catalog is not pulled
  into memory on every keystroke.
- Cover the generator transform, the generated asset, and the new fields with
  tests.
This commit is contained in:
vjrj 2026-07-10 02:00:47 +02:00
parent ba87bf2719
commit 959fa2551c
9 changed files with 68483 additions and 120 deletions

View file

@ -0,0 +1,56 @@
# Species catalog generator
Regenerates [`assets/catalog/species.json`](../assets/catalog/species.json) — the
bundled, offline, multilingual catalog of edible / cultivated plant species the
app seeds into the encrypted database at first run.
```sh
cd apps/app_seeds
dart run tool/gen_species_catalog.dart # writes species.json (version 3)
dart run tool/gen_species_catalog.dart --dry # prints stats, writes nothing
```
The generated JSON is **committed** to the repo — the app never queries the
network for it (local-first, offline).
## Sources & licensing
- **Names** come only from **Wikidata** labels and aliases. Wikidata is **CC0**
(public domain), so the bundled data carries **no attribution burden** and is
clean for redistribution under the app's AGPL-3.0 license.
- **`gbif_key`** links the [GBIF](https://www.gbif.org/) taxonomic backbone. It
is used **only as an identifier** (a fact, not copyrightable text) — no GBIF
vernacular text is bundled, so no CC-BY attribution is triggered.
- **Curated core crops** (names, family and `viability_years`) live in
[`curated_overrides.json`](curated_overrides.json), hand-maintained. For a
listed species the curated common names **win** (placed first, never dropped)
and its family/viability override Wikidata; Wikidata still adds more languages.
A curated species is always kept even if it would miss the relevance floor.
- **`viability_years`** (seed longevity) is **not** in Wikidata — it comes only
from the curated overrides (public-domain agricultural-extension figures).
Species without an entry simply get no age-based expiry warning.
## What it does
1. **Phase 1 — species list (SPARQL):** taxa with a binomial name that are a
*crop* (`P279* Q235352`), the source/ingredient of a *food*
(`P1672`/`P366`/reverse `P1582``P279* Q2095`), with scientific name
(`P225`) and GBIF key (`P846`).
2. **Phase 2 — names + family:** `rdfs:label` (primary) + `skos:altLabel`
(aliases), batched over the species and filtered to the bundled languages;
the family is walked up `P171*` to the family-rank ancestor.
3. **Transform (`buildCatalog`, pure & unit-tested):** dedup by scientific name;
drop non-vernacular noise (the binomial itself, author citations, `var.`/`×`
ranks, abbreviated initials); merge the curated overrides (authoritative,
first); require a real vernacular in **≥ 4 languages** as a relevance floor so
the long tail of obscure single-label taxa is dropped; sort; serialize.
The current run yields **~1200 species** (`assets/catalog/species.json` ≈ 1.5 MB).
## Bundled languages
`es, en, fr, de, it, pt, ca, gl, eu, ar, zh, ja, ru` — a multilingual **seed**,
not a ceiling (Latin + Arabic RTL + CJK + Cyrillic). Edit `kCatalogLanguages` in
[`gen_species_catalog.dart`](gen_species_catalog.dart) to widen coverage, then
regenerate. A species with no name in any bundled language is dropped (add its
language rather than assuming es/en).