feat(species): expand catalog to ~1200 edible species from Wikidata
Grow the bundled species catalog from 14 hand-curated entries to ~1200 edible/cultivated species, internationalized in 13 languages (es, en, fr, de, it, pt, ca, gl, eu, ar, zh, ja, ru — Latin + Arabic RTL + CJK + Cyrillic). - Add a reproducible generator (tool/gen_species_catalog.dart) that queries Wikidata (CC0, no attribution burden) in two phases, filters out non-vernacular noise (author citations, ranks, initials) and applies a relevance floor, then merges hand-curated, authoritative core-crop data (tool/curated_overrides.json: names, family, viability_years). GBIF is used only as an identifier. The generated species.json (v3) is committed. - Carry wikidata_qid and gbif_key through the parse/seed pipeline; the columns already existed, so no DB migration. - Rewrite seedBundled to one read + one batch (was a SELECT per species on every startup — a real cost at ~1200 rows) and keep it idempotent with backfill of the new reference fields. - Move species search filtering to SQL (LIKE) so a large catalog is not pulled into memory on every keystroke. - Cover the generator transform, the generated asset, and the new fields with tests.
This commit is contained in:
parent
ba87bf2719
commit
959fa2551c
9 changed files with 68483 additions and 120 deletions
56
apps/app_seeds/tool/README.md
Normal file
56
apps/app_seeds/tool/README.md
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
# Species catalog generator
|
||||
|
||||
Regenerates [`assets/catalog/species.json`](../assets/catalog/species.json) — the
|
||||
bundled, offline, multilingual catalog of edible / cultivated plant species the
|
||||
app seeds into the encrypted database at first run.
|
||||
|
||||
```sh
|
||||
cd apps/app_seeds
|
||||
dart run tool/gen_species_catalog.dart # writes species.json (version 3)
|
||||
dart run tool/gen_species_catalog.dart --dry # prints stats, writes nothing
|
||||
```
|
||||
|
||||
The generated JSON is **committed** to the repo — the app never queries the
|
||||
network for it (local-first, offline).
|
||||
|
||||
## Sources & licensing
|
||||
|
||||
- **Names** come only from **Wikidata** labels and aliases. Wikidata is **CC0**
|
||||
(public domain), so the bundled data carries **no attribution burden** and is
|
||||
clean for redistribution under the app's AGPL-3.0 license.
|
||||
- **`gbif_key`** links the [GBIF](https://www.gbif.org/) taxonomic backbone. It
|
||||
is used **only as an identifier** (a fact, not copyrightable text) — no GBIF
|
||||
vernacular text is bundled, so no CC-BY attribution is triggered.
|
||||
- **Curated core crops** (names, family and `viability_years`) live in
|
||||
[`curated_overrides.json`](curated_overrides.json), hand-maintained. For a
|
||||
listed species the curated common names **win** (placed first, never dropped)
|
||||
and its family/viability override Wikidata; Wikidata still adds more languages.
|
||||
A curated species is always kept even if it would miss the relevance floor.
|
||||
- **`viability_years`** (seed longevity) is **not** in Wikidata — it comes only
|
||||
from the curated overrides (public-domain agricultural-extension figures).
|
||||
Species without an entry simply get no age-based expiry warning.
|
||||
|
||||
## What it does
|
||||
|
||||
1. **Phase 1 — species list (SPARQL):** taxa with a binomial name that are a
|
||||
*crop* (`P279* Q235352`), the source/ingredient of a *food*
|
||||
(`P1672`/`P366`/reverse `P1582` → `P279* Q2095`), with scientific name
|
||||
(`P225`) and GBIF key (`P846`).
|
||||
2. **Phase 2 — names + family:** `rdfs:label` (primary) + `skos:altLabel`
|
||||
(aliases), batched over the species and filtered to the bundled languages;
|
||||
the family is walked up `P171*` to the family-rank ancestor.
|
||||
3. **Transform (`buildCatalog`, pure & unit-tested):** dedup by scientific name;
|
||||
drop non-vernacular noise (the binomial itself, author citations, `var.`/`×`
|
||||
ranks, abbreviated initials); merge the curated overrides (authoritative,
|
||||
first); require a real vernacular in **≥ 4 languages** as a relevance floor so
|
||||
the long tail of obscure single-label taxa is dropped; sort; serialize.
|
||||
|
||||
The current run yields **~1200 species** (`assets/catalog/species.json` ≈ 1.5 MB).
|
||||
|
||||
## Bundled languages
|
||||
|
||||
`es, en, fr, de, it, pt, ca, gl, eu, ar, zh, ja, ru` — a multilingual **seed**,
|
||||
not a ceiling (Latin + Arabic RTL + CJK + Cyrillic). Edit `kCatalogLanguages` in
|
||||
[`gen_species_catalog.dart`](gen_species_catalog.dart) to widen coverage, then
|
||||
regenerate. A species with no name in any bundled language is dropped (add its
|
||||
language rather than assuming es/en).
|
||||
Loading…
Add table
Add a link
Reference in a new issue