vjrj
959fa2551c
Grow the bundled species catalog from 14 hand-curated entries to ~1200 edible/cultivated species, internationalized in 13 languages (es, en, fr, de, it, pt, ca, gl, eu, ar, zh, ja, ru — Latin + Arabic RTL + CJK + Cyrillic). - Add a reproducible generator (tool/gen_species_catalog.dart) that queries Wikidata (CC0, no attribution burden) in two phases, filters out non-vernacular noise (author citations, ranks, initials) and applies a relevance floor, then merges hand-curated, authoritative core-crop data (tool/curated_overrides.json: names, family, viability_years). GBIF is used only as an identifier. The generated species.json (v3) is committed. - Carry wikidata_qid and gbif_key through the parse/seed pipeline; the columns already existed, so no DB migration. - Rewrite seedBundled to one read + one batch (was a SELECT per species on every startup — a real cost at ~1200 rows) and keep it idempotent with backfill of the new reference fields. - Move species search filtering to SQL (LIKE) so a large catalog is not pulled into memory on every keystroke. - Cover the generator transform, the generated asset, and the new fields with tests.
56 lines
2.9 KiB
Markdown
56 lines
2.9 KiB
Markdown
# Species catalog generator
|
||
|
||
Regenerates [`assets/catalog/species.json`](../assets/catalog/species.json) — the
|
||
bundled, offline, multilingual catalog of edible / cultivated plant species the
|
||
app seeds into the encrypted database at first run.
|
||
|
||
```sh
|
||
cd apps/app_seeds
|
||
dart run tool/gen_species_catalog.dart # writes species.json (version 3)
|
||
dart run tool/gen_species_catalog.dart --dry # prints stats, writes nothing
|
||
```
|
||
|
||
The generated JSON is **committed** to the repo — the app never queries the
|
||
network for it (local-first, offline).
|
||
|
||
## Sources & licensing
|
||
|
||
- **Names** come only from **Wikidata** labels and aliases. Wikidata is **CC0**
|
||
(public domain), so the bundled data carries **no attribution burden** and is
|
||
clean for redistribution under the app's AGPL-3.0 license.
|
||
- **`gbif_key`** links the [GBIF](https://www.gbif.org/) taxonomic backbone. It
|
||
is used **only as an identifier** (a fact, not copyrightable text) — no GBIF
|
||
vernacular text is bundled, so no CC-BY attribution is triggered.
|
||
- **Curated core crops** (names, family and `viability_years`) live in
|
||
[`curated_overrides.json`](curated_overrides.json), hand-maintained. For a
|
||
listed species the curated common names **win** (placed first, never dropped)
|
||
and its family/viability override Wikidata; Wikidata still adds more languages.
|
||
A curated species is always kept even if it would miss the relevance floor.
|
||
- **`viability_years`** (seed longevity) is **not** in Wikidata — it comes only
|
||
from the curated overrides (public-domain agricultural-extension figures).
|
||
Species without an entry simply get no age-based expiry warning.
|
||
|
||
## What it does
|
||
|
||
1. **Phase 1 — species list (SPARQL):** taxa with a binomial name that are a
|
||
*crop* (`P279* Q235352`), the source/ingredient of a *food*
|
||
(`P1672`/`P366`/reverse `P1582` → `P279* Q2095`), with scientific name
|
||
(`P225`) and GBIF key (`P846`).
|
||
2. **Phase 2 — names + family:** `rdfs:label` (primary) + `skos:altLabel`
|
||
(aliases), batched over the species and filtered to the bundled languages;
|
||
the family is walked up `P171*` to the family-rank ancestor.
|
||
3. **Transform (`buildCatalog`, pure & unit-tested):** dedup by scientific name;
|
||
drop non-vernacular noise (the binomial itself, author citations, `var.`/`×`
|
||
ranks, abbreviated initials); merge the curated overrides (authoritative,
|
||
first); require a real vernacular in **≥ 4 languages** as a relevance floor so
|
||
the long tail of obscure single-label taxa is dropped; sort; serialize.
|
||
|
||
The current run yields **~1200 species** (`assets/catalog/species.json` ≈ 1.5 MB).
|
||
|
||
## Bundled languages
|
||
|
||
`es, en, fr, de, it, pt, ca, gl, eu, ar, zh, ja, ru` — a multilingual **seed**,
|
||
not a ceiling (Latin + Arabic RTL + CJK + Cyrillic). Edit `kCatalogLanguages` in
|
||
[`gen_species_catalog.dart`](gen_species_catalog.dart) to widen coverage, then
|
||
regenerate. A species with no name in any bundled language is dropped (add its
|
||
language rather than assuming es/en).
|