# ADR 0002 — Retirement of Specialized Importers in Favor of Generic Import Registry

*Status: Adopted (2025-10-08)*

## Context

- Current importers (`TaxonomyImporter`, `PlotImporter`, `ShapeImporter`, `OccurrenceImporter`) directly manipulate SQLite, recalculate nested sets, and enforce three tables (`taxon_ref`, `plot_ref`, `shape_ref`).
- Transform/Export have been generalized (plugins + config) but remain coupled to these tables.
- The "Generic Import System" roadmap defines a declarative configuration (`entities.references/datasets`) and an Entity Registry to centralize schemas and metadata.
- We are still in alpha ⇒ no backward compatibility requirement.

## Decision

We are progressively retiring specialized importers in favor of a generic engine driven by the new configuration:

- Creation of a persistent **Entity Registry** describing each entity (type, physical table, links, aliases).
- A new import engine will orchestrate connectors, validations, hierarchies, and enrichments relying on DuckDB.
- Transform/Export/GUI will consume the Registry instead of querying fixed tables.
- The `core/components/imports/*` modules will be removed once functional equivalence is achieved.

## Consequences

### Positive

- Ability to describe/import any entity (third-party taxonomy, sites, habitats, etc.).
- Reduction of duplicated code (CSV/Geo validation, table creation, nested sets).
- Complete pipeline alignment (import → transform → export) on declarative configuration.
- Cleanup of plugin dependencies: they will no longer need to load `Config` or `Database` directly.

### Negative / Points of Attention

- Migration of existing plugins (`hierarchical_nav_widget`, `geospatial_extractor`, `top_ranking`, HTML exporters) to handle adjacency list or compatibility views.
- Update tests (unit, integration, CLI) that assume `taxon_ref` existence.
- Significant documentation work (new `import.yml`, examples, GUI guides).
- Risk of regression during switchover; plan fixtures and end-to-end tests.

### Actions

1. Finalize Pydantic models (`config_models`) ✅ (first draft in place).
2. Design Entity Registry + metadata storage (`niamoto_metadata.*` tables).
3. Implement generic import engine (DuckDB connectors, validations, execution plan).
4. Adapt Transform/Export to use Registry (remove `Config()` coupling in plugins).
5. Retire legacy importers and remove rigid SQLAlchemy models.
6. Update documentation (`docs/08-roadmaps/generic-import-ultrathink.md`, GUI README) and provide migration guide.

## Follow-up 2025-10-08
- ✅ Operational DuckDB registry: `registry.py`/`legacy_registry.py` expose entities to Transform/Export services and GUI.
- ✅ `direct_reference` loader and geospatial extractor migrated to registry; associated tests updated.
- 🔄 CLI `stats` command and loaders still depending on `sqlite_master` to be migrated before historical importer removal.
- 📌 Next step: remove `core/components/imports/*` and SQLAlchemy models once CLI is migrated and CLI tests are reinforced.