Data Pipeline & Quality

FilamentOps now treats catalog data as a structured pipeline, not just static storage. This gives each record lineage, quality scoring, confidence, and safe moderation before publishing.

Data Model Layers

Source Ingestion Flow

1. Load source files from vendor registry + active market outputs (`data/vendors/registry.json`, Bambu unified data, and vendor market links).
2. Normalize vendor/material/color/code tokens.
3. Build source indexes (source ID, internal code, material+color key).
4. Merge rows into canonical records with region/package coverage.
5. Join community suggestion signals.
6. Compute derived quality + verification state.
7. Write source/canonical/derived/staging/conflicts/change-log collections.

Normalization Rules

• Vendor canonicalization (for example, Bambu variants normalize to Bambu Lab).
• Material family normalization (PLA/PETG/ABS/ASA/TPU/PA/PC).
• Code normalization (uppercase alphanumeric with separators).
• Availability normalization to a stable state model: available | sold_out | not_listed | unknown | stale.
• Canonical key composition uses normalized vendor/material/color plus strongest identity (source ID/code/fallback).

Confidence & Verification

Each canonical record receives:

• Completeness score: weighted field coverage quality.
• Confidence score/level: completeness + source count + verification + community signals - conflict penalties.
• Verification status: verified, community_likely, or pending.
• Missing fields list: exact incomplete parts used by contextual prompts.

Conflict Detection & Staging

• Conflicts are generated when source identity signals disagree (for example, internal code mismatch).
• Records with low completeness, missing key identity, missing buy URL, or unresolved conflicts are put in staging.
• Moderation can publish staged rows, which marks verification to trusted state.
• Conflicts auto-resolve when no longer detected in later runs, while history is retained.

Website Integration

• Preset Catalog table: data quality chips (DQ%) and confidence now come from canonical pipeline runtime data.
• Filament detail: shows completeness, confidence, verification, conflict count, missing fields, and community signal counts.
• Context-aware prompts: Add Price/Add Store/Add Details actions are shown when missing fields are detected.
• Community linkage: approved/pending/rejected suggestion signal contributes to confidence and verification state.

Automatic Updates

• The pipeline auto-runs at backend startup.
• It continues on a scheduler (`DATA_PIPELINE_INTERVAL_MS`, default: 10 minutes).
• Bambu source sync can run independently; the pipeline ingests the latest active outputs automatically on the next run.

Pipeline API (Moderation/Admin)

• GET /api/data-pipeline/status - latest run + counters.
• POST /api/data-pipeline/run - force a run with reason/actor.
• GET /api/data-pipeline/runtime-map - filtered runtime rows used by UI.
• GET /api/data-pipeline/canonical - canonical records.
• GET /api/data-pipeline/staging - pending stage queue (role-gated).
• POST /api/data-pipeline/staging/:stageId/publish - publish staged item (role-gated).
• GET /api/data-pipeline/conflicts - open/resolved conflicts (role-gated).
• GET /api/data-pipeline/changes - change history (role-gated).

Public Data Access

• GET /api/public/data-hub - central discovery endpoint for active dataset file links and public runtime API links.
• Docs view: Public Data Hub (same links in a readable table format).
• Vendor registry file: /data/vendors/registry.json (used to surface non-Bambu datasets in the same endpoint).
• Primary active files include Bambu market outputs plus adapter datasets such as /data/vendors/<vendor>/catalog-active.json, /data/vendors/<vendor>/market/variant-links-active.json, and /data/vendors/<vendor>/market/store-snapshot-active.json.

Operational Notes

• If Mongo is unavailable, pipeline endpoints return unavailable state and UI falls back safely.
• Pipeline quality labels are advisory; moderation decisions remain role-gated.
• For diagnostics, use status + conflicts + staging endpoints together (they explain most data quality regressions).

Edit History

• 2026-04-16: Added local photo-model weights dataset (localPhotoModelWeights.json), validation coverage, and bounded auto-tune command (npm run diagnosis:local-model:train) wired into diagnosis pipeline.
• 2026-04-16: Added diagnosis data governance additions: trusted-source staging importer (allowlist only), schema validation with confidence/source/verified fields, eval-driven per-issue calibration tables, and automated rule-coverage gap reporting.
• 2026-03-30: Fixed non-Bambu adapter matching to read vendor identity from manufacturer/vendor/brand/maker, restoring SUNLU/eSUN/Polymaker/Overture catalog coverage.
• 2026-03-30: Added exact-vs-fallback buy-link flags for non-Bambu package rows so runtime UI can distinguish true variant links from search links.
• 2026-03-30: Added live non-Bambu vendor market probe flow (SUNLU/eSUN/Polymaker/Overture) with snapshot + override datasets feeding canonical runtime rows.
• 2026-03-30: Added vendor-agnostic registry coverage in Public Data Access (SUNLU + eSUN adapter outputs linked through data hub).
• 2026-03-30: Added public data discovery endpoint and linked Public Data Hub docs page for one-place dataset access.
• 2026-03-26: Added canonical data-pipeline docs (ingestion, normalization, verification/confidence, conflict handling, staging/publish flow, and UI integration).

Sources

Implemented backend/runtime modules and in-app catalog/detail integration in current build.