Home / Real cases end to end

Building a data asset, synthetic dataset or knowledge base

A synthetic training dataset and an internal knowledge base share the same four-operation architecture, a shared schema, a matching identifier, deduplication and validation before any downstream use.

Generating a thousand synthetic training examples and building an internal knowledge base for an agent share the same architecture under different names. Both go through four shared operations: batch ingestion or generation, deduplication, linking through a shared schema, then recall via a lightweight index.

A schema before the first record

The schema fixes the fields each record carries before the first one is created, whether it comes from batch generation or from ingesting existing documents. Each record also carries a matching identifier, stable from one pass to the next, which links a new entry to an already known entry without depending on the text content itself.

Deduplicate before validating, never after

Deduplication runs on the matching identifier and on a fingerprint of the content, before anyone reviews the batch. A duplicate not removed at this stage propagates into every validation that follows and skews the final count. The example below builds five fake records, including two deliberate duplicates, to show the mechanics without touching a real file.

# Données fabriquées ici même, aucun fichier réel n'est touché
enregistrements = [
    {"id": "corr-001", "texte": "ponceuse à bande"},
    {"id": "corr-002", "texte": "aspirateur d'atelier"},
    {"id": "corr-001", "texte": "ponceuse à bande"},
    {"id": "corr-003", "texte": "meuleuse d'angle"},
    {"id": "corr-002", "texte": "aspirateur d'atelier"},
]

vus = set()
uniques = []
for enregistrement in enregistrements:
    if enregistrement["id"] not in vus:
        vus.add(enregistrement["id"])
        uniques.append(enregistrement)

print(len(enregistrements), "enregistrements,", len(uniques), "uniques après déduplication")

Five fabricated entries, three kept after deduplication on the identifier: the same logic applies to five hundred or fifty thousand rows.

Batch processing that changes the cost, not the method

When generation or validation happens through repeated calls to a model, grouping these calls into a batch process rather than sending them one by one changes the cost without changing the method. This lesson's metric figure gives a figure for that gap. This same discipline of a shared schema and recall through a lightweight index then serves as the foundation for fan-out research, which verifies every claim before keeping it.