Building jev-curate: Fast Synthetic Dataset Sifter in Rust

Synthetic dataset pipelines have a silent failure mode: low-quality rows slip into training runs. LLM generators produce subtle hallucinations, boilerplate apologies, repetitive sentence patterns, and invalid JSON structures.

Traditional solutions fall into two bad extremes:

1. Regex heuristics: fast, but blind to semantic quality and coherence.
2. Generative LLM judges: accurate, but cost $3 to $15 per million tokens and take 800ms or more per row. Evaluating a 100,000-row dataset costs hundreds of dollars and runs for hours.

I built jev-curate to solve this with a third approach: TypeSafe AI's Jev model (jev-latest).

+------------------+        +----------------------+        +---------------------+
| Input Dataset    |        | jev-curate (Rust)    |        | TypeSafe AI (Jev)   |
| .jsonl / .parquet| -----> | Rayon Worker Pool    | <----> | /v1/systemone       |
+------------------+        | BufWriter Streaming  |        | 70-120ms, $0.042/M  |
                            +----------------------+        +---------------------+
                                       |
                                       +---> clean.jsonl (passed)
                                       +---> rejected.jsonl (with reason)

Why System One typed decisions

Jev is a System One decision model. It does not generate text, summarize essays, or write prose. It evaluates input state against criteria and returns typed primitives: Choice, Score, or Noul (probability) with calibrated confidence scores.

Key characteristics for dataset curation:

• Cost: $0.042 per million tokens. That is 70x to 350x cheaper than generative judges.
• Latency: 70ms to 150ms per evaluation.
• Calibrated confidence: Every verdict includes a confidence value between 0.0 and 1.0. A filter rule can require p_yes >= 0.80 to reject borderline rows.
• Speculative fan-out: Send dataset context once and batch multiple quality checks into a single API request.

Architecture

jev-curate is a single statically linked Rust binary.

src/
  main.rs         # CLI argument parsing (clap) and orchestration
  api.rs          # TypeSafe AI System One HTTP client with exponential backoff
  dataset.rs      # Streaming JSONL reader and batcher
  parquet_io.rs   # Parquet reader and BufWriter dataset streamer
  rules.rs        # Quality criteria definitions and threshold evaluator

1. Streaming disk I/O with BufWriter

A naive implementation buffers records in memory before writing clean and rejected outputs. On a 4GB Parquet file or multi-million line JSONL file, that causes OOM kills on laptops with 8GB RAM.

jev-curate streams rows straight to disk:

pub struct DatasetWriter {
    clean_writer: BufWriter<File>,
    rejected_writer: BufWriter<File>,
}

impl DatasetWriter {
    pub fn write_clean(&mut self, row: &str) -> Result<(), std::io::Error> {
        self.clean_writer.write_all(row.as_bytes())?;
        self.clean_writer.write_all(b"\n")
    }

    pub fn write_rejected(&mut self, row: &str, reason: &str) -> Result<(), std::io::Error> {
        let entry = serde_json::json!({
            "record": row,
            "rejection_reason": reason
        });
        serde_json::to_writer(&mut self.rejected_writer, &entry)?;
        self.rejected_writer.write_all(b"\n")
    }
}

Memory usage stays flat under 40MB RSS regardless of dataset size.

2. Spot audits via dry-run

Before running a pipeline over millions of records, developers need to verify their curation thresholds on a small sample.

Running jev-curate sift --input data.jsonl --dry-run --limit 20 samples 20 rows, displays the Jev verdict and calibrated confidence for each row in a colorized table, and exits without writing files to disk:

Row 1: PASS (p_yes: 0.94, confidence: 0.91)
Row 2: REJECT [hallucinated reference] (p_yes: 0.21, confidence: 0.88)
Row 3: PASS (p_yes: 0.87, confidence: 0.84)

CLI Usage

Install via cargo:

cargo install jev-curate

Sift a dataset:

jev-curate sift \
  --input synthetic_qa.jsonl \
  --output-clean clean.jsonl \
  --output-rejected rejected.jsonl \
  --confidence-threshold 0.80 \
  --concurrency 16

Parquet files work without conversion:

jev-curate sift \
  --input training_set.parquet \
  --output-clean clean.parquet \
  --output-rejected rejected.parquet

Benchmarks & Verification

During testing, the engine verified:

• JSONL parsing throughput: over 120,000 rows per second from local NVMe.
• Parquet column extraction: zero allocation on discarded columns.
• Test coverage: 4 unit and integration tests passing in 0.37 seconds.

Published as v0.1.0 on crates.io and GitHub under the MIT license.