Every RAG system has the same failure mode, and it is not the retriever. It is the corpus. You point a Bedrock Knowledge Base at an S3 bucket, ingest 40,000 documents, and three weeks later your assistant confidently answers with a policy that was superseded two years ago, or repeats the same paragraph four times because the same PDF was uploaded four times under different names. The embeddings were fine. The chunks were garbage going in.

The fix is boring and it works: clean the corpus before it reaches the vector store. In this tutorial we build a Glue 5.1 PySpark job that reads raw documents from an S3 raw/ prefix, normalizes and filters them, removes exact duplicates by content hash, removes near-duplicates with MinHash locality-sensitive hashing, and then runs the result through an AWS Glue Data Quality ruleset that acts as a gate. If the cleaned data does not meet the rules, the job fails and writes nothing. Only a corpus that passes lands in the clean/ prefix that your Knowledge Base ingests.

The non-obvious design choice is the gate. Most pipelines clean data and hope. This one treats the cleaned corpus as a contract: uniqueness, completeness, and minimum length are assertions the job must satisfy before it is allowed to publish, so a bad upstream change fails loudly at ETL time instead of silently at answer time.

You need an AWS account with permissions to create S3 buckets, IAM roles, and Glue jobs. You need the AWS CLI v2 configured (aws sts get-caller-identity should return your account), and Python 3.10+ locally with boto3 installed (pip install boto3) to drive the job. Comfort reading PySpark helps, but every block here is runnable as-is.

You do not need a running Bedrock Knowledge Base to finish this tutorial. The output is a clean S3 prefix; wiring it to a Knowledge Base is episode 2 of this series (S3 Vectors). Everything here is serverless, so there is nothing to keep running afterward and nothing to patch.

One version note: AWS shipped Glue 5.1 in December 2025 (Apache Spark 3.5.6, Python 3.11, boto3 1.40 on the workers). The plan for this series said "Glue 5"; we follow reality and use 5.1, which is the current default and is where the newer Spark ML and open-table-format libraries live.

Pick a Region and set a few shell variables. Everything below uses one bucket with three prefixes.

Create the Glue execution role. Glue needs to assume the role, read and write your bucket, and write CloudWatch logs.

Now seed a small, deliberately dirty corpus so you can watch the job earn its keep. Save this as docs.jsonl. Note that kb-002 is an exact duplicate of kb-001, kb-004 is a near-duplicate of kb-003 (one word changed), and kb-005 is a two-word stub that should be filtered out.

Smoke test the setup before writing any Spark: confirm the object landed and the role exists.

You should see docs.jsonl and an ARN like arn:aws:iam::<ACCOUNT_ID>:role/GlueRagCleanerRole. Hold onto that ARN.

Start the job script glue_clean_corpus.py. This first block is the standard Glue 5.1 boilerplate plus reading the JSONL into a Spark DataFrame. We pass the paths and the gate threshold as job arguments so the same script works for any corpus.

getResolvedOptions is the Glue way to read --key value arguments. The spark.read.json call reads newline-delimited JSON when each line is a JSON object, which is why we used .jsonl. If your source is Parquet or CSV, swap the reader; everything downstream operates on a DataFrame and does not care about the input format.

The reason we start with a print of the raw count is discipline. Every stage of this pipeline prints how many rows it drops, so when you run it you get a running tally of exactly where documents die. In a real corpus that tally is your first signal that an upstream export changed.

Raw documents carry HTML tags, collapsed or exploded whitespace, and empty or near-empty stubs. None of that belongs in an embedding. This block strips tags, normalizes whitespace, trims, and drops anything too short to be a useful chunk.

The <[^>]+> regex is a deliberately crude tag stripper, not an HTML parser. For a production corpus with nested markup you would preprocess with a real parser upstream, but for the tag noise that survives most exports this is enough and it runs at Spark speed across the whole dataset. The \s+ collapse matters more than it looks: models tokenize whitespace, so a document full of \n\n\n from a bad PDF extraction wastes context and skews chunk boundaries.

The length filter is the first quiet win. Our kb-005 stub ("See docs.") is 9 characters and gets dropped here. In a real corpus this filter removes navigation fragments, "click here" links, and empty pages that would otherwise become embeddings pointing at nothing. I use 40 characters as a floor for the demo; for prose chunks a floor of 200 to 500 is more realistic.

Exact duplicates are the cheapest to remove and the most common in real corpora, because the same document gets uploaded repeatedly under different filenames. Hash the normalized text and drop rows that share a hash.

Hashing the normalized text rather than the raw text is the important detail. kb-001 arrived wrapped in <p> tags and kb-002 did not, but after Step 2 both are byte-identical, so they hash to the same value and one is dropped. Had we hashed before normalizing, the tags would have made them look different and both would have survived. Clean first, then hash.

We also stamp each surviving row with a stable doc_id using monotonically_increasing_id(). The near-dedup join in the next step needs a stable key to decide which of two similar documents to keep, and the raw src_id may be missing or duplicated in messy data, so we mint our own.

Exact hashing misses the more insidious case: two documents that are 98 percent identical because one is a lightly edited copy of the other. Our kb-003 and kb-004 differ by a single word ("objects" versus "records"). Their hashes are completely different, so Step 3 kept both. To catch these we use MinHash locality-sensitive hashing from Spark ML, which estimates Jaccard similarity between token sets without comparing every pair directly.

approxSimilarityJoin returns pairs whose Jaccard distance is below the threshold, so 0.4 means we treat anything more than 60 percent similar as a duplicate. That threshold is the knob you will tune most: too low and you keep real dupes, too high and you delete legitimately distinct documents that happen to share vocabulary. Start at 0.4, inspect the pairs it flags on a sample, and adjust.

The self-join produces each duplicate pair twice plus a row matched with itself, so we filter to datasetA.doc_id < datasetB.doc_id. That keeps the lower doc_id of every similar pair and marks the higher one for removal, which deterministically drops one of kb-003/kb-004 while keeping the other. The final left_anti join is Spark's set-difference: keep every row in featurized whose doc_id is not in drop_ids.

One caveat worth stating plainly: approxSimilarityJoin is a self-join and gets expensive on large corpora. For millions of documents you bucket first (by source, date, or a coarse prefix) and run the join within buckets, not across the whole set. For the tens-of-thousands range this straightforward version is fine.

Here is the part that turns a cleaning script into a contract. AWS Glue Data Quality lets you assert rules in DQDL, a small declarative language, and evaluate them against a DynamicFrame. We convert our cleaned DataFrame, define the rules the corpus must satisfy, and evaluate.

Read the rules as English. There must be at least one row. The text column must never be null (IsComplete). Every content_hash must be unique, which is the assertion that dedup actually worked. Every value in text must be at least 40 characters long (ColumnLength). And text must be complete for at least 99 percent of rows (Completeness). IsUnique and Completeness here are doing something subtle: they re-verify the invariants Steps 2 through 4 were supposed to establish, so if a future refactor breaks dedup, the gate catches it rather than trusting the upstream code.

DQDL has a deep rule vocabulary beyond these five, including CustomSql for arbitrary SQL assertions, ColumnValues ... matches for regex checks, and freshness rules. The five above are the minimum viable gate for a RAG corpus; the DQDL reference lists the rest.

Now enforce the gate. EvaluateDataQuality.apply returns a DynamicFrame with one row per rule and an Outcome of Passed or Failed. We compute a pass ratio and refuse to write if it falls below the threshold passed in as --min_dq_score.

The raise is the whole point. If the corpus fails the rules, the job errors out, the write never runs, and your Knowledge Base keeps ingesting the last known-good clean/ prefix instead of a poisoned one. A failed gate is a successful outcome: it means you caught the problem at ETL time.

Upload the script and create the Glue 5.1 job from Python. Replace the role ARN with the one from Setup.

GlueVersion="5.1" pins the runtime to Spark 3.5.6 and Python 3.11. G.1X workers give one DPU each (4 vCPU, 16 GB); four of them is plenty for a demo and small enough to keep the bill negligible. The --enable-metrics argument turns on job metrics so you can see DPU usage in the console afterward.

The polling loop just waits for a terminal state. In production you would trigger this from an EventBridge rule on s3:ObjectCreated in raw/, or chain it into a Glue workflow, but for a first run watching the state transitions is the fastest way to learn where time goes.

When the run reaches SUCCEEDED, check the CloudWatch log for the running tally the script prints. You are looking for these lines, in order:

Five in, one stub filtered, one exact dupe removed, one near-dupe removed, leaving two clean documents that pass all five rules. Confirm the output landed:

You should see two JSON records, one for Lambda and one for S3, with no duplicates and no stub. To prove the gate bites, re-run with --min_dq_score set to 1.0 after loosening a rule so it fails (for example change ColumnLength "text" >= 40 to >= 5000). The run ends in FAILED, the log shows FAILED: ColumnLength ..., and s3://$BUCKET/clean/ is untouched from the previous good run. That untouched output is the contract working.

If the job fails immediately with ModuleNotFoundError: No module named 'awsgluedq', you are almost certainly running the script outside Glue (locally, or in a plain Spark session). The awsgluedq and awsglue modules only exist on the Glue runtime; run through create_job/start_job_run, not spark-submit on your laptop.

If you see AccessDenied on the S3 read or write, the execution role is missing the bucket in its inline policy, or you passed the wrong bucket name in the job arguments. Re-check that the S3Access policy Resource ARNs match $BUCKET exactly, including the /* variant for objects.

If approxSimilarityJoin runs for a long time or the job runs out of memory, your corpus is too large for a naive self-join. Bucket the data first and join within buckets, or raise NumberOfWorkers. For the five-row demo this never happens; it shows up around the hundreds-of-thousands mark.

If the DQ score is lower than you expect and IsUnique "content_hash" is the failing rule, dedup did not actually run before the gate, usually because the DataFrame you converted to a DynamicFrame was the pre-dedup one. Confirm you converted deduped, not cleaned or exact. This is exactly the class of bug the gate exists to catch.

If the job succeeds but clean/ is empty, the gate failed silently in your reading of the log. Search the log for Quality gate failed; a raised exception marks the run FAILED, so a SUCCEEDED run with empty output means the write path or the clean_path argument is wrong, not the gate.

First, make the gate richer. Add CustomSql "select count(*) from primary where text like '%lorem ipsum%'" = 0 to reject placeholder text, or a ColumnValues "text" matches rule to enforce a language or encoding constraint. Each rule you add is a class of corpus poison you will never debug at answer time again.

Second, wire the clean prefix into the rest of this series. The clean/ bucket is exactly the shape a Bedrock Knowledge Base on S3 Vectors wants as a data source, so point episode 2's Knowledge Base at it and you have an end-to-end pipeline where only quality-gated documents ever get embedded.

Third, automate the trigger. Replace the manual start_job_run with an EventBridge rule on s3:ObjectCreated:* under raw/, so every new upload runs the gate automatically and a bad batch fails before anyone notices. At that point the corpus stops being something you clean once and becomes something that cannot get dirty without alarming. What would your assistant answer differently if you could prove every document behind it passed five rules this morning?