How to Remove Outdated Documents from an AI Knowledge Base

When a policy changes or a service is withdrawn, do not close the task after removing the original document. Define the change as retiring an answer source: identify the obsolete version, determine which retrieval copies are in use, apply the approved update path, and collect evidence from the assistant that people actually use. This guide focuses on one source retirement, not initial knowledge-base setup or general answer scoring.

AI-Q Dynamics advertises versioned knowledge bases aligned with business policies and services.[11] The retirement record and tests below are recommendations for your content owner and implementer, not a statement about AI-Q's internal provider, index schema, or retention schedule. Microsoft documentation is included as a platform-specific example, not a required purchase or a claim that AI-Q uses Azure.

Removing a file is not the same as retiring an answer source

Distinguish three things in your review: the document you edit, the copy or passages made available to retrieval, and the answer the assistant displays. Ask your implementer to show how a change moves between those layers in your actual application. A missing file in a shared folder is not, by itself, evidence about what the assistant can still retrieve.

Microsoft's documentation for Azure Storage indexers in Azure AI Search says an indexer does not track object deletion in data sources automatically; the described storage workflows require explicit deletion handling.[3] That statement belongs to the documented Azure context. Do not assume another platform behaves the same way, and do not infer that a connector's ordinary refresh also handles removal.

Hypothetical example: your team replaces a service policy called policy-demo-v1 with an approved policy-demo-v2. Someone removes the old source file, but nobody checks the assistant's retrieval results. Instead of assuming completion, open a retirement record that identifies both versions and the questions most likely to expose the obsolete wording. Keep the example free of real customer data and avoid inventing policy terms just to fill out a test.

Create a source retirement record

Ask the person authorized to own the policy to approve the retirement. The technical operator should not decide which conflicting business rule is correct simply because one file has a later upload date. Establish the replacement's effective date and whether a transition period requires different answers for different circumstances. Where that distinction matters, have the content owner specify it before the implementer changes retrieval.

Copy the following fields into your existing change process. Use stable references where available, and record “not exposed by this platform” when a technical identifier cannot be obtained. That is more useful than leaving a blank that may later be mistaken for a completed check.

Owner and approver

Business content owner; implementer; authorized reviewer; approval reference.

Source location and ID

Repository or folder; source URL or internal reference; example ID: policy-demo-v1; relevant access scope.

Version and retirement reason

Version label; affected passages; why the wording is no longer approved; intended effective date.

Replacement or no replacement

Approved successor reference, such as policy-demo-v2; or explicit confirmation that no replacement exists.

Affected questions

Direct question; paraphrases; old document title; service name; question that should now require a handoff.

Execution and review

Approved change path; operator; planned review date; evidence location; fallback; unresolved copies; closure approver.

Scope retirement to the content that is actually obsolete. If only one section changed, ask whether the platform requires replacing the whole source or can update the relevant entries safely. Preserve unaffected approved content and its access boundaries. Do not widen permissions to make verification easier, and do not paste confidential source material into an unapproved public assistant for testing.

Find the copies used for retrieval

Ask for a source-to-retrieval map, not a generic diagram of an AI system. Begin with the old source ID and follow the actual configured ingestion route. Depending on your stack, the relevant items may include derived passages, chunks, one or more indexes, an ingestion job, or application-level caches. These are questions to investigate, not components every implementation necessarily contains.

• Derived entries: which passage or index identifiers point back to this source and version? Can the implementer enumerate them without relying only on a text search?
• Live destination: which configured retrieval store does the public or staff-facing assistant query? Does the test environment use a different one?
• Ingestion: what scheduled import, synchronization, or manual upload can recreate the old content? Is a duplicate upload or alternate folder still in scope?
• Cached material: does this application retain retrieved passages or generated answers in a cache relevant to the change? If so, what approved refresh or invalidation behavior applies?
• Citation mapping: how does a returned passage become a visible title or link? Can an old citation label remain even after the underlying content is replaced?

Include an evidence location and owner for each item. If the platform hides a layer, ask what supported diagnostics can establish its state and record the remaining uncertainty. Do not invent a cache-clearing command or claim a hidden store has been emptied. Also check whether another approved document repeats the same obsolete passage; retiring the original alone would not address that separate source.

Apply the correct update or deletion path

Microsoft documents separate Azure AI Search document actions for updating content and deleting index documents.[4] Use that distinction to ask your implementer which operation your own platform requires. Source editing and index deletion are separate operations in this example; neither should be substituted for the other without checking the actual configuration and documentation.

When replacing a policy: ask the content owner to approve the successor first. Have the implementer explain how the new version becomes available and how the old version stops being eligible for retrieval. Define what the assistant should do during any transition. Verify that a temporary overlap will not leave users receiving contradictory rules, and do not promise immediate propagation without evidence.

When removing without replacement: specify the answer boundary. For a withdrawn service, the approved behavior might be to state only a confirmed current status and offer a human contact. For an unanswered policy question, the appropriate result may be an explicit limitation and handoff rather than an improvised substitute. Ask the owner to approve the wording and avoid silently routing users to unrelated material.

Before execution, confirm the exact source and destination identifiers, the environment, the permissions, the approved scope, and how completion or failure will be observed. Have the implementer apply the provider-supported path; this checklist deliberately provides no generic destructive command. Record the result for the relevant entries and investigate unresolved items before approving the change as complete.

Prove the old source is no longer retrieved

Test both retrieval and the displayed answer. Start a new conversation for each scenario so prior conversation text does not obscure what the changed retrieval path is doing. Use the same interface, access role, and configuration that the intended audience uses. Keep the question, returned source IDs or passages where available, visible answer, citation targets, test time, and reviewer decision together.

Source retirement evidence matrix

Test	Expected evidence	Failure to investigate
Superseded policy	Ask a question previously answered by the old passage. Retrieve the approved replacement and review the answer against it.	The old passage, version, or policy wording remains in the retrieved context or answer.
Conflicting versions	Ask using terms from both versions. Confirm the current approved source governs and no obsolete version is retrieved.	The answer combines incompatible rules or cites both as current without the approved distinction.
Removed service	Ask directly for the retired offering. Confirm the owner-approved current-status response or handoff.	The answer still offers the removed service or retrieves an old promotional description.
Missing replacement	Ask a question now outside the approved material. Confirm an appropriate limitation and human route.	The assistant invents a replacement policy or treats an unrelated document as authority.
Stale citation in a new conversation	Open a fresh conversation, repeat a previously matching question, and inspect each cited title, version, and destination.	A citation points to the retired file, an obsolete version, or a missing document despite acceptable-looking prose.

For each scenario, include at least a direct question and a natural paraphrase appropriate to your users. If the expected answer is correct but retrieval still returns the old source, keep the change open: a good answer on that run does not resolve the obsolete retrieval input. Conversely, if retrieval is clean but the answer states an old rule, investigate the remaining answer path rather than falsely reporting that the source is still indexed.

Record the limits of your test set. Evidence that none of these runs retrieved old passages is not a guarantee that the assistant can never state outdated information. Where retrieval diagnostics are unavailable, clearly label the evidence as answer-and-citation review rather than claiming a direct index inspection occurred.

Close the change without overclaiming erasure

Require the reviewer to sign off on the source record, the implemented update path, the observed retrieval results, and the visible answers. List unresolved copies separately with owners and follow-up decisions. Set a review date appropriate to the workflow, including a check after the next configured ingestion run if that run could reintroduce an old upload.

Keep rollback safe. Ask for a reviewer-approved fallback or temporary human handoff if the replacement fails verification. Do not restore a known-wrong policy merely to make the assistant answer more questions. If rollback is appropriate for a technical change, specify which approved content remains eligible and which obsolete version must stay excluded.

Use narrow closure language: “The listed tests against the identified configuration did not retrieve the retired version, and the reviewer approved the replacement answers.” Source deletion is not proof of deletion from an index, logs, backups, prior chats, or model training data. Do not describe retrieval retirement as complete erasure or as a change to a base model's training. If broader retention or deletion obligations apply, handle those through a separately authorized review.

For the broader service context, see RAG and knowledge bases. If you are building the initial collection rather than retiring an existing source, use RAG setup best practices. Bring the retirement record and unresolved evidence questions to your implementer, or explore More implementation guides.