Before approving another cycle of prompt iteration, ask which failed requirement it will address. Unclear instructions, missing grounding and unchecked constraints call for different investigations. Require the team to name the suspected failure before choosing the intervention.
The evidence supports a conditional answer to the prompt engineering vs system design question. Microsoft’s RAG guidance places prompting downstream of retrieval. A bounded scheduling study reports gains from a verifier-backed compound architecture. Neither establishes that most AI product problems are architectural. Both give leaders a reason to examine what the prompt is being asked to take responsibility for.
Prompts are part of the system
There is no need to dismiss prompting to make a case for system design. A March 2023 software-engineering paper presents a prompt-pattern catalog and explores applications in requirements elicitation, rapid prototyping, code quality, refactoring and system design itself. Its abstract describes explored applications, not measured effectiveness against architectural interventions.
Keep prompt work scoped to a testable responsibility: the behavior an instruction should change and the evidence needed to retain that change. If the unresolved requirement concerns what information reaches the model, who authorizes an action or how correctness is checked, examine the surrounding system boundary too. Prompting and architecture need not compete for the same job.
Trace the path from grounding to output
Microsoft’s Azure Architecture Center makes the dependencies explicit for RAG: retrieve relevant grounding data, engineer the prompt, then evaluate the model’s end-to-end response. Its evaluation guidance also calls for completing and evaluating the preceding retrieval-system phases before evaluating prompts containing retrieved grounding data. This is RAG-specific vendor guidance, not a comparative finding about which intervention produces better outcomes.
Complete the preceding retrieval-system phases.
Check that their results meet expectations.
Use the retrieved grounding data.
Assess generated output end to end.
Guidance for RAG, not proof of a universal performance advantage.
In Microsoft’s RAG guidance, retrieval supplies the grounding for prompt construction, and evaluation covers both preceding phases and the generated response.Use these dependencies to structure a failure review. Inspect the retrieved material before attributing an inadequate answer to its instructions: was the necessary grounding present, and did the preceding phases meet their criteria? Then assess how the prompt used that context. This is a diagnostic recommendation, not an assumption that retrieval is always the limiting factor.
For the next investment decision, request evidence at both levels. Component evaluation should inform where to intervene; the end-to-end response should inform whether to retain the change. Do not use an isolated prompt test as a substitute for evaluating the RAG path.
Where verification changes the design
The November 2024 paper “Robust Planning with Compound LLM Architectures: An LLM-Modulo Approach” adds a bounded empirical case. Within their planning and scheduling framing, the authors characterize prior prompt-based approaches as capable within tested distributions but neither robust nor predictable. They report performance gains from their verifier-backed LLM-Modulo framework, using various models, across four scheduling domains.
The available abstract provides no numerical magnitude for those gains. The result concerns this particular compound design, whose correctness claim depends on a complete set of sound verifiers. It does not establish that adding a verifier makes an arbitrary AI product reliable.
The useful design question is whether a requirement should remain an instruction or also become an independently checked condition. For tasks with explicit correctness constraints, ask what a verifier could check, what would remain unchecked and what should happen after rejection. Treat verifier coverage as part of the proposed design, not as an assurance implied by the word “verification.”
Diagnose the constraint before choosing the intervention
The RAG guidance directs attention to input dependencies; the scheduling study directs it to correctness checks. Use that distinction to organize the next investigation. The decision aid below is an editorial recommendation, not a validated diagnostic model or a ranking of interventions.
Is the required behavior clearly specified with adequate context?
Does the necessary evidence reach prompt construction?
Which conditions can be checked, and which remain unchecked?
Where should authorization, rejection and escalation belong?
Recommended diagnostic questions. More than one boundary may need attention.
Match the failed requirement to the boundary worth inspecting before selecting a change.Continue prompt iteration when the team can name the behavior under test and evaluate it with adequate inputs. Broaden the investigation when the prompt is being asked to compensate for unresolved grounding, checking or workflow responsibilities. If the diagnosis remains uncertain, fund an investigation rather than commit to an architectural change.
Make technical ownership explicit at the chosen boundary. Ask the owner to state what will change, what remains outside its scope and how the complete workflow will be evaluated. That provides a concrete basis for reviewing the work without assuming either a better prompt or a more elaborate system is the answer.
Separate a promising change from production acceptance
For RAG, Microsoft’s guidance supports evaluating preceding retrieval phases and the end-to-end response. It does not supply a complete release standard or establish that a particular architecture delivers better production outcomes.
For release decisions, set broader acceptance criteria from the product’s operating requirements. Require evidence for the complete workflow, an owner for unresolved failures and an explicit decision about acceptable operational risk. Include maintainability, cost and latency in that review. These are operating recommendations: the available research does not quantify those trade-offs or provide production thresholds.
Approve the next experiment against a named constraint. Approve the release against the workflow’s acceptance criteria. A prompt improvement can justify further work without justifying deployment, and an architectural change should face the same distinction.
Written by the studio, from work in delivery
