How do you define success criteria for order entry automation?

Success criteria for order entry automation are the written, measurable conditions that define when the project is done, agreed by the customer and the vendor before the system touches a production order. A good success criteria document does five things: it defines exactly which orders are in scope, names every system the automation connects to, states how exceptions are handled, lists the business rules the system must enforce, and sets two separate finish lines. Go-Live is proof the system works. Phase 1 Complete is proof it works across the full agreed scope. The Project Management Institute's 2026 Pulse of the Profession found 31 percent of complex projects fail to achieve the full scope of their intended benefits. Most of them never wrote down what the full scope was.

Key takeaways

• PMI's 2026 research found 31 percent of complex projects fail to deliver their full intended benefits, and that projects managing complexity effectively succeed 88 percent of the time versus 14 percent for those that do not.
• CIO's 2026 feature on IT project failure names "ambiguity around measures of success" as a top cause. George Reed's test: "If we were already done, what does winning look like?"
• Y Meadows agrees a success criteria document with every customer within the first weeks after kickoff. It separates Go-Live (the system works on a minimum scope) from Phase 1 Complete (the system works on the full scope, measured in production).
• The document is where scope, exception rules, and business rules get written down. Most of that knowledge otherwise lives in the order desk's heads.

Most order automation projects define success this way: the statement of work lists the customers and order types in scope, the integration method, and a go-live date. Success is hitting the date with the scope. Then go-live happens, orders start flowing, and three months later the operations leader is asked whether the project worked and does not have an answer. The volume through the system is lower than expected. The team still keys some orders by hand. Nobody agreed what "worked" meant.

George Reed, CIO at auntEDNA.ai, tells CIO that "no project should be approved if you don't have targets for measurable results." Y Meadows agrees, and uses a success criteria document on every project. It is referenced in the statement of work, drafted at the kickoff workshop, and signed by both companies within ten business days of it. Here is what goes in it.

What is a success criteria document?

A short written agreement, signed by both parties, that states what the project will deliver and how both sides will know. It is not the statement of work. The SOW says what the vendor will build and what it costs. The success criteria say what the customer's operation will look like when the build has done its job, and they are specific enough that someone who runs the order desk can check them.

Y Meadows' version runs to ten sections, but they answer four questions. Which orders are in scope? What does the system connect to? What should it do when an order does not fit? Finally, what has to be true at each of two finish lines? The rest of this post walks through them in that order.

Reed's advice in CIO is that project owners "need to ask, 'If we were already done, what does winning look like?'" and then work backwards to milestones and leading indicators. The success criteria document is the written answer to that question.

Which orders are in scope for Phase 1?

The document defines scope on five dimensions, and an order has to fall inside all five to be processed automatically.

Channels: a dedicated orders inbox, orders buried in a general inbox, PDF-based EDI, fax-to-email, portal exports.

Order types: standard purchase orders, blanket and release orders, change orders, credit requests, quotes.

Customers: all of them, a named subset, or the top N by volume.

Segment: divisions, plants, or ERP company codes.

Product scope: stock items only, made-to-order, excluded categories such as hazmat or custom-configured lines.

Then it asks for one plain-language sentence that summarizes the result. Something like: all stock-inventory purchase orders from our top ten customers by volume, for the Industrial Products division, shipping domestically. If the business owner cannot write that sentence, scope is not defined yet.

Everything outside that sentence stays in the existing manual process until a future phase. Writing the exclusions down is as important as writing the inclusions. It is the difference between a project with a boundary and a project that expands until it stalls.

What does the system need to connect to?

Every integration point, with a named owner on the customer side. The ERP itself, including version, hosting, integration method, and whether a sandbox exists. The master data the automation matches against: customer master, ship-to addresses, item master, price lists, unit of measure crosswalks, salesperson and territory assignments, each with a source system and a refresh cadence. The inbound email configuration. Any other system in the path, such as a CRM, a WMS, or an approval workflow tool.

This section exists because integration delays are the most common reason a go-live date moves, and they are almost always a credentials or access problem rather than a technical one. Y Meadows' post on implementing AI order entry without disrupting your ERP covers the integration methods themselves. The success criteria document makes sure someone owns getting them turned on.

What happens when an order does not fit?

The document makes the customer choose, in advance, for each common exception. When the customer on a PO cannot be identified: leave it blank and send the order to the review queue, return it to the manual process, post it to the ERP on hold with a placeholder, or flag it and notify someone. The same choice for an item code that cannot be matched, and for a unit of measure that does not agree with the product master.

These are not technical decisions. They are decisions about how the order desk wants to work, and they are the ones that usually get made by accident, after go-live, by whoever happens to be looking at the queue. Making them at the kickoff workshop means the review queue behaves the way the team expects from the first day.

The same section captures the business rules the system must enforce before an order reaches the ERP. Required fields such as PO number, requested date, and ship-to. Validation rules such as price on the order agreeing with the price list, a quote being required and matching, certain items having to be ordered as a set, and duplicate PO detection. Additionally, the custom rules the team enforces that the accounting system does not. Those last ones are the rules that live in people's heads, and getting them onto paper is most of the value of the exercise.

What has to be true at Go-Live?

That the system works, proven on a minimum scope. Y Meadows' Go-Live criteria have three parts.

Technical readiness: the ERP integration is authenticated, the inbound channel is polling, master data is syncing, notifications reach the right people, and the review queue is accessible.

Functional readiness: validated during user acceptance testing on a representative sample of real orders: customers and items match against master data at an agreed accuracy threshold, required fields extract correctly, and exception and validation rules fire as defined.

Minimum scope: the subset of Phase 1 that must be processing successfully on day one, such as the first three of ten named customers, email orders only, stock inventory only.

Go-Live does not require the full Phase 1 scope. It requires proof. When the customer signs the Go-Live approval, a hyper-care support period of ten business days begins, during which the project team watches production closely and tunes rules on real volume.

That distinction matters because it lets the project go live early on a customer the team knows well, which is what the Y Meadows Success Roadmap calls the Quick Win. The order desk sees real orders from a familiar customer get read, matched, validated, and posted. Confidence builds on evidence rather than on a demo.

What has to be true at Phase 1 Complete?

That the full scope is live and performing, measured in production over a rolling window. The Phase 1 Completion criteria have two halves. Scope coverage: every in-scope customer, division, product type, channel, and geography from the scope section is active and processing. Production performance: overall order accuracy meets the agreed threshold, the straight-through processing rate (orders that post with no human review) meets the agreed percentage, average time from order receipt to ERP entry meets the agreed service level, exception and validation rules are operating as defined, and there are no unresolved critical defects.

Performance is measured over consecutive business days at full scope, not on the best day. The document also contains a ramp plan: Go-Live on the minimum scope, then named ramps adding customers or divisions, then Phase 1 Complete, each with a target date. The gap between Go-Live and Phase 1 Complete can be days or weeks, and the ramp plan is what keeps it from becoming months.

Side by side, the two finish lines look like this:

• What it proves: Go-Live proves the system works. Phase 1 Complete proves it works across the full agreed scope.
• Scope: Go-Live covers a minimum subset defined in advance, such as the first three named customers on email orders only. Phase 1 Complete covers everything in the scope statement.
• How it is tested: Go-Live uses technical readiness checks plus UAT on a representative sample of orders. Phase 1 Complete uses production performance over a rolling window of consecutive business days at full scope.
• What it triggers: Go-Live starts a ten-business-day hyper-care period. Phase 1 Complete transitions the project to ongoing support and the scoping of Phase 2.
• Who signs: in both cases the customer approves and Y Meadows acknowledges.

What should the criteria measure?

The things the business owner is evaluated on: accuracy, straight-through rate, turnaround, and whether the rules hold. Not the things the vendor's software reports on by default. Each needs a baseline, and discovery is where the baseline gets captured. If nobody knows how long an order takes today, the improvement cannot be shown.

Y Meadows' earlier post on why AI order entry pilots stall makes the related point that extraction accuracy alone misses where the time goes. The straight-through processing rate is the measure that captures it: an order that extracts perfectly but still needs a person to release it has not saved the person's time. That is why it sits in the Phase 1 Completion criteria alongside accuracy rather than instead of it.

Two things do not belong in the criteria. Vendor-specific technical measures the business owner cannot interpret, and targets nobody has a baseline for. If a number cannot be checked at the steering committee meeting by someone who runs the order desk, it is not a success criterion.

Who signs off, and when?

Three times. Both companies sign the success criteria document itself within ten business days of the kickoff workshop, which is the moment scope, rules, and finish lines become a shared commitment rather than a vendor's plan. The customer signs Go-Live approval when the Go-Live criteria are met, which starts hyper-care. The customer signs Phase 1 Completion when the Phase 1 criteria are met, which closes the implementation project.

The steering committee is where the second and third signatures happen. Y Meadows' post on what a steering committee does on an order automation project describes the cadence and composition. Each meeting checks where the project stands against both sets of criteria, and the sign-off is the meeting where the business owner confirms the operation looks the way the document said it would.

Sign-off is also where Phase 2 gets scoped. The customers and order types listed under Phase 1 exclusions become candidates. The Phase 1 production numbers become the new baseline. The same document structure gets filled in again, at the same table.

Bring your current order desk numbers, or the fact that you do not have them yet. Either way we can show you what a success criteria document for your operation would say. Book a demo. If you want the milestone-by-milestone view first, download the Success Roadmap.

Sources

• Mary K. Pratt, "Why IT projects still fail," CIO, August 31, 2026. cio.com
• Project Management Institute, "Pulse of the Profession 2026: Driving Success in Complex Projects," May 2026. pmi.org
• Y Meadows, Success Roadmap (implementation guide). use.ymeadows.com/roadmap