Key Takeaways

• Codex was worth testing because it appeared to reduce the need for separate direct API-key access to model providers while still keeping agent work close to the repository.

• The hard part was not getting another tool to create plausible files once. The hard part was preserving the behavior that made the workflow trustworthy across repeated runs.

• For this repository workflow, CrewAI remained the better production runner because its task model fit our need for ordered stages, artifact contracts, guardrail hooks, review separation, and repository-level observability.

• The broader lesson is not that one agent tool is universally better than another. When agent work becomes production work, evaluate the runner and the operational constraints around it, not only the model or the prompt.

  
  

It was the best of agents, it was the worst of agents…
it is, after all, the age of agents…

I have been working with agentic AI in earnest over the last several weeks. I started with CrewAI for no special reason – I had learned about it in an agentic AI course, so it was a natural choice to start experimenting. But I can to realize that there are several things about it that I really like, and which don’t seem to hold true for other agentic platforms.

My favorite things about CrewAI:

• All of the agent and task defitions, and the extra knowledge, can be stored in YAML file outside of the code proper, making both easier to maintain
• It gives a lot of control over parallelization, fan-out, and so on
• It lets you create deterministic guardrails easily
• It’s python-based and easy to integrate with Oracle AI Memory Core, which I have written about recently, and which I firmly believe dramatically improves performance, acccuracy, etc., for agentic systems
• You can easily use different models and effort/reasoning levels for different tasks – which let’s you optimize token use, and is great for LLM-as-a-judge tasks
• It has excellent observability, explainability and evaulation options available, e.g., Arize Pheonix has great integration
• And, perhaps most importantly, it faithfully respects the workflow and guardrail constraints, and it does so deterministically, not probabalistically
• And one more thing – for bonus points – it seems to be much easier to run multiple parallel instances of the agents to work on different jobs at the same time – compared to the Codex alternative described in this article

There came a time when it appeared that I may need to find an alternate plaform, primarily driven by availability of API access to models, and this lead me to explore Codex native agents, to be clear – not OpenAI Agent SDK, but the .agents/skills/... setup in Codex itself.

To my great surprise, especially since I was using the exact same model in both cases, I found that Codex seemed to lack many of the things that I really liked about CrewAI, and it was very difficult to get it to perform tasks reliably. It felt to me like it lacked self-awareness – it constantly made what I would call silly mistakes, it constantly ignored instructions, and it constantly introduced regressions.

So, when the API access situation constraint ceased to be a real constraint, it was a very easy decision to abandon the ill-fated Codex agent experiment, and go right back to CrewAI.

Just to give some context, the agentic system we’re talking about in this article has about 28 agents, 70 tasks, 12 different workflows/pipelines, over 1000 lines of deterministic validation functions (guardrails), shared, persistent memory, and produces several kinds of artifacts, including documents and code. It has some very well defined phases, but within those phases significant opportunity for parallelization. It does use a number of tools, and it also uses the Oracle AI Database Skills that I have written about recently.

By the way, I did recently write about my experiences working with Codex to develop real-world software, which I suggest is an interesting companion read to this piece.

Here’s our story.

Once an agent workflow starts producing useful results, the next question is not “Can the model do the task?” It is “Can the system do the task again next week, with enough evidence that someone else can trust the result?”

That question pushed us to compare two different centers of gravity for repository-based agent work: a CrewAI-based runner and a Codex-based workflow that initially looked easier to operate without separate direct model-provider access.

Codex got much farther than a quick proof of concept. It helped improve the system. It exposed weak assumptions, suggested useful validation checks, and made several implicit contracts more visible. But once model access no longer looked like a hard constraint, the production tradeoff changed, and we kept CrewAI as the runner for this workflow.

By runner, I mean the part of the system that turns work definitions into executed behavior. It sequences tasks, enforces contracts, records what happened, and gives operators enough evidence to decide whether a run is ready to trust.

The lesson was not “Codex failed.” The more useful lesson was this: Codex helped us improve the system. CrewAI was still better at being the system.

Side note: In Embabel, a Java-based agentic AI platform, like CrewAI – the workflow definitions and guardrails are deterministic, i.e., they are code – they are not the LLM making a decision. I think this is an essential ingredient for a successful agent implementation. It’s a case of using the right tool for the job – LLMs are great at some jobs, but that doesn’t mean they are great at every job.

Why a Codex-Based Workflow Was Worth Trying

The attraction was practical before it was architectural.

The first attraction was model access. At the time, Codex looked like a way to run agent work without arranging separate direct API-key access to underlying model providers. For a repeatable workflow, that would have removed a practical adoption problem.

That now appears not to be a hard constraint. Once that changed, the case for replacing the production runner became much less about access and much more about operational fit.

Codex still had a second attraction: it is documented as a coding agent that can read, edit, and run code. That made it a reasonable tool to test around a repository-heavy workflow where fixes often happen in code, configuration, validation scripts, task definitions, or supporting artifacts.

Repository proximity matters. When an agent workflow breaks, the problem is not always the prompt. Sometimes the issue is a brittle validator, a missing artifact check, an unclear task contract, or a convention that exists only in someone’s head. A coding agent close to the repository can be very useful in that repair loop.

Codex was useful in that neighborhood. It helped make assumptions explicit. It pushed us to describe roles, task expectations, validation references, and collaboration contracts more clearly. The clearer those contracts became, the better the Codex workflow became.

But improving a system is not the same job as running it. And once model access stopped being the forcing function, running the system well became the deciding question.

For this workflow, the production runner had to do more than help with implementation changes. It had to preserve ordering, outputs, review boundaries, guardrail behavior, and enough observability for an operator to understand a run without replaying the whole conversation.

That is where the tradeoff changed.

What the Runner Had to Preserve

The workflow we started with was not a single prompt and a style guide. It was a staged production system.

Some stages can run with more independence. Others need a deliberate order. A review is not useful if the thing being reviewed has not settled. A release bundle is not useful if it cannot find the expected outputs. A status report is not useful if it says a stage ran but cannot tell you what it produced.

That is where artifact contracts become important.

An artifact contract is the expectation that a task writes specific files, metadata, or outputs in predictable places and formats, so downstream stages can rely on them.

Here is a simplified example. This is not a production path from any repository:

Expected by the workflow: reviews/final_review.md
Written by the runner:   review/final_quality_report.md

The content might exist, but the contract is still broken. If a dashboard, metadata index, or downstream task expects one file path and the runner writes another, the workflow has created an operational problem.

That can sound fussy until you operate the same workflow more than once. Then it becomes essential infrastructure.

Stable file names are how dashboards render. Stable review paths are how people inspect a run. Stable metadata is how downstream tasks know what is ready. Stable validation reports are how we decide whether an output belongs in a release candidate.

So the replacement target was never “make another tool produce plausible output.”

The target was “make another runner execute the production workflow without losing the contracts that make the workflow operable.”

Everything else followed from that.

Structural Parity Was Easier Than Behavioral Parity

This became the central lesson of the evaluation: structural parity is not behavioral parity.

Structural parity means the workflow looks equivalent on paper. The right stages exist. The right task names appear. The right artifacts are requested. The final directory tree resembles the one you wanted.

Behavioral parity is harder. It means the workflow reliably behaves like the production system across runs. Tasks execute in a meaningful order. Review stages remain independent. Output contracts are followed consistently. Validation catches serious defects. Failure states are explicit. A weak stage does not quietly flow downstream as if everything succeeded.

For us, the hard part was not getting a Codex-based workflow to produce the right files once. The hard part was making the right behavior repeatable without live supervision constantly restating what counted as success.

That supervision is useful during design. You can inspect a weak output, tighten the instruction, add a validation step, and try again. Codex is productive in exactly that kind of loop.

Production workflows need more than attentive supervision. They need the runner to carry some of that burden.

A production runner needs to make distinctions visible. A review artifact should explain enough. Execution evidence should provide diagnostic value. Metadata should map the files downstream workflow expects. A stage name should correspond to distinct work, not only to a label in a directory tree.

A filename drifts, and suddenly an otherwise useful artifact becomes invisible to the part of the system that expected it.

Matching the nouns in a workflow is easier than matching the verbs. “Analyze,” “generate,” “review,” “validate,” and “package” are easy labels to reproduce. The behavior behind those labels is where the runner earns trust.

Observability Made the Difference Concrete

Observability was where the comparison stopped being philosophical.

For this repository workflow, observability does not mean “there is a log somewhere.” It means an operator can answer practical questions after the run:

• Which tasks ran?
• Which tasks produced which artifacts?
• Which files were expected, and which were missing?
• Which stages deserve closer inspection?
• Which model or provider configuration handled a stage, where that metadata is available?
• Did a review step do meaningful work?
• Can this run be compared with a previous run?

Our CrewAI-based implementation had grown around those questions.

CrewAI’s task documentation describes tasks with fields for expected output, context, output files, callbacks, guardrails, and structured outputs. Our repository then layers run metadata and artifact checks around that execution model.

That distinction is important. CrewAI does not magically record every piece of evidence any team might want. We built a workflow around CrewAI that records the evidence we care about. CrewAI’s task model gave us building blocks that made repository-level observability practical: named tasks, declared output files, task context, guardrail hooks, and sequential or asynchronous execution where the design calls for it.

That is the standard the runner has to meet. Execution evidence that is structurally complete but operationally thin may be enough to make a dashboard row render, but it is not enough to build operator confidence.

A production runner should make the boring questions easy. What ran? What failed? What changed? What did it produce? What should I inspect next?

If the answer is “read the whole conversation and infer it,” the workflow is still too dependent on supervision.

Independent Checks Need Real Substance

Explainability in this kind of workflow is not model interpretability in the abstract. It is the ability to explain how the final output came to be.

What input shaped the plan? What did the technical review flag? What did validation check? What quality concerns were found? What changed before the final output? Why is the final output better than the first version?

A review file is not the same thing as a review.

For this workflow, we needed artifacts that explained what was checked, what changed, what remained uncertain, and which stage made that decision. Otherwise, the workflow could look complete while leaving us with very little evidence that independent review had happened.

Our CrewAI-based staged workflow gave us a cleaner path to that evidence. Technical review can be separate from general validation. Quality checks can be separate from improvement work. Validation can happen after the thing being validated is stable enough to check. Packaging can focus on whether the right public artifacts exist without pretending to be a technical review.

Separation alone does not guarantee quality. A weak review is still weak if it runs in its own task.

But independent stages create places where quality can be measured, compared, and improved. They also make failures more visible. If a review artifact is thin, we can see that the review stage underperformed. If an assessment misses a serious defect, we can tighten the rubric. If validation leaves uncertainty, we can decide whether to improve or hold the output.

For this workflow, the runner has to preserve that independence. If stages collapse into broad undifferentiated generation work, the output may contain files but still lack the evidence we need.

Better consistency checks help, and Codex helped us identify several of them. They did not replace the confidence we already had in the CrewAI evidence trail.

That mattered because the workflow was not only about generating text or code. It was about being able to explain why the result was ready.

Why CrewAI Still Fits This Production Workflow

For this repository, we chose CrewAI because it fit the production shape of the work.

The important pieces were tasks, context dependencies, output files, guardrails, guardrail retry behavior, and execution modes. Those map closely to what the workflow already needed.

CrewAI remained the better fit as the production runner, but not for free. It asks you to model the work as a workflow: agents, tasks, outputs, guardrails, and run artifacts. For a quick coding problem, that can feel like ceremony. For this production workflow, that ceremony was the point.

Here is a deliberately simplified YAML-like sketch, not exact CrewAI syntax and not copied from production. It shows the kind of contract shape we care about:

task:
  name: technical_review
  agent: technical_reviewer
  context:
    - candidate_output
    - validation_notes
  expected_output: reviewed technical findings
  output_file: reviews/technical_review.md
  guardrail: require_findings_or_explicit_no_findings

The value is not that YAML is magical. The value is that the workflow contract is reviewable.

A task can declare what it depends on, who should run it, what it is expected to produce, where the output should land, and what kind of guardrail should apply. The repository still has to validate that the expected files exist and contain useful evidence, but the contract has a visible place to live.

That matters when multiple people maintain a workflow. YAML-centered definitions can be reviewed in pull requests. They can be diffed. They can be discussed without reading the entire runtime implementation. They give reviewers and maintainers a shared object to inspect.

Execution order was another deciding factor.

Some stages must be sequential. Context should inform planning. Planning should guide production. Production should precede review. Improvements should respond to review. Packaging should happen after the required outputs and sidecars exist. CrewAI supports sequential execution, and our workflow uses that where dependencies matter.

At the same time, not everything needs to be serialized. Some supporting work can run asynchronously if the dependencies are clear. CrewAI gives us a mechanism for asynchronous tasks, while our workflow decides where parallel execution is safe.

That balance is exactly what a production runner should support: strong order where it matters, concurrency where it does not.

For this repository, CrewAI also gave us a clearer place to attach task-level guardrails. A guardrail is not a slogan. It is a workflow mechanism. In our implementation, guardrails are where we can encode checks for missing required sections, malformed outputs, broken artifact contracts, review stages with no findings and no explicit “no findings” statement, or validation reports that lack expected evidence.

CrewAI provides the hook; the workflow still has to supply the judgment.

Per-stage model configuration also mattered. We did not want one global “agent intelligence” setting for the whole workflow. Research synthesis, technical review, validation, quality checks, and packaging have different cost, risk, and reasoning profiles. In our CrewAI implementation, tasks are assigned to agents configured for those stages, so model and provider choices are part of the workflow design.

None of this means CrewAI is effortless. It is a framework, and frameworks create surface area. Bad task definitions still produce bad runs. Weak guardrails still miss defects. Observability only helps if we record the right evidence.

But for this repository, CrewAI let us encode more operational concerns in the workflow runner rather than relying on the surrounding conversation to restate them.

What Codex Still Does Very Well

We did not come away liking Codex less. We came away with a sharper sense of where it belongs in this workflow.

I want to be clear that we really, really tried to make Codex successful – we did a lot of reasearch, we learned about how other people have used it, we consulted best practices and reviewed other agentic implementations to work out where we were going wrong. This was not a casual test, we made real, concerted effort to get this to work satisfactorily before finally coming to the conclusion that it was not the right choice for this system

Codex remains useful close to the repository. In our workflow, it helped with code edits, repair loops, test ideas, validation checks, dependency updates, and implementation review. When a contract is vague, Codex can help make it concrete. When a validation artifact is missing, Codex can help design it. When the workflow has implementation bugs, Codex is useful in the repair loop.

The evaluation also improved our thinking.

The task-card style made us write down collaboration contracts more clearly. The comparison forced us to separate “file exists” from “stage did useful work.” The consistency checks that emerged from the exercise are worth carrying back into the CrewAI workflow.

That is a win.

There is also a broader lesson here for teams evaluating agent systems. For us, a coding agent was a strong place to evolve the workflow, while CrewAI remained the better place to run this repeatable system.

That is where we landed.

Codex helped us build and repair the system. CrewAI remained the better fit for running it.

The Decision, for Now

The decision is straightforward: CrewAI remains the production runner for this repository workflow. Codex remains part of how we improve the workflow.

We will keep using Codex where it shines: near code, near tests, near implementation repairs, near task-contract cleanup, and near validation design. We will keep carrying good ideas back from the comparison. We will keep tightening the CrewAI workflow with the consistency checks the work made obvious.

But the production run itself stays in CrewAI for now.

The reason is not that CrewAI is universally better than Codex. It is that this workflow is more than a conversation and more than a patch. It is a multi-stage production system with artifact contracts, review boundaries, execution dependencies, guardrails, observability, and assembly rules.

In that environment, the runner matters.

Other teams should make this decision based on their own workflow shape. If your workflow is mostly exploratory, coding-heavy, and interactive, Codex may be the right center of gravity. If your agent work is mostly pull-request repair, bug fixing, refactoring, or test generation, putting the agent close to the repository can be exactly right.

If your workflow is a repeatable production system, ask different questions.

What must run in order? Which artifacts are contractual? Which stages need independent review? How do you know a stage did meaningful work? What happens when a file is missing? Can you compare one run with the next? Can you recover from a partial failure? Can an operator trust the evidence without replaying the whole conversation?

Those questions pushed us back to CrewAI.

Conclusion: The Runner Has to Keep the Promises

Agent quality matters. Model quality matters. Prompt quality matters. But once agent work becomes production workflow, the runner matters too.

We learned that describing a workflow is easier than running one. Codex helped us describe, inspect, repair, and improve the system. It pushed us to make implicit contracts explicit. It generated useful ideas we will keep.

That is not a small contribution.

But the workflow also needed trustable operational properties: ordered execution, stable artifacts, useful diagnostics, review independence, guardrail hooks, repository-level evidence, retry hooks where appropriate, and per-stage control. For this repository, CrewAI gave us more of that in the runner structure we could build around.

That is the decision heuristic I would take from this evaluation: do not only ask whether an agent can name your process, draft your files, or produce the right shape once. Ask whether the runner can keep the promises your process depends on when nobody is watching every step.