Recently, I was lucky enough to be invited to give a short presentation on blended teams, and I wanted to share some of my thoughts with you, dear reader!
What is a “blended team”? I am defining it as a team that has both human and AI workers who are collaborating on the same goals and tasks.
As you know, AI is good at some things, and not so good at others. You could say “when it’s good, it’s very, very good; but when it’s bad, it’s very, very bad” (wink to Henry Wadsworth Longfellow).
Adding AI to a human team does not guarantee better performance.
The diagrams below, from arXiv:2507.09089v2 [cs.AI]Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity – Joel Becker, Nate Rush, Elizabeth Barnes, David Rein, present an interesting finding from the domain of software development: on average, experts tend to dramatically over-estimate how much AI assistance will speed up development. This study found that the estimates ranged from a 10% to 50% reduction in time – but the actual observed results were that tasks took up to 40% longer when AI assistants were used.
While developers estimated that AI assistance would enable them to reduce the time required for a task, as shown in the diagram below from the same study, the result was that they in fact took significantly longer when using AI.
Why is this the case? What’s going on here?
I’d like to offer the hypothesis that AI (and Generative AI in particular) is very good at doing things that have been done before, but much worse at doing something novel. This makes a lot of sense (to me, at least) if you consider that these models have been trained on very large corpora of existing code, and that they work by probabalistic prediction of the next token. So a model that has seen many examples of how to implement a particular type of requirement is likely to do a very good job of implementing that requirement. But, if you ask a model to implement something new, that it has not seen any, or many, examples of – it’s not going to do as good a job on that.
Another important thing to consider is the quality of the code the models were trained on. Charles Babbage called it way back in 1820 – garbage in, garbage out. And every developer knows, this is still very relevant today. If you consider that most of the code that the models were trained on is not what we would consider “good” – then you might not be surprised that the code they generate is also not “good”. Here’s some charts I pulled out of a paper showing distributions of some measurements of goodness across a wide sample of open-source code. The whole paper is very interesting and well worth a read.
Jin, Siyuan; Zhang, Mianmian; Guo, Yekai; He, Yuejiang; Li, Ziyuan; Chen, Bichao; Zhu, Bing; and Xia, Yong. “Software Code Quality Measurement: Implications from Metric Distributions.” 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS), 2023, pp. 488–496. https://doi.org/10.1109/QRS60937.2023.00054
Quantity is not quality
Consider another example – AI assisted detection of security vulnerabilities. Models like Mythos and Fable can detect many more vulnerabilities that we ever did before them using human review and our pre-existing tools. But on analysis, we find that a lot of those issues detected (at least in my experience) were poor quality, false positives, or just duplicates of issues that we found with our existing tools. An enormous amount of human work is needed to review the output of these models and separate the signal from the noise.
With the widespread use of AI in this area, there also came a massive amount of work for software developers to review, categorize and address these issues. We’ve seen a spike in CVEs since the availability of these models, as shown in the diagram below, created from public CVE data reported by cve.org – note that this data set is up to end of June 2026. No one really knows if that trend will continue upwards – my educated guess is that all of the activity by security researchers and developers will result in a lot of issues being fixed, and we will possibly see that level off and start to drop again, as the industry as a whole gets more focused on fixing issues, and adapts our development, testing and release processes to catch more issues before shipping code.
We should think about tasks, not jobs, that AI can do.
There’s a lot of talk about AI replacing jobs, but I think it is much better to think about tasks that AI can do, not whole jobs. We all know that the job description rarely, if ever, actually describes the real job. There are a lot of tasks that people do that don’t make it into the job description. I don’t like to think about using AI to replace a person’s job, but instead to think about which individual tasks we could use AI for – tasks where we would get an improvement (in time, quality, etc.) and which would let the person focus more time on the more difficult (less automatable) tasks.
So what tasks are AI models good at? Here’s a synthesis from a number of studies (see the citations at the end of this article):
In my own personal experience, I’ve found that AI is really good at things like doing research, fact checking, writing code (for known patterns, and with strong guardrails), scanning for vulerabilities, and less good at things like writing prose, creating graphics, making decisions, and so on. Human verification is essential for all AI outputs, but especially whenever errors are costly, context is hidded or judgement defines success.
Here’s a finding from a study that looked at usage of Copilot for various tasks and how successful that usage was for different kinds of tasks.
arXiv:2507.07935v6[cs.AI] Working with AI: Measuring the Applicability of Generative AI to Occupations – Kiran Tomlinson, Sonia Jaffe, Will Wang, Scott Counts, Siddharth Suri
AI does not remove work – it changes the mix
A very interesting study by Becker et al clearly demonstrates how the amount of time spent of different tasks changes when AI is used. Interestingly, the total amount of work is not necessarily the same, this looks only at the percentages, not the total.
arXiv:2507.09089v2[cs.AI] Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity – Joel Becker, Nate Rush, Elizabeth Barnes, David Rein
This is supported by other studies which found that the total amount of time increased. Here are some findings from one such study in the domain of open-source software development:
And here are some findings from a study on professional writing – total time reduced but the amount of time spent of each part of the work changed:
Generation speed is not delivery speed
Even if AI can reduce the amount of time it takes to write code from weeks to minutes – that does not mean I can ship the code and pass on that time gain to my end users. I still need to have people review the code, and I still need to do all the other security, compliance, and release management tasks. So while an individual task might be faster, my wall clock time is not necessarily impacted.
I love a poster I saw at a ballet school which read “Hard work is not for everyone” – I think that’s a very true statement. Using AI is easy, anyone can do it. But using AI well is a very different thing – and I think it is fair to say that most people probably won’t invest the time and effort to learn how to use AI well. This means that a lot of AI applications are going to be sub-par. You can easily find evidence for this – from the hype merchants publishing YouTube videos claiming to show you how to build an AI application in “five minutes”, to the high visibility failures like those showcased in “12 famous AI disasters”, Thor Olavsrud, CIO Magazine https://www.cio.com/article/190888/5-famous-analytics-and-ai-disasters.html and I suspect that you have personal experience of talking to an AI and finding its ability to actually “help” to be very poor.
Trust is vitally important in AI, and reputation damage and financial penalties are very real risks.
You won’t get it right the first time
Using AI “well” means things like understanding how the models work, comparing the performance of different models and hyper parameter settings for your specific use case and possibly corpora, implementing guardrails and retries, using techniques like planning and reflection, building and using evaluations, knowing how and when to use different kinds of memory, using skills, and so on. No matter how good you are, I’d argue that you won’t get it right the first time, and you need a continuous improvement loop.
It’s important to collect inputs and outputs, and then to do human reviews, and provide feedback about what is good and what bad. This data can be used to implement improvements, so that in essence the application ‘learns’ over time how to do the task right.
The handoff is critical
You need a mechanism to route and track tasks. AI can work faster than people. You need a way to queue up tasks so that it can pick form the queue based on whatever criteria are appropirate, maybe the token quota for example. If a task is not clear, it should be able to ask questions before starting work – so you need a way to allow it to assign a task back to a human to get more information. You should use techniques like guardrails and adversial reviews (ideally with a different model) to control quality.
So when is AI going to excel?
I think that this is a good litmus test to apply when trying to identify tasks that AI will be able to excel at:
Smaller, well-defined tasks with clear context, strong guardrails, and a strong definition of “right”
Tasks that do not require novel thought or complex decision making
Clean boundaries and handoffs between tasks
Repeatable, high-volume tasks
Continuous improvement feedback loop
We have a huge amount of opportunity ahead of us. Here’s an interesting finding from an Anthropic study that compares the opportunity in various industries to the actual observed usage today:
There’s lots of very active research in this area I’ve shared a number of papers that I’ve read myself as well as some of my conclusions from my own personal experience. I’d love to hear your thoughts and experience, please hit me up in the comments.
[1] Noy, Shakked, and Whitney Zhang. “Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence.” Science 381(6654), 2023. https://doi.org/10.1126/science.adh2586
[2] Brynjolfsson, Erik, Danielle Li, and Lindsey R. Raymond. “Generative AI at Work.” Quarterly Journal of Economics 140(2), 2025. First circulated as NBER Working Paper 31161, 2023. https://www.nber.org/papers/w31161
[3] Ishikane, Masahiro, Yuki Kataoka, Yasushi Tsujimoto, Yuki Moriyama, Yukimasa Matsuzawa, and Norio Ohmagari. “Efficacy of a Large Language Model Data Extraction System in Evidence Reviews for Emerging Infectious Diseases: A Randomized Crossover Trial.” Open Forum Infectious Diseases 13(7), 2026. https://pmc.ncbi.nlm.nih.gov/articles/PMC13393321/
[4] Khraisha, Qusai, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. “Can Large Language Models Replace Humans in Systematic Reviews? Evaluating GPT-4’s Efficacy in Screening and Extracting Data from Peer-Reviewed and Grey Literature in Multiple Languages.” Research Synthesis Methods 15(4), 2024. https://doi.org/10.1002/jrsm.1715
[6] Becker, Joel, Nate Rush, Elizabeth “Beth” Barnes, and David Rein. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR / arXiv preprint, 2025. https://arxiv.org/abs/2507.09089
[8] Kumar, Deepak. “SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback.” arXiv preprint, 2026. https://arxiv.org/abs/2603.26130
[9] Ribalta-Albado, Maria, and Pere-Pau Vázquez. “Evaluating LLMs’ Abilities to Create Charts, a Systematic Approach.” Computers & Graphics 135, 2026. https://doi.org/10.1016/j.cag.2026.104544
[10] Collier, Zachary A., Richard J. Gruss, and Alan S. Abrahams. “How Good Are Large Language Models at Product Risk Assessment?” Risk Analysis 45(4), 2025 (published online in 2024). https://doi.org/10.1111/risa.14351
Before asking whether Go or Java is faster, both services need to answer the same HTTP question. That means the same endpoints, request fields, validation rules, work-factor semantics, response JSON, and error behavior.
The benchmark contract defines equivalent externally specified work, not identical CPU instructions. extraWork drives a deterministic operation whose result affects a returned checksum, so conformance tests can catch many skipped or altered implementations.
JSON comparison should be strict about schema, values, types, and error codes, but not brittle about object member order or whitespace. Framework-default HTML, text errors, stack traces, and runtime metadata are contract failures for body-bearing responses.
Shared acceptance tests are the gate into later implementation and benchmark work. The Java/Helidon and Go services must pass the same contract before any benchmark result is admissible.
You can make two services look almost identical and still measure different work.
One implementation accepts an unknown JSON field and ignores it. The other rejects the request. One silently clamps a payload size that is too large. The other returns a validation error. One computes a checksum from generated bytes. The other computes it from a different buffer, or runs a loop that does not affect the returned result. One returns a framework-default text error. The other returns JSON.
If those services go into a benchmark, the chart is already compromised. The result might look like Go versus Java, but part of the difference could be request parsing, validation policy, payload generation, error handling, or a work loop that does not mean the same thing in both implementations.
So before asking whether Go or Java is faster, we need to make sure both services are answering the same question.
The first article in this series set a fairness rule: no benchmark result is admissible until the method and evidence chain are strong enough to support it. This series is scoped to a later single-machine benchmark method, but the exact operating system, kernel, CPU, memory, runtime versions, and isolation settings belong in the execution-method article and benchmark manifest, not in this service-contract article. Here, we define the language-neutral service behavior that the Java service and the Go service must satisfy at the HTTP boundary before either one is tuned, load-tested, or compared.
For developers building service-tier APIs around Oracle AI Vector Search, retrieval, embedding, reranking, or Oracle AI Database workflows, this boundary matters. A small application service often sits between clients, model calls, and Oracle AI Database. If two versions of that service validate different JSON, generate different payloads, or return different correctness artifacts, later runtime measurements are not about the runtime anymore. They are about drift.
This article does not publish benchmark results. It does not choose a faster runtime. It does not implement either service. It does not design the full load generator or run the benchmark package. We are defining the service behavior that later code must satisfy.
What This Article Defines, and What It Does Not Measure
A language-neutral service contract is the externally visible behavior both implementations must provide, expressed without depending on Java or Go internals. For this series, that means the behavior a client can observe over HTTP: endpoint paths, methods, request headers, request JSON, validation rules, response JSON, status codes, and error shapes.
HTTP gives us the shared vocabulary for methods, status codes, content, and representations. JSON gives us a shared request and response format. The contract uses those pieces to define observable behavior explicitly rather than letting framework defaults decide.
This article defines:
The endpoint set and request media type
The JSON request and response shapes
The request body limit
Payload-size and deterministic payload-generation rules
The meaning of extraWork
Validation rules, duplicate-field behavior, and unknown-field behavior
Success and error response schemas
JSON equivalence rules
Cross-implementation acceptance-test categories
It does not measure throughput, latency, memory, CPU, startup, warmup, garbage collection, scheduler behavior, Oracle AI Database query performance, model inference, or vector-search performance. Those questions belong later in the series, after the service contract, implementations, load generator, execution method, and analysis method are stable.
Later latency measurements also need their own load model, request scheduling policy, timeout policy, warmup plan, sampling method, and treatment of coordinated omission.
That distinction is not academic. It is the difference between a benchmark that can be challenged constructively and a benchmark that depends on trust.
What Endpoint Is Under Test?
For this benchmark contract, both services expose the same two endpoints: GET /health and POST /work.
GET /health returns 200 OK, Content-Type: application/json, and the exact JSON object shown below. It exists to let scripts and acceptance tests confirm that the service is up and speaking JSON. It is not part of the measured work path. The response must not include extra fields.
{
"status":"ok"
}
No timestamp. No process ID. No framework name. No Java or Go version. Health is a readiness check for this contract, not a diagnostic endpoint.
HEAD /health follows HTTP HEAD semantics for the GET /health resource: it returns the same status and headers that GET /health would return, without a response body. The conformance suite treats that as protocol behavior, not as part of the measured work path.
POST /work is the endpoint under test. It accepts a JSON request body and returns JSON on success and on body-bearing failures.
For this benchmark contract, the request Content-Type must be application/json. For common-client compatibility, the service also accepts application/json; charset=utf-8. The JSON media type registration does not define a charset parameter; accepting that parameter here is a deliberate contract choice and does not change JSON processing. A missing Content-Type on POST /work is treated as an unsupported media type. The service returns 415 Unsupported Media Type with the agreed JSON error shape.
Successful and body-bearing error responses use Content-Type: application/json.
The request body for POST /work is also bounded. If the /work request content exceeds 4096 bytes after HTTP transfer decoding, the service returns 413 Content Too Large with the agreed JSON error shape. This limit is intentionally small because the request body contains only control fields, not the generated benchmark payload.
The rule is simple: for requests that can carry a response body, JSON in, JSON out.
That means success responses are JSON. Validation errors are JSON. Malformed request errors are JSON. Unsupported media type errors are JSON. Oversized request errors are JSON. Unsupported method errors are JSON, except where HEAD semantics require no response body. Unknown route errors are JSON. Framework-default HTML pages, plain-text errors, stack traces, redirects, or implementation-specific error documents are not acceptable for this benchmark contract.
Unsupported methods are also part of the boundary. If a client sends GET /work, the service returns 405 Method Not Allowed, includes Allow: POST, and returns the agreed JSON error shape. HEAD /work is unsupported and returns 405 Method Not Allowed with Allow: POST and no response body. If a client sends an unsupported body-bearing method to /health, the service returns 405 Method Not Allowed, includes an Allow header containing GET and HEAD, and returns the agreed JSON error shape. If a client sends POST /missing, the service returns a consistently defined 404 Not Found JSON error.
For the primary pure-service benchmark, request logging, TLS, compression, and tracing are off. Those are real production concerns, and later slices can measure them deliberately. They are not mixed into the primary service-equivalence contract. If a later scenario enables one of them, both implementations must use an equivalent configuration and pass conformance again.
HTTP status-code and method behavior in this contract follows the semantics defined by RFC 9110: HTTP Semantics. The endpoint names, response fields, and exact validation policy are benchmark contract choices.
What Does Each Request Contain?
The request body for POST /work is compact. It does not send a large payload over the wire. Instead, it tells the service how to generate deterministic bytes and how much deterministic work to apply.
The body must be a single top-level JSON object. Top-level arrays, strings, numbers, booleans, and null are invalid request shapes and return 400 Bad Request. Multiple JSON documents in one request body are also invalid.
A valid request looks like this:
{
"requestId":"case-0001",
"payloadSize":1024,
"seed":42,
"extraWork":10
}
Each field has a specific job.
requestId is a stable client-provided identifier. The service echoes it in a successful work response. It must be a JSON string with length 1..128.
payloadSize is the requested deterministic payload size in bytes. For this contract, it must be an integer between 0 and 131072, inclusive. That upper bound gives us payloads up to 128 KiB without putting large payload bodies into article examples or small conformance fixtures. A value below 0, a value above the maximum, a string such as "1024", or a fractional number fails validation.
seed is the input to deterministic payload generation. It must be an integer between 0 and 2147483647, inclusive. Keeping it in a conservative integer range avoids JSON number interoperability problems and makes the same fixture easy to represent in both Java and Go.
extraWork is a bounded work factor. It must be an integer between 0 and 100, inclusive. A value of 0 is valid and means “generate the payload and compute the initial correctness artifact, but apply no extra repeated work.”
Numeric fields must be JSON integers, not quoted numeric strings or fractional values.
Unknown fields are rejected. That may feel strict, but it is useful strictness. If the Go service silently accepts "payload_size" while the Java service rejects it, the two services no longer have the same contract. If one parser ignores "debug": true and the other fails, the benchmark has hidden drift. During conformance, extra fields are a 400 Bad Request.
Duplicate member names in the top-level /work request object are also invalid for this contract. The request schema does not contain nested objects; if nested objects are added later, the duplicate-member rule must be extended explicitly.
JSON syntax and interoperability rules are defined by RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format. RFC 8259 describes JSON objects as unordered collections, says object names should be unique, and notes that behavior can be unpredictable when parsers receive duplicate names. This benchmark contract removes that ambiguity by rejecting duplicate top-level request fields before field validation.
The services should not silently accept a request body such as this:
{
"requestId":"case-0001",
"payloadSize":1024,
"payloadSize":2048,
"seed":42,
"extraWork":10
}
Test fixtures use unique field names. Conformance tests include duplicate-field cases to make sure both services fail them consistently.
How Is Payload Size Controlled?
The request does not carry the full payload. The service generates the payload deterministically from payloadSize and seed.
That choice keeps the benchmark focused. If the client sends a large JSON string, we are measuring request-body transport, JSON string parsing, escaping behavior, memory copies, and possibly different decoder strategies. Those can be useful in another benchmark, but they are not the shape we need here.
For this contract, the input tells the service how many bytes to generate. Both implementations must use the same language-neutral payload-generation algorithm. They must not call a runtime-specific random API and assume the output will match.
The contract can be expressed this way:
state = unsigned64(seed)
for i from 0 to payloadSize - 1:
state = (state * 6364136223846793005 + 1442695040888963407) modulo 2^64
buffer[i] = (state >> 56) & 0xff
Both implementations must use the same constants, unsigned 64-bit wraparound behavior, byte extraction, and iteration count. The same payloadSize and seed must produce the same byte sequence in Java and Go.
That gives acceptance tests a clean target. A test case can say, “For payloadSize = 1024 and seed = 42, both services must generate the same buffer and return the same checksum after the same work factor.” We do not need to inspect the buffer over HTTP; the returned correctness artifact shows whether the service produced the specified result.
What Does the Work Factor Mean?
This is the part of the contract most likely to make or break the benchmark.
extraWork is not a delay knob. It is not “run a loop for a while.” It is not “do whatever is convenient in each language.” It is the number of times the service applies a specified deterministic operation to the generated payload buffer, and the operation must affect the returned checksum.
A language-neutral sketch looks like this:
buffer = generatePayload(payloadSize, seed)
state = SHA-256(buffer)
repeat extraWork times:
state = SHA-256(state || buffer)
checksum = lowercaseHex(state)
The || symbol means byte concatenation: the current 32-byte hash state followed by the generated buffer. The final checksum is the lowercase hexadecimal encoding of the final SHA-256 state.
This does not claim that Java and Go execute identical machine instructions. They will not. The JIT compiler, Go compiler, libraries, memory layout, CPU intrinsics, allocator behavior, and runtime scheduling can differ. Java and Go may also use different SHA-256 implementations. The contract requires equivalent externally specified work: both services generate the same buffer, apply the same deterministic operation the same number of times, and expose the resulting checksum so tests can verify the externally specified result.
That last point is important. If extraWork does not affect the response, an implementation could accidentally skip the loop and still pass a superficial status-code test. Because extraWork affects the returned checksum, conformance tests can detect many skipped or altered implementations of the specified work. That does not prove instruction-level equivalence; it proves that the observable result matches the contract.
Checksum examples in this article are format placeholders, not test vectors. Conformance fixtures must use complete expected checksum values computed from the published payload-generation and work-factor algorithms.
What Does a Successful Response Look Like?
A successful POST /work response returns 200 OK, Content-Type: application/json, and a stable JSON body.
The success schema has this shape. The checksum value below is a format placeholder, not an expected value for the earlier request:
The placeholder checksum is not a test vector. A real conformance fixture must use a complete lowercase SHA-256 hex string independently computed from the published payload-generation and work-factor algorithms.
The fields are deliberately limited.
requestId must match the request. payloadSize must match the accepted request value. extraWork must match the accepted request value. checksum must be derived from the deterministic payload and the work-factor operation. ok is true for a successful work response.
The response intentionally does not echo seed; the seed is validated and used to derive checksum, but it is not part of the success response schema.
The response does not include runtime metadata. No thread ID. No goroutine ID. No hostname. No process ID. No memory statistics. No framework name. No timestamp. No JDK or Go version. Those can be useful in logs, manifests, diagnostic endpoints, or benchmark artifacts, but they do not belong in the correctness response.
Why so strict? Because correctness responses are compared across implementations. If Java returns "framework": "Helidon" and Go returns "framework": "net/http", either the comparison has to ignore fields or the services fail. Ignoring fields too broadly can hide real differences. The cleaner choice is to keep the response schema focused on the contract.
How Should Validation Failures Behave?
Validation behavior is part of the benchmark, not an implementation detail.
If invalid input changes which requests are accepted, how much work is performed, or what response body is serialized, it can contaminate later measurements. So the contract defines both status code and JSON shape for responses that carry a body.
A field validation error looks like this:
{
"error":{
"code":"INVALID_REQUEST",
"message":"payloadSize must be between 0 and 131072",
"field":"payloadSize"
}
}
For this contract, request-shape and scalar validation failures use 400 Bad Request. That includes missing required fields, wrong JSON types, out-of-range numeric values, unknown fields, duplicate top-level request fields, and invalid requestId values.
Malformed JSON also returns 400 Bad Request, with the error code MALFORMED_JSON and a field value of null. A body containing multiple JSON documents is treated the same way.
Unsupported request media types return 415 Unsupported Media Type. Request bodies larger than 4096 bytes return 413 Content Too Large. Unsupported body-bearing methods return 405 Method Not Allowed, include an Allow header listing supported methods for that target resource, and return the agreed JSON error shape. HEAD responses follow HTTP no-body semantics. Unknown routes return 404 Not Found.
A non-field error uses the same envelope:
{
"error":{
"code":"UNSUPPORTED_MEDIA_TYPE",
"message":"Content-Type must be application/json or application/json; charset=utf-8",
"field":null
}
}
The contract uses these machine-readable error codes:
MALFORMED_JSON for syntactically invalid JSON or multiple JSON documents in one request body.
INVALID_REQUEST for a valid JSON body with an invalid top-level shape, missing required field, wrong type, out-of-range value, invalid requestId, unknown field, or duplicate top-level request field.
UNSUPPORTED_MEDIA_TYPE for missing or unsupported Content-Type on POST /work.
CONTENT_TOO_LARGE for a /work request body larger than 4096 bytes.
METHOD_NOT_ALLOWED for unsupported methods on known routes.
NOT_FOUND for unknown routes.
Conformance fixtures compare HTTP status, machine-readable code, field, and JSON shape when the response carries a body. The message value must be a non-empty string, but basic conformance does not require exact message equality unless a fixture explicitly opts into it. If one service returns INVALID_REQUEST and the other returns bad_request for the same fixture, that is a contract failure.
When a request has multiple problems, conformance tests use this precedence order unless a fixture explicitly states otherwise:
This order keeps routing and protocol failures separate from JSON-shape failures. Basic conformance fixtures should still prefer one failure per request, but the precedence rule prevents accidental drift when multi-fault cases appear.
The error object is a small custom JSON shape. It is not claiming RFC 9457 Problem Details for HTTP APIs compliance. If a future version adopts Problem Details, it should do so explicitly, including the appropriate media type and fields.
What Makes Two JSON Responses Equivalent?
JSON response comparison should be strict enough to catch real differences and flexible enough to avoid false failures from incidental serialization details.
A response with the same fields in a different order is equivalent:
That object is semantically equivalent to the earlier success-response shape if all required fields, values, and JSON types match. Object member order and insignificant whitespace are not part of correctness.
But these differences must fail conformance:
"payloadSize": "1024" instead of "payloadSize": 1024
Missing checksum
Different requestId
Different extraWork
A checksum that does not match the deterministic payload and work factor
A different machine-readable error code for the same validation failure
Extra runtime-specific fields when the contract forbids them
A framework-default HTML or plain-text error body where JSON is required
Byte-for-byte JSON comparison is too brittle. A loose “does the response contain a few expected fields somewhere?” comparison is too weak. The target is structural comparison: required field names, JSON types, values, error codes, and status behavior must match.
The language-neutral contract defines the same request, validation rules, deterministic work, and JSON response before Java/Helidon and Go implementations can enter benchmark execution.
How Do Acceptance Tests Prove Equivalence?
Acceptance tests are the gate between contract and benchmark.
The test runner is not the important part yet. It could eventually be a small custom tool, a script, or a test suite in a language chosen for the benchmark harness. What matters is that the same cases run against both implementations and compare the same expected behavior.
The acceptance cases should cover the ordinary path and the annoying edges.
For the happy path, the suite needs a valid minimal request, representative small and medium payloads, the maximum allowed payloadSize, extraWork = 0, and the maximum allowed extraWork. The maximum-payload case does not need to combine maximum payloadSize with maximum extraWork in a basic conformance run. The point is to prove the boundary values and deterministic behavior, not to turn contract validation into a stress test.
For determinism, the same request sent repeatedly must return the same stable fields and checksum. The same test vector sent to Java and Go must return JSON-equivalent responses. Object member ordering must not affect the comparison.
For validation, the suite needs missing required fields, wrong JSON types, negative payloadSize, payloadSize = 131073, invalid requestId, negative extraWork, extraWork above 100, malformed JSON, multiple JSON documents, duplicate fields, unknown fields, unsupported media type, oversized body, unsupported method, and unknown route.
For error equivalence, both services must return the same status code, same machine-readable error code, same required error fields when a body is allowed, and no framework-default HTML, stack traces, or plain-text errors.
For protocol behavior, GET /health must return stable JSON, HEAD /health must follow HEAD semantics for the health resource, POST /work must require JSON, response Content-Type must be consistent with the contract, and 405 responses must include the appropriate Allow header.
Passing these tests does not prove performance fairness by itself. It proves the functional contract. Performance fairness also needs implementation review, allocation analysis, tuning discipline, load-generator validation, environment capture, run protocol, and statistical analysis. Those appear later in the series.
But this is the gate that comes first. Java and Go can differ internally. They can use different libraries, runtime strategies, allocators, diagnostics, and tuning flags. If they cannot produce the same observable behavior for the same fixtures, they do not enter benchmark execution.
The Handoff to the Implementations
The contract gives the implementation articles a narrow job: build services that satisfy the same externally visible behavior.
The Java/Helidon implementation and the Go implementation can choose idiomatic libraries, data structures, and runtime configuration where the contract leaves room. They cannot change endpoint paths, request fields, validation rules, deterministic payload generation, extraWork semantics, success response fields, error response shape, or JSON equivalence rules to make one implementation easier.
That may sound restrictive, but it is what makes the later comparison useful. Once both services pass the same acceptance suite, the benchmark can move on to the next questions: how the services are implemented, how the load generator is validated, how the environment is captured, how runs are executed, and how results are analyzed. Until then, there is no Go-versus-Java result to interpret. There is only a contract to satisfy.
References
The contract choices in this article use standards and product documentation as context, but the endpoint names and field names are benchmark design decisions.
RFC 9110: HTTP Semantics defines the HTTP vocabulary for methods, status codes, content, and representation metadata used by the contract.
RFC 9457: Problem Details for HTTP APIs provides context for standardized HTTP API error responses, although this article uses a smaller contract-specific error shape.
This first article does not publish benchmark results. It defines the contract that later Go net/http and Java Helidon SE runs must satisfy before any chart is worth discussing.
Fairness in this series means testable equivalence: the same endpoint behavior, payload rules, validation and error handling, plus declared CPU, memory, logging, TLS, and process-topology choices.
The primary benchmark is a bounded single-machine Linux service-under-test benchmark. It is relevant to developers building AI services with Oracle AI Database, but it does not measure database execution, model inference, Kubernetes, autoscaling, service mesh behavior, or cloud-provider performance unless those appear as separately labeled slices.
If we include multiple JDK builds, a distinct request-handling mode, a Leyden-related startup or warmup path, or GraalVM Native Image, each one will be labeled as its own runtime shape with exact versions, flags, generated artifacts, and run conditions.
You have probably seen this pattern before. A benchmark shows a compact Go service next to a compact Java service. The chart is interesting. The conclusion is tempting. Then the comments get more useful than the chart.
How many cores were assigned? Was the load generator the bottleneck? Was the Go implementation doing the same work as the Java implementation? What did memory look like? What did garbage collection do? Which JDK build was used? Did virtual threads matter, or were they just present? If the selected JDK includes startup or warmup features associated with Project Leyden work, did they change startup, warmup, first-request behavior, footprint, or steady-state behavior? Were the runs long enough? Were there enough repeats? Were the statistics strong enough to support the claim?
Those are not annoying questions. They are the right questions.
This series follows up on that kind of Go-versus-Java service comparison: a Go net/http service on one side and a Java Helidon SE service on the other. That shape is worth investigating, but only if the method is strong enough to carry the result. The follow-up needs to answer the questions that usually get buried under the chart: CPU and core disclosure, memory use, GC behavior, JDK build clarity, request-handling behavior where applicable, startup or warmup features where applicable, load-generator validity, implementation equivalence, longer runs, more repeats, and statistical treatment.
So Part 1 does not publish new numbers. No old CSV, old chart, or previous result claim is evidence for this run. This article defines the rules a later run must satisfy before any result deserves attention.
If you are building AI applications that use Oracle AI Database, that distinction still matters—but the primary benchmark measures the application-tier service boundary, not database behavior. A retrieval API, embedding gateway, reranking service, feature-enrichment service, or application-tier vector-search facade can spend meaningful time in ordinary service code before and after it touches a database or a model. Runtime efficiency, tail behavior, warmup, memory pressure, and operational visibility still affect cost and reliability.
But if we mix database latency, model inference, service mesh overhead, and runtime behavior into one chart too early, we will not know what we measured.
The first move is to narrow the question.
Why This Follow-Up Starts With a Benchmark Contract
A benchmark contract is the set of rules that determines what is measured, which variants are included, how implementation equivalence is checked, how runs are accepted or rejected, and how analysis is performed before charts appear.
That sounds formal, but the idea is simple: write down the method before you see the result.
Without a contract, benchmark discussion drifts quickly. One side argues that the Go service is smaller. Another side argues that the Java runtime has better optimization and diagnostics. Someone asks whether the request-handling model changed the result. Someone else asks whether the load generator was saturated. Eventually the conversation becomes a proxy argument about language preference instead of a concrete discussion about a specific workload.
Our thesis is deliberately narrower: a useful Go-versus-Java service benchmark starts with a contract, not a chart. We will define equivalent work, fair runtime variants, measurement rules, and acceptance gates before publishing results, so later performance claims can be judged against a declared method rather than accepted on trust.
The contract has to answer a few practical questions before any chart is allowed:
What service behavior must the Go and Java implementations provide?
Which runtime shapes are being compared?
Which choices are fixed for the primary run, such as logging, TLS, CPU allocation, memory policy, and process topology?
What correctness tests prove that both implementations do the same externally visible work?
What evidence shows that the load generator has headroom?
Which warmup and steady-state windows define the measurement?
How many repeats are required, and what makes a run invalid?
Which raw artifacts connect the final chart back to source, build, runtime, machine, command, and analysis rule?
That list is the benchmark contract in plain language. A script producing numbers is not enough. A valid result has to trace back to the source revision, build, runtime, machine, command, measurement window, raw output, and analysis rule that produced it.
That dependency chain is the whole point of the series.
We define the benchmark contract in Articles 1–8 before publishing fresh results in Article 9 and interpreting the safe conclusions in Article 10.
The benchmark manifest for accepted runs will record the exact Go, JDK, Helidon, Linux, hardware, and tool versions used, including details that affect interpretation such as kernel version, CPU policy, and whether the service is running directly on the host or inside a constrained environment.
Which Go and Java Runtime Shapes Are Included
The comparison in this series is not “Go versus Java” in the abstract. That framing is too broad to be useful.
The Go side will use Go’s standard-library net/http package. That keeps the baseline focused on the built-in HTTP server path and avoids introducing third-party router or middleware choices into the primary comparison. Real Go services often include routers, middleware, validation libraries, and observability agents, and that is fine. This benchmark starts with the standard-library service boundary because it keeps the baseline small enough to audit.
The Java side will use Helidon SE, the selected Java service framework for this series, so the benchmark compares one declared Java service shape rather than the broader Java framework ecosystem. Spring Boot, Quarkus, Micronaut, and other stacks matter in real systems, but adding them here would change the question. This series is about Go net/http and Java Helidon SE under a declared service contract.
We will not report the Java side as just “Java.” If multiple JDK builds are included, each Java result will identify the JDK distribution or build provider, version, build identifier, command line, JVM flags, and support or licensing label used for that slice where that label affects interpretation. The question is not whether a label is faster in the abstract. The question is how the same Helidon SE source behaves under the specific JDK builds declared in the manifest.
Virtual threads are part of that Java service-shape discussion, but they need careful framing. JEP 444 delivered virtual threads in JDK 21. They are most relevant when the service model uses many concurrent blocking operations, especially I/O-bound work. They let developers write straightforward blocking-style code while supporting high concurrency more efficiently in workloads that spend time waiting.
They should not be treated as a general throughput or latency improvement for CPU-bound handlers. CPU-bound handlers still need CPU. If a request handler is mostly serialization, checksum work, or allocation pressure, the virtual-thread story may look different from a service that spends much of its time waiting on blocking I/O.
So we will treat the Java request-handling model as something to verify and measure, not something to assume. We will document the selected Helidon version, JDK version, server configuration, and request-handling model before applying labels such as platform threads, virtual threads, or default Helidon behavior. If that combination provides a distinct, supported, reproducible virtual-thread service shape, that slice will be labeled separately. If virtual threads are simply part of the selected Helidon shape, the manifest will say so instead of pretending that “Java” is a single opaque runtime.
Project Leyden and related startup or warmup features get the same separate-label treatment, and only when the selected JDK provides a supported, reproducible path. Project Leyden aims to improve startup time, time to peak performance, and footprint. Related JDK work, such as JEP 483: Ahead-of-Time Class Loading & Linking, is relevant to startup and warmup questions, but availability depends on the selected JDK version and build.
Leyden-related JDK features and GraalVM Native Image are different technologies. GraalVM Native Image ahead-of-time compiles Java applications into native executables. Leyden-related JDK features such as ahead-of-time class loading and linking are a different path. If either is included, we will label it separately, record the exact toolchain, flags, generated artifacts, and measurement scope, and avoid folding those results into the default JVM result.
For any startup or warmup feature associated with Leyden work, we will look especially at startup, warmup, footprint, and first-request behavior. If it appears to affect more than startup or warmup, the analysis will need artifacts to support that claim.
What Fair Means in This Benchmark
Fairness is not a slogan. It is a set of checks.
For this series, fair means the Go and Java services perform the same externally visible work: the same endpoint semantics, payload rules, validation behavior, error behavior, response shape, logging policy for the primary run, TLS policy for the primary run, declared CPU allocation, declared memory policy, and process topology.
That last part matters. One Go process versus one Java process is a different topology from several processes per runtime. A service with request logging disabled is different from one formatting a line on every request. Plain HTTP is different from TLS. A colocated load generator is different from a load generator running with enough isolated headroom to stay out of the server’s way. None of these choices is automatically wrong, but hidden choices make results hard to interpret.
Equivalent work comes before tuning. Each implementation must provide the same externally visible behavior before runtime-specific optimization is considered: request parsing, response generation, validation, error behavior, and any configured dependency interaction.
If the Go handler accidentally performs extra allocation in a work loop that the Java handler avoids, that is not a Go runtime result; it is a benchmark implementation bug until proven otherwise. If the Java service returns a different error body, validates different parameter ranges, or serializes a different payload, that is not a runtime result either.
The contract will require correctness tests before performance runs. Those tests need to compare outputs across implementations using stable request cases, not generated numeric row identifiers or one-off local observations. If an implementation changes for equivalence, the changed source becomes the benchmark source. Old results do not carry forward.
Fairness also means symmetrical restraint. We should not tune one runtime aggressively while leaving the other at casual defaults unless that is explicitly labeled as a tuned-versus-untuned comparison. We should not enable diagnostics inside the primary measurement path for one runtime and not the other unless the overhead is measured and accepted. We should not declare a run valid if the load generator is the thing being measured.
A useful benchmark should make unfairness difficult to hide.
What the Single-Machine Benchmark Measures
The primary benchmark will measure a declared single-machine Linux service-under-test workload. The service process or processes being measured will run on one declared Linux machine with a declared CPU, memory, and process policy. The load-generator placement will also be declared and validated in the run protocol because the client side can affect the result. If a separate client host is used for headroom, it remains part of the measurement harness, not a claim about distributed deployment.
Within that boundary, the benchmark will measure service behavior under controlled workload shapes: payload size, work factor, concurrency, CPU allocation, process topology, warmup, steady state, and resource pressure. It will report throughput and latency distribution together, because average response time without distribution data hides too much. It will also report CPU use, memory use, runtime behavior, startup, warmup, and steady-state behavior.
Warmup means the period before the measurement window where the service is allowed to initialize, compile hot paths, populate runtime structures, and settle into a more representative operating shape. Steady state means the measurement window after warmup, when the benchmark records the primary service behavior for that run.
For a JIT runtime, warmup is obviously important. For a non-JIT runtime, warmup can still matter because caches, connection state, allocator behavior, and CPU-frequency behavior can change early-run measurements.
Java diagnostic runs will include runtime telemetry where appropriate, such as GC behavior, allocation pressure, thread behavior, and selected Java Flight Recorder data. JDK tools such as jcmd can help capture runtime state in diagnostic runs, with exact commands recorded in artifacts. Profiling and diagnostics can add overhead, so diagnostic runs must stay separate from primary measurement runs when that overhead matters.
Go diagnostic runs will also capture runtime and diagnostic information where appropriate, using Go tooling such as pprof, runtime statistics, and tracing. Again, the point is not to drown the reader in profiles. The point is to avoid explaining a throughput or latency shape by guessing.
The load generator is part of the benchmark, not an afterthought. A service benchmark is not valid just because the server process is busy. The client side also needs headroom. If the load generator is CPU-saturated, cannot sustain the requested rate, or distorts latency measurement, the run is evidence about the test harness, not about Go versus Java service behavior.
The selected load generator must be validated for headroom and latency-measurement behavior. A fixed-rate tool such as wrk2, or another documented tool selected later, can help address coordinated-omission risk, but the benchmark must still prove that the client is not CPU-, network-, connection-, or scheduling-limited. Latency reporting will need distribution detail, not just a single average or p99. Tools and formats such as HdrHistogram are useful here because they preserve a fuller percentile spectrum.
The primary pure-service results will stay separate from secondary slices. TLS is a useful slice. Request logging is a useful slice. A real dependency, possibly an Oracle AI Database call, can be a useful slice. But those slices answer different questions.
If we add an Oracle AI Database dependency later, the result may reflect driver behavior, connection pooling, query shape, database execution, network locality, and runtime behavior together. That can be useful, but it should not be confused with the primary pure-service benchmark.
One more boundary matters: this is an end-to-end HTTP service benchmark, not a JVM microbenchmark. JMH is useful for JVM microbenchmarks, but it is not a replacement for measuring a service path through HTTP handling, payload work, runtime behavior, load generation, and resource use.
What the Benchmark Does Not Measure
The boundary is narrow on purpose.
We keep the primary benchmark inside a single-machine service boundary, so later results are useful without pretending to answer deployment, cloud, or distributed-systems questions.
The primary benchmark will not support conclusions about Kubernetes scheduling, autoscaling, service mesh behavior, multi-node networking, cloud-provider performance, distributed reliability, rolling updates, node cold starts, or production fleet operations. It will not compare cloud providers. It will not make claims about how a full distributed system behaves.
It also does not measure all Java frameworks or all Go frameworks. It does not measure universal language superiority. It does not measure model inference. It does not measure Oracle AI Database-dependent behavior, including database-side vector search, in the primary run. It does not tell you how a full retrieval-augmented generation stack behaves under production traffic.
That does not make the benchmark small in a bad way. It makes the benchmark interpretable.
If you are building an AI application on Oracle AI Database, you often need both kinds of information. You need database and model benchmarks for database and model questions. You also need application-tier service benchmarks for runtime, concurrency, warmup, memory, and operational questions. We start with the service boundary so that later, if a dependency slice is added, we can see what changed.
A benchmark that answers one question clearly is more useful than a benchmark that vaguely gestures at ten.
How the Series Gets From Contract to Results
This is a ten-part series because the method needs to be visible before the result.
Part 1, this article, defines the benchmark contract and the scope boundary. Part 2 defines the language-neutral service requirements: endpoint behavior, payload generation, work factor, response shape, validation, errors, and cross-implementation acceptance tests. Those requirements come before Go or Java implementation detail.
Parts 3 and 4 then constrain the implementations. The Java article covers the Helidon SE service shape, request handling for the selected Helidon and JDK combination, JDK runtime inputs if multiple builds are included, startup or warmup features where applicable, and Java diagnostics. The Go article covers the net/http implementation, the equivalence review, allocation behavior, Go runtime settings, and Go diagnostics.
Part 5 focuses on load generation. That includes request model, fixed-rate behavior, coordinated-omission risk, client headroom, and the evidence needed to show the load generator is not the bottleneck. Part 6 defines benchmark methodology: run protocol, CPU and memory policy, warmup, steady-state windows, invalid-run rules, resource capture, and traceability. Part 7 turns the method into execution artifacts: environment, versions, commands, manifests, run order, and raw outputs.
Part 8 defines analysis before results: canonical data shape, aggregation, uncertainty reporting, plot rules, outlier handling, and invalid-run handling. Only after those steps does Part 9 publish fresh benchmark results. Part 10 interprets what those results mean inside the declared boundary.
That order is deliberate. If the analysis rules are written after seeing the charts, the reader has to wonder whether the rules were chosen to make the charts look better. If the implementation contract is written after tuning, the reader has to wonder whether equivalence was enforced or assumed. If the load generator is validated after a surprising result, the reader has to wonder whether the harness is being rescued.
Part 1 publishes no benchmark results. Later result articles will include or link to manifests that identify the source revision, build commands, runtime versions, machine configuration, load-generator configuration, raw outputs, and analysis scripts used for each accepted run. Each accepted run must be traceable to:
The exact Go and Java source revision
The build commands and generated artifacts
The Go version, JDK distribution or build provider, JDK build identifier, and Helidon version
The runtime flags and process topology
The load-generator command and configuration
The Linux environment manifest for the machine
The warmup and measurement windows
The raw latency, throughput, CPU, memory, and runtime diagnostic files
The analysis script or notebook that produced the chart
This is how old charts and orphaned CSV files stay out of the evidence chain. They may explain why we are asking better questions, but they are not proof for the new run.
Acceptance Gates Before Results
A run does not count just because it completed.
The benchmark needs acceptance gates that reject bad evidence before it influences conclusions. Some gates are functional: the Go and Java services must return equivalent responses for the same valid requests and equivalent errors for invalid requests. Some gates are environmental: the machine, runtime versions, CPU topology, memory, operating-system state, and process topology must be captured. Some gates are methodological: warmup and steady-state windows must be declared before the run, repeat rules must be defined, and invalid-run reasons must be applied consistently.
A run does not count if the Go and Java services return different responses, the selected JDK build is not recorded, the load generator is saturated, the warmup period is missing, a repeat is discarded without a declared reason, resource data is missing, or raw artifacts cannot be traced back to source, build, runtime, and machine state.
That list is not exhaustive, but it shows the posture. Invalidity comes from contract violations, not from surprise. A run is not rejected because one runtime looks unexpectedly good or unexpectedly bad. It is rejected because it failed a rule that existed before the result.
Repeat aggregation also needs discipline. If multiple runs are collected, the series must say how they are combined and how uncertainty is shown. If a run is unstable, the instability is either a result to explain or a contract failure to document. It is not something to hide with a prettier chart.
Resource metrics must be interpreted with throughput and latency, not as a separate appendix nobody reads. A service that reaches high throughput by consuming much more CPU is different from one that reaches the same throughput with less CPU. A low-latency shape with high memory pressure has a different operational story from a low-latency shape with stable memory. GC pauses, allocation rate, heap behavior, and warmup matter because they explain why a service behaves the way it does under load.
This is especially relevant for AI application services. The expensive part of an AI system is not always the HTTP handler, but inefficient service code can still waste CPU that you would rather spend on database work, model calls, embedding pipelines, or concurrent users. The benchmark needs to report enough resource context to make that tradeoff visible.
The Questions This Series Will Answer
By the end of the series, you should be able to answer a tighter set of questions than “Which language is faster?”
Under this single-machine service-under-test workload, how do Go net/http and Java Helidon SE behave as payload size, work factor, concurrency, CPU allocation, and process topology change?
Does workload shape change the result? A small response with little work may stress the runtime differently from a larger payload, a more allocation-heavy path, or a higher-concurrency request pattern.
How much CPU and memory does each variant consume to produce its service behavior? Throughput without resource context is incomplete.
What are the Java runtime characteristics: GC frequency, pause behavior, heap behavior, allocation pressure, thread behavior, startup, warmup, and steady-state profile?
What are the Go runtime characteristics: allocation behavior, heap behavior, scheduler behavior, GC impact, startup, warmup, and steady-state profile?
Does the selected Java request-handling model matter for this workload? If it does, where does it show up: latency distribution, CPU use, memory use, concurrency behavior, warmup, or operational visibility?
If more than one JDK build is included, what changes, if anything, when the same Java source is tested with the same declared JVM settings and separately recorded JDK distribution, version, and build identifier?
Where do startup or warmup features associated with Leyden work help, if they are included: startup, warmup, first-request behavior, footprint, steady state, or not at all under this workload?
If GraalVM Native Image is included as a separate slice, what changes when the Java application is built and measured as a native executable rather than as the default JVM run?
Did both implementations perform the same work-factor calculation and payload handling under the language-neutral service contract? If an equivalence issue is found and corrected, which corrected source revision is used for the main run?
Does adding focused service behavior, such as TLS, request logging, or a real dependency, change the interpretation? If a dependency slice is added, which part of the result belongs to the service runtime and which part belongs to the dependency path?
Which conclusions are safe inside the single-machine service-under-test boundary, and which remain out of scope?
Those are the questions worth answering. Not because they produce a more dramatic chart, but because they make the chart harder to misunderstand.
References and Methodology Notes
These references support the terminology and scope used in this kickoff article:
The most useful thing this first article can do is refuse to rush.
Go offers a standard-library HTTP server and a common single-binary deployment style. Modern Java offers Helidon SE as one focused service framework option, virtual threads for suitable blocking-concurrency workloads, extensive JDK diagnostics, multiple JDK build choices, and ongoing work around startup, warmup, and footprint.
None of that decides the benchmark.
The benchmark has to decide the benchmark, and it can only do that if the work is equivalent, the runtime shapes are explicit, the load generator is valid, the environment is captured, warmup and steady state are separated, repeats are handled consistently, and uncertainty is reported honestly.
That is the contract. Next we define the language-neutral service requirements so Go and Java are asked to do the same work before either implementation gets tuned.
Grounded AI applications are hard to trust when the evidence is split across similar text, structured rows, and relationships. Flight disruption questions are a good example: a delayed traveler needs timing math, airport relationships, route alternatives, policy language, and a plain explanation that shows how the answer was reached. A basic RAG application can retrieve policy text, but it does not naturally understand that an inbound flight, a connection airport, an outbound segment, and an alternate route form a connected problem.
This tutorial builds a compact travel disruption assistant with Oracle AI Database 26ai, SQL property graphs, Oracle AI Vector Search, Python, and OpenAI. We will load staged tutorial CSVs shaped after OurAirports and BTS on-time performance fields, build a relational schema, define a SQL property graph, compare three retrieval modes, and generate a grounded answer to a missed-connection question.
The completed demo is intentionally small enough to inspect. It is not an operational airline recommendation system, and the included risk score is a deterministic tutorial ranking feature rather than a predictive model. The point is to show how graph-aware retrieval changes the evidence we can give an assistant.
What Is GraphRAG With Oracle AI Database 26ai?
GraphRAG combines graph traversal with retrieval-augmented generation. Instead of retrieving only text chunks that look semantically similar to the question, the application also follows relationships between entities: airports, flights, itinerary segments, connection rules, and candidate routes.
Oracle AI Database 26ai is a useful place to build this pattern because the same database can hold relational tables, vectors, and SQL property graph definitions. In this tutorial, policy chunks use vector retrieval, route facts stay in relational tables, and connected travel reasoning uses graph queries over the same stored data.
The demo calls OpenAI directly for embeddings and answer generation. The tutorial code keeps the OpenAI calls isolated in openai_provider.py, so the retrieval logic, schema, and graph queries remain clear.
Embeddings convert policy text and the user’s question into vectors so the database can compare meaning rather than only keywords. The stored policy vectors and the query vector must come from the same embedding model, because dimensions and numeric behavior are model-specific. If the OpenAI embedding model or configured embedding dimension changes, rebuild the stored policy vectors before comparing new queries with old rows.
Key Features of Oracle AI Database 26ai for GraphRAG
Oracle AI Database lets us keep structured, vector, and graph evidence close to the data. The demo uses a few features together rather than treating them as separate systems.
SQL property graphs
A SQL property graph exposes selected relational tables as vertices and edges. We model airports, passenger itineraries, and itinerary segments as vertices, then use route summaries, flight instances, and helper edge tables as relationships.
Oracle AI Vector Search
Policy chunks are stored with a VECTOR column and searched with VECTOR_DISTANCE(). The tutorial dataset has only a few policy chunks, so we keep the vector path simple and do not create an index in the runnable lab.
Relational constraints
Connection buffers, scheduled times, route risk scores, and passenger preferences are easier to inspect as relational rows. We use SQL for the pieces that should stay deterministic and explainable.
LLM-grounded answers
The final response is generated from retrieved evidence rather than from a free-form prompt. The OpenAI provider wrapper receives the question, evidence blocks, and explanation path, then returns an answer that the demo can inspect.
How to Get Started with Oracle AI Database 26ai GraphRAG
We will use the companion demo package generated with this tutorial. The public article shows the commands and representative snippets; the complete runnable files are in demo/demo_code_package.md and in the extracted demo workspace that accompanies this tutorial package.
Prerequisites:
Python 3.12.
Docker or a compatible Compose runtime for the included Oracle AI Database 26ai Free container.
Access to Oracle Container Registry with the database/free repository terms accepted.
An OpenAI API key, an OpenAI generation model, an OpenAI embedding model, and the embedding dimension used for the vector column.
The staged tutorial CSV files included in data/raw.
The companion demo provisions Oracle AI Database 26ai Free locally with compose.yaml. The container startup mounts sql/10-setup-user.sql, creates a least-privilege graphrag_demo user in FREEPDB1, and leaves the Python checkpoint scripts to create only application-owned tutorial objects. You still supply your own OpenAI key and model identifiers, but you do not need a pre-created Oracle schema. The Python examples use python-oracledb thin mode by default; thick mode is useful when an environment requires Oracle Client libraries or wallet-specific client features.
Create the environment from the extracted demo project root:
docker compose up -d
cp .env.example .env
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
Edit .env and replace only the OpenAI placeholders. The Oracle values already match the local Compose database and demo user created by the init SQL script. Do not commit .env, credentials, private endpoints, wallet files, or API keys:
Before you run the checkpoint scripts, look at the shared helper file. Most of the demo imports demo_common.py, so understanding it makes the rest of the workflow feel like a system instead of a series of magic commands.
@dataclass(frozen=True)
classSettings:
oracle_user: str
oracle_password: str
oracle_dsn: str
openai_api_key: str
openai_model: str
openai_embedding_model: str
vector_dimension: int
The Settings object separates database connectivity, OpenAI generation, and OpenAI embeddings. That separation matters because each part changes at a different pace. If the embedding dimension changes, the vector table must be rebuilt. If the generation model changes, the schema and graph queries can stay the same.
The environment loader also validates a bug-prone value early:
defload_settings() -> Settings:
# Keep the vector table shape aligned with the embedding model.
raiseRuntimeError("OPENAI_EMBEDDING_DIMENSION must be an integer")fromexc
ifdimension<=0:
raiseRuntimeError("OPENAI_EMBEDDING_DIMENSION must be greater than zero")
This is more than defensive programming. Vector search depends on a fixed dimension. A query vector with one dimension count cannot be compared against stored vectors with another, so a typo in .env should stop the run before any database rows are loaded.
The CSV fixtures are small tutorial extracts. They follow the field shape needed for the lab, but they are not real-time operational flight feeds.
Before loading them, inspect the shape. The staged airport file is intentionally tiny:
airport_code
airport_name
airport_type
municipality
region_code
JFK
John F Kennedy International
large_airport
New York
US-NY
ORD
Chicago O’Hare International
large_airport
Chicago
US-IL
SFO
San Francisco International
large_airport
San Francisco
US-CA
DEN
Denver International
large_airport
Denver
US-CO
DFW
Dallas Fort Worth International
large_airport
Dallas-Fort Worth
US-TX
The staged flight file keeps the delay and distance columns that the route summary uses later:
flight_date
carrier
flight_number
origin
dest
dep_delay
arr_delay
cancelled
distance
2025-01-03
OC
410
JFK
ORD
45
23
0
740
2025-01-04
OC
410
JFK
ORD
15
18
0
740
2025-01-03
OC
821
ORD
SFO
10
14
0
1846
2025-01-04
OC
821
ORD
SFO
62
70
0
1846
2025-01-03
OC
603
ORD
DEN
8
11
0
888
GraphRAG Travel Disruption Assistant Demo Project
The demo answers this question:
My flight from JFK to ORD is delayed by 45 minutes. I have a connecting flight from ORD to SFO. Can I still make it, and what should I do?
We will build three retrieval modes:
Mode A: plain policy RAG.
Mode B: policy RAG plus relational itinerary and route facts.
Mode C: policy RAG plus relational facts plus SQL property graph traversal.
The retrieval-mode diagram shows why the rest of the demo is organized as a comparison. Each mode adds a kind of evidence: policy text, structured itinerary timing, and finally the graph path that exposes the DEN alternative.
The route graph is not abstract. It connects a delayed JFK to ORD segment, an at-risk ORD to SFO segment, and a ranked ORD to DEN to SFO alternative.
The map outline is derived from Natural Earth public-domain vector data, and the airport nodes use the latitude and longitude values from the staged airport fixture.
Basic
Step 1: Check the environment
First, run the environment check. It verifies required environment variables and database connectivity without printing secrets.
python 00_check_environment.py
Expected output:
Required database variables: present
OpenAI configuration: present
Database connection: OK
Database identity: <OracleAIDatabasebanner>
Environment check: OK
The check does not print secret values. It only confirms that the names needed by demo_common.py and openai_provider.py are present.
The check script is intentionally small. It calls the same settings loader used by later scripts and runs a banner query against the database. Treat it as a contract test for the workspace: if this step fails, fix the environment, dependency installation, or database connectivity before debugging graph SQL or answer generation.
Step 2: Create the schema
Next, create the database objects. The schema includes airports, flight instances, route summaries, passenger itineraries, itinerary segments, policy chunks, and helper edge tables.
python 01_create_schema.py
The script starts by dropping old demo objects. That makes the lab repeatable while you are learning:
This is a tutorial reset pattern, not a production migration pattern. In production you would use controlled migrations and narrower privileges. Here, the point is to let you recreate the same schema, reload the same data, and compare retrieval modes without hand cleanup.
The create_tables() function creates three kinds of objects:
Entity tables such as airports, passenger_itineraries, and itinerary_segments.
Evidence tables such as flight_instances, route_summaries, and policy_chunks.
Helper edge tables such as itinerary_has_segment_edges, segment_connect_edges, and segment_alternative_route_edges.
The policy_chunks table is where vector search enters the model:
CREATETABLE policy_chunks (
chunk_id VARCHAR2(50)PRIMARYKEY,
policy_key VARCHAR2(80),
title VARCHAR2(200),
content VARCHAR2(2000),
is_synthetic NUMBER(1),
source_note VARCHAR2(500),
embedding VECTOR(<embedding dimension>, FLOAT32)
);
The vector dimension comes from OPENAI_EMBEDDING_DIMENSION, so the table is aligned with the embedding model supplied by the environment. The table also keeps readable policy text next to the vector so the final answer can cite text that a reader can inspect.
The property graph helper edges keep the model explicit. For example, itinerary-to-segment and segment-to-segment relationships are represented as dedicated edge tables rather than reusing a vertex table as an edge source.
CREATETABLE segment_connect_edges (
edge_id NUMBER GENERATED ALWAYS ASIDENTITYPRIMARYKEY,
from_segment_id VARCHAR2(40),
to_segment_id VARCHAR2(40),
connection_airport VARCHAR2(3),
scheduled_buffer_minutes NUMBER,
projected_buffer_minutes NUMBER,
required_buffer_minutes NUMBER
);
That edge table carries the central business fact in the tutorial: how much connection time the traveler is projected to have and how much the tutorial rule requires. By putting those values on an edge, the graph path can explain the connection risk as a relationship, not only as a standalone row.
Expected output:
Dropped old demo objects: OK
Created tables: 11
policy_chunks.embedding data type: VECTOR
Schema creation: OK
Step 3: Load staged airport and flight data
Now load the staged airport and flight CSVs. The airport file uses a small subset shaped after OurAirports fields. The flight file uses a compact sample shaped after BTS on-time performance fields.
python 02_load_airports.py
python 03_load_flights.py
The flight load script filters unusable records, stores scheduled and actual timing fields, and computes route-level summary values. The route risk score is deliberately simple: it combines p90 arrival delay, cancellation rate, and diversion rate into an inspectable ranking feature.
The route summary logic is intentionally visible in Python rather than hidden behind a stored procedure:
fromstatisticsimportmedian
for(origin,dest),rowsingrouped.items():
# Each group becomes one route summary, such as JFK->ORD or ORD->SFO.
# Keep the score inspectable: delay risk plus cancellation/diversion penalties.
risk=(
common.p90(delays)
+100*(cancelled/len(rows))
+100*(diverted/len(rows))
)
# This tuple becomes one row in route_summaries and one graph edge.
route_tuple=(
f"{origin}_{dest}",
origin,
dest,
len(rows),
median(scheduled),
median(actual),
sum(dep_delays)/len(dep_delays),
sum(delays)/len(delays),
common.p90(delays),
cancelled/len(rows),
diverted/len(rows),
float(rows[0]["Distance"]),
risk,
)
The route summary also stores median scheduled elapsed time, median actual elapsed time, average departure delay, average arrival delay, p90 arrival delay, cancellation rate, diversion rate, distance, and the tutorial risk score. That score is not a prediction model. It is a transparent ranking feature that lets the demo compare route evidence without hiding the math.
The percentile helper is intentionally easy to inspect:
This step creates the passenger itinerary, the delayed inbound segment, the outbound connection, candidate route alternatives, and fictional policy chunks.
python 04_build_demo_context.py
The policy documents are tutorial data, not airline policy. They cover same-carrier rebooking, missed connections, baggage transfer, demo minimum connection time, and explanation boundaries.
This script is where the connected use case becomes visible. It creates:
ITIN-100, the synthetic passenger itinerary.
SEG-1, the delayed JFK to ORD inbound segment.
SEG-2, the ORD to SFO outbound segment at risk.
A segment_connect_edges row that says SEG-1 connects to SEG-2 at ORD.
A segment_alternative_route_edges row that points from SEG-2 to DEN as the ranked alternative.
This row does two jobs. First, it connects the at-risk outbound segment to the Denver alternative in the graph. Second, it stores the explanation text and computed candidate score that the final answer can summarize. The fixture has one tutorial alternative, so it receives rank 1; the score still comes from route summaries instead of a hard-coded constant. That is the GraphRAG pattern in miniature: the LLM should not invent the reroute. It should summarize retrieved evidence.
Finally, the script stages fictional policy chunks:
We insert the chunks before embedding them because the database row is the unit of retrieval. Step 6 fills the vector column, but the row already contains the title, text, and source note that the answer can cite.
Expected output:
Route summaries loaded: <route_summary_count>
Projected ORD buffer: 22
Policy chunks staged: 5
Demo context build: OK
Step 5: Create the SQL property graph
Create the graph after the vertex and edge tables are populated:
python 05_create_property_graph.py
The graph uses airports, itineraries, and itinerary segments as vertices. Route summaries and flight instances become airport-to-airport edges. Helper tables represent itinerary structure and route alternatives.
The complete graph DDL in the demo package includes route, flight, itinerary-to-segment, segment-to-segment, segment-origin, segment-destination, and segment-alternative edge tables. The shortened snippet above shows the modeling pattern.
The graph query asks for a path, not just a row lookup:
SELECT*
FROM GRAPH_TABLE(travel_disruption_graph
MATCH
(seg IS segment)-[origin_edge IS departs_from]->(origin IS airport)
-[first_leg IS has_route_to]->(mid IS airport)
-[second_leg IS has_route_to]->(dest IS airport),
(seg)-[alt IS has_alternative_route]->(mid)
WHERE seg.segment_id ='SEG-2'
AND origin.airport_code ='ORD'
AND mid.airport_code ='DEN'
AND dest.airport_code ='SFO'
COLUMNS (
seg.segment_id AS disrupted_segment_id,
origin.airport_code AS origin_airport,
mid.airport_code AS alternative_connection_airport,
dest.airport_code AS destination_airport,
alt.candidate_rank AS candidate_rank,
alt.route_risk_score AS candidate_route_risk_score,
first_leg.route_risk_score AS first_leg_route_risk_score,
second_leg.route_risk_score AS second_leg_route_risk_score
)
);
Representative result:
disrupted_segment_id
origin_airport
alternative_connection_airport
destination_airport
candidate_rank
SEG-2
ORD
DEN
SFO
1
Before we move into vector retrieval, inspect the graph shape itself. The figure below uses circular vertices, labeled relationship pills, and connected edge property cards so the SQL graph definition is easier to connect to the query.
The route map earlier shows where the airports sit geographically. This model view shows what the database exposes to GRAPH_TABLE(): circular vertices are the things we traverse, relationship labels name the edges we follow, and the connected edge property cards show the evidence used to explain or rank the path.
Step 6: Embed policy chunks
Now generate policy embeddings with OpenAI and store them in policy_chunks.embedding.
python 06_embed_policy_chunks.py
The OpenAI provider wrapper keeps model calls out of the retrieval scripts. The retrieval scripts call a small interface instead of embedding provider code directly:
raiseRuntimeError("OpenAI returned an invalid embedding payload")
forvectorinvectors:
iflen(vector)!=self.settings.vector_dimension:
raiseRuntimeError("OpenAI embedding dimension does not match OPENAI_EMBEDDING_DIMENSION")
returnvectors
This keeps provider-specific code out of the retrieval path. The schema, graph query, and prompt builder only ask for embeddings or a generated answer. The dimension check is especially important because it catches a common vector bug before invalid rows are stored.
When the embedding script writes vectors, it converts Python floats into a database vector literal:
That helper makes the handoff visible: the OpenAI provider returns Python lists of floats, and Oracle receives a serialized vector representation that this tutorial passes through TO_VECTOR() for readability. Before adapting the pattern for production or large loads, verify the current python-oracledb vector binding guidance for your driver and database version; a target environment may prefer a driver-native vector binding path.
Oracle AI Database 26ai Free includes native vector support for this local lab, so no separate extension step is shown. We use VECTOR_DISTANCE() instead of shorthand vector distance operators such as <->, <=>, and <#> because the selected metric is visible in the SQL and easier for learners to inspect.
After embedding, a policy row has both readable text and a vector:
policy_key
title
content excerpt
embedding shape
connection_buffer
Connection buffer
Use a minimum domestic connection buffer of 40 minutes…
VECTOR(<embedding dimension>, FLOAT32)
same_carrier_rebook
Same-carrier rebooking
For missed connections caused by delay…
VECTOR(<embedding dimension>, FLOAT32)
The actual vector is a numeric array. The first few values are enough to show the shape:
The numbers are not meant to be read one by one. What matters is that the policy chunks and the question use the same OpenAI embedding model and dimension, so Oracle can compare them with VECTOR_DISTANCE().
The small learning dataset has only a few policy chunks and intentionally does not create a vector index. For larger datasets, evaluate HNSW neighbor-graph indexes and IVF or IVFFlat neighbor-partition indexes with the current Oracle AI Vector Search documentation, your embedding dimension, your latency goals, and your recall requirements.
Advanced
Step 7: Compare retrieval modes
Run the comparison script to see what each retrieval mode can and cannot explain:
python 07_compare_retrieval_modes.py
Plain policy RAG can retrieve missed-connection policy language, but it cannot calculate the connection window. Hybrid vector plus relational retrieval can calculate the window and expose route scores. GraphRAG adds the connected path that explains why one alternative is better.
The comparison script is the main teaching tool in the demo. It does not ask the LLM which mode is better. It retrieves evidence for each mode and prints the missing pieces so you can see why graph-aware retrieval changes the answer.
The plain RAG query is only vector search over policy text:
"ORDER BY VECTOR_DISTANCE(embedding, TO_VECTOR(:1), COSINE) "
"FETCH FIRST 2 ROWS ONLY",
[query_vector],
)
policy_rows=common.rows_to_dicts(cur)
The rows_to_dicts() helper turns cursor rows into dictionaries with lowercase column names. That makes later prompt construction explicit: the answer receives fields such as title, content, projected_buffer_minutes, and candidate_rank rather than anonymous tuple positions.
"WHERE segment_id = 'SEG-2' AND candidate_rank = 1"
)
alternative=common.rows_to_dicts(cur)[0]
This intermediate mode proves that ordinary relational retrieval already solves part of the problem. You do not need graph traversal to know that 22 minutes is less than 40. You need graph traversal when you want to explain how the at-risk segment, the candidate airport, and the route evidence connect.
Representative result:
projected_buffer_minutes
required_buffer_minutes
conclusion
22
40
connection is short by 18 minutes
Mode C adds the graph path, which is why it can cite the connected alternative instead of merely saying “look for a rebooking.”
The graph retrieval follows the edges created from the helper tables. It starts at the disrupted outbound segment, follows the segment origin edge to ORD, traverses route summaries from ORD to DEN and DEN to SFO, and also checks that the segment has a ranked alternative edge to DEN:
cur.execute(
"SELECT * FROM GRAPH_TABLE(travel_disruption_graph "
"MATCH (seg IS segment)-[origin_edge IS departs_from]->(origin IS airport) "
"-[first_leg IS has_route_to]->(mid IS airport) "
"-[second_leg IS has_route_to]->(dest IS airport), "
"(seg)-[alt IS has_alternative_route]->(mid) "
"WHERE seg.segment_id = 'SEG-2' "
"AND origin.airport_code = 'ORD' "
"AND mid.airport_code = 'DEN' "
"AND dest.airport_code = 'SFO' "
"COLUMNS ("
"seg.segment_id AS disrupted_segment, "
"origin.airport_code AS origin_airport, "
"mid.airport_code AS alternative_connection_airport, "
"dest.airport_code AS destination_airport, "
"alt.candidate_rank AS candidate_rank, "
"alt.route_risk_score AS candidate_route_risk_score, "
"first_leg.route_risk_score AS first_leg_route_risk_score, "
"second_leg.route_risk_score AS second_leg_route_risk_score"
")) ORDER BY candidate_rank"
)
alternative_path=common.rows_to_dicts(cur)
This is where the demo moves from “retrieve some rows” to “retrieve a connected explanation.” The graph path proves SEG-2 -> ORD -> DEN -> SFO, while the alternative edge proves that DEN is the ranked candidate midpoint for the disrupted segment. The edge properties carry rank and route scores, so the answer summarizes retrieved evidence instead of inventing a reroute.
This router is intentionally basic. Its purpose is to show the retrieval design decision, not to solve natural-language routing. Policy questions can use vector retrieval. Questions about connection risk, alternatives, route paths, or itinerary structure need graph evidence. A production implementation might replace this with a tested classifier, but the principle is the same: choose retrieval tools based on the evidence needed to answer safely.
Step 8: Generate the grounded answer
Finally, generate the passenger-facing answer from Mode C evidence:
python 08_answer_question.py
Representative answer:
You are unlikely to make the current connection. The projected arrival at ORD leaves less than the demo 40-minute connection buffer before the ORD to SFO departure. The ranked tutorial alternative is ORD to DEN to SFO because the demo graph connects the at-risk segment to DEN and the candidate score is computed from the staged route summaries.
The answer includes an explanation path:
Passenger itinerary
-> inbound segment JFK to ORD
-> projected arrival delay
-> connection at ORD
-> required connection buffer
-> outbound segment ORD to SFO
-> alternative route ORD to DEN to SFO
-> policy chunk: same-carrier rebooking
The answer script builds the prompt from retrieved evidence, not from the user’s question alone:
prompt=f"""Answer with grounded travel disruption guidance only.
Use these headings exactly: Recommendation, Connection feasibility, Evidence path,
Policy basis, Alternatives considered, Confidence and caveat, Data freshness disclaimer.
Use only supplied graph facts, relational facts, and policy chunks. If evidence is missing,
State that this is tutorial data, not real-time operational airline advice.
"""
answer=provider.generate(prompt)
The headings force the generated answer to separate recommendation, feasibility, evidence, policy basis, alternatives, confidence, and data freshness so reviewers can spot unsupported advice.
The generation method uses the same narrow OpenAI wrapper as embeddings:
raiseRuntimeError("OpenAI returned an empty generation result")
returntext.strip()
That keeps the tutorial focused on GraphRAG rather than provider plumbing. The rest of the code only knows that the provider can embed text and generate grounded text.
Run the static validator and optional cleanup when finished:
python 09_validate_demo.py
python 10_cleanup_demo.py
The validator checks file presence, syntax, credential hygiene, local Compose setup, init SQL, and static project consistency. Live database execution uses the included Oracle AI Database 26ai Free container plus your OpenAI configuration.
Tips And Troubleshooting
Keep the tutorial schema small while learning the pattern. It is easier to inspect a six-airport graph than to debug a full operational feed before the retrieval modes are clear.
Use environment variables for all credentials and model identifiers. The demo package’s .env.example is intentionally placeholder-only, and the scripts should never print passwords, tokens, wallet contents, or private endpoint values.
When graph creation fails, check that the helper edge tables are populated and that the source and destination keys reference the intended vertex tables. The tutorial graph deliberately avoids using the same relational table as both a vertex table and an edge table in the same relationship.
When vector results look surprising, inspect the retrieved doc_name, chunk_title, and distance values before changing prompts. For a larger policy corpus, add indexing only after validating the embedding model, vector dimension, metric, and expected recall behavior.
For production services, create a python-oracledb connection pool with oracledb.create_pool() instead of opening one connection per question. Batch embedding calls where your OpenAI rate limits and latency goals support batching, and monitor cost, rate limits, and latency before scaling bulk loads or query fan-out.
Telemetry is optional for this tutorial run. If your environment emits spans to shared infrastructure, keep attributes free of secrets and treat delivery confirmation as operational evidence rather than a prerequisite for understanding the GraphRAG pattern.
Oracle AI Vector Search Notes
This tutorial uses the included Oracle AI Database 26ai Free Compose setup for local execution. The mounted init SQL creates a least-privilege graphrag_demo user in FREEPDB1; the Python checkpoints then create only application-owned tutorial objects.
Oracle AI Database 26ai-compatible environments include native support for the VECTOR data type and VECTOR_DISTANCE(). If you adapt the demo to another environment, keep the runtime user least-privilege and verify the vector and SQL property graph features before running the checkpoints.
The demo uses small deterministic vectors so every ranking step is inspectable. In production, keep vector generation provider-configurable: use the approved embedding provider or embedding API for the environment, and generate document vectors and query vectors with the same embedding model so their vector dimensions and distance relationships are comparable.
Metric choice depends on the embedding model and scoring goal: cosine is useful when vector direction matters more than magnitude, Euclidean is useful when absolute distance and vector magnitude are meaningful, and dot-product-style scores are useful only when the model and normalization strategy make score direction explicit.
For larger datasets, evaluate vector index choices in the target environment. HNSW indexes are a good fit when recall and low-latency nearest-neighbor lookups matter and additional memory is acceptable. IVF or IVFFlat-style neighbor partition indexes are useful when partitioned search and memory control are more important. The right choice depends on workload size, latency goals, recall goals, and metric alignment.
This tutorial keeps the runnable lab focused on the small policy_chunks table and does not publish exact vector-index DDL. Before adding an index example to production documentation, validate the current Oracle AI Vector Search syntax in the target environment and use table names from the real schema being documented.
Conclusion
We built a travel disruption assistant that shows why GraphRAG matters for connected questions. Plain policy retrieval finds relevant words, hybrid retrieval adds structured facts, and the graph-aware mode supplies the path that ties the passenger, flights, airports, connection buffer, route alternatives, and policy evidence together.
This pattern fits applications where the answer depends on both semantic text and explainable relationships. It is not a substitute for validated operational systems, live airline data, or production decision policies. The next useful step is to run the extracted local demo, inspect the Mode A/B/C evidence side by side, and then adapt the schema to one additional connected scenario.
Related resources:
Oracle AI Database – Start with the database platform used for the relational, vector, and graph portions of the lab.
GraphRAG helps when enterprise answers depend on relationships, not just similar text. A passage can mention the right product or policy and still be wrong for the customer, asset, contract, region, date, entitlement, or user.
Start with a minimum viable knowledge graph. Model the smallest set of entities, relationships, and linked sources that improves one real retrieval workflow before expanding into a broader enterprise graph.
Oracle AI Database 26ai can support a database-centered GraphRAG pattern. SQL property graphs, Oracle AI Vector Search, relational tables, JSON metadata, and SQL predicates can work together while the application or retrieval service orchestrates the workflow.
The useful output is an evidence bundle. A practical GraphRAG system should return passages, graph paths, structured facts, source identifiers, provenance, and authorization-filtered context that an AI application can use safely.
A useful AI answer needs more than similar text
A support engineer asks what sounds like a straightforward question:
Which maintenance policy applies to ACME’s pump 17 after the seal failure?
A vector search workflow may get close. It might retrieve a product manual for Pump X9, a support note about seal failure, a policy document about pump maintenance, and a contract clause that mentions replacement work.
That is a good start. It is not yet a good answer.
The policy may mention the right product but apply only in another region. The support note may describe the right failure mode but be tied to a retired product version. The contract clause may use similar language but belong to another customer. A retrieved passage can be semantically relevant and still be wrong for this customer, this asset, this contract, this entitlement, this incident date, or this user.
That is the practical gap GraphRAG is meant to close.
GraphRAG is a retrieval-augmented generation pattern that adds graph-based relationship context to semantic retrieval. Instead of sending only similar passages to a model, a GraphRAG system can also retrieve connected entities, relationship paths, structured facts, source identifiers, and provenance.
In this article, Oracle GraphRAG means a GraphRAG-style architecture pattern built with Oracle AI Database 26ai capabilities. It is not a separate product name. The examples use Oracle AI Database 26ai capabilities such as SQL property graphs and Oracle AI Vector Search.
The practical pattern is simple: use vector search to find candidate text, use graph traversal to understand which entities and relationships matter, use SQL predicates to apply structured filters and authorization, and send the AI workflow an evidence bundle rather than a loose pile of similar passages.
For the ACME question, the retrieval layer should be able to answer questions like these before it sends context to a model:
Does ACME Manufacturing own PUMP-17?
Which product is PUMP-17 an instance of?
Which contract is active for ACME?
Which entitlement does that contract grant?
Does the entitlement cover Pump X9?
Which policy governs Pump X9 in EMEA on the incident date?
Which support document cites the policy?
Is the requesting user allowed to see this contract, policy, asset, and document context?
That is the shift. We are not just looking for text about seal failure. We are looking for evidence that applies.
Where this article fits in the graph series
This article builds on a few recent RedStack posts about graph features in Oracle AI Database 26ai. If you want the hands-on starting point, begin with Build your first SQL property graph in Oracle AI Database 26ai. That post shows the basic shape of creating and querying a graph.
The ACME/PUMP-17 example below is a fictional product-support scenario used to illustrate the architecture pattern.
Why vector similarity is not always applicable evidence
Vector search is valuable because language is messy. Users rarely ask questions using the exact words in your documents. A support engineer may ask about “pump 17,” while the manual says “Pump X9 installed asset,” the policy says “sealed pump assembly,” and the case note says “mechanical seal failure.”
Oracle AI Vector Search helps with that semantic gap. It lets applications store vector embeddings with business data and query for semantic similarity. That is a strong starting point for retrieval-augmented generation.
The hard part begins when similar text has to be tested against business context. Similarity helps find candidate evidence. It does not prove that the evidence applies.
For our support example, a policy passage may be highly similar to the question. It may mention Pump X9 and seal failure. But the answer can still be wrong if ACME does not have the right entitlement, the contract is inactive, the policy applies only outside EMEA, the policy was not effective on the incident date, or the user cannot access the contract details.
That is why the retrieval path needs relationships among customers, assets, products, contracts, entitlements, policies, regions, dates, support documents, and incidents.
Those relationships are often already present in operational systems. They appear as foreign keys, relationship tables, status columns, date ranges, document metadata, entitlement rules, and access predicates. A knowledge graph becomes useful when it makes those relationships retrievable for the AI workflow.
A knowledge graph represents important business entities and the relationships among them. In enterprise AI, that graph might connect customers to contracts, assets to products, policies to regions, suppliers to components, employees to certifications, or incidents to affected services.
The graph is not valuable because it is abstract. It is valuable because the relationships change which evidence should be retrieved, how the answer should be explained, or what the workflow should do next.
There is a boundary worth stating plainly. Graph context supports retrieval and explanation. It does not replace business rules, data quality work, authorization, review workflows, or application-level evaluation. If a policy decision requires approval, the graph should help assemble evidence for that decision. It should not pretend to be the decision-maker.
The same pattern shows up outside product support. In supplier risk, the answer may depend on suppliers, facilities, contracts, certifications, incidents, components, and affected products. In policy compliance, it may depend on regulations, controls, policies, exceptions, evidence documents, and business units. In workforce planning, it may depend on employees, roles, certifications, projects, skills, and learning material.
Graph-enriched retrieval is useful when relationships change which evidence is applicable, how the answer is explained, or what action the workflow should take next.
What is a minimum viable knowledge graph?
A minimum viable knowledge graph is the smallest set of entities, relationships, and linked sources that improves one real AI retrieval workflow. It is not an enterprise-wide ontology. It starts with one use case, one domain, and the relationships that change whether an answer is correct, applicable, or explainable.
That definition matters because graph projects are easy to overbuild. Once you start drawing entities and relationships, everything feels connected to everything else. Customers connect to accounts, accounts connect to contracts, contracts connect to products, products connect to suppliers, suppliers connect to facilities, facilities connect to shipments, and suddenly your “first graph” has become a long-running modeling program.
Start smaller.
The first graph should earn the right to grow. Start with the path that makes one answer better.
For the support scenario, the first graph might contain customer, asset, product, contract, entitlement, policy, and support document entities. The first relationships might be just as small: customer owns asset, asset is an instance of product, customer has contract, contract grants entitlement, entitlement covers product, policy governs product, and support document cites policy.
That is enough to test whether a policy applies to a customer asset. It is also small enough for a development team to reason about.
Figure 2. A minimum viable knowledge graph starts with the relationships that change answer quality for one use case.
The practical test for each relationship is simple:
Would this relationship change which evidence we retrieve, how we rank it, or how we explain the answer?
If the answer is no, it can probably wait.
A good first graph has one domain, one AI use case, a small set of entity types, a small set of relationship types, a few trusted documents or records, a measurable retrieval question, and a way to compare results against known answers.
Product support is a good starter domain because support questions often depend on customer assets, product versions, contracts, entitlements, policies, and support documents. Policy compliance, supplier risk, asset maintenance, customer service, and workforce skills can also work well.
The point is not to avoid ambition. The point is to make progress. You can add suppliers, facilities, incidents, exceptions, regulatory controls, service history, and case outcomes later. Add them when a real retrieval failure shows that the graph is missing something.
How Oracle AI Database 26ai capabilities map to GraphRAG
Oracle AI Database 26ai can support this pattern because graph, vector, relational, JSON, and SQL capabilities can be used together in a database-centered architecture. That does not mean the database automatically builds the GraphRAG application. It means developers have a practical foundation for keeping retrieval close to the business data that gives evidence its meaning.
A SQL property graph exposes selected database objects as vertices and edges with labels and properties, so applications can query relationship patterns while still working inside a SQL-centered database architecture. In Oracle AI Database 26ai, CREATE PROPERTY GRAPH defines a property graph over supported database objects, and GRAPH_TABLE can match graph patterns and project results as rows and columns.
For the support workflow, the graph can expose customers, assets, products, contracts, entitlements, policies, support documents, and incidents. The graph query retrieves the path that explains applicability: ACME owns PUMP-17; PUMP-17 is an instance of Pump X9; ACME has an active contract; the contract grants Premium maintenance; Premium maintenance covers Pump X9; and the policy governs Pump X9 in EMEA on the incident date.
Oracle AI Vector Search does a different job. It provides vector storage and similarity search capabilities in Oracle AI Database 26ai, allowing applications to retrieve semantically similar passages, documents, notes, or entity summaries where embeddings have been stored and indexed for retrieval. In this workflow, vector search might retrieve policy chunks, support document chunks, product manual excerpts, case notes, contract clause summaries, or entity summaries about seal failure.
Relational tables remain central. They hold operational facts such as customer records, contract status, asset inventory, product versions, policy effective dates, tenancy, and approval state. SQL is the control plane for applicability: region, date, status, tenant scope, product lifecycle state, data classification, user access predicates, and business rules.
JSON is useful where flexible metadata helps. A document chunk may carry JSON metadata for page number, section title, source system, extraction confidence, classifier output, or retrieval trace. An application may serialize an evidence bundle as JSON before sending it to an LLM or another service. Oracle AI Database 26ai supports JSON data management alongside relational data, which is useful when metadata changes faster than the core operational schema.
Security has to travel with the retrieval path. If the user cannot access a contract, policy, support case, document chunk, graph path, or structured fact, that item should stay out of retrieval results, evidence bundles, prompts, logs, traces, and responses.
Database-layer controls such as privileges and row-level predicates can help, and the Oracle AI Database Security Guide covers the broader database security model. The application still has to pass user context, call the right retrieval service, apply authorization-aware predicates, and avoid leaking restricted context into prompts, logs, traces, or responses.
The architecture boundary is important. Oracle AI Database 26ai provides the graph, vector, relational, JSON, and SQL foundation. The application or retrieval service orchestrates entity resolution, embedding calls or lookup, graph traversal, result fusion, prompt assembly, response handling, evaluation, and workflow-specific approval logic.
A practical Oracle GraphRAG flow
A practical Oracle GraphRAG flow turns a user question into an evidence bundle. The retrieval service does the orchestration, while Oracle AI Database 26ai capabilities provide graph, vector, relational, JSON, and SQL context for the bundle.
Figure 1. The application or retrieval service orchestrates the GraphRAG flow. Oracle AI Database 26ai supplies graph, vector, relational, JSON, and SQL capabilities used to assemble the evidence bundle.
In a typical flow:
The application captures the user question, identity, tenant or customer context, and any known entities from the UI.
The application embeds or interprets the question.
Oracle AI Vector Search retrieves candidate passages, chunks, records, or summaries.
The application resolves relevant entities such as customer, asset, product, policy, contract, incident, or region.
A SQL property graph query traverses relationships among those entities.
SQL predicates apply region, date, status, entitlement, tenant, data classification, and authorization-aware filters.
The retrieval service assembles an evidence bundle with passages, graph paths, structured facts, source identifiers, and provenance.
The evidence bundle is sent to an LLM, assistant, search workflow, recommendation service, decision-support UI, or bounded agent.
The application returns an answer, recommendation, or next action with source references and workflow controls.
An evidence bundle is the context assembled for an AI workflow. It gives an LLM, assistant, search application, recommendation service, decision-support workflow, or agent a grounded basis for an answer or next action.
That “grounded basis” wording is intentional. The evidence bundle is not a guarantee that the generated answer is correct. It is a better input to the workflow. The application still needs evaluation, response controls, user experience design, and human review where the domain requires it.
Walkthrough: policy context for a customer asset
Let’s make the pattern concrete with a small fictional product-support slice. This is a walkthrough, not a full copy-paste tutorial. The important part is the retrieval shape: the graph retrieves the applicability path, while vector search retrieves semantically relevant passages that the application can filter against that path.
Here is the data slice:
Customer: ACME Manufacturing
Customer key: acme-mfg
Policy-applicability region: EMEA
Asset: PUMP-17
Product: Pump X9
Contract: ACME-2026-PLATINUM
Contract status: ACTIVE
Entitlement: Premium maintenance
Policy: Pump X9 Seal Failure Policy
Policy code: pump-x9-seal-failure-emea
Incident type: seal_failure
Support doc: Pump X9 Maintenance Guide
Document code: pump-x9-maint-guide
Event date: 2026-03-14
A vector search for the engineer’s question may retrieve passages about pump seals, maintenance procedures, and policy language. The graph’s job is narrower. It should retrieve the path that proves which policy is applicable to this customer asset.
In plain text, the useful path looks like this:
ACME Manufacturing
owns PUMP-17
is an instance of Pump X9
ACME Manufacturing
has ACME-2026-PLATINUM
grants Premium maintenance
covers Pump X9
Pump X9
is governed by Pump X9 Seal Failure Policy
applies in EMEA
applies to seal_failure
is cited by Pump X9 Maintenance Guide
A SQL property graph query can make that path retrievable once the underlying graph, labels, edge definitions, properties, and predicates are defined.
For this simplified walkthrough, the customer’s region is the policy-applicability region. In a production support system, that predicate might use the asset install site, service territory, contract region, or incident location instead.
The following block is SQL/PGQ-style pseudocode, not a complete runnable script. It is meant to show the relationships and predicates that belong in the retrieval path. A runnable version would require tested table definitions, sample data, privileges, a CREATE PROPERTY GRAPH statement, and target-environment syntax.
The authorization checks are intentionally shown as architecture requirements rather than runnable SQL. Do not send a graph path, document chunk, or structured fact to the model unless the requesting user is allowed to see it.
A conceptual result from that traversal would identify:
Customer: ACME Manufacturing
Asset:PUMP-17
Product: Pump X9
Contract:ACME-2026-PLATINUM
Entitlement: Premium maintenance
Policy: Pump X9 Seal Failure Policy
Policy code:pump-x9-seal-failure-emea
Policy region: EMEA
Effective from: 2026-01-01
Source document: Pump X9 Maintenance Guide
Source code:pump-x9-maint-guide
This result gives the AI workflow something more useful than a similar passage. It gives the path that explains applicability:
Pump X9 Seal Failure Policy applies to ACME Manufacturing's PUMP-17
because ACME owns the asset, the asset is an instance of Pump X9,
ACME has an active premium maintenance contract, the entitlement covers
Pump X9, the policy governs Pump X9 in EMEA for seal failure, and the
maintenance guide cites the policy.
Now vector search can plug in cleanly. Oracle AI Vector Search can retrieve passages about seal failure from policy chunks, support documents, or product manuals where those passages have embeddings available. The graph traversal identifies which policies and products apply to this customer asset. SQL predicates then keep only passages that match the applicable policy, incident type, region, effective date, entitlement, tenant scope, and authorization rules.
Vector search is still doing important work. The graph does not replace semantic retrieval; it helps decide which semantically relevant passages are applicable to this business situation.
The evidence bundle might look like this at the application boundary:
{
"question":"Which maintenance policy applies to ACME's pump 17 after the seal failure?",
"reason":"The support document cites the applicable policy."
}
],
"passages":[
{
"source_code":"pump-x9-maint-guide",
"policy_code":"pump-x9-seal-failure-emea",
"title":"Pump X9 Maintenance Guide",
"section":"Seal failure procedure"
}
],
"structured_facts":{
"contract_status":"ACTIVE",
"policy_effective_from":"2026-01-01",
"incident_type":"seal_failure",
"event_date":"2026-03-14"
},
"provenance":[
{
"source_code":"pump-x9-maint-guide",
"source_type":"support_document",
"title":"Pump X9 Maintenance Guide"
},
{
"source_code":"pump-x9-seal-failure-emea",
"source_type":"policy",
"title":"Pump X9 Seal Failure Policy"
}
]
}
This JSON is not a built-in database response. It is an application pattern. The retrieval service decides how to assemble, filter, log, and pass this context to the next consumer.
The access_context field is illustrative. The important point is that authorization filtering happens before the bundle is logged, sent to a model, or returned to a caller.
The walkthrough proves a modest but useful point: a small graph path can retrieve applicability context. It does not build a full LLM app, implement production authorization, solve entity resolution, or evaluate generated answers. That is fine. The first step is to make the relationship retrievable.
Turn graph-enriched retrieval into a knowledge service
The useful boundary is not “the graph.” The useful boundary is a service that returns trusted context for a workflow.
If we bury the graph query inside one chatbot, the pattern will not travel very far. The next search app, agent, recommendation service, or case-management screen will rebuild the same logic. A better design is to expose graph-enriched retrieval as an application knowledge service.
A platform team might expose a bounded service shape like this:
Behind that boundary, the implementation could use SQL lookups, GRAPH_TABLE, Oracle AI Vector Search, JSON metadata, relational filters, and authorization-aware predicates. The caller should not need to know the graph schema or the vector indexing strategy.
A conceptual request might look like this:
{
"customer_key":"acme-mfg",
"asset_tag":"PUMP-17",
"incident_type":"seal_failure",
"event_date":"2026-03-14"
}
And a conceptual response could return the applicable policy, the relationship paths, and the source references:
{
"applicable_policy":{
"policy_code":"pump-x9-seal-failure-emea",
"title":"Pump X9 Seal Failure Policy",
"region":"EMEA",
"incident_type":"seal_failure",
"effective_from":"2026-01-01"
},
"relationship_paths":[
"ACME Manufacturing owns PUMP-17",
"PUMP-17 is an instance of Pump X9",
"ACME Manufacturing has active contract ACME-2026-PLATINUM",
"ACME-2026-PLATINUM grants Premium maintenance",
"Premium maintenance covers Pump X9",
"Pump X9 Seal Failure Policy governs Pump X9 in EMEA for seal failure",
"Pump X9 Maintenance Guide cites Pump X9 Seal Failure Policy"
],
"sources":[
{
"source_code":"pump-x9-maint-guide",
"title":"Pump X9 Maintenance Guide",
"section":"Seal failure procedure"
}
]
}
The service contract is conceptual. It is not a built-in Oracle endpoint; it is an application boundary that hides graph schema, vector indexing strategy, and authorization logic from callers.
Figure 3. Once the first graph path is useful, expose graph-enriched retrieval as reusable services instead of rebuilding context assembly in every application.
The service layer can start small. Entity lookup resolves user terms to trusted database entities. Relationship traversal returns paths connecting customers, assets, products, contracts, policies, and documents. Policy-context retrieval returns policies applicable to a customer, asset, product, region, date, or incident. Provenance lookup returns source documents, sections, codes, and lineage. Context assembly builds evidence bundles for questions.
An agent retrieval tool is another useful boundary. It gives an agent bounded access to context without giving it raw enterprise data access.
This is where a GraphRAG demo starts to look like a system. The graph is still small, but the boundary is reusable.
Where GraphRAG helps in enterprise AI workflows
Graph-enriched retrieval helps when relationships change which evidence is applicable, how the answer is explained, or what action the workflow should take next.
For semantic search and question answering, a support assistant can retrieve passages about a product failure, then use graph context to check the customer’s asset, contract, entitlement, product, and region. The workflow can return source-backed evidence for the applicable policy instead of a generic policy summary.
For knowledge discovery, a product team can ask which policies, support cases, and incidents are connected to a recurring asset failure. Vector retrieval may find relevant notes. Graph traversal can connect those notes to products, assets, policies, regions, and affected customers.
For recommendations, a service workflow can recommend the next troubleshooting step based on product, asset history, known failure modes, entitlement, and trusted support documents. The recommendation is more useful when the system can explain the path that led to it.
For decision support, a case manager can see why a policy applies before approving an exception. The evidence bundle can show the applicable contract, entitlement, policy, support document, event date, and source references. That does not remove the need for review. It makes the review easier.
For AI agents, the pattern is especially important. An agent should have bounded retrieval tools instead of open-ended access to raw enterprise tables. Give the agent a tool that returns policy context, graph paths, source passages, and provenance. Let the application decide what actions are allowed, what needs approval, and what gets logged.
This pattern is not necessary for every AI system. If a workflow only needs to find a few public documents, vector or hybrid search may be enough. Use GraphRAG when relationship context materially changes retrieval quality or workflow behavior.
What to build next
The next step is not to schedule a six-month ontology program. The next step is to pick one useful workflow and make one relationship path retrievable.
Start with a sequence like this:
Pick one domain, such as product support, policy compliance, supplier risk, asset maintenance, customer service, or workforce skills.
Pick one AI use case where users already ask questions today.
Write five real questions users ask in that workflow.
Identify the relationships that change the answer.
Model the smallest useful set of entities and edges.
Link trusted documents or chunks to graph entities.
Add vector search where semantic retrieval improves the workflow.
Use SQL predicates for status, date, region, tenant, entitlement, and authorization.
Evaluate against known questions and expected evidence.
Expand only when failures show a missing entity, relationship, source, or filter.
The first evaluation should be practical. Did the system retrieve the right source passages? Did it find the right graph path? Did it apply the right region, date, status, entitlement, tenant, and authorization filters? Did it avoid restricted records? Did it return source references that a human can inspect?
You can also compare the graph-enriched result with a vector-only result. The question is not whether the graph feels more sophisticated. The question is whether it retrieves better evidence for the workflow.
If the system fails because the customer-to-asset relationship is missing, add it. If it fails because policy effective dates are not modeled, add them. If it fails because support documents are not linked to products or policies, add that link. Let failures guide the graph’s growth.
That keeps the graph honest. It grows because it improves retrieval, explanation, recommendation, or decision support.
Conclusion
GraphRAG becomes useful when relationships become retrievable evidence. Similar passages matter, but enterprise AI often needs more: customer context, asset ownership, product identity, active contracts, entitlements, policy scope, region, effective dates, permissions, source references, and provenance.
Oracle AI Database 26ai is a practical foundation for this pattern because SQL property graphs, Oracle AI Vector Search, relational data, JSON metadata, and SQL predicates can work together in a database-centered architecture. The application still orchestrates the workflow, but the retrieval layer can stay close to the governed business data that gives evidence its meaning.
The first project should be small. Build a minimum viable knowledge graph for one workflow. Use vector search to find candidate passages. Use graph traversal and SQL predicates to test applicability. Assemble an evidence bundle that an LLM, assistant, search app, recommendation service, decision-support workflow, or bounded agent can use.
Pick one workflow, find the relationship that changes the answer, and build the smallest graph path that makes that relationship retrievable.
Choose HikariCP when your requirement is standard Spring Boot JDBC pooling. HikariCP is a strong, widely used choice because Spring Boot auto-configuration supports it directly and exposes HikariCP settings under spring.datasource.hikari.*.
Choose Oracle Universal Connection Pool when Oracle-specific connection management matters. UCP is the better fit when the application’s production contract includes Oracle Database high availability-related behavior, service-aware connection management, Fast Application Notification, Fast Connection Failover, or runtime load-balancing behavior in a supported configuration.
Choose by operational fit, not generic speed claims. HikariCP and UCP solve overlapping but not identical problems, and a pool that behaves well for one workload may behave differently under bursty requests, longer transactions, retrieval workloads, or background jobs.
Test with the workload shape your users will actually hit. A local lab that runs the same Spring Boot application with two pool profiles helps you observe saturation, timeout behavior, and connection pressure before production.
Why Connection Pooling Becomes a Production Problem
A Spring Boot application can look perfectly healthy on your laptop and still run into trouble as soon as real traffic arrives.
At first, the symptoms are easy to misread. A few requests get slower. Then some requests fail with connection timeout errors. Then the database team asks why one service is trying to open more sessions than anyone expected.
AI application patterns can make this show up sooner because a single user request may perform several database operations: retrieval, metadata reads, vector search, chat-memory writes, usage updates, or audit inserts. From the outside, that is still one request. Inside the application, it may touch Oracle Database several times.
The connection pool is where that work becomes visible.
In a Spring Boot application, the connection pool controls how concurrent application work becomes Oracle Database connection and session demand. HikariCP and Oracle Universal Connection Pool (UCP) are both good choices for Oracle-backed Spring Boot applications. HikariCP fits teams that want standard Spring Boot JDBC pooling: connection reuse, explicit pool limits, and clear timeout behavior. UCP fits teams that want Oracle-optimized connection management, especially when high availability-related integration is part of the requirement.
Version basis: the examples below assume Linux and Bash, JDK 21 or newer, a supported Spring Boot line, Oracle Database Free started with Docker Compose, and k6 installed on the Docker host for the local load harness. Use the Oracle Spring Boot Starter for UCP version that matches your Spring Boot line.
The practical answer is simple: choose HikariCP when you want straightforward Spring Boot JDBC pooling and your platform is built around the Spring Boot default pool; choose Oracle UCP when Oracle-specific connection management, service-aware behavior, or high availability-related requirements matter; and test with your workload shape instead of choosing from generic speed claims.
Let’s make that concrete.
What a Connection Pool Does in a Spring Boot Oracle Application
A connection pool keeps a controlled set of physical database connections available for application code. Instead of opening a new database connection for every request, the application borrows a connection from the pool, uses it, and returns it.
The main terms are worth being precise about.
A physical connection is the actual JDBC connection from the application process to Oracle Database. In many common configurations, that connection corresponds to a database session.
A database session is the server-side Oracle Database state associated with a connected client. Pool sizing matters because too many application connections can create too much database session demand.
A borrowed connection is a connection currently checked out by application code.
A returned connection is back in the pool and available for another request.
The maximum pool size is the upper bound on how many connections from that pool can exist at the same time, including borrowed and idle connections.
The minimum idle or minimum pool size setting controls how many connections the pool tries to keep ready for use, depending on the pool implementation.
A connection timeout or borrow wait timeout controls how long a request waits when every connection is busy.
Pool saturation happens when all available connections are borrowed and new requests must wait.
Connection validation is how the pool checks whether a connection is still usable before giving it to application code.
That is the mechanical view. The design view is more important:
The connection pool is where application concurrency becomes database connection and session demand.
A connection pool turns concurrent application work into a controlled number of Oracle Database connections and, in common configurations, sessions. When all pool connections are borrowed, later requests wait or time out.
Here is the quick arithmetic every team should do before production:
4 application instances × maximum pool size 5 = up to 20 database connections
That does not mean the service supports only five users per instance. It means one application instance can hold up to five database connections at the same time. If you run four replicas, that setting can become up to 20 database connections from this one service before you count other applications, background jobs, administration sessions, or monitoring.
This is an upper-bound conversation starter, not a complete capacity formula. It is still useful because it makes the hidden multiplier visible.
A larger pool may reduce waiting in a local test, but it can also increase database session pressure. A smaller pool may protect the database, but it can also create unnecessary request wait time and timeout errors.
The pool is a gate. It protects Oracle Database from unbounded application concurrency, and it becomes a visible bottleneck when the application asks for more concurrent database work than the pool allows.
HikariCP in a Spring Boot Oracle Application
HikariCP is a strong, well-supported choice in many Spring Boot JDBC applications because Spring Boot auto-configuration supports it directly. In a Spring Boot JDBC application, HikariCP is selected when HikariCP is available and no other pool is explicitly selected.
That makes HikariCP a natural fit for many Oracle-backed Spring Boot services. If your requirement is standard connection reuse, explicit pool limits, and clear timeout behavior, HikariCP keeps the application configuration small while still requiring real production sizing and validation.
In the sample application, the HikariCP profile uses a small pool on purpose so saturation is easy to observe:
Start with maximum-pool-size. In this profile, one application instance can hold up to five physical database connections from this pool. If all five are borrowed, the next request waits.
Next, look at connection-timeout. In HikariCP, this controls how long a request waits to borrow a connection before the pool gives up. HikariCP time values are expressed in milliseconds, so 2000 means two seconds. In the sample, the value is intentionally short so you can see timeout behavior without running a long test.
minimum-idle controls how many idle connections HikariCP tries to maintain. It is set here to keep two idle connections ready in the local lab. When minimum-idle is not set, HikariCP uses a fixed-size pool equal to maximum-pool-size; HikariCP’s documentation recommends that fixed-size approach for many production deployments. Your startup behavior, steady-state traffic, and database session budget should drive the final choice.
max-lifetime controls when physical connections are retired and replaced. Set HikariCP max-lifetime several seconds shorter than any database, network, or infrastructure-imposed connection lifetime that applies in your environment. Firewalls, load balancers, NAT gateways, database profiles, and managed service policies can all affect long-lived connections.
idle-timeout is also worth deciding explicitly when the pool is allowed to shrink. HikariCP’s default is 600000 milliseconds. It controls when idle connections above minimum-idle can be removed, and it should be reviewed with max-lifetime, Oracle profile settings, and any network idle limits.
validation-timeout controls how long connection validation can take. HikariCP’s default is 5000 milliseconds; the sample uses 1000 only to keep local failure behavior visible. The oracle.jdbc.defaultConnectionValidation data source property keeps the example aligned with lightweight Oracle JDBC validation. Avoid adding a custom connectionTestQuery just because an old example used one; HikariCP prefers JDBC4 Connection.isValid() when the driver supports it.
Oracle Maximum Availability Architecture guidance for HikariCP also calls out Oracle JDBC configuration details that are easy to miss. For production configuration, prefer the Oracle DataSource class approach when you need to carry Oracle JDBC properties deliberately, use JDBC4 validation rather than a hand-written validation query, and review Oracle JDBC connection properties such as oracle.net.CONNECT_TIMEOUT with the same care you give pool timeouts.
HikariCP is a strong choice when the pool’s job is to reuse connections, limit concurrent sessions, fail predictably when saturated, and expose behavior the application team can monitor through familiar Spring Boot patterns.
It is not a reason to skip production sizing. A simple pool can still be badly sized.
Oracle Universal Connection Pool is a JDBC connection pool for Oracle Database applications. It provides standard pooling behavior and adds Oracle Database-specific connection management features.
That combination is why UCP matters. It is the Oracle Database-focused pool for applications where the database topology and client pool are part of the same operational design.
If your Spring Boot application runs in an environment where the pool is expected to participate in Oracle Database operational behavior, put Oracle UCP on equal footing in the design discussion. That may include production designs involving Oracle Real Application Clusters (Oracle RAC), Fast Application Notification (FAN), Fast Connection Failover (FCF), Runtime Connection Load Balancing (RCLB), service-aware connection management, or related high availability patterns.
Those terms are easy to blur together, so keep the first definitions simple. Oracle RAC is an Oracle Database high availability architecture in which multiple database instances can provide access to a database through services. FAN propagates database service and instance events that UCP can consume for Fast Connection Failover when the required Oracle high availability and Oracle Notification Service configuration is in place. FCF uses those events so the client connection pool can react more quickly to certain failures by detecting and removing dead connections from the pool. Runtime Connection Load Balancing lets supported client configurations use database-provided advisory information when choosing where to create or borrow connections.
Enabling FCF is not just a naming choice in the application YAML. It requires fast-connection-failover-enabled: true, service-name based connections, and an Oracle Notification Service endpoint that is configured and reachable from the application host in the target topology.
These capabilities depend on the right database services, Oracle JDBC and UCP configuration, Oracle Notification Service and FAN setup where applicable, network configuration, and validation in the target topology. UCP can participate in Oracle-specific connection management; it does not remove the need for application-level exception handling, retry decisions, transaction design, or workload testing.
The local sample in this article uses UCP as a connection pool so you can compare basic pooling behavior under the same workload shape. It is not a replacement for high availability testing in a topology designed for that purpose.
The sample uses a ucp runtime profile. Use the Oracle Spring Boot Starter for UCP version selected for your Spring Boot line, and configure UCP through its own property model rather than translating HikariCP settings one for one.
A minimal UCP profile uses the same database connection information as the HikariCP profile, then sets UCP-specific pool controls. The important part looks like this:
In this lab, the ucp profile creates a UCP PoolDataSource explicitly and reads these spring.datasource.ucp.* values. The connection-pool-name gives DBAs and operators a useful name to find in monitoring and database session views.
The property names and units matter. In this sample, HikariCP uses a millisecond-valued connection-timeout, while the UCP profile maps connection-wait-timeout to seconds when configuring PoolDataSource#setConnectionWaitTimeout.
The sample deliberately sets initial-pool-size to 2 so local saturation is easy to see. If you omit initial-pool-size, verify the starter’s default behavior for the exact version you selected instead of assuming the pool starts at the size you expect.
When validate-connection-on-borrow is enabled, also decide the Oracle JDBC validation level deliberately. Validation behavior and cost depend on properties such as oracle.jdbc.defaultConnectionValidation, so carry a tested value such as LOCAL or another level chosen for the target environment.
Configure the profile around the behavior you need:
the maximum number of physical connections this application instance can open
the initial and minimum pool sizes
how long a request waits when the pool is exhausted
whether a connection is validated before it is borrowed
which inactive, abandoned, or lifetime settings fit the environment
whether your production topology needs fast-connection-failover-enabled, Oracle Notification Service configuration, service-name based URLs, abandoned connection timeout settings, or user-based borrowing with getConnection(user, password)
The ucp profile keeps the same workload shape as the HikariCP profile: the same database, endpoints, delayed request path, and load harness. That gives you an apples-to-apples way to observe local pool pressure without turning the article into a UCP high availability tutorial.
Oracle UCP becomes especially valuable when those basic settings are only part of the story. If the production environment expects the client pool to react to Oracle Database service events, planned maintenance, instance changes, or load-balancing advisories, choose and validate UCP with the DBA and platform teams.
The useful question is not “Which pool is always faster?” It is “Which pool matches the operational contract this application has to meet?”
Choose HikariCP when the application needs standard Spring Boot JDBC pooling and your team wants the behavior Spring Boot commonly provides out of the box. That is a common, solid choice. You want connection reuse, an explicit pool limit, predictable wait timeout behavior, and observability. Your platform may already have HikariCP standards, metrics, alerts, and incident playbooks. If those standards meet the application’s operational requirements, HikariCP is a good fit.
Choose Oracle UCP when Oracle-specific connection management is part of the requirement. That usually means the pool is not just an application-local optimization. It is part of the Oracle Database operational design. If production uses Oracle RAC or a service-based Oracle Database architecture, and the application is expected to participate in FAN, FCF, RCLB, or similar service-aware behavior, bring UCP into the design early with the DBA and platform teams.
HikariCP can still be configured with Oracle JDBC and Oracle Maximum Availability Architecture recommendations. The distinction is not that HikariCP is outside the Oracle story; it is that UCP is the Oracle Database-focused pool to evaluate when the pool itself needs UCP-specific capabilities such as FAN, FCF, or RCLB integration in a supported topology.
Follow the platform standard when your organization already has one that is tested and meets the requirement. A standard is more than a dependency choice. It should include timeout policy, connection string patterns, monitoring, session budgets, alerts, and an incident playbook. If the application needs something the standard does not provide, bring that requirement to the platform discussion.
A simple decision flow works well:
If your application needs standard Spring Boot JDBC pooling and your platform already uses HikariCP successfully, choose HikariCP.
If your application needs Oracle-specific connection management or high availability-related integration, choose Oracle UCP.
If your DBA or platform team has a standard for Oracle-backed services, follow that standard unless your application has a clear reason to differ.
In either case, test with the same workload shape your application expects in production.
Choose the pool that matches the application’s operational contract. HikariCP is a strong path for standard Spring Boot pooling; UCP is a strong path when Oracle-specific connection management or high availability-related behavior is required.
Treat performance as workload-specific. A pool that behaves well for short lookup queries may behave differently for longer transactions, bursty retrieval workloads, embedding metadata writes, or scheduled batch work. The local harness below helps you see pool pressure and timeout behavior, but it is not a universal benchmark.
Capability Comparison in Plain English
HikariCP and Oracle UCP overlap in the basics. Both can reuse JDBC connections, set maximum pool size, control how long a request waits for a connection, validate connections, expose operational signals, and make connection pressure visible when demand exceeds capacity.
The difference is what each pool is trying to be.
Use HikariCP when the application needs standard Spring Boot JDBC pooling, compact configuration, and the behavior your platform already expects from Spring Boot defaults. Its configuration vocabulary is familiar to many Spring teams: maximum-pool-size, connection-timeout, minimum-idle, max-lifetime, idle-timeout, and validation-timeout.
Use Oracle UCP when the pool is part of the Oracle Database operational design. UCP adds Oracle-specific connection management controls such as named pools, UCP-specific timeout and abandoned-connection settings, service-aware connection behavior, FCF and FAN integration in supported topologies, and RCLB-related behavior when the database, Oracle Notification Service, service names, driver, and pool settings are all configured for it.
The practical comparison is not “basic versus advanced” or “fast versus slow.” It is “standard Spring Boot pooling contract” versus “Oracle-specific database operations contract.” Both are strong choices when they match the contract. The lab design below focuses only on the shared pooling basics so the first run stays small and reproducible.
Try it Yourself: One Spring Boot App with Two Pool Profiles
Both profiles connect to the same Oracle Database Free container, expose the same endpoints, and use the same load harness. That lets you change the pool while keeping the workload shape the same.
The local lab design runs the same Spring Boot application with either the hikari or ucp profile, then sends the same delayed workload to observe pool pressure.
Keep the repository layout intentionally small:
oracle-spring-pool-lab/
src/main/java/com/oracle/demo/poollab/...
src/main/resources/
application.yml
application-hikari.yml
application-ucp.yml
compose.yaml
pom.xml
scripts/
start-db.sh
run-hikari.sh
run-ucp.sh
load.sh
restart-db.sh
setup/
01-create-app-user.sh
load/
pool-saturation.js
README.md
Pin the core runtime pieces rather than relying on transitive drift:
the Spring Boot patch version used by the sample
JDK 21 or newer
the Oracle Spring Boot Starter for UCP version that matches your Spring Boot line
the resolved Oracle JDBC, UCP, HikariCP, and Oracle Notification Service artifacts
the Oracle Database Free image tag used by the lab
k6 installed on the Docker host for the local load harness
For the HikariCP profile, the Maven dependency set can stay close to normal Spring Boot JDBC. The sample uses Spring Boot web and JDBC starters plus Oracle Spring Boot Starter for UCP; the Oracle JDBC and pool artifacts then come from the resolved dependency tree:
Keep the dependency set clear enough that one pool, not both, owns the active DataSource. In this lab, the hikari and ucp profiles create their DataSource beans explicitly from profile-specific configuration. Record the final dependency tree in the repository so future upgrades are deliberate.
The application exposes three simple endpoints:
GET /health
GET /query
GET /query-with-delay?ms=...
The most important endpoint is /query-with-delay.
There is a subtle trap here. If you run a JdbcTemplate query and then call Thread.sleep(), the connection may already be returned to the pool before the sleep happens. That would not create useful pool pressure.
For this demo, the endpoint borrows a connection explicitly and sleeps before the connection is returned:
thrownewIllegalStateException("Interrupted during demo delay",interrupted);
}
StringdatabaseTime=queryDatabaseTime(connection);
returnMap.of(
"status","ok",
"profile",activeProfile(),
"pool",properties.getPoolLabel(),
"databaseTime",databaseTime,
"requestedDelayMs",ms,
"effectiveDelayMs",delayMs,
"acquisitionMs",acquisitionMs
);
}
}
The delay is intentionally inside the try-with-resources block, so the connection remains borrowed until the block exits. The endpoint also caps the requested delay because this is a local saturation demo, not a general-purpose sleep endpoint.
This is a teaching pattern for the local lab, not a production recommendation. Production code should avoid holding database connections while doing unrelated work.
Prerequisites for the Local Connection Pool Lab
The companion lab uses these prerequisites:
Linux with Bash
Docker and Docker Compose
JDK 21 or newer
Maven, unless you use the repository wrapper script
enough local memory and disk space for the Oracle Database Free container
k6 installed on the Docker host and run from that host, not from a separate k6 container
The repository’s compose.yaml uses Oracle Database Free and exposes the FREEPDB1 service on local port 1521. The setup script creates the local pool_app user. Validate the service name, username, and password against the exact image and initialization path used by the lab if you change them.
Local Run Flow for HikariCP and Oracle UCP
Start by copying the local sample environment file:
cp .env.example .env
The .env file contains local demo placeholders such as ORACLE_DB_PASSWORD. Do not reuse them for shared development, staging, or production environments.
Start the database from the repository root:
./scripts/start-db.sh
The script starts the Oracle Database Free container and runs the repository’s initialization path for the demo user. If you change the image, service name, username, or password, update both the database initialization files and the Spring profile configuration.
Wait until the database container is healthy before starting the application. You can check the container state with:
The exact timestamp and acquisition timing will differ on your machine. What matters is that effectiveDelayMs reflects the requested delay cap and that acquisitionMs rises when the pool is under pressure.
Now run the load harness:
VUS=20DURATION=30s DELAY_MS=750 ./scripts/load.sh
Stop the app with Ctrl+C, then run the Oracle UCP profile:
./scripts/run-ucp.sh
Run the same load again:
VUS=20DURATION=30s DELAY_MS=750 ./scripts/load.sh
The sample includes a scripts/load.sh wrapper that runs the same k6 script against whichever pool profile is active. k6 is a load-testing tool from Grafana Labs; install it on the Docker host and run the wrapper from that host so the script can reach the Spring Boot application at http://localhost:8080. If you run k6 from somewhere else, override TARGET_URL explicitly.
With a pool size around 5, 20 virtual users, and a delay around 750 ms, the sample design is intended to create pool pressure. You should see latency rise and may see failed requests if borrow wait time exceeds the configured pool timeout.
When a borrow request cannot be satisfied within the configured wait timeout, the pool throws an exception and the request fails unless your application handles it differently. In the lab, look for failed HTTP responses in k6 and pool timeout exceptions in the application logs.
That is the point of the demo.
What the Local Demo Shows—and What It Does Not Prove
A local load run shows how a Spring Boot connection pool behaves when application concurrency exceeds the configured pool capacity. It can help you observe request latency, failed requests, borrow timeout behavior, and connection pressure under one controlled workload shape.
A load run gives you two useful local signals without adding pool-specific code:
the k6 summary, which shows request latency and failed requests
the application logs, which show borrow timeout or database connection errors
The k6 summary gives you a request-level view:
HikariCP:
http_req_duration avg=2.24s p(95)=2.67s
http_req_failed 21.56%
http_reqs 269
pool_lab_success_200 211
pool_lab_controlled_503 58
UCP:
http_req_duration avg=1.76s p(95)=2.54s
http_req_failed 36.63%
http_reqs 333
pool_lab_success_200 211
pool_lab_controlled_503 122
Use those numbers carefully. They show how this sample behaves on your machine under one controlled workload shape. They do not prove that HikariCP or Oracle UCP is universally faster.
The important saturation shape is the same even when pools expose metrics differently: the pool reaches its configured maximum, no idle connections are available, and later requests wait until a connection is returned or the borrow timeout expires.
Try changing one thing at a time.
Increase DELAY_MS, and the endpoint holds connections longer.
Increase VUS, and more requests compete for the same pool.
Increase the maximum pool size, and local waiting may drop, but database connection and session demand rises.
Shorten the borrow wait timeout, and saturation becomes visible faster.
This is the lesson you want before production: pool behavior is system behavior. The pool, application concurrency, request duration, database capacity, timeout policy, and number of app instances all interact.
For AI applications, this is especially worth testing before a feature launch. A retrieval endpoint that performs vector search plus metadata reads plus chat memory writes may have a different connection profile than a simple CRUD endpoint. Measure the path your users will actually hit.
This local demo design does not prove production high availability behavior. Oracle RAC, Oracle Data Guard, Application Continuity, Transparent Application Continuity, FAN, FCF, RCLB, and service draining require the right database topology, services, driver settings, network configuration, and workload validation.
Optional Local Recovery Exercise
An optional local recovery exercise restarts the database container while the application is running:
./scripts/restart-db.sh
After the database container is available again, try a normal query:
curl-s http://localhost:8080/query
During the restart, requests may fail. Logs may show database connection errors. Borrow attempts may time out. After the database accepts connections again and the pool has discarded or replaced broken connections, simple requests should succeed.
This is useful because it lets you see the application’s local failure symptoms. Keep the scope clear: a local Oracle Database Free container restart is not the same as validating Oracle RAC failover, Oracle Data Guard role transitions, FAN, FCF, RCLB, service draining, Application Continuity, or Transparent Application Continuity.
Use this exercise as a quick resilience observation. Use an environment designed for high availability testing when those features are part of the production requirement.
Practical Defaults and Production Conversations
The sample values are teaching values. They are not production recommendations.
Before production, treat pool configuration as an application-and-database design conversation. Developers, DBAs, and platform engineers should agree on what the service is allowed to do and how it should fail under pressure.
Start with the pool size. If one application instance has a maximum pool size of 20 and you run ten replicas, that service can open up to 200 connections from those pools before you count other services, jobs, or administrative sessions. That might be fine. It might also be the thing that pushes a shared database service into trouble.
Make the maximum pool size explicit. Align the total across replicas with the database session budget. Include background workers and scheduled jobs in the estimate. AI applications often have ingestion pipelines, embedding jobs, or agent workflows that run outside the main HTTP request path, and those also need connection budgets.
Use explicit wait timeouts. A request that waits forever for a connection is hard to diagnose and unpleasant for users. A request that fails quickly with a clear signal is easier to alert on and easier to handle with a retry policy when the operation is safe to retry.
Watch request latency and pool wait behavior together. If request latency rises while the pool is saturated, the application is probably waiting on database connections. If pool usage is low while latency rises, the bottleneck is somewhere else.
Review connection lifetime settings with the platform team. Firewalls, load balancers, NAT gateways, database profiles, and managed service policies can all affect long-lived connections. The pool’s lifetime and validation settings should fit that environment.
Be careful with retries. A retry can help when the failure is transient and the operation is idempotent. It can make saturation worse when many clients retry immediately against the same exhausted pool. Retry policy belongs with timeout policy, not as an afterthought.
Use leak detection as a diagnostic tool, not a design strategy. The real fix is to return connections quickly and reliably. In Java, that usually means try-with-resources or framework-managed transaction boundaries that you understand.
For Oracle UCP-specific production work, bring the database topology into the conversation early. If Oracle RAC, FAN, FCF, RCLB, Application Continuity, or Transparent Application Continuity are in scope, validate them with the database services, driver settings, network configuration, and workload patterns that production will actually use.
For HikariCP-specific production work, make sure the team has clear guidance for pool sizing, timeout behavior, validation, metrics, and incident response. HikariCP integrates cleanly with Spring Boot, but it still needs operational standards.
A good production conversation covers:
expected request concurrency and workload shape
number of application instances and background workers
maximum pool size per instance
total possible database sessions from the service
Oracle Database service name and connection string
timeout behavior at the pool, HTTP server, gateway, and client
retry policy and idempotency
required Oracle high availability features
pool metrics, alerts, and database session monitoring
what to do when the pool saturates or the database is unavailable
That last point matters. A setting without an operational response is only half a design.
Choose by Operational Fit, Then Test
HikariCP is a strong choice for many Spring Boot JDBC applications that connect to Oracle Database. It is directly supported by Spring Boot auto-configuration, has a compact configuration model, and fits applications that need standard pooling behavior.
Oracle UCP is a strong choice when Oracle-specific connection management or high availability-related integration is part of the application’s operational contract. If the production design expects the pool to participate in Oracle Database service behavior, choose and validate UCP with your DBA and platform teams.
The companion lab design gives you a practical way to see the basics. Run the same Spring Boot app with the hikari profile and the ucp profile. Send the same delayed workload. Watch latency, errors, application logs, and timeout behavior.
Then take those observations into the production conversation.
Keep the pool size explicit, validate the dependency tree and local scripts when versions change, and choose the pool that matches the system you are actually building.
Codex was worth testing because it appeared to reduce the need for separate direct API-key access to model providers while still keeping agent work close to the repository.
The hard part was not getting another tool to create plausible files once. The hard part was preserving the behavior that made the workflow trustworthy across repeated runs.
For this repository workflow, CrewAI remained the better production runner because its task model fit our need for ordered stages, artifact contracts, guardrail hooks, review separation, and repository-level observability.
The broader lesson is not that one agent tool is universally better than another. When agent work becomes production work, evaluate the runner and the operational constraints around it, not only the model or the prompt.
It was the best of agents, it was the worst of agents… it is, after all, the age of agents…
I have been working with agentic AI in earnest over the last several weeks. I started with CrewAI for no special reason – I had learned about it in an agentic AI course, so it was a natural choice to start experimenting. But I can to realize that there are several things about it that I really like, and which don’t seem to hold true for other agentic platforms.
My favorite things about CrewAI:
All of the agent and task defitions, and the extra knowledge, can be stored in YAML file outside of the code proper, making both easier to maintain
It gives a lot of control over parallelization, fan-out, and so on
It lets you create deterministic guardrails easily
It’s python-based and easy to integrate with Oracle AI Memory Core, which I have written about recently, and which I firmly believe dramatically improves performance, acccuracy, etc., for agentic systems
You can easily use different models and effort/reasoning levels for different tasks – which let’s you optimize token use, and is great for LLM-as-a-judge tasks
It has excellent observability, explainability and evaulation options available, e.g., Arize Pheonix has great integration
And, perhaps most importantly, it faithfully respects the workflow and guardrail constraints, and it does so deterministically, not probabalistically
And one more thing – for bonus points – it seems to be much easier to run multiple parallel instances of the agents to work on different jobs at the same time – compared to the Codex alternative described in this article
There came a time when it appeared that I may need to find an alternate plaform, primarily driven by availability of API access to models, and this lead me to explore Codex native agents, to be clear – not OpenAI Agent SDK, but the .agents/skills/... setup in Codex itself.
To my great surprise, especially since I was using the exact same model in both cases, I found that Codex seemed to lack many of the things that I really liked about CrewAI, and it was very difficult to get it to perform tasks reliably. It felt to me like it lacked self-awareness – it constantly made what I would call silly mistakes, it constantly ignored instructions, and it constantly introduced regressions.
So, when the API access situation constraint ceased to be a real constraint, it was a very easy decision to abandon the ill-fated Codex agent experiment, and go right back to CrewAI.
Just to give some context, the agentic system we’re talking about in this article has about 28 agents, 70 tasks, 12 different workflows/pipelines, over 1000 lines of deterministic validation functions (guardrails), shared, persistent memory, and produces several kinds of artifacts, including documents and code. It has some very well defined phases, but within those phases significant opportunity for parallelization. It does use a number of tools, and it also uses the Oracle AI Database Skills that I have written about recently.
By the way, I did recently write about my experiences working with Codex to develop real-world software, which I suggest is an interesting companion read to this piece.
Here’s our story.
Once an agent workflow starts producing useful results, the next question is not “Can the model do the task?” It is “Can the system do the task again next week, with enough evidence that someone else can trust the result?”
That question pushed us to compare two different centers of gravity for repository-based agent work: a CrewAI-based runner and a Codex-based workflow that initially looked easier to operate without separate direct model-provider access.
Codex got much farther than a quick proof of concept. It helped improve the system. It exposed weak assumptions, suggested useful validation checks, and made several implicit contracts more visible. But once model access no longer looked like a hard constraint, the production tradeoff changed, and we kept CrewAI as the runner for this workflow.
By runner, I mean the part of the system that turns work definitions into executed behavior. It sequences tasks, enforces contracts, records what happened, and gives operators enough evidence to decide whether a run is ready to trust.
The lesson was not “Codex failed.” The more useful lesson was this: Codex helped us improve the system. CrewAI was still better at being the system.
Side note: In Embabel, a Java-based agentic AI platform, like CrewAI – the workflow definitions and guardrails are deterministic, i.e., they are code – they are not the LLM making a decision. I think this is an essential ingredient for a successful agent implementation. It’s a case of using the right tool for the job – LLMs are great at some jobs, but that doesn’t mean they are great at every job.
Why a Codex-Based Workflow Was Worth Trying
The attraction was practical before it was architectural.
The first attraction was model access. At the time, Codex looked like a way to run agent work without arranging separate direct API-key access to underlying model providers. For a repeatable workflow, that would have removed a practical adoption problem.
That now appears not to be a hard constraint. Once that changed, the case for replacing the production runner became much less about access and much more about operational fit.
Codex still had a second attraction: it is documented as a coding agent that can read, edit, and run code. That made it a reasonable tool to test around a repository-heavy workflow where fixes often happen in code, configuration, validation scripts, task definitions, or supporting artifacts.
Repository proximity matters. When an agent workflow breaks, the problem is not always the prompt. Sometimes the issue is a brittle validator, a missing artifact check, an unclear task contract, or a convention that exists only in someone’s head. A coding agent close to the repository can be very useful in that repair loop.
Codex was useful in that neighborhood. It helped make assumptions explicit. It pushed us to describe roles, task expectations, validation references, and collaboration contracts more clearly. The clearer those contracts became, the better the Codex workflow became.
But improving a system is not the same job as running it. And once model access stopped being the forcing function, running the system well became the deciding question.
For this workflow, the production runner had to do more than help with implementation changes. It had to preserve ordering, outputs, review boundaries, guardrail behavior, and enough observability for an operator to understand a run without replaying the whole conversation.
That is where the tradeoff changed.
What the Runner Had to Preserve
The workflow we started with was not a single prompt and a style guide. It was a staged production system.
Some stages can run with more independence. Others need a deliberate order. A review is not useful if the thing being reviewed has not settled. A release bundle is not useful if it cannot find the expected outputs. A status report is not useful if it says a stage ran but cannot tell you what it produced.
That is where artifact contracts become important.
An artifact contract is the expectation that a task writes specific files, metadata, or outputs in predictable places and formats, so downstream stages can rely on them.
Here is a simplified example. This is not a production path from any repository:
Expected by the workflow: reviews/final_review.md
Written by the runner: review/final_quality_report.md
The content might exist, but the contract is still broken. If a dashboard, metadata index, or downstream task expects one file path and the runner writes another, the workflow has created an operational problem.
That can sound fussy until you operate the same workflow more than once. Then it becomes essential infrastructure.
Stable file names are how dashboards render. Stable review paths are how people inspect a run. Stable metadata is how downstream tasks know what is ready. Stable validation reports are how we decide whether an output belongs in a release candidate.
So the replacement target was never “make another tool produce plausible output.”
The target was “make another runner execute the production workflow without losing the contracts that make the workflow operable.”
Everything else followed from that.
Structural Parity Was Easier Than Behavioral Parity
This became the central lesson of the evaluation: structural parity is not behavioral parity.
Structural parity means the workflow looks equivalent on paper. The right stages exist. The right task names appear. The right artifacts are requested. The final directory tree resembles the one you wanted.
Behavioral parity is harder. It means the workflow reliably behaves like the production system across runs. Tasks execute in a meaningful order. Review stages remain independent. Output contracts are followed consistently. Validation catches serious defects. Failure states are explicit. A weak stage does not quietly flow downstream as if everything succeeded.
For us, the hard part was not getting a Codex-based workflow to produce the right files once. The hard part was making the right behavior repeatable without live supervision constantly restating what counted as success.
That supervision is useful during design. You can inspect a weak output, tighten the instruction, add a validation step, and try again. Codex is productive in exactly that kind of loop.
Production workflows need more than attentive supervision. They need the runner to carry some of that burden.
A production runner needs to make distinctions visible. A review artifact should explain enough. Execution evidence should provide diagnostic value. Metadata should map the files downstream workflow expects. A stage name should correspond to distinct work, not only to a label in a directory tree.
A filename drifts, and suddenly an otherwise useful artifact becomes invisible to the part of the system that expected it.
Matching the nouns in a workflow is easier than matching the verbs. “Analyze,” “generate,” “review,” “validate,” and “package” are easy labels to reproduce. The behavior behind those labels is where the runner earns trust.
Observability Made the Difference Concrete
Observability was where the comparison stopped being philosophical.
For this repository workflow, observability does not mean “there is a log somewhere.” It means an operator can answer practical questions after the run:
Which tasks ran?
Which tasks produced which artifacts?
Which files were expected, and which were missing?
Which stages deserve closer inspection?
Which model or provider configuration handled a stage, where that metadata is available?
Did a review step do meaningful work?
Can this run be compared with a previous run?
Our CrewAI-based implementation had grown around those questions.
CrewAI’s task documentation describes tasks with fields for expected output, context, output files, callbacks, guardrails, and structured outputs. Our repository then layers run metadata and artifact checks around that execution model.
That distinction is important. CrewAI does not magically record every piece of evidence any team might want. We built a workflow around CrewAI that records the evidence we care about. CrewAI’s task model gave us building blocks that made repository-level observability practical: named tasks, declared output files, task context, guardrail hooks, and sequential or asynchronous execution where the design calls for it.
That is the standard the runner has to meet. Execution evidence that is structurally complete but operationally thin may be enough to make a dashboard row render, but it is not enough to build operator confidence.
A production runner should make the boring questions easy. What ran? What failed? What changed? What did it produce? What should I inspect next?
If the answer is “read the whole conversation and infer it,” the workflow is still too dependent on supervision.
Independent Checks Need Real Substance
Explainability in this kind of workflow is not model interpretability in the abstract. It is the ability to explain how the final output came to be.
What input shaped the plan? What did the technical review flag? What did validation check? What quality concerns were found? What changed before the final output? Why is the final output better than the first version?
A review file is not the same thing as a review.
For this workflow, we needed artifacts that explained what was checked, what changed, what remained uncertain, and which stage made that decision. Otherwise, the workflow could look complete while leaving us with very little evidence that independent review had happened.
Our CrewAI-based staged workflow gave us a cleaner path to that evidence. Technical review can be separate from general validation. Quality checks can be separate from improvement work. Validation can happen after the thing being validated is stable enough to check. Packaging can focus on whether the right public artifacts exist without pretending to be a technical review.
Separation alone does not guarantee quality. A weak review is still weak if it runs in its own task.
But independent stages create places where quality can be measured, compared, and improved. They also make failures more visible. If a review artifact is thin, we can see that the review stage underperformed. If an assessment misses a serious defect, we can tighten the rubric. If validation leaves uncertainty, we can decide whether to improve or hold the output.
For this workflow, the runner has to preserve that independence. If stages collapse into broad undifferentiated generation work, the output may contain files but still lack the evidence we need.
Better consistency checks help, and Codex helped us identify several of them. They did not replace the confidence we already had in the CrewAI evidence trail.
That mattered because the workflow was not only about generating text or code. It was about being able to explain why the result was ready.
Why CrewAI Still Fits This Production Workflow
For this repository, we chose CrewAI because it fit the production shape of the work.
The important pieces were tasks, context dependencies, output files, guardrails, guardrail retry behavior, and execution modes. Those map closely to what the workflow already needed.
CrewAI remained the better fit as the production runner, but not for free. It asks you to model the work as a workflow: agents, tasks, outputs, guardrails, and run artifacts. For a quick coding problem, that can feel like ceremony. For this production workflow, that ceremony was the point.
Here is a deliberately simplified YAML-like sketch, not exact CrewAI syntax and not copied from production. It shows the kind of contract shape we care about:
The value is not that YAML is magical. The value is that the workflow contract is reviewable.
A task can declare what it depends on, who should run it, what it is expected to produce, where the output should land, and what kind of guardrail should apply. The repository still has to validate that the expected files exist and contain useful evidence, but the contract has a visible place to live.
That matters when multiple people maintain a workflow. YAML-centered definitions can be reviewed in pull requests. They can be diffed. They can be discussed without reading the entire runtime implementation. They give reviewers and maintainers a shared object to inspect.
Execution order was another deciding factor.
Some stages must be sequential. Context should inform planning. Planning should guide production. Production should precede review. Improvements should respond to review. Packaging should happen after the required outputs and sidecars exist. CrewAI supports sequential execution, and our workflow uses that where dependencies matter.
At the same time, not everything needs to be serialized. Some supporting work can run asynchronously if the dependencies are clear. CrewAI gives us a mechanism for asynchronous tasks, while our workflow decides where parallel execution is safe.
That balance is exactly what a production runner should support: strong order where it matters, concurrency where it does not.
For this repository, CrewAI also gave us a clearer place to attach task-level guardrails. A guardrail is not a slogan. It is a workflow mechanism. In our implementation, guardrails are where we can encode checks for missing required sections, malformed outputs, broken artifact contracts, review stages with no findings and no explicit “no findings” statement, or validation reports that lack expected evidence.
CrewAI provides the hook; the workflow still has to supply the judgment.
Per-stage model configuration also mattered. We did not want one global “agent intelligence” setting for the whole workflow. Research synthesis, technical review, validation, quality checks, and packaging have different cost, risk, and reasoning profiles. In our CrewAI implementation, tasks are assigned to agents configured for those stages, so model and provider choices are part of the workflow design.
None of this means CrewAI is effortless. It is a framework, and frameworks create surface area. Bad task definitions still produce bad runs. Weak guardrails still miss defects. Observability only helps if we record the right evidence.
But for this repository, CrewAI let us encode more operational concerns in the workflow runner rather than relying on the surrounding conversation to restate them.
What Codex Still Does Very Well
We did not come away liking Codex less. We came away with a sharper sense of where it belongs in this workflow.
I want to be clear that we really, really tried to make Codex successful – we did a lot of reasearch, we learned about how other people have used it, we consulted best practices and reviewed other agentic implementations to work out where we were going wrong. This was not a casual test, we made real, concerted effort to get this to work satisfactorily before finally coming to the conclusion that it was not the right choice for this system
Codex remains useful close to the repository. In our workflow, it helped with code edits, repair loops, test ideas, validation checks, dependency updates, and implementation review. When a contract is vague, Codex can help make it concrete. When a validation artifact is missing, Codex can help design it. When the workflow has implementation bugs, Codex is useful in the repair loop.
The evaluation also improved our thinking.
The task-card style made us write down collaboration contracts more clearly. The comparison forced us to separate “file exists” from “stage did useful work.” The consistency checks that emerged from the exercise are worth carrying back into the CrewAI workflow.
That is a win.
There is also a broader lesson here for teams evaluating agent systems. For us, a coding agent was a strong place to evolve the workflow, while CrewAI remained the better place to run this repeatable system.
That is where we landed.
Codex helped us build and repair the system. CrewAI remained the better fit for running it.
The Decision, for Now
The decision is straightforward: CrewAI remains the production runner for this repository workflow. Codex remains part of how we improve the workflow.
We will keep using Codex where it shines: near code, near tests, near implementation repairs, near task-contract cleanup, and near validation design. We will keep carrying good ideas back from the comparison. We will keep tightening the CrewAI workflow with the consistency checks the work made obvious.
But the production run itself stays in CrewAI for now.
The reason is not that CrewAI is universally better than Codex. It is that this workflow is more than a conversation and more than a patch. It is a multi-stage production system with artifact contracts, review boundaries, execution dependencies, guardrails, observability, and assembly rules.
In that environment, the runner matters.
Other teams should make this decision based on their own workflow shape. If your workflow is mostly exploratory, coding-heavy, and interactive, Codex may be the right center of gravity. If your agent work is mostly pull-request repair, bug fixing, refactoring, or test generation, putting the agent close to the repository can be exactly right.
If your workflow is a repeatable production system, ask different questions.
What must run in order? Which artifacts are contractual? Which stages need independent review? How do you know a stage did meaningful work? What happens when a file is missing? Can you compare one run with the next? Can you recover from a partial failure? Can an operator trust the evidence without replaying the whole conversation?
Those questions pushed us back to CrewAI.
Conclusion: The Runner Has to Keep the Promises
Agent quality matters. Model quality matters. Prompt quality matters. But once agent work becomes production workflow, the runner matters too.
We learned that describing a workflow is easier than running one. Codex helped us describe, inspect, repair, and improve the system. It pushed us to make implicit contracts explicit. It generated useful ideas we will keep.
That is not a small contribution.
But the workflow also needed trustable operational properties: ordered execution, stable artifacts, useful diagnostics, review independence, guardrail hooks, repository-level evidence, retry hooks where appropriate, and per-stage control. For this repository, CrewAI gave us more of that in the runner structure we could build around.
That is the decision heuristic I would take from this evaluation: do not only ask whether an agent can name your process, draft your files, or produce the right shape once. Ask whether the runner can keep the promises your process depends on when nobody is watching every step.
Helidon and Spring Boot services can work as peers when they share platform contracts instead of framework internals.
CloudBank V5 demonstrates those contracts through Eureka discovery, JWT/OIDC security, APISIX routing, and OpenTelemetry telemetry.
The Helidon customer service and Spring Boot account service each stay idiomatic while participating in the same operational system.
A live CloudBank V5 deployment shows the mixed services registered in Eureka and visible in SigNoz telemetry.
I have been working with a microservices application that includes both Helidon and Spring Boot services. That mix is not unusual. Real systems accumulate useful services over time. Different teams choose different frameworks for good reasons. Some services benefit from Spring Boot conventions, Spring Security, and the Spring ecosystem. Others are a good fit for Helidon, MicroProfile APIs, and a smaller service runtime.
The interesting question is not which framework should win. In an application that already has both, the practical question is how the services collaborate.
This article looks at that collaboration pattern through CloudBank V5, an example banking application that includes a Helidon customer service and Spring Boot services such as account. CloudBank and OBaaS give us a concrete deployment and operations environment, but they are not the point of the article. The point is interoperability: how Helidon and Spring Boot services can behave like one system when they agree on the right contracts.
In the CloudBank example, the shared contracts are:
service discovery through Eureka
authentication and authorization through Spring Authorization Server, OAuth2, OpenID Connect metadata, and JWTs
external API routing and route-level scope checks through APISIX
common telemetry through OpenTelemetry auto-instrumentation and SigNoz
Each service remains idiomatic inside its own framework. The integration happens at the system boundaries.
The Collaboration Model
Mixed-framework systems work best when the shared surface is small and explicit. The goal is not to make a Helidon service look like a Spring Boot service, or the other way around. The goal is to make both services understandable to the platform and to each other.
In CloudBank, that means a request can enter through APISIX, carry a JWT issued by the authorization server, resolve service targets through Eureka, reach either the Helidon customer service or the Spring Boot account service, and emit telemetry into the same observability pipeline.
That is the architecture in one picture. The gateway does not need to know the backend programming model. The registry does not need every service to use the same application framework. The authorization server publishes a token contract. The observability pipeline consumes telemetry in a common format.
The services collaborate because they share platform contracts, not because they share framework internals.
Discovery: One Registry for Different Runtimes
Service discovery is the first integration plane. A mixed-framework system needs a way to identify services without binding callers to pod addresses, Kubernetes internals, or framework-specific clients.
CloudBank uses Eureka as the service registry. The Spring Boot account service reaches Eureka through normal Spring configuration. Its application configuration gives the service a stable Spring application name and imports shared settings:
spring:
application:
name: account
config:
import: classpath:common.yaml
The shared Spring configuration supplies the Eureka client behavior:
eureka:
instance:
hostname: ${spring.application.name}
preferIpAddress: true
client:
service-url:
defaultZone: ${eureka.service-url}
fetch-registry: true
register-with-eureka: true
enabled: true
The Helidon customer service reaches the same registry through its deployment configuration. Its Helm values identify the service as a Helidon workload and enable Eureka registration:
fullnameOverride:"customer-helidon"
obaas:
releaseName:"obaas"
framework:"HELIDON"
eureka:
enabled: true
instance:
appname:"helidon-customer-service"
This is a good example of healthy interoperability. The two services do not need the same configuration syntax. They need the same operational outcome: each service has a stable identity and registers with the same discovery system.
That shared registry matters at the edge. CloudBank’s APISIX routes use Eureka discovery for upstreams. A route can target the ACCOUNT or CUSTOMER service identity without caring whether the implementation behind that identity is Spring Boot or Helidon.
The design rule is simple: let each framework configure discovery in its natural way, but make the service identity stable and visible outside the process.
In the live CloudBank deployment used for this article, Eureka shows Spring Boot ACCOUNT and Helidon CUSTOMER-HELIDON registered in the same service registry:
Security: Use the JWT as the Shared Trust Object
Security is the integration plane where framework differences can become painful if the architecture is not careful. Spring Security and MicroProfile JWT do not expose the same programming model. That is fine. They do not need to.
The shared object is the JWT.
CloudBank uses azn-server, a Spring Authorization Server service, to issue OAuth2 access tokens. APISIX exposes public authorization metadata and key endpoints:
/.well-known/* -> AZN-SERVER
/oauth2/* -> AZN-SERVER
Protected API routes require scopes. The route script creates read, write, admin, test, and transfer routes for the CloudBank services. For the customer and account thread, the relevant pattern is:
APISIX uses the OpenID Connect plugin in bearer-only mode. The route configuration points to authorization server discovery metadata and preserves the bearer token for the backend service:
The gateway check is important, but it is only the first layer. The backend service still validates the token and enforces resource-specific rules. That is what keeps authorization close to the data.
The Helidon customer service uses MicroProfile JWT. Its configuration points JWT verification at the same JWKS endpoint that the rest of the system uses:
Inside the service, the resource can work with the authenticated principal and token claims using Helidon and MicroProfile APIs. A compact version of the scope-checking pattern looks like this:
The Spring Boot account service validates the same kind of token through Spring Security. CloudBank’s shared Spring configuration points the resource server at the same JWKS location:
The frameworks are different, but the contract is the same:
the issuer is the authorization server
the verification keys are published through JWKS
the token carries the principal and scopes
APISIX checks broad route-level authorization
each service checks resource-specific authorization
That is a much cleaner integration point than trying to share framework-specific security code across runtimes.
Routing: A Consistent Edge Without Flattening the Services
APISIX gives the mixed system one external API edge. That edge can expose consistent routes and enforce coarse-grained route scopes while leaving each service free to use its own controller model.
The route script creates upstreams by service identity and discovery type:
{
"uri":"/api/v1/customer*",
"upstream":{
"service_name":"CUSTOMER",
"type":"roundrobin",
"discovery_type":"eureka"
}
}
That route is framework-neutral. APISIX needs a URI, an upstream service name, a discovery mechanism, and plugins. It does not need to know about JAX-RS annotations, Spring MVC annotations, CDI beans, or Spring components.
This gives the system a useful split of responsibility:
APISIX controls the public surface area.
Eureka resolves service identities.
Spring Authorization Server defines the token contract.
Helidon and Spring Boot services enforce the rules close to their resources.
CloudBank also shows why that split matters. The route script blocks external access to account journal endpoints by creating a deny-style route for /api/v1/account/journal*. The account service can still have internal endpoints for service-to-service work, but the gateway does not accidentally publish them as part of the public API.
This is the kind of boundary that lets teams mix frameworks without creating an accidental tangle. The outside API is deliberate. Internal implementation remains local.
APIs: Share Behavior, Not Annotations
The customer and account services do not need common controller annotations to interoperate. They need common API behavior.
The Helidon customer service exposes its API with JAX-RS style resources:
@Path("/api/v1/customer")
@Authenticated
publicclassCustomerResource{
@GET
@Produces(MediaType.APPLICATION_JSON)
publicResponsegetCustomers(){
...
}
@POST
@Consumes(MediaType.APPLICATION_JSON)
publicResponsecreateCustomer(Customercustomer){
...
}
}
The Spring Boot account service exposes its API with Spring MVC:
Those programming models are different. The interoperability question lives one level up:
What does the authenticated principal represent?
Which scopes allow reads, writes, admin operations, tests, and transfers?
Which paths are public?
Which paths are internal?
Which service owns each business rule?
What JSON shapes and status codes do callers see?
Once those answers are stable, framework annotations become implementation details. A Helidon service can protect customer records using MicroProfile JWT and JAX-RS. A Spring Boot service can protect account records using Spring Security and Spring MVC. Clients and gateways interact with the API contract, not the controller framework.
This is also where integration tests should focus. A good mixed-framework test does not assert that both services are built the same way. It asserts that the same token identity produces consistent business behavior at both endpoints, that the same scope vocabulary is honored, and that callers see predictable HTTP results. That kind of test protects the contract without freezing either team inside the other team’s framework choices.
Observability: One Operational View
Interoperability is not complete if the services are only integrated at request time. They also need to be operable together.
CloudBank uses OBaaS OpenTelemetry support as the demonstration path. The Spring Boot account values enable telemetry:
otel:
enabled: true
The same values file identifies account as a Spring Boot workload:
obaas:
framework:"SPRING_BOOT"
The Helidon customer values identify the customer service as a Helidon workload:
obaas:
framework:"HELIDON"
The useful idea is platform-level instrumentation. Application teams should not have to invent a different tracing pipeline for each framework. With the OBaaS Java auto-instrumentation path, the platform can inject instrumentation and send traces, metrics, and logs toward the shared observability backend.
For day-two operations, this is a major part of the integration story. When a support engineer follows a request, the framework boundary should not become an observability boundary. A request to customer and a request to account should be visible in the same telemetry system, with service names, status codes, latency, and logs available from one operational view.
The SigNoz Services dashboard shows the same operational view from the telemetry side. In this run, the services list includes account, customer, and customer-helidon, with latency, error-rate, and operations-per-second columns coming from the shared OpenTelemetry pipeline:
SigNoz also receives trace data from the CloudBank request path. The deployment did not produce a single application trace containing both Spring Boot account and Helidon customer-helidon; the services are demonstrated here as separate participants in the same trace store. The first SigNoz capture shows a representative Helidon customer trace:
The second SigNoz capture shows a representative Spring Boot account trace from the same backend:
The same run validated the security path with live HTTP calls:
gateway_metadata=200
account_no_token=401
account_read_token=200
helidon_no_token=401
helidon_read_token=200
Those checks are intentionally contract-level checks. They do not prove that Helidon and Spring Boot use the same security implementation. They prove something more useful for interoperability: the same issuer and scope vocabulary can protect both service styles, and a scoped token can reach both the Spring Boot account API and the Helidon customer API.
Design Lessons Beyond CloudBank
CloudBank V5 is a convenient example, not a special rulebook. The same pattern applies to other systems that mix Helidon and Spring Boot.
First, give every service a stable identity. That identity should appear in discovery, route configuration, logs, traces, and dashboards. If the platform can identify the service consistently, the implementation framework can stay behind the boundary.
Second, standardize on a token contract. A shared issuer, JWKS endpoint, principal meaning, and scope vocabulary are more important than shared security code. Helidon can validate the token with MicroProfile JWT. Spring Boot can validate it with Spring Security. The architecture stays coherent because both services trust the same issuer and interpret the same claims.
Third, put public exposure at the gateway and data protection in the service. APISIX is a good place to define public routes and route-level scope requirements. It is not a substitute for service-side authorization. The backend still owns resource-level decisions.
Fourth, make telemetry a platform contract. Even when requests do not cross directly between Helidon and Spring Boot services, both frameworks should emit OpenTelemetry data into the same backend so operators can inspect each service from one place.
Finally, resist the urge to flatten the frameworks. Shared libraries can help inside a family of services, as CloudBank does with common Spring configuration for Spring Boot services. But Helidon and Spring Boot interoperability is stronger when the cross-framework contract is based on HTTP, JWTs, discovery, routing, and telemetry instead of a forced common programming model.
Closing
Helidon and Spring Boot can coexist cleanly when the architecture treats frameworks as local implementation choices and platform contracts as the shared language.
In CloudBank V5, the Helidon customer service and the Spring Boot account service collaborate through ordinary, durable boundaries: Eureka for discovery, Spring Authorization Server for JWTs and OpenID Connect metadata, APISIX for consistent routing and route scopes, and OpenTelemetry for a shared operational view.
That is the lesson I would carry into any mixed-framework system. Do not integrate frameworks by making them imitate each other. Integrate services by making the boundaries explicit.
Embabel lets Java developers express agent behavior as typed actions and goals.
The GOAP-style planner uses available objects and action signatures to find a path to the goal.
OCI Generative AI can be introduced through a starter without rewriting the agent’s action code.
Deterministic Java code can prepare trusted context before the model writes the language-heavy part of the response.
Observability matters because agent behavior is a sequence of decisions and calls; Jaeger gives you a concrete way to inspect that sequence.
I was lucky enough to see a presentation from Rod Johnson at a meetup in New York recently. Rod is the creator of Spring, and his new project, Embabel, is fascinating. It’s a different kind of take on building agentic AI applications. I am particularly attracted to its use of method signatures as a way to deterministically create a goal, and the ease with which you can mix deterministic Java code with probabalistic LLM reasoning. It plays nice in existing Spring applications, uses the same conventions as Spring, so it’s already familiar to you if you come from that ecosystem.
Embabel is a very interesting project – because it does not ask Java developers to choose between “real code” and “AI code.” It treats the LLM as one participant in a typed application flow. Ordinary Java methods still do the parts that should be deterministic. The model does the parts where language, synthesis, and judgment are useful. Embabel sits between those pieces and works out how to get from the input you have to the goal you asked for.
That is a useful place to be if you already live on the JVM. A lot of enterprise software is not waiting to be rewritten into a new stack just because LLMs arrived. It already has Spring services, domain objects, tests, configuration, and deployment habits. Embabel plugs into that world and gives agentic code a shape Java developers can reason about: methods, parameters, return types, goals, and tests.
In this article we will use a new OCI Generative AI starter for Embabel. The starter lets an Embabel application use OCI Generative AI models through the same model abstraction the rest of the framework expects. The demo is a small travel briefing agent, but the interesting part is the architecture. It puts three kinds of work into one planned flow:
Parse a user request into a typed Java record.
Gather deterministic local facts with ordinary Java code.
Ask OCI Generative AI to turn those facts into a typed briefing.
The point is not the travel domain. The point is the shape of the application: deterministic code and probabilistic model calls in one planned flow, with Java types describing what each step consumes and produces.
What Embabel Is Doing
Embabel models an agent around a few core concepts: actions, goals, conditions, domain objects, and plans. The developer writes action methods. Each action method advertises what it needs through its parameters and what it produces through its return type. A goal describes a useful final state. At runtime, Embabel can infer a plan from the available objects, the actions it knows about, and the goal it needs to reach.
That makes the Java method signature more than a convenience. It is part of the agent contract. The method signature says, in a way the framework and the compiler can both see, “if you can give me a TravelRequest, I can give you LocalFacts.”
In an Embabel application, a method signature is not just implementation detail. It is part of the map the planner uses to decide what can happen next.
"The Chicago Riverwalk is a good base for architecture walks.",
"The lakefront trail gives an easy outdoor reset between indoor stops.",
"River North and Fulton Market have reliable coffee options near transit."
)
);
}
This action does not call an LLM. It does not need to. It accepts a TravelRequest and returns LocalFacts. When a later action needs LocalFacts, Embabel has a way to produce them.
Now compare that with the action that writes the final answer:
@AchievesGoal(description="A concise travel briefing has been prepared")
@Action
publicTravelBriefingwriteBriefing(
TravelRequestrequest,
LocalFactsfacts,
Aiai
){
varprompt="""
Write a concise two-day travel briefing in Markdown.
Use the supplied local facts. Keep the plan practical.
Mention why the deterministic facts and the LLM-written narrative both matter.
This method is still typed. It still takes explicit inputs. It still returns a Java record. Inside the method, the Ai helper calls the configured LLM and asks it to create a TravelBriefing. The model has room to write, but the application still decides where the model is allowed to act and what shape the result must have.
Why Goals Matter
Embabel’s default planning approach is based on Goal-Oriented Action Planning. In practical terms, GOAP lets the framework ask: given the objects currently available, which action can move the process closer to a desired goal?
For this demo, the starting object is a UserInput. The goal is a TravelBriefing. Embabel can see that:
parseRequest(UserInput) can produce TravelRequest.
gatherLocalFacts(TravelRequest) can produce LocalFacts.
writeBriefing(TravelRequest, LocalFacts, Ai) can produce TravelBriefing.
So the path is straightforward. In a larger application, this same mechanism becomes more valuable. You can add actions without rewriting a giant orchestration method. If a new action produces a type that another goal needs, it becomes part of the planning vocabulary.
That is the key difference between “call an LLM from Java” and “build an agentic Java application.” The former is a client call. The latter is a system of typed capabilities.
There is another practical benefit here: the plan is inspectable. If an agent run surprises you, you can look at the objects that were present, the actions that were eligible, and the goal Embabel was trying to satisfy. That fits the way many Java teams already debug applications. You do not have to treat the whole system as one giant prompt. You can ask smaller questions. Did the parser produce the right domain object? Did the deterministic action add the facts the model needed? Did the LLM action receive the right prompt? Did the final object satisfy the goal?
That debugging posture is one reason this style works well for enterprise Java demos. It keeps the model important, but it does not make the model responsible for everything. The more important the application is, the more you want boring pieces around the surprising piece: typed inputs, explicit outputs, repeatable tests, configuration that can be reviewed, and traces that show what happened.
Adding OCI Generative AI
The OCI starter is intentionally small from the application developer’s point of view. The demo depends on the starter and imports the Embabel dependency BOM:
The local OCI GenAI starter autoconfiguration registers OCI-backed chat and embedding model beans. The model definitions live in oci-genai-models.yml; the application can then select a default model through Embabel configuration:
For local development, the OCI starter can use the normal OCI config file authentication path. The important design detail is that the agent code does not change when the model connection is configured. The action asks for the default LLM; the platform supplies the OCI-backed model.
That separation keeps the demo from becoming a provider-specific tangle. The agent action is written in terms of Embabel’s Ai abstraction and the target Java type. OCI Generative AI appears in configuration and model loading. The Java method still says, “given this request and these facts, produce a TravelBriefing.” That is a good boundary. Application code owns the domain. Platform configuration owns the model connection.
The same boundary also helps with testing. The integration test does not need to contact OCI Generative AI to prove that Embabel can discover the actions, build the plan, and pass deterministic facts into the LLM step. It stubs the model response and verifies the application behavior around the call. Local deterministic tests catch ordinary mistakes quickly: missing beans, wrong action signatures, a prompt that forgot the facts, or a goal that cannot be reached.
The Demo Agent
Here are the domain records:
publicrecordTravelRequest(
Stringdestination,
intdays,
List<String> interests,
StringoriginalText
){
}
publicrecordLocalFacts(
Stringdestination,
List<String> facts
){
}
publicrecordTravelBriefing(
Stringdestination,
StringtripStyle,
Stringmarkdown
){
}
These records are not ceremony. They are the backbone of the agent. They tell Embabel what exists in the process at each step, and they tell the LLM exactly what object it must produce when the flow reaches the probabilistic part of the application.
The first action turns a free-form request into a simple typed object:
This is deliberately deterministic. A real application might use a database lookup, a rules engine, a customer profile, or a known catalog. Here, it is just enough code to show the boundary: not every step belongs in the model.
The final action uses the LLM, but it does so with a typed result:
Temperature is low because this is a practical briefing. We want enough language ability to make the result readable, not a wildly creative itinerary.
Notice what is not happening in this design. The prompt is not being asked to rediscover the whole workflow. It does not decide whether local facts should be gathered. It does not choose the action order. It receives a constrained job at the point where language generation is useful. The rest of the workflow remains normal Java. That is the difference between an agent that is merely “LLM-shaped” and an agent that can live inside an application architecture.
That split also gives reviewers a cleaner way to reason about risk. Deterministic code can be reviewed like deterministic code. The LLM prompt can be reviewed as a bounded language task. The trace can show whether the handoff between those two worlds happened where expected.
Running It
Start Jaeger first:
docker compose up -d
The example application assume you have install the OCI CLI and set up a profile so you can authenticate to OCI. Assuming you have done that, run the application with your OCI compartment and region:
mvn spring-boot:run
-Dspring-boot.run.arguments="Plan a two day visit to Chicago for a Java developer who likes architecture, coffee, and the lake."
The console output includes the question-driven answer produced by the agent:
=== Embabel OCI GenAI travel briefing ===
Destination: Chicago
Trip style: Java developer
# Chicago Briefing: A Developer's Two-Day Itinerary
This plan balances structured sightseeing with time for discovery, focusing on architecture, coffee, and the lakefront. It's built on deterministic facts (the reliable APIs of your trip) and a narrative layer (the application logic that creates a seamless user experience).
### Day 1: River, Loop & Classic Structures
* **Morning:** Start your day in River North. Grab a high-quality coffee at a local spot like Intelligentsia to fuel your exploration. The area is well-connected by transit.
* **Mid-day:** Head to the Chicago Riverwalk for the Chicago Architecture Foundation Center's River Cruise. It's the most efficient way to see dozens of iconic buildings and understand the city's history.
* **Afternoon:** After the cruise, walk along the Riverwalk for different ground-level perspectives of the canyon-like urban core.
* **Late Afternoon:** Walk south into the Loop to see foundational skyscrapers up close, including the Rookery Building and the Monadnock Building.
### Day 2: Lakefront, Modern Design & Fulton Market
* **Morning:** Explore Fulton Market, a former industrial district now known for tech offices and a vibrant food scene. Start with coffee at a neighborhood roaster.
* **Mid-day:** Head east to the Lakefront Trail for an outdoor reset. Rent a Divvy bike or walk a segment north of Millennium Park for skyline and water views.
* **Afternoon:** Make your way to Millennium Park. See Cloud Gate and Frank Gehry's Pritzker Pavilion.
* **Evening:** Return to Fulton Market for dinner.
The test suite also validates the same typed agent flow with mocked LLM output. That is useful because it proves the planner can get from UserInput to TravelBriefing and that the prompt receives the deterministic facts before you spend any cloud tokens.
mvn test
A successful run shows Embabel planning and executing the three typed actions, then Maven reports a clean test result:
Open Jaeger at http://localhost:16686 and search for embabel-oci-travel-agent. The useful trace is not only “there was a request.” It shows whether the agent run, action spans, and LLM call line up with the flow you intended to demonstrate. That makes tracing a sanity check for the agent design, not just an operations checkbox.
In the trace, the most useful thing to inspect is the shape of the run. You want to see the deterministic actions before the model-backed action, and you want the LLM call to appear under the action that writes the briefing. If the trace shows the model call happening before the facts are gathered, the problem is not the model. It is the application flow.
Here’s a different view of the trace, called a “flame graph”:
OpenTelemetry tracing makes the Helidon AI request path inspectable without changing the assistant into a different application.
Jaeger gives us a local trace viewer for the demo; the instrumentation is application-created spans around the work we care about.
A stable recipe lookup is better than depending on a generated recipe id in a follow-on demo.
The trace proves the instrumented path ran in order. It does not prove that every model answer is correct.
In the first article in this follow-on series, the Helidon Eats app learned how to answer a recipe question with OpenAI, LangChain4j, Oracle AI Database vector search, and the same recipe data from the published Helidon Eats demo. In the second article, the assistant gained a memory model: working memory in JSON, semantic memory in vectors, episodic memory as events, procedural memory as rules, and a SQL property graph to connect the user, ingredients, and recipes.
That is enough behavior that a plain JSON response is no longer enough to explain what happened.
If the assistant says, “Try Tangy Rhubarb Salsa,” I want to know more than whether the final sentence sounds useful. Did the app embed the question? Did Oracle AI Database run the recipe vector search? Did the memory lookups run? Did the prompt get assembled after those lookups? Did LangChain4j call OpenAI chat? Which step took time?
That is what tracing is for.
Helidon SE can participate in OpenTelemetry, including configuration and APIs for tracing support, and the Helidon documentation describes its OpenTelemetry support as a preview feature in the Helidon 4.4.1 line Helidon OpenTelemetry docs. For this demo, I keep the instrumentation deliberately explicit. The application creates spans around the assistant path and exports them to Jaeger all-in-one over OTLP HTTP. Jaeger is the local viewer; the spans are created by our Helidon application.
Keep one repeatable request
Before adding tracing, I want a stable request path.
The request is the same one we have used throughout the demo:
GET /ask?q=what%20can%20I%20make%20with%20rhubarb
The answer should exercise the same pieces each time: question embedding, Oracle recipe vector search, working memory lookup, semantic memory vector search, procedural rule lookup, prompt build, and OpenAI chat.
There is a small but important database detail here. The recipe rows come from the already-published Helidon Eats article. The recipe ids are generated when the data is loaded. That means a hard-coded id is a poor anchor for a follow-on article. It may be correct in one validation database and wrong in another.
The demo now resolves the recipe from its stable content instead:
WITH target_recipe AS(
SELECT*
FROM recipe
WHERE recipe_title ='Tangy Rhubarb Salsa'
AND category ='Appetizers And Snacks'
AND subcategory ='Salsa'
FETCHFIRST1ROWONLY
)
SELECT recipe_id, recipe_title
FROM target_recipe;
That gives the rest of the demo a durable anchor. The generated id can vary, but the title/category/subcategory row is the one the published data set is meant to contain. The smoke test checks that the row exists exactly where the follow-on demo expects it.
This matters for observability because traces are easier to compare when the domain path is stable. OpenAI can still phrase an answer differently. That is fine. The application path should still be the same.
Add Jaeger to the local stack
The Docker Compose file keeps Oracle AI Database and adds Jaeger all-in-one:
services:
jaeger:
image: jaegertracing/all-in-one:1.76.0
environment:
COLLECTOR_OTLP_ENABLED:"true"
ports:
-"16686:16686"
-"4318:4318"
-"4317:4317"
oracle:
image: gvenzl/oracle-free:23.26.2-slim-faststart
ports:
-"15211:1521"
Jaeger documents the all-in-one image as a quick local way to run the collector and query UI together, with the UI on 16686 and OTLP ports on 4317 and 4318Jaeger getting started. That gives us a local trace viewer that is easy to start beside the database container. It keeps setup small while still giving every span a place to land.
The application points the OpenTelemetry exporter at the HTTP endpoint:
I am using OTLP HTTP here because it keeps the Java exporter configuration small. The Jaeger container also exposes the gRPC OTLP port if you prefer that path.
Create the tracer once
The app creates one small TracingSupport helper during startup. When OTEL_EXPORTER_OTLP_ENDPOINT is set, it builds an OpenTelemetry SDK tracer provider and an OTLP HTTP span exporter. When the variable is not set, it returns a no-op tracer.
For a tutorial app, SimpleSpanProcessor is easy to reason about. Each finished span is exported immediately. For a production service, I would usually use batching and a collector strategy, but that is not the point here.
The helper exposes two operations the rest of the app uses:
Spanspan(Stringname,SpanKindkind){
returntracer.spanBuilder(name)
.setSpanKind(kind)
.startSpan();
}
SpaninternalSpan(Stringname){
returnspan(name,SpanKind.INTERNAL);
}
That is intentionally simplistic. The useful part is not the helper. The useful part is deciding which work units deserve spans.
Then the interesting work gets its own child spans:
openai.embedding.question
oracle.recipe.vector_search
oracle.memory.working_lookup
oracle.memory.semantic_vector_search
oracle.memory.procedural_lookup
assistant.prompt.build
openai.chat.completion
This is the part that makes the trace readable. The spans are named for application work, not for implementation trivia. A reader can open the trace and follow the request path: embed the question, retrieve recipes, read memory, build the prompt, call OpenAI.
The Oracle vector search span records small, safe attributes:
That is enough to show that the vector search ran and returned Tangy Rhubarb Salsa as the first hit. It does not put the full recipe text in telemetry.
OpenTelemetry’s generative AI semantic conventions are useful, but they are marked as development, so I keep the mapping small and easy to change OpenTelemetry GenAI spans. I also avoid recording full prompts and full responses in spans. For this demo, counts, model names, selected recipe title, and memory hit counts are enough.
Bridge LangChain4j events into the trace
LangChain4j already gives us listener hooks for selected chat model implementations. The observability documentation describes ChatModelListener callbacks for request, response, and error events LangChain4j observability docs.
The demo keeps the existing listener, but now it also writes events onto the current OpenTelemetry span:
This does not mean LangChain4j magically traces the whole application. It means the model listener gives the application a clean place to attach chat model events to the openai.chat.completion span.
That distinction is worth keeping. Observability is better when it is honest about the boundary. Helidon serves the route. The app creates spans around the AI work. LangChain4j gives model hooks. Oracle AI Database performs SQL lookups and vector search. OpenAI handles embedding and chat calls.
Run the traced request
Start the containers:
docker compose up -d oracle jaeger
The Oracle container mounts startup/ as /container-entrypoint-startdb.d, so the database setup runs on each container start. The startup script loads the predecessor Helidon Eats recipe data when the food.recipe table does not already contain the Tangy Rhubarb Salsa anchor, applies the additive AI schema, and runs the smoke checks.
docker compose exec oracle
sqlplus -s food/Welcome12345##@FREEPDB1
@/work/sql/20-smoke-checks.sql
That smoke check should show one Tangy Rhubarb Salsa anchor, 500 recipe chunks, eight semantic memories, eight episodic events, four procedural rules, and the SQL property graph:
The answer can vary in wording, but the response should include the selected memory values and a recipe answer grounded in Tangy Rhubarb Salsa.
Then open Jaeger at:
http://localhost:16686
Search for the helidon-eats-ai service and open the GET /ask trace.
The trace summary above comes from the captured Jaeger trace. It shows the request-time path: the route span, the assistant span, the question embedding, Oracle vector search, three memory lookups, prompt build, and OpenAI chat call.
Read the trace as evidence
The trace answers a different question than the final JSON response.
The final response tells us what the assistant said. The trace tells us how the application got there.
In the captured request, the trace shows:
GET /ask
assistant.answer
openai.embedding.question
oracle.recipe.vector_search
oracle.memory.working_lookup
oracle.memory.semantic_vector_search
oracle.memory.procedural_lookup
assistant.prompt.build
openai.chat.completion
The Oracle vector span records two recipe hits and names Tangy Rhubarb Salsa as the first hit. The semantic memory span records one selected semantic memory. The working and procedural memory spans record that those values were present. The prompt-build span records the prompt length, not the prompt body. The OpenAI chat span records the model and response length, and it includes LangChain4j request and response events.
That is a useful level of evidence. It proves the instrumented path ran, in order, for that request. It proves the app did not skip memory retrieval before calling chat. It proves Oracle vector search ran before prompt assembly. It proves the model call happened after the grounded context was selected.
It does not prove the answer is always correct. It does not prove retrieval quality for every question. It does not prove anything about uninstrumented code. It does not give server-side timing inside OpenAI or Oracle. It gives application-side evidence for the spans we created.
That is still a big improvement over guessing.
Validate with the API too
The Jaeger UI is the nicest way to inspect the trace, but the API is useful for validation. A dashboard screenshot can show that a human saw the trace. A small API check can prove that the expected spans are present.
For the stable request, I like checking the service list first:
That returns the trace data as JSON. The fields are a little verbose, but they are deterministic enough for a smoke check. The request trace should contain these operation names:
GET /ask
assistant.answer
openai.embedding.question
oracle.recipe.vector_search
oracle.memory.working_lookup
oracle.memory.semantic_vector_search
oracle.memory.procedural_lookup
assistant.prompt.build
openai.chat.completion
That check is not glamorous, but it is a useful habit. It keeps the trace claim grounded. If the trace is missing oracle.memory.semantic_vector_search, then the request did not prove semantic-memory retrieval. If the trace is missing openai.chat.completion, then the app might have returned a setup response, failed before the model call, or hit a different path. The API check makes those cases visible.
It also separates two kinds of evidence.
The SQL smoke check proves the database state: 500 chunks, eight semantic memories, eight episodic events, four procedural rules, one working-memory row, and a graph edge from Tangy Rhubarb Salsa to rhubarb.
The /ask response proves the live route can call OpenAI and return an answer with selected memory.
The Jaeger trace proves the instrumented request path: embedding, vector search, memory lookup, prompt build, and chat completion.
Each one catches a different class of mistake.
Together, they make the demo much easier to trust.
Those are three different checks, and I want all three. When a demo combines AI, database retrieval, and memory, a single “it returned JSON” check is too thin.
Choose attributes carefully
Span attributes are where observability can become either very helpful or very messy.
The recipe vector search span records recipe.hit.count and recipe.first_hit.title. That is enough to confirm that the query returned two chunks and that the first one was Tangy Rhubarb Salsa. It does not record the entire chunk text. The semantic memory span records memory.semantic.hit.count, not the full memory document. The prompt-build span records prompt.length, not the prompt body.
Those choices are deliberate.
The most useful attributes are the ones that explain application decisions without turning telemetry into another data store. If the assistant starts returning odd answers, recipe.hit.count=0 is a strong clue. If memory.working.present=false, the request did not have the planning goal we expected. If prompt.length suddenly jumps from a couple thousand characters to tens of thousands, the prompt builder probably started including too much context.
Those are debugging signals. They are not secrets, and they are not the whole user conversation.
The same idea applies to the OpenAI spans. Recording gen_ai.system=openai, gen_ai.request.model=gpt-4o-mini, and embedding.vector.dimension=1536 is useful. Recording the API key would be a disaster. Recording the full prompt might be acceptable only in a local experiment with deliberate redaction. The default demo does not do that.
This is also why I prefer application-created spans for the teaching version. Automatic instrumentation is valuable, but explicit spans force us to name the application decisions we care about. For this assistant, those decisions are not hidden in the network stack. They are the recipe retrieval, memory selection, prompt construction, and model call.
Keep the trace useful
The most tempting mistake is to turn tracing into another place to dump everything. That usually makes traces noisier and less useful.
For this assistant, I would keep the default telemetry small:
model name
vector dimension
recipe hit count
first recipe title
semantic memory hit count
prompt length
response length
route name
tenant/session identifiers that are safe for the demo
I would not put API keys, full prompts, full recipe context, or complete model responses into spans. If a team needs prompt capture for a local experiment, make it explicit, temporary, and redacted.
The stable recipe lookup also remains useful as the assistant grows. If you change the retrieval limit, memory filter, prompt template, or model, run the same rhubarb request and compare the trace. The exact answer text may move around, but the application path should still be understandable.
That gives us a nice close to this part of the series.
Article 1 grounded the assistant in Oracle AI Database vector search. Article 2 gave it memory. Article 3 makes the request path visible. The assistant is still small, but it now has the pieces I want before adding more ambitious agent behavior: trusted data, useful memory, and a trace that shows how the answer was assembled.
1,364,617 people have been kind enough to visit our humble blog. Others get our posts by RSS or email or through syndicators. We hope you took away something of value. Please come again!
Copyright 2009-2026 Mark Nelson and other contributors. All Rights Reserved. The views expressed in this blog are our own and do not necessarily reflect the views of Oracle Corporation. All content is provided on an ‘as is’ basis, without warranties or conditions of any kind, either express or implied, including, without limitation, any warranties or conditions of title, non-infringement, merchantability, or fitness for a particular purpose. You are solely responsible for determining the appropriateness of using or redistributing and assume any risks.
You must be logged in to post a comment.