RedStack

Key Takeaways

• This article defines what Java means for this benchmark: a Helidon SE service running on explicitly identified Oracle-built JDK variants, with a declared request-handling model, JVM policy, diagnostics, and manifest evidence.
• Virtual threads are treated as a request-handling model, not a performance claim. A later benchmark must prove which model was active and measure whether this workload benefits from it.
• Runtime labels require exact build evidence. A later result is not accepted if it only says “Java” or “Java 21” without provider, version, build, architecture, flags, and environment details.
• This article defines the Java contract only. The runnable benchmark package and benchmark results come later, after the series defines the full service contract, load generation, execution method, and analysis method.

You can make “Java” mean almost anything in a benchmark.

It might mean a plain JDK HTTP server, Spring Boot, Quarkus, Micronaut, Vert.x, Netty, or Helidon. It might mean platform threads, virtual threads, a reactive handler model, a native executable, a checkpoint/restore workflow, or an ordinary JVM run with JIT warmup.

It might mean default heap ergonomics, fixed heap settings, one garbage collector, another garbage collector, request logging on, TLS on, or a tuned command line nobody wrote down.

If a later chart labels one bar Java, that chart is already too vague.

This article defines the Java side before we measure it. For this benchmark, the Java service shape is Helidon SE. Measured Java evidence in this series is limited to explicitly identified Oracle-built JDK variants. The request-handling model must be declared. Startup and warmup boundaries must be separated. JVM tuning must be framed as a hypothesis. Diagnostics must be captured before results are accepted.

No benchmark result is published here. No winner is declared here. We are defining the Java/Helidon contract that later source code and benchmark runs must satisfy.

Where This Article Fits in the Series

This is Part 3 of a ten-part, single-machine benchmark series. Part 1 set the rule: method first, measurement later. A run is not useful because it completed; it is useful only if the service behavior, runtime identity, environment, load generator, run protocol, and analysis method are all traceable.

Part 2 defines the language-neutral service behavior. Both implementations expose the same health and work endpoints, follow the same request and response contract, and use the same posture for logging, TLS, compression, and related service settings in the primary pure-service benchmark.

Those requirements are not Java requirements or Go requirements. They are the HTTP contract both services must satisfy before either one is tuned or measured.

This article binds that neutral service contract to the Java implementation shape. It answers a practical question: when a later article says Java/Helidon, what exactly does that mean?

Part 4 applies the same discipline to the Go side. It defines the Go net/http implementation shape, runtime settings, profiling plan, and extraWork equivalence review. Parts 5 through 8 define load generation, execution, reproduction, and analysis. Results do not arrive until Part 9, with final interpretation in Part 10.

That ordering matters. If we tune Java first and define the Java contract afterward, the reader has to wonder whether the contract was shaped around the result. If we define the contract now, before source generation and before measurement, later Java results become auditable.

Later results must be generated fresh from the validated contract and preserved with the artifacts needed to audit them.

What “Java” Means in This Benchmark

For this benchmark, “Java” means a Helidon SE service that implements the finalized Article 02 endpoint contract exactly. The later implementation should make the Helidon SE WebServer routing and server configuration reviewable rather than relying on undocumented defaults.

That sentence intentionally excludes a lot. It does not say all Java services. It does not say all Java frameworks. It does not say Helidon represents Spring Boot, Quarkus, Micronaut, Vert.x, Netty, or a plain JDK server. Those are valid production choices, but they are outside the Java service shape selected for this benchmark.

Helidon SE is selected because its programmatic style can make routing and server configuration explicit in the benchmark source package. That helps reviewability for this particular contract. We want a reader to inspect the later source package and see the choices that matter, not infer them from a larger application stack.

The runnable benchmark must record the released Helidon SE version and resolved dependency set actually used for each run.

A later accepted Java run must identify the runtime shape in a manifest. This YAML is only an illustrative sketch, not the final manifest schema and not evidence that this article builds a service:

java_runtime:
  implementation_id: "java-helidon-se"
  framework: "Helidon SE"
  manifest_must_record:
    framework:
      - helidon_version
      - resolved_dependency_set
    jdk:
      - provider
      - version_output_file
      - runtime_properties_file
      - system_settings_file
      - effective_flags_file
      - architecture
    request_handling:
      - selected_threading_model
      - evidence_that_model_was_active
    execution_policy:
      - heap_policy
      - gc_policy
      - startup_boundary
      - warmup_boundary
      - measurement_boundary
    diagnostics:
      - gc_log
      - jfr_recording_when_enabled
      - process_metrics
      - request_metrics
    artifacts:
      - raw_result_files
      - summary_files
      - manifest_links

The important point is not the field names. The important point is the chain of evidence. A later Java result must tie together the framework, dependency set, JDK build, request-handling model, JVM flags, heap and GC posture, diagnostics, warmup window, measurement window, and raw artifacts.

The Java side becomes auditable only when the framework, JDK identity, request-handling model, JVM policy, diagnostics, and manifest are treated as one contract.

Java Runtime-Shape Map

The Java benchmark result is not valid until the Helidon SE service shape, JDK variant, request-handling model, JVM flags, diagnostics, and manifest evidence are captured from the validated implementation.

Figure 1. We define the Java runtime shape before measurement: Article 2’s service contract flows into Helidon SE, an exact JDK variant, a declared request-handling model, JVM policy, diagnostics, and a manifest before results are allowed.

The diagram is a reminder that the benchmark does not start at the load generator. It starts with the service contract. The Java implementation must satisfy that contract, record its runtime identity, and capture enough diagnostics to explain later measurements.

The Java Service Contract

The Helidon SE service must implement the finalized Article 2 language-neutral contract without adding framework-specific drift.

That means the Java service must expose the Article 2 endpoints and behavior exactly, and the later Go service must be held to the same finalized contract. The Java service must use the same request body limit, accept the same JSON media types, reject the same invalid request shapes, enforce the same numeric bounds, reject unknown and duplicate top-level request fields, return the same success fields, and return the same JSON error shape for body-bearing failures.

The primary benchmark posture also stays the same: request logging off, TLS off, compression off, and tracing off unless a later secondary slice explicitly enables one of them for both implementations. A Helidon default cannot become a hidden benchmark input.

This is where framework details matter. JSON parsing can differ in duplicate-field handling, numeric coercion, unknown-field policy, and error reporting. If the selected JSON library does not reject duplicate top-level fields by default, the Java implementation must add an explicit parsing or validation step that does. Routing can differ in how unsupported methods are handled. Server defaults can affect request size limits, connection behavior, response headers, exception mapping, and log output.

None of those details necessarily makes one framework better or worse. Hidden differences simply make results harder to trust.

The later Java source package must make these choices explicit enough for conformance tests to fail when the service drifts. A framework-default HTML error page is not a small cosmetic issue in this benchmark; it is a contract failure. Accepting "payloadSize": "1024" as a string when the contract requires an integer JSON number is also a contract failure. Returning an extra diagnostic field in /health is a contract failure.

Benchmarks often rush toward tuning. Here, correctness and equivalence come first. A fast service that answers a different question is not evidence.

Runtime Variants and Manifest Policy

Runtime labels are not vague labels in this series. They are benchmark inputs that must be backed by exact evidence.

For the measured Java evidence in this comparison, accepted Java runtime variants must be built by Oracle and labeled precisely. Non-Oracle JDK distributions are outside this measured comparison unless a later article explicitly defines a new scope and records them as separate, verified evidence. A result is not accepted unless the run records the exact provider, version, build string, architecture, effective JVM flags, operating system, and available processors as seen by the JVM.

These command shapes show the kind of runtime evidence the benchmark package needs to preserve:

mkdir -p artifacts
java -version > artifacts/java-version.txt 2>&1
java 
  -XshowSettings:properties 
  -version > artifacts/java-runtime-properties.txt 2>&1
java 
  -XshowSettings:system 
  -version > artifacts/java-system-settings.txt 2>&1
java 
  -XX:+PrintFlagsFinal 
  -version > artifacts/jvm-flags.txt 2>&1

Those files preserve the basic runtime identity. artifacts/java-version.txt captures the version string, runtime build, and VM name. artifacts/java-runtime-properties.txt captures properties such as java.vendor, java.version, java.runtime.version, java.vm.name, os.name, and os.arch. artifacts/java-system-settings.txt captures system and container-related settings exposed by the selected JDK. artifacts/jvm-flags.txt captures effective JVM flags, including defaults and explicitly supplied options.

These commands do not prove benchmark correctness. They only show the kind of runtime evidence the later benchmark must preserve. -XshowSettings:system should be validated for each selected JDK and Linux environment because the exact system and container fields exposed by the launcher can vary.

If the benchmark runs inside a constrained environment, the manifest must also record CPU and memory limits visible to the JVM. JVM ergonomics use available processor and memory information. If the service sees two processors but the host has many more, that is not a footnote; it is part of the benchmark input.

The series will not infer a JDK advantage from branding. Runtime variants must run the same service contract, under the same run protocol, with comparable diagnostics, before any comparison is allowed. If a later result differs, the first question will be: what exact runtime, flags, machine state, and artifacts produced that difference?

Virtual Threads Are a Request-Handling Model, Not a Result Claim

Virtual threads matter for the Java side because they can change how the service handles concurrent requests. JEP 444 defines virtual threads as a final feature in JDK 21 and describes them as lightweight threads managed by the JDK rather than the operating system.

That does not make virtual threads a magic performance switch.

Virtual threads are especially relevant when a service uses a thread-per-task style and spends time blocked on I/O. They can make blocking-style code easier to write and scale for suitable workloads.

But they do not make CPU-bound work run faster. If the handler is mostly JSON parsing, deterministic payload generation, checksum work, allocation pressure, or CPU-bound simulation, virtual threads may not improve throughput. They may still affect scheduling behavior, memory pressure, thread diagnostics, and latency distribution.

The contract treats virtual threads as a declared request-handling model. A platform-thread versus virtual-thread comparison is allowed only if the selected Helidon SE release documents both modes in a way that keeps the framework version, endpoint code, server architecture, JVM flags, and other benchmark inputs constant. Otherwise, the benchmark should declare the supported Helidon request-handling model and omit the artificial comparison.

The important rule is isolation: if platform-thread and virtual-thread variants are compared, the benchmark must prove that only the intended request-handling variable changed.

Here is the contract idea as a small sketch, not a Helidon configuration file:

request_handling:
  selected_model: "declared for the run"
  comparison_model: "included only when supported and isolated"
  result_claim_allowed_from_configuration_alone: false
  required_evidence:
    - selected_helidon_configuration
    - jdk_version_and_flags
    - threading_model_declaration
    - representative_thread_diagnostics_when_enabled

Diagnostics matter here. Blocking behavior, synchronized regions, native calls, and pinned virtual threads can affect scalability. Pinning is not automatically a correctness bug, but it can explain why a virtual-thread service behaves differently from the mental model. Representative Java diagnostic runs should capture enough evidence to inspect thread behavior when the methodology calls for it.

So the benchmark will not say “virtual threads are faster.” It will ask a narrower question: under the Article 02 workload, with the selected Helidon SE service and selected JDK build, does this request-handling model change throughput, latency distribution, CPU use, memory use, warmup behavior, or diagnosability?

That question is measurable. The slogan is not.

Startup and Warmup Boundaries

The baseline Java execution model is an ordinary JVM run with a defined warmup boundary and a defined measurement boundary.

That sounds boring, and boring is good here. A JIT runtime needs warmup time. The service may initialize framework structures, compile hot paths, settle allocation behavior, and reach a different steady-state profile than it showed in the first few requests. Later methodology articles define the warmup interval, measurement interval, repeat count, invalid-run rules, and analysis rules.

Any Project Leyden-related startup or warmup path is handled separately and only if the selected JDK exposes a supported, reproducible workflow for the benchmark. Project Leyden focuses on startup time, time to peak performance, and footprint. If the selected JDK provides a supported Leyden-related workflow, such as ahead-of-time class loading and linking for that JDK, the benchmark may include it as a separately labeled startup and warmup variant with recorded commands, generated artifacts, checksums, and training input. If it does not, the variant is omitted.

The contract is simple:

startup_warmup_policy:
  baseline:
    mode: "ordinary-jvm"
    included: true
    warmup_boundary: "defined by later benchmark methodology"
  leyden_related_variant:
    included: "only when supported by the selected JDK"
    required_evidence:
      - jdk_version_and_build
      - exact_record_create_run_commands
      - generated_artifact_filenames
      - generated_artifact_checksums
      - training_input_description
      - regeneration_rule_per_application_build
  graalvm_native_image:
    included_in_this_jvm_comparison: false
    note: "Separate execution model; not silently mixed into JVM results."

This separation prevents a common confusion. Leyden-related HotSpot and JDK work is not the same thing as GraalVM Native Image. GraalVM Native Image compiles a Java application ahead of time into a native executable. That changes the build and runtime model. It may be worth measuring in a separate slice, but it is not silently part of the ordinary JVM comparison in this article.

Checkpoint and restore workflows belong in the same category: separate execution model unless explicitly defined. They should not slip into the primary Java/Helidon comparison just because they affect startup or warmup.

If a later article includes a supported Leyden-related path, the result must identify the generated artifacts, training traffic, checksums, commands, and regeneration rules. Startup features without artifact discipline quickly become unrepeatable anecdotes.

JVM Tuning Hypotheses

JVM tuning in this series starts as a hypothesis, not a conclusion.

A fixed heap may make memory behavior easier to compare, but it can increase GC frequency under some workloads. Ergonomic heap sizing may better reflect default production behavior, but it can make memory comparisons harder if the runtime adapts differently under constraints. A pause-time target can influence latency behavior, but it is a goal, not a hard guarantee. A different collector may help some workloads and hurt others. A setting that improves one percentile can move cost into CPU, allocation rate, RSS, warmup, or throughput.

So the primary Java baseline should be intentionally plain and reproducible before variants are added. A reasonable Java run profile might fix the selected JDK build, fix the selected Helidon SE version, declare heap policy, use a documented baseline collector such as G1 when selected, enable GC logging when methodology requires it, and capture enough runtime evidence to explain what happened.

This illustrative launch shape shows diagnostic intent. It is not a final benchmark runner, and target/java-service.jar is a placeholder for a later built service artifact rather than an artifact produced by this article:

mkdir -p logs artifacts
java 
  -Xms512m 
  -Xmx512m 
  -XX:+UseG1GC 
  -Xlog:gc*,safepoint:file=logs/gc.log:time,uptime,level,tags 
  -XX:StartFlightRecording=filename=logs/run.jfr,settings=profile,dumponexit=true 
  -jar target/java-service.jar

If used later against a real service artifact, this style of run would produce a GC log and a Java Flight Recorder recording for diagnostic analysis. That does not make the command optimized. It only makes the run easier to inspect.

The later methodology decides which diagnostics are enabled during primary measurement and which are enabled only for representative diagnostic runs. Java Flight Recorder and profilers are useful, but they are not free. The benchmark has to define overhead policy rather than pretending diagnostics have no effect.

JVM tuning also needs to stay connected to the workload. An application service may spend time in HTTP handling, JSON serialization, validation, request shaping, vector-search calls, embedding calls, connection pools, or model gateways. The primary benchmark in this series isolates a pure service path first. If a later dependency slice adds a database call, the Java tuning question changes because blocking I/O, driver behavior, connection pooling, and database latency enter the picture.

That is another reason not to overfit JVM flags before the benchmark contract is complete.

Diagnostics Required Before Results

Diagnostics are not extra paperwork. They are how we avoid guessing.

If a later chart shows a latency spike, we need to know whether the service was CPU-saturated, allocating heavily, collecting frequently, warming up, blocked, pinned, hitting a request limit, logging unexpectedly, or being distorted by the load generator.

If a throughput difference appears between two runtime variants, we need to know whether the same flags, heap policy, and request-handling model were actually used. If memory differs, we need heap and RSS context, not a single number detached from GC behavior.

Every accepted Java run needs enough evidence to connect the observed HTTP behavior to the runtime state that produced it. That evidence includes runtime identity, framework identity, endpoint conformance, JVM behavior, process metrics, request metrics, and manifest links back to the raw artifacts.

Runtime identity means java -version, runtime properties, system settings, effective JVM flags, operating system, architecture, and available processors as seen by the JVM. Framework identity means the Helidon SE version, dependency set, and source revision used to build the service. Endpoint conformance means the Java service passed the Article 02 contract tests before measurement. JVM behavior means heap policy and GC evidence captured under the methodology’s diagnostics policy. Process and request metrics must cover the warmup and measurement windows, not just a single end-of-run snapshot.

Representative diagnostic runs can add more detail. Java Flight Recorder is especially useful on selected JDK builds where it is available and enabled for the benchmark methodology, because it can expose JVM events relevant to allocation, GC, I/O, threads, and runtime behavior. JDK Mission Control can later inspect Java Flight Recorder recordings when diagnostic analysis is needed.

async-profiler can add sampling views for CPU and allocation behavior, but it is a separate tool with platform and permission requirements. It should be used deliberately, not assumed to be present everywhere.

The goal is not to make every benchmark run heavy. The goal is to make every accepted result explainable. Primary measurement runs and representative diagnostic runs can have different profiles, as long as the methodology records which artifacts belong to which purpose.

The later load-generator article must also define whether load is open-loop or closed-loop, how client saturation is detected, how timing is recorded, how errors are counted, and how coordinated-omission risk is handled before any latency comparison is accepted. Java diagnostics can explain the server side only if the client side is controlled well enough to trust the measurements.

Acceptance Criteria for the Later Java Implementation

This article does not generate a Helidon project, build a JAR, run a load generator, or produce benchmark results. It defines the Java side of the contract. The runnable benchmark package is deferred until the series defines the full service contract, implementation plans, load-generator validation, execution method, reproduction requirements, and analysis method.

A later Java result is not accepted unless the Java implementation and run artifacts satisfy these gates:

• The service implements the finalized Article 02 endpoint contract exactly, including JSON request rules, success responses, error responses, media-type handling, request body limit, logging posture, and TLS posture.
• The service uses Helidon SE, and the manifest records the exact framework version and resolved dependency set used in the run.
• The run records the exact JDK provider, version, build, architecture, runtime properties, system settings, effective JVM flags, heap policy, GC policy, available processors visible to the JVM, and any CPU or memory limits imposed by the execution environment.
• The measured Java runtime variants are explicitly identified Oracle-built JDK variants unless a later scope change defines and validates additional runtime evidence separately.
• The request-handling model is declared as platform threads, virtual threads, or another explicitly documented Helidon- and JDK-supported model, with evidence that the intended model was active.
• Platform-thread and virtual-thread variants are included only when the selected Helidon SE release documents both modes in a way that allows the request-handling model to change without unrelated framework, endpoint, server-architecture, or JVM-policy changes.
• Ordinary JVM execution, Leyden-related startup and warmup behavior, GraalVM Native Image, and checkpoint and restore workflows remain separate execution models unless a later article defines a distinct variant.
• The run captures GC logs when required by methodology, process metrics, request metrics, warmup and measurement timestamps, and manifest links to artifacts.
• Representative diagnostic runs include Java Flight Recorder where the methodology enables it, and selected profiler slices where platform support and permissions allow them.
• The result does not compare against Go until the Go contract, load-generator contract, execution protocol, and analysis method are complete.

The smallest evidence-capture shape looks like this:

mkdir -p artifacts logs
java -version > artifacts/java-version.txt 2>&1
java 
  -XX:+PrintFlagsFinal 
  -version > artifacts/jvm-flags.txt 2>&1

Passing those commands is not enough to validate a benchmark. It only demonstrates the posture: runtime identity and effective flags are captured before results are reviewed.

If a later Java run is missing runtime identity, framework identity, request-handling evidence, warmup boundaries, diagnostics required by the methodology, or endpoint conformance, the run is not surprising. It is invalid.

Handoff to the Go Contract

The Java side is now specified enough to constrain later implementation work.

Java means Helidon SE. Runtime labels require exact JDK evidence. Virtual threads are a request-handling model. Leyden-related behavior is conditional and separate. JVM tuning is a hypothesis. Diagnostics and manifests gate results.

The next article applies the same discipline to Go.

That matters because Go has its own ways to drift. A compact net/http service can be easier to inspect, but it can still allocate differently, parse differently, validate differently, return different errors, or perform a different amount of work in the extraWork path. The Go implementation and tuning plan must be held against the same Article 2 service contract.

Java is specified here. Go is specified next. Results come later.

Conclusion: Make Java Auditable Before Measuring It

A defensible Go-versus-Java benchmark cannot leave Java undefined.

For this series, Java is a Helidon SE implementation of the Article 2 service contract, run on explicitly identified JDK variants, with a declared request-handling model, documented startup and warmup boundaries, testable JVM tuning hypotheses, required diagnostics, and manifest-backed evidence.

That does not make Java faster or slower. It makes the later Java result reviewable.

If you are building application services that depend on databases, model gateways, vector search, or other external systems, that is the useful habit to carry forward. Before comparing runtimes, define the service boundary. Before tuning, prove equivalence. Before publishing charts, capture the runtime evidence. Then, when results arrive, you can argue about the data instead of arguing about what was measured.

References and Methodology Notes

• Helidon documentation and Helidon SE 4 introduction for the selected Java framework shape and Helidon SE concepts.
• JEP 444: Virtual Threads for virtual-thread semantics and the important caveat that virtual threads are not faster threads.
• Java SE documentation for JDK tools, launcher options, Java Flight Recorder, and JVM behavior.
• Oracle JDK downloads for runtime distribution and build selection at benchmark time.
• Project Leyden and JEP 483: Ahead-of-Time Class Loading & Linking for Leyden-related startup and warmup context.
• GraalVM Native Image documentation for the separate native-executable model that is not silently included in ordinary JVM results.
• Java Flight Recorder API documentation and JDK Mission Control for diagnostic recording and analysis.
• async-profiler for optional representative CPU, allocation, and wall-clock profiling slices after platform compatibility is validated.

Recently, I was lucky enough to be invited to give a short presentation on blended teams, and I wanted to share some of my thoughts with you, dear reader!

What is a “blended team”? I am defining it as a team that has both human and AI workers who are collaborating on the same goals and tasks.

As you know, AI is good at some things, and not so good at others. You could say “when it’s good, it’s very, very good; but when it’s bad, it’s very, very bad” (wink to Henry Wadsworth Longfellow).

Vaccaro et al., Nature Human Behaviour, 2024 · When combinations of humans and AI are useful: A systematic review and meta-analysisSystematic review and meta-analysis synthesized 106 experiments and found that human – AI combinations underperformed the stronger standalone performer on average. The combination wins only when the task, evidence standard, reviewer, and escalation point are explicit.

The diagrams below, from arXiv:2507.09089v2 [cs.AI] Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity – Joel Becker, Nate Rush, Elizabeth Barnes, David Rein, present an interesting finding from the domain of software development: on average, experts tend to dramatically over-estimate how much AI assistance will speed up development. This study found that the estimates ranged from a 10% to 50% reduction in time – but the actual observed results were that tasks took up to 40% longer when AI assistants were used.

While developers estimated that AI assistance would enable them to reduce the time required for a task, as shown in the diagram below from the same study, the result was that they in fact took significantly longer when using AI.

Why is this the case? What’s going on here?

I’d like to offer the hypothesis that AI (and Generative AI in particular) is very good at doing things that have been done before, but much worse at doing something novel. This makes a lot of sense (to me, at least) if you consider that these models have been trained on very large corpora of existing code, and that they work by probabalistic prediction of the next token. So a model that has seen many examples of how to implement a particular type of requirement is likely to do a very good job of implementing that requirement. But, if you ask a model to implement something new, that it has not seen any, or many, examples of – it’s not going to do as good a job on that.

Another important thing to consider is the quality of the code the models were trained on. Charles Babbage called it way back in 1820 – garbage in, garbage out. And every developer knows, this is still very relevant today. If you consider that most of the code that the models were trained on is not what we would consider “good” – then you might not be surprised that the code they generate is also not “good”. Here’s some charts I pulled out of a paper showing distributions of some measurements of goodness across a wide sample of open-source code. The whole paper is very interesting and well worth a read.

Jin, Siyuan; Zhang, Mianmian; Guo, Yekai; He, Yuejiang; Li, Ziyuan; Chen, Bichao; Zhu, Bing; and Xia, Yong. “Software Code Quality Measurement: Implications from Metric Distributions.” 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS), 2023, pp. 488–496. https://doi.org/10.1109/QRS60937.2023.00054

Consider another example – AI assisted detection of security vulnerabilities. Models like Mythos and Fable can detect many more vulnerabilities that we ever did before them using human review and our pre-existing tools. But on analysis, we find that a lot of those issues detected (at least in my experience) were poor quality, false positives, or just duplicates of issues that we found with our existing tools. An enormous amount of human work is needed to review the output of these models and separate the signal from the noise.

With the widespread use of AI in this area, there also came a massive amount of work for software developers to review, categorize and address these issues. We’ve seen a spike in CVEs since the availability of these models, as shown in the diagram below, created from public CVE data reported by cve.org – note that this data set is up to end of June 2026. No one really knows if that trend will continue upwards – my educated guess is that all of the activity by security researchers and developers will result in a lot of issues being fixed, and we will possibly see that level off and start to drop again, as the industry as a whole gets more focused on fixing issues, and adapts our development, testing and release processes to catch more issues before shipping code.

There’s a lot of talk about AI replacing jobs, but I think it is much better to think about tasks that AI can do, not whole jobs. We all know that the job description rarely, if ever, actually describes the real job. There are a lot of tasks that people do that don’t make it into the job description. I don’t like to think about using AI to replace a person’s job, but instead to think about which individual tasks we could use AI for – tasks where we would get an improvement (in time, quality, etc.) and which would let the person focus more time on the more difficult (less automatable) tasks.

So what tasks are AI models good at? Here’s a synthesis from a number of studies (see the citations at the end of this article):

In my own personal experience, I’ve found that AI is really good at things like doing research, fact checking, writing code (for known patterns, and with strong guardrails), scanning for vulerabilities, and less good at things like writing prose, creating graphics, making decisions, and so on. Human verification is essential for all AI outputs, but especially whenever errors are costly, context is hidded or judgement defines success.

Here’s a finding from a study that looked at usage of Copilot for various tasks and how successful that usage was for different kinds of tasks.

arXiv:2507.07935v6 [cs.AI]  Working with AI: Measuring the Applicability of Generative AI to Occupations – Kiran Tomlinson, Sonia Jaffe, Will Wang, Scott Counts, Siddharth Suri

A very interesting study by Becker et al clearly demonstrates how the amount of time spent of different tasks changes when AI is used. Interestingly, the total amount of work is not necessarily the same, this looks only at the percentages, not the total.

arXiv:2507.09089v2 [cs.AI] Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity – Joel Becker, Nate Rush, Elizabeth Barnes, David Rein

This is supported by other studies which found that the total amount of time increased. Here are some findings from one such study in the domain of open-source software development:

And here are some findings from a study on professional writing – total time reduced but the amount of time spent of each part of the work changed:

Even if AI can reduce the amount of time it takes to write code from weeks to minutes – that does not mean I can ship the code and pass on that time gain to my end users. I still need to have people review the code, and I still need to do all the other security, compliance, and release management tasks. So while an individual task might be faster, my wall clock time is not necessarily impacted.

I love a poster I saw at a ballet school which read “Hard work is not for everyone” – I think that’s a very true statement. Using AI is easy, anyone can do it. But using AI well is a very different thing – and I think it is fair to say that most people probably won’t invest the time and effort to learn how to use AI well. This means that a lot of AI applications are going to be sub-par. You can easily find evidence for this – from the hype merchants publishing YouTube videos claiming to show you how to build an AI application in “five minutes”, to the high visibility failures like those showcased in “12 famous AI disasters”, Thor Olavsrud, CIO Magazine https://www.cio.com/article/190888/5-famous-analytics-and-ai-disasters.html and I suspect that you have personal experience of talking to an AI and finding its ability to actually “help” to be very poor.

Trust is vitally important in AI, and reputation damage and financial penalties are very real risks.

Using AI “well” means things like understanding how the models work, comparing the performance of different models and hyper parameter settings for your specific use case and possibly corpora, implementing guardrails and retries, using techniques like planning and reflection, building and using evaluations, knowing how and when to use different kinds of memory, using skills, and so on. No matter how good you are, I’d argue that you won’t get it right the first time, and you need a continuous improvement loop.

It’s important to collect inputs and outputs, and then to do human reviews, and provide feedback about what is good and what bad. This data can be used to implement improvements, so that in essence the application ‘learns’ over time how to do the task right.

You need a mechanism to route and track tasks. AI can work faster than people. You need a way to queue up tasks so that it can pick form the queue based on whatever criteria are appropirate, maybe the token quota for example. If a task is not clear, it should be able to ask questions before starting work – so you need a way to allow it to assign a task back to a human to get more information. You should use techniques like guardrails and adversial reviews (ideally with a different model) to control quality.

So when is AI going to excel?

I think that this is a good litmus test to apply when trying to identify tasks that AI will be able to excel at:

• Smaller, well-defined tasks with clear context, strong guardrails, and a strong definition of “right”
• Tasks that do not require novel thought or complex decision making
• Clean boundaries and handoffs between tasks
• Repeatable, high-volume tasks
• Continuous improvement feedback loop

We have a huge amount of opportunity ahead of us. Here’s an interesting finding from an Anthropic study that compares the opportunity in various industries to the actual observed usage today:

Labor market impacts of AI: A new measure and early evidence, Massenkoff, McCrory 2026 https://www.anthropic.com/research/labor-market-impacts

There’s lots of very active research in this area I’ve shared a number of papers that I’ve read myself as well as some of my conclusions from my own personal experience. I’d love to hear your thoughts and experience, please hit me up in the comments.

[1] Noy, Shakked, and Whitney Zhang. “Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence.” Science 381(6654), 2023. https://doi.org/10.1126/science.adh2586

[2] Brynjolfsson, Erik, Danielle Li, and Lindsey R. Raymond. “Generative AI at Work.” Quarterly Journal of Economics 140(2), 2025. First circulated as NBER Working Paper 31161, 2023. https://www.nber.org/papers/w31161

[3] Ishikane, Masahiro, Yuki Kataoka, Yasushi Tsujimoto, Yuki Moriyama, Yukimasa Matsuzawa, and Norio Ohmagari. “Efficacy of a Large Language Model Data Extraction System in Evidence Reviews for Emerging Infectious Diseases: A Randomized Crossover Trial.” Open Forum Infectious Diseases 13(7), 2026. https://pmc.ncbi.nlm.nih.gov/articles/PMC13393321/

[4] Khraisha, Qusai, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. “Can Large Language Models Replace Humans in Systematic Reviews? Evaluating GPT-4’s Efficacy in Screening and Extracting Data from Peer-Reviewed and Grey Literature in Multiple Languages.” Research Synthesis Methods 15(4), 2024. https://doi.org/10.1002/jrsm.1715

[5] Kalliamvakou, Eirini. “Research: Quantifying GitHub Copilot’s Impact on Developer Productivity and Happiness.” GitHub Blog, 2022 (updated 2024). Experiment conducted with Sida Peng and Aadharsh Kannan. https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/

[6] Becker, Joel, Nate Rush, Elizabeth “Beth” Barnes, and David Rein. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR / arXiv preprint, 2025. https://arxiv.org/abs/2507.09089

[7] Nadra, Samir, and James P. Purdy. “Is genAI a Good Editor of Academic Writing?” Computers and Composition 80, 2026. https://doi.org/10.1016/j.compcom.2026.102993

[8] Kumar, Deepak. “SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback.” arXiv preprint, 2026. https://arxiv.org/abs/2603.26130

[9] Ribalta-Albado, Maria, and Pere-Pau Vázquez. “Evaluating LLMs’ Abilities to Create Charts, a Systematic Approach.” Computers & Graphics 135, 2026. https://doi.org/10.1016/j.cag.2026.104544

[10] Collier, Zachary A., Richard J. Gruss, and Alan S. Abrahams. “How Good Are Large Language Models at Product Risk Assessment?” Risk Analysis 45(4), 2025 (published online in 2024). https://doi.org/10.1111/risa.14351

Key Takeaways

• Before asking whether Go or Java is faster, both services need to answer the same HTTP question. That means the same endpoints, request fields, validation rules, work-factor semantics, response JSON, and error behavior.
• The benchmark contract defines equivalent externally specified work, not identical CPU instructions. extraWork drives a deterministic operation whose result affects a returned checksum, so conformance tests can catch many skipped or altered implementations.
• JSON comparison should be strict about schema, values, types, and error codes, but not brittle about object member order or whitespace. Framework-default HTML, text errors, stack traces, and runtime metadata are contract failures for body-bearing responses.
• Shared acceptance tests are the gate into later implementation and benchmark work. The Java/Helidon and Go services must pass the same contract before any benchmark result is admissible.

You can make two services look almost identical and still measure different work.

One implementation accepts an unknown JSON field and ignores it. The other rejects the request. One silently clamps a payload size that is too large. The other returns a validation error. One computes a checksum from generated bytes. The other computes it from a different buffer, or runs a loop that does not affect the returned result. One returns a framework-default text error. The other returns JSON.

If those services go into a benchmark, the chart is already compromised. The result might look like Go versus Java, but part of the difference could be request parsing, validation policy, payload generation, error handling, or a work loop that does not mean the same thing in both implementations.

So before asking whether Go or Java is faster, we need to make sure both services are answering the same question.

The first article in this series set a fairness rule: no benchmark result is admissible until the method and evidence chain are strong enough to support it. This series is scoped to a later single-machine benchmark method, but the exact operating system, kernel, CPU, memory, runtime versions, and isolation settings belong in the execution-method article and benchmark manifest, not in this service-contract article. Here, we define the language-neutral service behavior that the Java service and the Go service must satisfy at the HTTP boundary before either one is tuned, load-tested, or compared.

For developers building service-tier APIs around Oracle AI Vector Search, retrieval, embedding, reranking, or Oracle AI Database workflows, this boundary matters. A small application service often sits between clients, model calls, and Oracle AI Database. If two versions of that service validate different JSON, generate different payloads, or return different correctness artifacts, later runtime measurements are not about the runtime anymore. They are about drift.

This article does not publish benchmark results. It does not choose a faster runtime. It does not implement either service. It does not design the full load generator or run the benchmark package. We are defining the service behavior that later code must satisfy.

What This Article Defines, and What It Does Not Measure

A language-neutral service contract is the externally visible behavior both implementations must provide, expressed without depending on Java or Go internals. For this series, that means the behavior a client can observe over HTTP: endpoint paths, methods, request headers, request JSON, validation rules, response JSON, status codes, and error shapes.

HTTP gives us the shared vocabulary for methods, status codes, content, and representations. JSON gives us a shared request and response format. The contract uses those pieces to define observable behavior explicitly rather than letting framework defaults decide.

This article defines:

• The endpoint set and request media type
• The JSON request and response shapes
• The request body limit
• Payload-size and deterministic payload-generation rules
• The meaning of extraWork
• Validation rules, duplicate-field behavior, and unknown-field behavior
• Success and error response schemas
• JSON equivalence rules
• Cross-implementation acceptance-test categories

It does not measure throughput, latency, memory, CPU, startup, warmup, garbage collection, scheduler behavior, Oracle AI Database query performance, model inference, or vector-search performance. Those questions belong later in the series, after the service contract, implementations, load generator, execution method, and analysis method are stable.

Later latency measurements also need their own load model, request scheduling policy, timeout policy, warmup plan, sampling method, and treatment of coordinated omission.

That distinction is not academic. It is the difference between a benchmark that can be challenged constructively and a benchmark that depends on trust.

What Endpoint Is Under Test?

For this benchmark contract, both services expose the same two endpoints: GET /health and POST /work.

GET /health returns 200 OK, Content-Type: application/json, and the exact JSON object shown below. It exists to let scripts and acceptance tests confirm that the service is up and speaking JSON. It is not part of the measured work path. The response must not include extra fields.

{
  "status": "ok"
}

No timestamp. No process ID. No framework name. No Java or Go version. Health is a readiness check for this contract, not a diagnostic endpoint.

HEAD /health follows HTTP HEAD semantics for the GET /health resource: it returns the same status and headers that GET /health would return, without a response body. The conformance suite treats that as protocol behavior, not as part of the measured work path.

POST /work is the endpoint under test. It accepts a JSON request body and returns JSON on success and on body-bearing failures.

For this benchmark contract, the request Content-Type must be application/json. For common-client compatibility, the service also accepts application/json; charset=utf-8. The JSON media type registration does not define a charset parameter; accepting that parameter here is a deliberate contract choice and does not change JSON processing. A missing Content-Type on POST /work is treated as an unsupported media type. The service returns 415 Unsupported Media Type with the agreed JSON error shape.

Successful and body-bearing error responses use Content-Type: application/json.

The request body for POST /work is also bounded. If the /work request content exceeds 4096 bytes after HTTP transfer decoding, the service returns 413 Content Too Large with the agreed JSON error shape. This limit is intentionally small because the request body contains only control fields, not the generated benchmark payload.

The rule is simple: for requests that can carry a response body, JSON in, JSON out.

That means success responses are JSON. Validation errors are JSON. Malformed request errors are JSON. Unsupported media type errors are JSON. Oversized request errors are JSON. Unsupported method errors are JSON, except where HEAD semantics require no response body. Unknown route errors are JSON. Framework-default HTML pages, plain-text errors, stack traces, redirects, or implementation-specific error documents are not acceptable for this benchmark contract.

Unsupported methods are also part of the boundary. If a client sends GET /work, the service returns 405 Method Not Allowed, includes Allow: POST, and returns the agreed JSON error shape. HEAD /work is unsupported and returns 405 Method Not Allowed with Allow: POST and no response body. If a client sends an unsupported body-bearing method to /health, the service returns 405 Method Not Allowed, includes an Allow header containing GET and HEAD, and returns the agreed JSON error shape. If a client sends POST /missing, the service returns a consistently defined 404 Not Found JSON error.

For the primary pure-service benchmark, request logging, TLS, compression, and tracing are off. Those are real production concerns, and later slices can measure them deliberately. They are not mixed into the primary service-equivalence contract. If a later scenario enables one of them, both implementations must use an equivalent configuration and pass conformance again.

HTTP status-code and method behavior in this contract follows the semantics defined by RFC 9110: HTTP Semantics. The endpoint names, response fields, and exact validation policy are benchmark contract choices.

What Does Each Request Contain?

The request body for POST /work is compact. It does not send a large payload over the wire. Instead, it tells the service how to generate deterministic bytes and how much deterministic work to apply.

The body must be a single top-level JSON object. Top-level arrays, strings, numbers, booleans, and null are invalid request shapes and return 400 Bad Request. Multiple JSON documents in one request body are also invalid.

A valid request looks like this:

{
  "requestId": "case-0001",
  "payloadSize": 1024,
  "seed": 42,
  "extraWork": 10
}

Each field has a specific job.

requestId is a stable client-provided identifier. The service echoes it in a successful work response. It must be a JSON string with length 1..128.

payloadSize is the requested deterministic payload size in bytes. For this contract, it must be an integer between 0 and 131072, inclusive. That upper bound gives us payloads up to 128 KiB without putting large payload bodies into article examples or small conformance fixtures. A value below 0, a value above the maximum, a string such as "1024", or a fractional number fails validation.

seed is the input to deterministic payload generation. It must be an integer between 0 and 2147483647, inclusive. Keeping it in a conservative integer range avoids JSON number interoperability problems and makes the same fixture easy to represent in both Java and Go.

extraWork is a bounded work factor. It must be an integer between 0 and 100, inclusive. A value of 0 is valid and means “generate the payload and compute the initial correctness artifact, but apply no extra repeated work.”

Numeric fields must be JSON integers, not quoted numeric strings or fractional values.

Unknown fields are rejected. That may feel strict, but it is useful strictness. If the Go service silently accepts "payload_size" while the Java service rejects it, the two services no longer have the same contract. If one parser ignores "debug": true and the other fails, the benchmark has hidden drift. During conformance, extra fields are a 400 Bad Request.

Duplicate member names in the top-level /work request object are also invalid for this contract. The request schema does not contain nested objects; if nested objects are added later, the duplicate-member rule must be extended explicitly.

JSON syntax and interoperability rules are defined by RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format. RFC 8259 describes JSON objects as unordered collections, says object names should be unique, and notes that behavior can be unpredictable when parsers receive duplicate names. This benchmark contract removes that ambiguity by rejecting duplicate top-level request fields before field validation.

The services should not silently accept a request body such as this:

{
  "requestId": "case-0001",
  "payloadSize": 1024,
  "payloadSize": 2048,
  "seed": 42,
  "extraWork": 10
}

Test fixtures use unique field names. Conformance tests include duplicate-field cases to make sure both services fail them consistently.

How Is Payload Size Controlled?

The request does not carry the full payload. The service generates the payload deterministically from payloadSize and seed.

That choice keeps the benchmark focused. If the client sends a large JSON string, we are measuring request-body transport, JSON string parsing, escaping behavior, memory copies, and possibly different decoder strategies. Those can be useful in another benchmark, but they are not the shape we need here.

For this contract, the input tells the service how many bytes to generate. Both implementations must use the same language-neutral payload-generation algorithm. They must not call a runtime-specific random API and assume the output will match.

The contract can be expressed this way:

state = unsigned64(seed)
for i from 0 to payloadSize - 1:
    state = (state * 6364136223846793005 + 1442695040888963407) modulo 2^64
    buffer[i] = (state >> 56) & 0xff

Both implementations must use the same constants, unsigned 64-bit wraparound behavior, byte extraction, and iteration count. The same payloadSize and seed must produce the same byte sequence in Java and Go.

That gives acceptance tests a clean target. A test case can say, “For payloadSize = 1024 and seed = 42, both services must generate the same buffer and return the same checksum after the same work factor.” We do not need to inspect the buffer over HTTP; the returned correctness artifact shows whether the service produced the specified result.

What Does the Work Factor Mean?

This is the part of the contract most likely to make or break the benchmark.

extraWork is not a delay knob. It is not “run a loop for a while.” It is not “do whatever is convenient in each language.” It is the number of times the service applies a specified deterministic operation to the generated payload buffer, and the operation must affect the returned checksum.

A language-neutral sketch looks like this:

buffer = generatePayload(payloadSize, seed)
state = SHA-256(buffer)
repeat extraWork times:
    state = SHA-256(state || buffer)
checksum = lowercaseHex(state)

The || symbol means byte concatenation: the current 32-byte hash state followed by the generated buffer. The final checksum is the lowercase hexadecimal encoding of the final SHA-256 state.

This does not claim that Java and Go execute identical machine instructions. They will not. The JIT compiler, Go compiler, libraries, memory layout, CPU intrinsics, allocator behavior, and runtime scheduling can differ. Java and Go may also use different SHA-256 implementations. The contract requires equivalent externally specified work: both services generate the same buffer, apply the same deterministic operation the same number of times, and expose the resulting checksum so tests can verify the externally specified result.

That last point is important. If extraWork does not affect the response, an implementation could accidentally skip the loop and still pass a superficial status-code test. Because extraWork affects the returned checksum, conformance tests can detect many skipped or altered implementations of the specified work. That does not prove instruction-level equivalence; it proves that the observable result matches the contract.

Checksum examples in this article are format placeholders, not test vectors. Conformance fixtures must use complete expected checksum values computed from the published payload-generation and work-factor algorithms.

What Does a Successful Response Look Like?

A successful POST /work response returns 200 OK, Content-Type: application/json, and a stable JSON body.

The success schema has this shape. The checksum value below is a format placeholder, not an expected value for the earlier request:

{
  "requestId": "case-0001",
  "payloadSize": 1024,
  "extraWork": 10,
  "checksum": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "ok": true
}

The placeholder checksum is not a test vector. A real conformance fixture must use a complete lowercase SHA-256 hex string independently computed from the published payload-generation and work-factor algorithms.

The fields are deliberately limited.

requestId must match the request. payloadSize must match the accepted request value. extraWork must match the accepted request value. checksum must be derived from the deterministic payload and the work-factor operation. ok is true for a successful work response.

The response intentionally does not echo seed; the seed is validated and used to derive checksum, but it is not part of the success response schema.

The response does not include runtime metadata. No thread ID. No goroutine ID. No hostname. No process ID. No memory statistics. No framework name. No timestamp. No JDK or Go version. Those can be useful in logs, manifests, diagnostic endpoints, or benchmark artifacts, but they do not belong in the correctness response.

Why so strict? Because correctness responses are compared across implementations. If Java returns "framework": "Helidon" and Go returns "framework": "net/http", either the comparison has to ignore fields or the services fail. Ignoring fields too broadly can hide real differences. The cleaner choice is to keep the response schema focused on the contract.

How Should Validation Failures Behave?

Validation behavior is part of the benchmark, not an implementation detail.

If invalid input changes which requests are accepted, how much work is performed, or what response body is serialized, it can contaminate later measurements. So the contract defines both status code and JSON shape for responses that carry a body.

A field validation error looks like this:

{
  "error": {
    "code": "INVALID_REQUEST",
    "message": "payloadSize must be between 0 and 131072",
    "field": "payloadSize"
  }
}

For this contract, request-shape and scalar validation failures use 400 Bad Request. That includes missing required fields, wrong JSON types, out-of-range numeric values, unknown fields, duplicate top-level request fields, and invalid requestId values.

Malformed JSON also returns 400 Bad Request, with the error code MALFORMED_JSON and a field value of null. A body containing multiple JSON documents is treated the same way.

Unsupported request media types return 415 Unsupported Media Type. Request bodies larger than 4096 bytes return 413 Content Too Large. Unsupported body-bearing methods return 405 Method Not Allowed, include an Allow header listing supported methods for that target resource, and return the agreed JSON error shape. HEAD responses follow HTTP no-body semantics. Unknown routes return 404 Not Found.

A non-field error uses the same envelope:

{
  "error": {
    "code": "UNSUPPORTED_MEDIA_TYPE",
    "message": "Content-Type must be application/json or application/json; charset=utf-8",
    "field": null
  }
}

The contract uses these machine-readable error codes:

• MALFORMED_JSON for syntactically invalid JSON or multiple JSON documents in one request body.
• INVALID_REQUEST for a valid JSON body with an invalid top-level shape, missing required field, wrong type, out-of-range value, invalid requestId, unknown field, or duplicate top-level request field.
• UNSUPPORTED_MEDIA_TYPE for missing or unsupported Content-Type on POST /work.
• CONTENT_TOO_LARGE for a /work request body larger than 4096 bytes.
• METHOD_NOT_ALLOWED for unsupported methods on known routes.
• NOT_FOUND for unknown routes.

Conformance fixtures compare HTTP status, machine-readable code, field, and JSON shape when the response carries a body. The message value must be a non-empty string, but basic conformance does not require exact message equality unless a fixture explicitly opts into it. If one service returns INVALID_REQUEST and the other returns bad_request for the same fixture, that is a contract failure.

When a request has multiple problems, conformance tests use this precedence order unless a fixture explicitly states otherwise:

• Match the route first.
• Validate the HTTP method for that route.
• Validate request Content-Type for POST /work.
• Enforce the 4096-byte request-body limit.
• Decode exactly one JSON document.
• Reject duplicate top-level request fields.
• Validate required fields, unknown fields, JSON types, scalar ranges, and requestId rules.

This order keeps routing and protocol failures separate from JSON-shape failures. Basic conformance fixtures should still prefer one failure per request, but the precedence rule prevents accidental drift when multi-fault cases appear.

The error object is a small custom JSON shape. It is not claiming RFC 9457 Problem Details for HTTP APIs compliance. If a future version adopts Problem Details, it should do so explicitly, including the appropriate media type and fields.

What Makes Two JSON Responses Equivalent?

JSON response comparison should be strict enough to catch real differences and flexible enough to avoid false failures from incidental serialization details.

A response with the same fields in a different order is equivalent:

{
  "ok": true,
  "checksum": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "extraWork": 10,
  "payloadSize": 1024,
  "requestId": "case-0001"
}

That object is semantically equivalent to the earlier success-response shape if all required fields, values, and JSON types match. Object member order and insignificant whitespace are not part of correctness.

But these differences must fail conformance:

• "payloadSize": "1024" instead of "payloadSize": 1024
• Missing checksum
• Different requestId
• Different extraWork
• A checksum that does not match the deterministic payload and work factor
• A different machine-readable error code for the same validation failure
• Extra runtime-specific fields when the contract forbids them
• A framework-default HTML or plain-text error body where JSON is required

Byte-for-byte JSON comparison is too brittle. A loose “does the response contain a few expected fields somewhere?” comparison is too weak. The target is structural comparison: required field names, JSON types, values, error codes, and status behavior must match.

The language-neutral contract defines the same request, validation rules, deterministic work, and JSON response before Java/Helidon and Go implementations can enter benchmark execution.

How Do Acceptance Tests Prove Equivalence?

Acceptance tests are the gate between contract and benchmark.

The test runner is not the important part yet. It could eventually be a small custom tool, a script, or a test suite in a language chosen for the benchmark harness. What matters is that the same cases run against both implementations and compare the same expected behavior.

The acceptance cases should cover the ordinary path and the annoying edges.

For the happy path, the suite needs a valid minimal request, representative small and medium payloads, the maximum allowed payloadSize, extraWork = 0, and the maximum allowed extraWork. The maximum-payload case does not need to combine maximum payloadSize with maximum extraWork in a basic conformance run. The point is to prove the boundary values and deterministic behavior, not to turn contract validation into a stress test.

For determinism, the same request sent repeatedly must return the same stable fields and checksum. The same test vector sent to Java and Go must return JSON-equivalent responses. Object member ordering must not affect the comparison.

For validation, the suite needs missing required fields, wrong JSON types, negative payloadSize, payloadSize = 131073, invalid requestId, negative extraWork, extraWork above 100, malformed JSON, multiple JSON documents, duplicate fields, unknown fields, unsupported media type, oversized body, unsupported method, and unknown route.

For error equivalence, both services must return the same status code, same machine-readable error code, same required error fields when a body is allowed, and no framework-default HTML, stack traces, or plain-text errors.

For protocol behavior, GET /health must return stable JSON, HEAD /health must follow HEAD semantics for the health resource, POST /work must require JSON, response Content-Type must be consistent with the contract, and 405 responses must include the appropriate Allow header.

Passing these tests does not prove performance fairness by itself. It proves the functional contract. Performance fairness also needs implementation review, allocation analysis, tuning discipline, load-generator validation, environment capture, run protocol, and statistical analysis. Those appear later in the series.

But this is the gate that comes first. Java and Go can differ internally. They can use different libraries, runtime strategies, allocators, diagnostics, and tuning flags. If they cannot produce the same observable behavior for the same fixtures, they do not enter benchmark execution.

The Handoff to the Implementations

The contract gives the implementation articles a narrow job: build services that satisfy the same externally visible behavior.

The Java/Helidon implementation and the Go implementation can choose idiomatic libraries, data structures, and runtime configuration where the contract leaves room. They cannot change endpoint paths, request fields, validation rules, deterministic payload generation, extraWork semantics, success response fields, error response shape, or JSON equivalence rules to make one implementation easier.

That may sound restrictive, but it is what makes the later comparison useful. Once both services pass the same acceptance suite, the benchmark can move on to the next questions: how the services are implemented, how the load generator is validated, how the environment is captured, how runs are executed, and how results are analyzed. Until then, there is no Go-versus-Java result to interpret. There is only a contract to satisfy.

References

The contract choices in this article use standards and product documentation as context, but the endpoint names and field names are benchmark design decisions.

• RFC 9110: HTTP Semantics defines the HTTP vocabulary for methods, status codes, content, and representation metadata used by the contract.
• RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format defines JSON syntax and explains why object member ordering is not a good byte-level equivalence target.
• IANA media type registration for application/json provides the media-type registration context for the request Content-Type policy.
• RFC 9457: Problem Details for HTTP APIs provides context for standardized HTTP API error responses, although this article uses a smaller contract-specific error shape.
• NIST FIPS 180-4: Secure Hash Standard provides context for SHA-256, the hash function used in the deterministic work-factor sketch.
• Oracle AI Database provides the database platform context for AI application services and database-backed workflows.
• Oracle AI Vector Search provides the high-level context for service-tier APIs around vector search and AI application workflows.

Key Takeaways

• This first article does not publish benchmark results. It defines the contract that later Go net/http and Java Helidon SE runs must satisfy before any chart is worth discussing.
• Fairness in this series means testable equivalence: the same endpoint behavior, payload rules, validation and error handling, plus declared CPU, memory, logging, TLS, and process-topology choices.
• The primary benchmark is a bounded single-machine Linux service-under-test benchmark. It is relevant to developers building AI services with Oracle AI Database, but it does not measure database execution, model inference, Kubernetes, autoscaling, service mesh behavior, or cloud-provider performance unless those appear as separately labeled slices.
• If we include multiple JDK builds, a distinct request-handling mode, a Leyden-related startup or warmup path, or GraalVM Native Image, each one will be labeled as its own runtime shape with exact versions, flags, generated artifacts, and run conditions.

You have probably seen this pattern before. A benchmark shows a compact Go service next to a compact Java service. The chart is interesting. The conclusion is tempting. Then the comments get more useful than the chart.

How many cores were assigned? Was the load generator the bottleneck? Was the Go implementation doing the same work as the Java implementation? What did memory look like? What did garbage collection do? Which JDK build was used? Did virtual threads matter, or were they just present? If the selected JDK includes startup or warmup features associated with Project Leyden work, did they change startup, warmup, first-request behavior, footprint, or steady-state behavior? Were the runs long enough? Were there enough repeats? Were the statistics strong enough to support the claim?

Those are not annoying questions. They are the right questions.

This series follows up on that kind of Go-versus-Java service comparison: a Go net/http service on one side and a Java Helidon SE service on the other. That shape is worth investigating, but only if the method is strong enough to carry the result. The follow-up needs to answer the questions that usually get buried under the chart: CPU and core disclosure, memory use, GC behavior, JDK build clarity, request-handling behavior where applicable, startup or warmup features where applicable, load-generator validity, implementation equivalence, longer runs, more repeats, and statistical treatment.

So Part 1 does not publish new numbers. No old CSV, old chart, or previous result claim is evidence for this run. This article defines the rules a later run must satisfy before any result deserves attention.

If you are building AI applications that use Oracle AI Database, that distinction still matters—but the primary benchmark measures the application-tier service boundary, not database behavior. A retrieval API, embedding gateway, reranking service, feature-enrichment service, or application-tier vector-search facade can spend meaningful time in ordinary service code before and after it touches a database or a model. Runtime efficiency, tail behavior, warmup, memory pressure, and operational visibility still affect cost and reliability.

But if we mix database latency, model inference, service mesh overhead, and runtime behavior into one chart too early, we will not know what we measured.

The first move is to narrow the question.

Why This Follow-Up Starts With a Benchmark Contract

A benchmark contract is the set of rules that determines what is measured, which variants are included, how implementation equivalence is checked, how runs are accepted or rejected, and how analysis is performed before charts appear.

That sounds formal, but the idea is simple: write down the method before you see the result.

Without a contract, benchmark discussion drifts quickly. One side argues that the Go service is smaller. Another side argues that the Java runtime has better optimization and diagnostics. Someone asks whether the request-handling model changed the result. Someone else asks whether the load generator was saturated. Eventually the conversation becomes a proxy argument about language preference instead of a concrete discussion about a specific workload.

Our thesis is deliberately narrower: a useful Go-versus-Java service benchmark starts with a contract, not a chart. We will define equivalent work, fair runtime variants, measurement rules, and acceptance gates before publishing results, so later performance claims can be judged against a declared method rather than accepted on trust.

The contract has to answer a few practical questions before any chart is allowed:

• What service behavior must the Go and Java implementations provide?
• Which runtime shapes are being compared?
• Which choices are fixed for the primary run, such as logging, TLS, CPU allocation, memory policy, and process topology?
• What correctness tests prove that both implementations do the same externally visible work?
• What evidence shows that the load generator has headroom?
• Which warmup and steady-state windows define the measurement?
• How many repeats are required, and what makes a run invalid?
• Which raw artifacts connect the final chart back to source, build, runtime, machine, command, and analysis rule?

That list is the benchmark contract in plain language. A script producing numbers is not enough. A valid result has to trace back to the source revision, build, runtime, machine, command, measurement window, raw output, and analysis rule that produced it.

That dependency chain is the whole point of the series.

We define the benchmark contract in Articles 1–8 before publishing fresh results in Article 9 and interpreting the safe conclusions in Article 10.

The benchmark manifest for accepted runs will record the exact Go, JDK, Helidon, Linux, hardware, and tool versions used, including details that affect interpretation such as kernel version, CPU policy, and whether the service is running directly on the host or inside a constrained environment.

Which Go and Java Runtime Shapes Are Included

The comparison in this series is not “Go versus Java” in the abstract. That framing is too broad to be useful.

The Go side will use Go’s standard-library net/http package. That keeps the baseline focused on the built-in HTTP server path and avoids introducing third-party router or middleware choices into the primary comparison. Real Go services often include routers, middleware, validation libraries, and observability agents, and that is fine. This benchmark starts with the standard-library service boundary because it keeps the baseline small enough to audit.

The Java side will use Helidon SE, the selected Java service framework for this series, so the benchmark compares one declared Java service shape rather than the broader Java framework ecosystem. Spring Boot, Quarkus, Micronaut, and other stacks matter in real systems, but adding them here would change the question. This series is about Go net/http and Java Helidon SE under a declared service contract.

We will not report the Java side as just “Java.” If multiple JDK builds are included, each Java result will identify the JDK distribution or build provider, version, build identifier, command line, JVM flags, and support or licensing label used for that slice where that label affects interpretation. The question is not whether a label is faster in the abstract. The question is how the same Helidon SE source behaves under the specific JDK builds declared in the manifest.

Virtual threads are part of that Java service-shape discussion, but they need careful framing. JEP 444 delivered virtual threads in JDK 21. They are most relevant when the service model uses many concurrent blocking operations, especially I/O-bound work. They let developers write straightforward blocking-style code while supporting high concurrency more efficiently in workloads that spend time waiting.

They should not be treated as a general throughput or latency improvement for CPU-bound handlers. CPU-bound handlers still need CPU. If a request handler is mostly serialization, checksum work, or allocation pressure, the virtual-thread story may look different from a service that spends much of its time waiting on blocking I/O.

So we will treat the Java request-handling model as something to verify and measure, not something to assume. We will document the selected Helidon version, JDK version, server configuration, and request-handling model before applying labels such as platform threads, virtual threads, or default Helidon behavior. If that combination provides a distinct, supported, reproducible virtual-thread service shape, that slice will be labeled separately. If virtual threads are simply part of the selected Helidon shape, the manifest will say so instead of pretending that “Java” is a single opaque runtime.

Project Leyden and related startup or warmup features get the same separate-label treatment, and only when the selected JDK provides a supported, reproducible path. Project Leyden aims to improve startup time, time to peak performance, and footprint. Related JDK work, such as JEP 483: Ahead-of-Time Class Loading & Linking, is relevant to startup and warmup questions, but availability depends on the selected JDK version and build.

Leyden-related JDK features and GraalVM Native Image are different technologies. GraalVM Native Image ahead-of-time compiles Java applications into native executables. Leyden-related JDK features such as ahead-of-time class loading and linking are a different path. If either is included, we will label it separately, record the exact toolchain, flags, generated artifacts, and measurement scope, and avoid folding those results into the default JVM result.

For any startup or warmup feature associated with Leyden work, we will look especially at startup, warmup, footprint, and first-request behavior. If it appears to affect more than startup or warmup, the analysis will need artifacts to support that claim.

What Fair Means in This Benchmark

Fairness is not a slogan. It is a set of checks.

For this series, fair means the Go and Java services perform the same externally visible work: the same endpoint semantics, payload rules, validation behavior, error behavior, response shape, logging policy for the primary run, TLS policy for the primary run, declared CPU allocation, declared memory policy, and process topology.

That last part matters. One Go process versus one Java process is a different topology from several processes per runtime. A service with request logging disabled is different from one formatting a line on every request. Plain HTTP is different from TLS. A colocated load generator is different from a load generator running with enough isolated headroom to stay out of the server’s way. None of these choices is automatically wrong, but hidden choices make results hard to interpret.

Equivalent work comes before tuning. Each implementation must provide the same externally visible behavior before runtime-specific optimization is considered: request parsing, response generation, validation, error behavior, and any configured dependency interaction.

If the Go handler accidentally performs extra allocation in a work loop that the Java handler avoids, that is not a Go runtime result; it is a benchmark implementation bug until proven otherwise. If the Java service returns a different error body, validates different parameter ranges, or serializes a different payload, that is not a runtime result either.

The contract will require correctness tests before performance runs. Those tests need to compare outputs across implementations using stable request cases, not generated numeric row identifiers or one-off local observations. If an implementation changes for equivalence, the changed source becomes the benchmark source. Old results do not carry forward.

Fairness also means symmetrical restraint. We should not tune one runtime aggressively while leaving the other at casual defaults unless that is explicitly labeled as a tuned-versus-untuned comparison. We should not enable diagnostics inside the primary measurement path for one runtime and not the other unless the overhead is measured and accepted. We should not declare a run valid if the load generator is the thing being measured.

A useful benchmark should make unfairness difficult to hide.

What the Single-Machine Benchmark Measures

The primary benchmark will measure a declared single-machine Linux service-under-test workload. The service process or processes being measured will run on one declared Linux machine with a declared CPU, memory, and process policy. The load-generator placement will also be declared and validated in the run protocol because the client side can affect the result. If a separate client host is used for headroom, it remains part of the measurement harness, not a claim about distributed deployment.

Within that boundary, the benchmark will measure service behavior under controlled workload shapes: payload size, work factor, concurrency, CPU allocation, process topology, warmup, steady state, and resource pressure. It will report throughput and latency distribution together, because average response time without distribution data hides too much. It will also report CPU use, memory use, runtime behavior, startup, warmup, and steady-state behavior.

Warmup means the period before the measurement window where the service is allowed to initialize, compile hot paths, populate runtime structures, and settle into a more representative operating shape. Steady state means the measurement window after warmup, when the benchmark records the primary service behavior for that run.

For a JIT runtime, warmup is obviously important. For a non-JIT runtime, warmup can still matter because caches, connection state, allocator behavior, and CPU-frequency behavior can change early-run measurements.

Java diagnostic runs will include runtime telemetry where appropriate, such as GC behavior, allocation pressure, thread behavior, and selected Java Flight Recorder data. JDK tools such as jcmd can help capture runtime state in diagnostic runs, with exact commands recorded in artifacts. Profiling and diagnostics can add overhead, so diagnostic runs must stay separate from primary measurement runs when that overhead matters.

Go diagnostic runs will also capture runtime and diagnostic information where appropriate, using Go tooling such as pprof, runtime statistics, and tracing. Again, the point is not to drown the reader in profiles. The point is to avoid explaining a throughput or latency shape by guessing.

The load generator is part of the benchmark, not an afterthought. A service benchmark is not valid just because the server process is busy. The client side also needs headroom. If the load generator is CPU-saturated, cannot sustain the requested rate, or distorts latency measurement, the run is evidence about the test harness, not about Go versus Java service behavior.

The selected load generator must be validated for headroom and latency-measurement behavior. A fixed-rate tool such as wrk2, or another documented tool selected later, can help address coordinated-omission risk, but the benchmark must still prove that the client is not CPU-, network-, connection-, or scheduling-limited. Latency reporting will need distribution detail, not just a single average or p99. Tools and formats such as HdrHistogram are useful here because they preserve a fuller percentile spectrum.

The primary pure-service results will stay separate from secondary slices. TLS is a useful slice. Request logging is a useful slice. A real dependency, possibly an Oracle AI Database call, can be a useful slice. But those slices answer different questions.

If we add an Oracle AI Database dependency later, the result may reflect driver behavior, connection pooling, query shape, database execution, network locality, and runtime behavior together. That can be useful, but it should not be confused with the primary pure-service benchmark.

One more boundary matters: this is an end-to-end HTTP service benchmark, not a JVM microbenchmark. JMH is useful for JVM microbenchmarks, but it is not a replacement for measuring a service path through HTTP handling, payload work, runtime behavior, load generation, and resource use.

What the Benchmark Does Not Measure

The boundary is narrow on purpose.

We keep the primary benchmark inside a single-machine service boundary, so later results are useful without pretending to answer deployment, cloud, or distributed-systems questions.

The primary benchmark will not support conclusions about Kubernetes scheduling, autoscaling, service mesh behavior, multi-node networking, cloud-provider performance, distributed reliability, rolling updates, node cold starts, or production fleet operations. It will not compare cloud providers. It will not make claims about how a full distributed system behaves.

It also does not measure all Java frameworks or all Go frameworks. It does not measure universal language superiority. It does not measure model inference. It does not measure Oracle AI Database-dependent behavior, including database-side vector search, in the primary run. It does not tell you how a full retrieval-augmented generation stack behaves under production traffic.

That does not make the benchmark small in a bad way. It makes the benchmark interpretable.

If you are building an AI application on Oracle AI Database, you often need both kinds of information. You need database and model benchmarks for database and model questions. You also need application-tier service benchmarks for runtime, concurrency, warmup, memory, and operational questions. We start with the service boundary so that later, if a dependency slice is added, we can see what changed.

A benchmark that answers one question clearly is more useful than a benchmark that vaguely gestures at ten.

How the Series Gets From Contract to Results

This is a ten-part series because the method needs to be visible before the result.

Part 1, this article, defines the benchmark contract and the scope boundary. Part 2 defines the language-neutral service requirements: endpoint behavior, payload generation, work factor, response shape, validation, errors, and cross-implementation acceptance tests. Those requirements come before Go or Java implementation detail.

Parts 3 and 4 then constrain the implementations. The Java article covers the Helidon SE service shape, request handling for the selected Helidon and JDK combination, JDK runtime inputs if multiple builds are included, startup or warmup features where applicable, and Java diagnostics. The Go article covers the net/http implementation, the equivalence review, allocation behavior, Go runtime settings, and Go diagnostics.

Part 5 focuses on load generation. That includes request model, fixed-rate behavior, coordinated-omission risk, client headroom, and the evidence needed to show the load generator is not the bottleneck. Part 6 defines benchmark methodology: run protocol, CPU and memory policy, warmup, steady-state windows, invalid-run rules, resource capture, and traceability. Part 7 turns the method into execution artifacts: environment, versions, commands, manifests, run order, and raw outputs.

Part 8 defines analysis before results: canonical data shape, aggregation, uncertainty reporting, plot rules, outlier handling, and invalid-run handling. Only after those steps does Part 9 publish fresh benchmark results. Part 10 interprets what those results mean inside the declared boundary.

That order is deliberate. If the analysis rules are written after seeing the charts, the reader has to wonder whether the rules were chosen to make the charts look better. If the implementation contract is written after tuning, the reader has to wonder whether equivalence was enforced or assumed. If the load generator is validated after a surprising result, the reader has to wonder whether the harness is being rescued.

Part 1 publishes no benchmark results. Later result articles will include or link to manifests that identify the source revision, build commands, runtime versions, machine configuration, load-generator configuration, raw outputs, and analysis scripts used for each accepted run. Each accepted run must be traceable to:

• The exact Go and Java source revision
• The build commands and generated artifacts
• The Go version, JDK distribution or build provider, JDK build identifier, and Helidon version
• The runtime flags and process topology
• The load-generator command and configuration
• The Linux environment manifest for the machine
• The warmup and measurement windows
• The raw latency, throughput, CPU, memory, and runtime diagnostic files
• The analysis script or notebook that produced the chart

This is how old charts and orphaned CSV files stay out of the evidence chain. They may explain why we are asking better questions, but they are not proof for the new run.

Acceptance Gates Before Results

A run does not count just because it completed.

The benchmark needs acceptance gates that reject bad evidence before it influences conclusions. Some gates are functional: the Go and Java services must return equivalent responses for the same valid requests and equivalent errors for invalid requests. Some gates are environmental: the machine, runtime versions, CPU topology, memory, operating-system state, and process topology must be captured. Some gates are methodological: warmup and steady-state windows must be declared before the run, repeat rules must be defined, and invalid-run reasons must be applied consistently.

A run does not count if the Go and Java services return different responses, the selected JDK build is not recorded, the load generator is saturated, the warmup period is missing, a repeat is discarded without a declared reason, resource data is missing, or raw artifacts cannot be traced back to source, build, runtime, and machine state.

That list is not exhaustive, but it shows the posture. Invalidity comes from contract violations, not from surprise. A run is not rejected because one runtime looks unexpectedly good or unexpectedly bad. It is rejected because it failed a rule that existed before the result.

Repeat aggregation also needs discipline. If multiple runs are collected, the series must say how they are combined and how uncertainty is shown. If a run is unstable, the instability is either a result to explain or a contract failure to document. It is not something to hide with a prettier chart.

Resource metrics must be interpreted with throughput and latency, not as a separate appendix nobody reads. A service that reaches high throughput by consuming much more CPU is different from one that reaches the same throughput with less CPU. A low-latency shape with high memory pressure has a different operational story from a low-latency shape with stable memory. GC pauses, allocation rate, heap behavior, and warmup matter because they explain why a service behaves the way it does under load.

This is especially relevant for AI application services. The expensive part of an AI system is not always the HTTP handler, but inefficient service code can still waste CPU that you would rather spend on database work, model calls, embedding pipelines, or concurrent users. The benchmark needs to report enough resource context to make that tradeoff visible.

The Questions This Series Will Answer

By the end of the series, you should be able to answer a tighter set of questions than “Which language is faster?”

Under this single-machine service-under-test workload, how do Go net/http and Java Helidon SE behave as payload size, work factor, concurrency, CPU allocation, and process topology change?

Does workload shape change the result? A small response with little work may stress the runtime differently from a larger payload, a more allocation-heavy path, or a higher-concurrency request pattern.

How much CPU and memory does each variant consume to produce its service behavior? Throughput without resource context is incomplete.

What are the Java runtime characteristics: GC frequency, pause behavior, heap behavior, allocation pressure, thread behavior, startup, warmup, and steady-state profile?

What are the Go runtime characteristics: allocation behavior, heap behavior, scheduler behavior, GC impact, startup, warmup, and steady-state profile?

Does the selected Java request-handling model matter for this workload? If it does, where does it show up: latency distribution, CPU use, memory use, concurrency behavior, warmup, or operational visibility?

If more than one JDK build is included, what changes, if anything, when the same Java source is tested with the same declared JVM settings and separately recorded JDK distribution, version, and build identifier?

Where do startup or warmup features associated with Leyden work help, if they are included: startup, warmup, first-request behavior, footprint, steady state, or not at all under this workload?

If GraalVM Native Image is included as a separate slice, what changes when the Java application is built and measured as a native executable rather than as the default JVM run?

Did both implementations perform the same work-factor calculation and payload handling under the language-neutral service contract? If an equivalence issue is found and corrected, which corrected source revision is used for the main run?

Does adding focused service behavior, such as TLS, request logging, or a real dependency, change the interpretation? If a dependency slice is added, which part of the result belongs to the service runtime and which part belongs to the dependency path?

Which conclusions are safe inside the single-machine service-under-test boundary, and which remain out of scope?

Those are the questions worth answering. Not because they produce a more dramatic chart, but because they make the chart harder to misunderstand.

References and Methodology Notes

These references support the terminology and scope used in this kickoff article:

• Go net/http package documentation
• Go diagnostics overview
• Helidon documentation
• OpenJDK project site
• JDK builds from OpenJDK
• Oracle JDK downloads
• JEP 444: Virtual Threads
• Java virtual threads guide
• Project Leyden
• JEP 483: Ahead-of-Time Class Loading & Linking
• GraalVM Native Image documentation
• wrk2 fixed-rate HTTP load generator
• HdrHistogram
• JMH Java Microbenchmark Harness

Conclusion: Method First, Then Measurement

The most useful thing this first article can do is refuse to rush.

Go offers a standard-library HTTP server and a common single-binary deployment style. Modern Java offers Helidon SE as one focused service framework option, virtual threads for suitable blocking-concurrency workloads, extensive JDK diagnostics, multiple JDK build choices, and ongoing work around startup, warmup, and footprint.

None of that decides the benchmark.

The benchmark has to decide the benchmark, and it can only do that if the work is equivalent, the runtime shapes are explicit, the load generator is valid, the environment is captured, warmup and steady state are separated, repeats are handled consistently, and uncertainty is reported honestly.

That is the contract. Next we define the language-neutral service requirements so Go and Java are asked to do the same work before either implementation gets tuned.