Key Takeaways

• A benchmark does not become trustworthy when the load test ends. It becomes more trustworthy when raw artifacts are normalized, invalid runs stay visible, repeated runs are aggregated consistently, and uncertainty stays attached to the published comparison.
• This article defines the analysis contract for this benchmark series. Later articles use the same units, reporting rules, and chart constraints instead of inventing logic after seeing the numbers.
• We do not treat throughput, tail latency, CPU, memory, runtime diagnostics, startup, and warmup as one scoreboard. Each metric family answers a different engineering question, so we report them separately.
• The results article may compare variants within defined scenario and variant cells, but it may not declare an overall language winner. Exclusions, instability, and uncertainty remain visible wherever we make a comparison.

If you have ever read a benchmark post and thought, the chart is clean, but I still do not trust it, this is probably the missing layer.

Most benchmark arguments sound like methodology arguments, but a lot of the real trust problem shows up one step later, in analysis. The load test finishes. A directory fills up with raw outputs, logs, host metrics, and maybe runtime telemetry. Then someone has to decide what a “run” means, what gets excluded, how repeats get summarized, which latency numbers matter, and what kind of chart makes the comparison easiest to understand.

That is where a benchmark can drift from careful measurement into selective storytelling.

This article fixes the analysis contract before we publish results. Earlier entries in the series defined the run protocol, invalid-run logic, execution artifacts, and reproduction path. This article answers a narrower question: how do repeated benchmark runs become a published comparison that a skeptical engineer can inspect without guessing what got smoothed away?

The answer is disciplined rather than complicated. We commit now to the unit of analysis, the canonical dataset, the aggregation rules, the reporting rules, and the allowed plot types. We do that before the results article and before the interpretation article.

That matters because benchmark trust is not won by having more telemetry. It is won by making the path from raw artifacts to public claims easy to inspect.

We turn raw benchmark artifacts into reported comparisons through a fixed path: normalize each run, validate acceptance rules, aggregate by cell, surface exclusions and uncertainty, and only then choose allowed plots.

Why benchmark analysis methodology determines trust

It is easy to treat methodology as the part that happens before measurement: define the service, define the workload, validate the load generator, run the matrix, keep the host stable. All of that matters, and we have already spent several articles making it explicit.

But once the measurement phase is over, the benchmark still is not done. At that point, you have evidence, not a result. The result only appears after you answer analysis questions that are easy to wave away and surprisingly easy to bias.

What exactly is the unit we compare? One run, one average, one percentile, one scenario, one runtime variant, or one bundle of unlike metrics?

What happens when one repeat fails startup, another has incomplete telemetry, and six more look fine?

Do we show the best repeat, the mean of accepted repeats, the median, a tail-focused summary, or some mashup of all three?

Do we keep startup and steady state separate, or quietly blend them into one performance story?

Do we show throughput with uncertainty signals and latency as a selected p99, or do we carry uncertainty through both?

Those are not cosmetic questions. They determine how much trust the reader should place in the claim.

That is why we are fixing the analysis rules now instead of letting the results article discover them on the fly. Later in the series, if a chart appears, it should be possible to ask three immediate questions and get concrete answers:

• What exact cell does this chart summarize?
• How many repeats were attempted and accepted?
• What uncertainty or instability should stop me from overreading the difference?

If a chart cannot answer those questions, it does not belong in this series.

Benchmark terms we will use consistently

Before we talk about schema or aggregation, we need stable terms. These are series terms, not universal benchmark law, but we will use them consistently from here on.

A scenario is one workload definition. It captures payload size, concurrency target, workload mode, and the other settings that define what kind of test this is.

A variant is one runtime shape under test. Examples in this series include a Go service built on the standard-library net/http package and a Java service built with Helidon. These are benchmarked implementation shapes, not exhaustive representatives of either ecosystem.

A run is one executed measurement attempt for one scenario and one variant.

A repeat is one numbered run within a set of repeated attempts for the same scenario and variant combination. In practice, a repeat is usually just a run viewed in the context of its sibling attempts.

A cell is the combination of one scenario and one variant. This is the main unit of aggregation. We do not compare a loose pile of runs. We compare cells built from repeated runs under one fixed scenario and one fixed variant.

An accepted run is a run that passed the predeclared validation rules and is eligible for aggregation.

An invalid run is a run that failed those rules. Invalid does not mean invisible. It means excluded from the aggregated summaries and still reported with a reason class.

An unstable cell is a cell that meets the minimum bar for reporting but still triggers one or more warning conditions, such as higher-than-allowed repeat spread, missing metric slices, or incomplete evidence for one metric family.

Warmup is the interval where the service and runtime are getting into shape but we are not yet treating the behavior as steady measurement. For Java that can include JIT and class-loading effects; for Go it can still include allocator, scheduler, and cache stabilization. Either way, warmup is not the same thing as the main measurement window.

Steady state is the defined analysis window where service behavior is treated as the main comparable runtime state for throughput, latency, CPU, memory, and related metrics.

That vocabulary is intentionally boring. If those terms slide around between articles, the results slide around with them.

What the canonical benchmark dataset contains

A result chart should never be a pile of hand-entered summary numbers with the interesting mess stripped away. Our analysis starts from a canonical dataset that links the raw evidence to the derived metrics we actually publish.

The top-level shape is runtime-neutral. Go and Java both need to fit the same record model. Java-specific telemetry such as GC logs or Java Flight Recorder recordings can appear when captured, but they are additive fields, not proof that the schema belongs only to the JVM side.

At minimum, one normalized run record needs to connect scenario metadata, variant metadata, environment details, raw artifact references, analysis windows, derived metrics, and validation state.

Here is an intentionally schematic JSON example that shows the contract. Treat the field names, metric names, file paths, and IDs as illustrative only, not as the finalized benchmark schema.

{
  "scenario_id": "<scenario>",
  "variant_id": "<variant>",
  "run_id": "<timestamp>-<variant>-r03",
  "repeat_index": 3,
  "environment": {
    "host_id": "<bench-host>",
    "os": "linux"
  },
  "runtime": {
    "language": "<language>",
    "framework": "<framework>",
    "build_mode": "<mode>"
  },
  "load_profile": {
    "connections": 64,
    "target_rps": 40000,
    "warmup_s": 60,
    "measurement_s": 180
  },
  "artifacts": {
    "loadgen_output": "artifacts/<scenario>/<variant>/<repeat>/loadgen-output.json",
    "service_log": "artifacts/<scenario>/<variant>/<repeat>/service.log",
    "runtime_log": "artifacts/<scenario>/<variant>/<repeat>/runtime.log",
    "runtime_trace": "artifacts/<scenario>/<variant>/<repeat>/runtime-trace.dat"
  },
  "windows": {
    "startup": {
      "readiness_ms": 842
    },
    "warmup": {
      "start_s": 0,
      "end_s": 60
    },
    "steady_state": {
      "start_s": 60,
      "end_s": 240
    }
  },
  "derived_metrics": {
    "throughput_rps": 39520.4,
    "latency_p50_ms": 1.9,
    "latency_p95_ms": 4.8,
    "latency_p99_ms": 9.7,
    "error_rate_pct": 0.0,
    "cpu_utilization_pct": 87.4,
    "rss_mb": 612,
    "runtime_pause_total_ms": 128.0
  },
  "validation": {
    "accepted": true,
    "exclusion_reasons": []
  }
}

The important thing here is not the exact spelling of the fields. It is the separation of concerns.

The record points to raw files instead of pretending the summaries appeared by magic. It separates startup, warmup, and steady-state windows so we do not flatten them together later, and those windows come from the predefined run protocol rather than being tuned after the numbers are in. It keeps derived metrics distinct from artifact references. And it carries validation state inside the same normalized object instead of hiding exclusions in a spreadsheet note.

That structure is especially useful on the Java side. The Java implementation in this series is one specific framework and runtime shape, not “Java” in the abstract. On HotSpot-based runs, the artifact set may also include service logs, GC logs, and Java Flight Recorder data where configured. Those artifacts are useful, but only if they are tied to a run ID and normalized into the same record model as the Go-side evidence rather than used as a separate storytelling layer.

The example record uses Linux as the host OS label, but the actual results package should identify the tested host environment with more precision. These are single-machine results, and kernel, distribution, CPU settings, and other host details can materially affect behavior.

Our analysis contract requires a manifest-style mapping from archived files to normalized inputs so the path from source artifact to reported metric stays inspectable.

A compact directory sketch can be as simple as this:

artifacts/
  <scenario>/
    <variant>/
      <repeat>/
        manifest.json
        loadgen-output.json
        service.log
        runtime-log-or-trace.dat

And a matching manifest might look like this at a conceptual level. The keys and filenames below are illustrative placeholders, not verified repository literals:

{
  "run_id": "<timestamp>-<variant>-r03",
  "scenario_id": "<scenario>",
  "variant_id": "<variant>",
  "raw_artifacts": {
    "loadgen_output": "loadgen-output.json",
    "service_log": "service.log",
    "runtime_log": "runtime.log",
    "runtime_trace": "runtime-trace.dat"
  },
  "normalization_inputs": {
    "throughput_source": "loadgen_output",
    "latency_source": "loadgen_output",
    "startup_source": "service_log",
    "runtime_diagnostics_source": "runtime_trace"
  }
}

This example shows the minimum traceability we want: raw artifacts are archived, the mapping to normalized metrics is explicit, and optional Java telemetry is supported without making the whole dataset JVM-specific.

Our canonical dataset keeps one repeated run inspectable end to end by linking scenario and variant metadata, environment details, raw artifacts, analysis windows, derived metrics, and validation status in one normalized record.

How repeated runs become reported comparisons

This is the point where many benchmark posts quietly switch from measurement to rhetoric.

Our aggregation path is strict:

run -> cell -> scenario and variant comparison

That means one run never becomes a headline by itself. A run belongs to a repeat set. Accepted repeats are aggregated into a cell. Cells are compared only within the same scenario boundary.

So if you are looking at one workload scenario, the comparison is between variants within that scenario. We are not allowed to blend unlike scenarios into one runtime score. That would answer no coherent engineering question.

We also summarize metric families separately.

Startup and readiness stay separate because they answer, “How quickly can this process become usable?” In this series, startup and readiness are only comparable when they come from the same per-run lifecycle defined in the benchmark protocol. We do not mix fresh-process startup measurements with reused-process steady-state repeats and call them one startup story.

Warmup behavior stays separate because it answers, “How quickly does this runtime shape settle?” Steady-state throughput stays separate because it answers, “How much work is delivered once the run reaches its main measurement phase?” Latency percentiles stay separate because tail behavior is not interchangeable with throughput. CPU efficiency and memory footprint are separate again because resource cost is part of the runtime shape, not background noise. Runtime diagnostics such as GC logs or Java Flight Recorder traces stay contextual and explanatory rather than turning into a hidden deciding factor.

That leads to one rule worth stating plainly:

We do not treat throughput, tail latency, CPU, memory, runtime diagnostics, and startup as one scoreboard.

In practice, that means later figures and captions should compare like with like: throughput against throughput, latency against latency, startup against startup, and resource-cost metrics in their own views.

Why be so strict? Because composite benchmark scoring hides tradeoffs the reader actually cares about. A variant can start faster and still have weaker steady-state throughput. Another can deliver stronger throughput and still consume more CPU per unit of work. A third can look strong on average and still have a wider p99 spread. Those are not contradictions. They are different dimensions of runtime behavior.

For the Java side, this matters even more if multiple Java runtime shapes appear. The Helidon-based benchmark variant is one Java framework shape, not “Java” as a whole. If the series later includes another JDK-based workflow with different startup or warmup behavior, that remains its own variant. The analysis contract forbids us from pooling unlike runtime modes into one Java bucket and then talking as if the combined number represents the platform fairly.

In practical terms, a reported comparison in the results article might say that one variant shows higher steady-state throughput in a given scenario while another reaches readiness sooner or stabilizes earlier in warmup. That is allowed. Saying one of them “wins overall” is not.

How we handle invalid runs, excluded repeats, and unstable cells

A benchmark result is not just a number. It is also an accounting of how much evidence survived long enough to support that number.

Every reported cell summary needs to make four things visible:

• total repeats attempted
• repeats accepted
• repeats excluded by reason class
• a central value plus a visible uncertainty or instability signal for the reported metrics

If a cell is unstable, that needs to be visible too. If telemetry is incomplete for one metric family, that should show up as a missing slice, not vanish into silence. That does not always mean the whole run is invalid. Missing optional telemetry may leave a run usable for core throughput and latency reporting while making it unavailable for a diagnostic metric family. Missing required comparison artifacts is a different case and can invalidate the run entirely under the series acceptance rules.

Here is an illustrative cell-summary example showing the reporting shape we expect. The interval fields here show reporting shape only; they do not prescribe a specific statistical method by themselves.

{
  "cell_id": "<scenario>/<variant>",
  "runs_attempted": 8,
  "runs_accepted": 6,
  "runs_excluded": 2,
  "exclusion_reason_counts": {
    "startup_failure": 1,
    "artifact_incomplete": 1
  },
  "reported_metrics": {
    "throughput_rps_median": 39480.0,
    "throughput_rps_interval": [38990.0, 39910.0],
    "latency_p99_ms_median": 9.9,
    "latency_p99_ms_interval": [9.1, 11.4],
    "rss_mb_median": 608,
    "rss_mb_interval": [601, 620]
  },
  "stability_flags": [
    "moderate_tail_variance"
  ],
  "notes": [
    "steady-state metrics computed only from accepted runs",
    "excluded runs remain visible in the published summary"
  ]
}

This is a small example, but it carries the full contract.

First, it tells you how many times we tried. That matters because “median of six accepted runs” means something different when it came from six attempts versus eight attempts with two exclusions.

Second, it tells you what got excluded and why. A startup failure and an incomplete artifact are not the same problem, and they should not disappear behind a cleaned-up summary.

Third, it reports uncertainty signals alongside central values. We are not going to publish a tiny difference in throughput or p99 and pretend the reader should treat that as settled truth without any variation context. Repeated benchmark runs vary, and the summary needs to make that visible.

Fourth, it can flag instability even when the cell is still worth showing. An unstable cell is not automatically worthless. It is a sign that the claim attached to it needs to stay narrow.

Here is how we want a human reader to interpret that example:

Attempted eight repeats. Accepted six. Excluded two, and the reasons are visible. The reported throughput and p99 come only from accepted runs. Tail behavior still varies enough that we should not use this cell as the basis for sweeping claims.

There is also a small-sample rule underneath all of this. This article fixes the reporting shape, not the full statistical appendix, but later results still need to meet a minimum bar: if a metric is compared, the chart or adjacent caption must show accepted-repeat counts and some visible uncertainty or instability signal for that metric family. If accepted evidence is too thin for a normal cell summary, we mark the cell as insufficient evidence instead of presenting it as a clean head-to-head result. If a cell does not meet the series minimum for comparability, it may still be shown as incomplete evidence, but not as a claimed winner.

This article keeps invalid runs, accepted counts, and uncertainty visible, but load-generator validity questions such as coordinated-omission handling must be documented separately in the series methodology and tool configuration.

A reported cell summary shows how many repeats we attempted, how many were accepted, why any were excluded, which metrics are summarized, and whether instability limits strong claims.

Which uncertainty signals we report and why

Uncertainty is part of the result, not a footnote.

A point estimate by itself is easy to quote and easy to overread. That is exactly why later articles need more than a single median, average, or percentile value in isolation. Repeated runs vary even when the benchmark protocol is stable, so the published comparison has to carry some visible signal of that variation.

Our reporting rule is simple:

• each compared metric family needs a central summary value
• each compared metric family also needs a visible uncertainty or instability signal
• accepted-repeat counts must stay close to the reported value
• excluded-repeat counts and reasons remain visible at the cell level

That does not mean every figure turns into a statistics lecture. It means the reader should be able to tell whether a small gap is supported by stable repeated evidence or sits inside a noisy range that weakens the claim.

It also means we are careful with what interval language implies. An interval estimate is not proof. It is a compact way to show that repeated measurements vary and that narrow or wide spread changes how strongly we should read the comparison.

This article fixes the reporting shape, not the exact interval formula. The results article needs to name one series-wide method and apply it consistently across compared cells. The same applies to instability thresholds: they need one published rule for the series, not case-by-case judgment after the numbers are visible.

The practical effect is straightforward: later result claims can be stronger when accepted repeats are stable and uncertainty is tight, and narrower when the cell remains noisy or partially incomplete.

Which plots are allowed for each metric family

Plot choice is part of the method. It is not decoration added after the data already “speaks for itself.”

Different metric families need different chart shapes because they answer different questions.

Throughput comparisons work well as bars or points with visible uncertainty because the reader needs a direct side-by-side view and a clear sense of whether small gaps are likely meaningful.

Latency tails need tail-aware views. Percentile lines or tail-focused point comparisons are appropriate because p95 and p99 behavior should stay visible instead of being flattened into one average-latency story.

Startup and readiness need timeline or phase-duration views. A startup chart should not sit on the same visual axis as steady-state throughput because those are unlike quantities answering unlike questions.

Warmup behavior needs time-series views. That is the honest way to show convergence, early instability, or time-to-peak differences.

CPU efficiency is better served by scatter or normalized views that pair delivered work with CPU cost. Raw CPU percentages alone are often too context-poor.

Memory footprint works as bars or points with intervals, as long as it is not mixed into a composite throughput-latency-resource scoreboard.

Just as important, some chart patterns are off-limits in this series because they mislead more than they clarify.

We will not use stacked bars that imply throughput, latency, CPU, and memory add up to one thing. They do not.

We will not use mixed-axis mashups that visually dramatize one metric by scaling another into the background.

We will not publish a single scoreboard that combines startup, p99 latency, throughput, CPU, memory, and runtime diagnostics into one rank.

And we will not hide exclusions or uncertainty in figure notes while putting the cleanest-looking number in the chart title.

We choose plot types by metric family so readers can inspect throughput, tail latency, startup, warmup, CPU efficiency, and memory behavior without collapsing unlike measures into one score.

What later articles will and will not be allowed to claim

By this point, the contract should feel narrow enough to be useful.

The results article may compare variants within defined scenario and cell boundaries. It may say that one variant showed higher steady-state throughput for a given scenario. It may say another reached readiness sooner. It may say a variant’s p99 interval was tighter or that a cell remained unstable enough to weaken the claim. It may use Java runtime telemetry as explanatory context when the artifact trail supports it. For example, GC logs or Java Flight Recorder data can help explain whether tail disturbances align with runtime events instead of leaving the reader with a vague runtime story.

But later articles may not do several tempting things.

They may not declare an overall language winner.

They may not merge startup, throughput, tail latency, CPU, and memory into one ranking.

They may not hide invalid or excluded runs just because the accepted set tells a cleaner story.

They may not generalize from the benchmarked Java framework variant to all Java frameworks, or from a Go service built on net/http to Go as a whole.

They may not treat separately benchmarked JDK distributions as interchangeable just because they are closely related upstream. Even when two JDK distributions are derived from OpenJDK source, separately benchmarked distributions remain separate variants for reporting.

They may not treat a JVM run, a native image, a checkpoint-and-restore workflow, or any other materially different runtime path as the same thing just because they come from related tooling or source bases.

A short way to remember the rule is this:

Allowed claim: “Within this scenario, variant A shows higher steady-state throughput, while variant B reaches readiness sooner and shows a tighter p99 interval.”

Disallowed claim: “Variant A wins overall.”

That line may sound strict, but it is the difference between a benchmark that helps engineering decisions and one that mostly fuels argument.

What changes after this article

Up to now, the series established the benchmark scope, the service contract, the implementation shapes, the load-generation logic, the run protocol, and the reproduction path. What changes here is that the analysis rules are now fixed.

That matters because once results appear, readers should not have to reverse-engineer the logic behind each figure. You should already know what a cell is, what the canonical dataset carries, how accepted runs are aggregated, what exclusions look like, why startup and steady state stay separate, and why no one score is coming to rescue a complicated comparison.

You should also know what this article does not settle. Keeping exclusions and uncertainty visible improves the honesty of the reporting, but it is not a substitute for tool-specific load-generator behavior, runtime-specific diagnostics, or careful host-environment disclosure. Those still need to appear where the benchmark evidence is published.

If you are the kind of reader who distrusts benchmark posts by default, this is the article that should make the next one easier to read. Not because it proves any runtime is better, but because it limits what we are allowed to claim when the numbers arrive.

So when you read the results article, read it with this contract in mind. Ask whether every chart stays inside a scenario boundary. Ask whether exclusions and accepted counts are visible. Ask whether uncertainty is shown where the article wants you to care about a difference. And ask whether the interpretation respects the fact that startup, warmup, throughput, latency, CPU, memory, and runtime telemetry are related signals, not one giant score.

That is the discipline this series needs if the conclusions are going to be worth anything.

Conclusion

Benchmark trust is easy to lose because the final chart compresses a lot of judgment into one visual. This article makes those judgments explicit before the results article gets a chance to hide them.

Our analysis contract is simple to state even if it takes work to follow. We normalize each run into a canonical record. We keep raw artifacts linked to derived metrics. We validate runs and keep excluded ones visible by reason class. We aggregate accepted repeats into scenario and variant cells. We report metric families separately. We show uncertainty and instability instead of pretending they are background noise. And we forbid composite-score storytelling.

That does not make the later results automatically correct. It does make them much harder to overclaim.

In the next article, we apply this contract to the actual benchmark outputs. The important part is that the rules are already on the page.

References

• Gil Tene, How NOT to Measure Latency and related coordinated-omission materials from HdrHistogram: https://hdrhistogram.github.io/HdrHistogram/
• HdrHistogram project: https://github.com/HdrHistogram/HdrHistogram
• Georges, Buytaert, and Eeckhout, Statistically Rigorous Java Performance Evaluation, USENIX OOPSLA: https://www.usenix.org/conference/oopsla13/statistically-rigorous-java-performance-evaluation
• Neil J. Gunther, Performance Matters, ACM Queue: https://queue.acm.org/detail.cfm?id=2855183
• Java Flight Recorder documentation: https://docs.oracle.com/en/java/javase/25/troubleshoot/troubleshoot-performance-issues-using-jfr.html
• Helidon project home: https://helidon.io/
• Go net/http package documentation: https://pkg.go.dev/net/http
• Oracle JDK FAQ: https://www.oracle.com/java/technologies/javase/jdk-faqs.html
• GraalVM Native Image documentation: https://www.graalvm.org/latest/reference-manual/native-image/

Key Takeaways

• A benchmark result is only useful if another engineer can inspect the source package, rerun the command path, and review the generated artifacts. Reproducibility comes before interpretation.

• This package treats the Go net/http service and the Java Helidon SE service symmetrically at the harness level. Both are built from source through package scripts, validated against the same public contract, and expected to write evidence under the same artifact tree.

• The run manifest is the main handoff artifact for this package. It connects toolchain versions, service configuration, validation status, matrix configuration, and generated file locations so later analysis can trace supported claims back to the run artifacts that support them.

• This article covers benchmark execution and reproduction. It does not compare latency, throughput, resource use, language superiority, or Oracle AI Database performance.

  
  

Service-runtime choices matter when an HTTP endpoint sits in front of a model call, retrieval step, vector search request, enrichment pipeline, or another application workload. Teams naturally ask whether Go, Java, framework choice, JDK choice, warmup, or runtime configuration will change the shape of that service under load.

Before any performance chart deserves attention, though, the artifact behind it has to be runnable.

That is the job of this article.

In this part of the single-machine benchmark series, we move from design to execution. Earlier articles defined the benchmark goals, service contract, Go implementation shape, Java Helidon SE implementation shape, load-generator design, and run methodology. Here we start from the benchmark source package, run the prerequisite and validation gates, build both services, run the benchmark matrix, and inspect the artifacts that later articles will analyze.

This is not the results article. It is the reproduction article.

This reproduction path is scoped to a local Linux environment using Bash and the package scripts. Other environments, including macOS, Windows, WSL, containers, and CI runners, may work only where the package explicitly documents and validates them.

Version basis: the command path and sample output reflect the package layout used for this article, so script wording, manifest field names, artifact names, and dependency versions can vary by package release.

What this article helps you reproduce

The benchmark source package is the unit of trust. Not a chart. Not a slide. Not an isolated binary. The package includes the Go service source, the Java Helidon SE service source, shared contracts, fixtures, scripts, package-local tooling, manifest templates, and artifact directories. The reproduction run writes generated evidence under artifacts/.

In this walkthrough, we use the package-root command path to show:

• Where the source files, contracts, fixtures, and scripts live.
• How to verify the local Linux/Bash environment before building anything.
• How to capture toolchain and host facts.
• How to validate the shared service contract.
• How to build both services from source.
• How to run smoke checks on loopback.
• How to execute the benchmark matrix without publishing benchmark numbers.
• How to inspect the run manifest and generated artifacts.

The package root is the top-level directory of the materialized benchmark source package. In this package, it is the directory containing scripts/, contracts/, fixtures/, services/, tools/, manifests/, and artifacts/.

Run every command in this article from the package root unless a section says otherwise. The validation scripts load contract files relative to that root, and the benchmark scripts write generated evidence into the package’s artifacts/ tree.

If a command is run from the wrong directory, the scripts fail fast rather than silently loading a different contract or writing artifacts somewhere unexpected. That is deliberate. The working directory is part of the reproduction contract.

We reproduce the benchmark by moving from the source package through prerequisite checks, environment capture, validation, builds, smoke checks, matrix execution, artifact capture, and manifest review before interpreting any results.

What this article does not claim

This article does not compare Go and Java latency. It does not compare throughput. It does not compare CPU use, memory use, garbage collection behavior, JDK distributions, or deployment cost. It does not say one language is better than the other.

It also does not measure Oracle AI Database performance. This package validates the benchmark execution harness. If another benchmark slice adds an Oracle AI Database dependency, that slice needs its own driver configuration, credential handling, manifest fields, validation gates, and scope statement.

For this article, a good run is not a fast run. A good run is one where the package builds, validates, executes, and captures evidence without skipping a gate.

Source package layout

Start by orienting yourself in the package. The first command is not a build. It is a quick check that you are standing in the right place.

pwd
find . -maxdepth 2 -type d | sort

The top-level shape should look like this:

.
./artifacts
./artifacts/benchmark
./artifacts/env
./artifacts/logs
./artifacts/manifests
./artifacts/validation
./contracts
./docs
./fixtures
./fixtures/errors
./fixtures/valid
./manifests
./scripts
./services
./services/go-http
./services/java-helidon
./tools
./tools/benchctl

The Go implementation is intended to use Go’s standard-library net/http package, with its service entry point under services/go-http/cmd/server/.

The Java service lives under services/java-helidon/. It is a Helidon SE service, with its Helidon version pinned in services/java-helidon/pom.xml. The Java build packages the service JAR and runs it with the classpath resolved by the package’s Maven build.

The shared service contract lives at contracts/work-contract.json. The valid and error fixtures live under fixtures/valid/ and fixtures/errors/. The package-local benchmark and contract tooling lives under tools/benchctl/, so the orchestration path is inspectable as source.

Generated evidence goes under artifacts/. That includes environment facts, validation output, service logs, benchmark matrix output, and manifests.

The split matters. Source inputs and generated evidence need to be easy to separate. Later analysis should be able to connect a supported claim to a manifest, validation output, generated file, or log instead of relying on memory or console scrollback.

Step 1: Verify prerequisites

Run the prerequisite check before you build or launch either service.

./scripts/check-prereqs.sh

A successful run should include these categories of checks and end with a PASS result. Exact log wording can vary by package version:

[check-prereqs] package root: PASS
[check-prereqs] Go toolchain: PASS
[check-prereqs] Java runtime: PASS
[check-prereqs] Java compiler: PASS
[check-prereqs] Maven command: PASS
[check-prereqs] required scripts: PASS
[check-prereqs] contracts directory: PASS
[check-prereqs] artifacts writable: PASS
[check-prereqs] result: PASS

This step checks that the expected commands and paths are visible from the package root. It looks for tools such as Bash, Go, java, javac, Maven, curl, and common Linux utilities used by the scripts. It also verifies required package paths such as contracts/work-contract.json, fixtures/, services/go-http/, services/java-helidon/, and tools/benchctl/.

The prerequisite check is a preflight gate, not a full build. If the package does not vendor all dependencies, the Java build needs access to the configured Maven repositories or a pre-populated local Maven cache, and the Go build needs access to any required Go modules or a pre-populated Go module cache. Dependency-resolution problems usually become visible at the later build gates.

A passing prerequisite check means the package can start the reproduction path. It does not mean the benchmark is valid, and it does not mean the Java and Go services are behaviorally comparable. Those claims require later gates.

Helidon 4 requires Java 21 or later. The prerequisite check records the Java commands it finds; the Java build and runtime validation gates still have to pass.

Step 2: Capture environment facts

Benchmarks become hard to trust when the environment is described from memory. The package records environment facts before the benchmark matrix runs.

./scripts/capture-env.sh

A successful run writes environment and manifest files and ends with a PASS result. The output should include files in these categories:

[capture-env] wrote artifacts/env/go-version.txt
[capture-env] wrote artifacts/env/java-version.txt
[capture-env] wrote artifacts/env/javac-version.txt
[capture-env] wrote artifacts/env/os-release.txt
[capture-env] wrote artifacts/manifests/environment-manifest.json
[capture-env] result: PASS

Now inspect the files that were created:

find artifacts -maxdepth 3 -type f | sort

At this point you should see environment files like these:

artifacts/env/architecture.txt
artifacts/env/cpu.txt
artifacts/env/go-version.txt
artifacts/env/java-version.txt
artifacts/env/javac-version.txt
artifacts/env/kernel.txt
artifacts/env/maven-version.txt
artifacts/env/memory.txt
artifacts/env/os-release.txt
artifacts/manifests/environment-manifest.json

The environment manifest records toolchain and host facts used by a specific benchmark execution. It does not try to be a complete machine inventory. It records the facts that matter to reproduction: Go version, Java runtime version, Java compiler version, Maven version, OS and kernel details, architecture, CPU summary, memory summary, package version, and the package-local tool version when that tool has already been built.

Record the exact JDK distribution, version, and build string from java -version, not only the Java feature version. JDK distribution and build details are benchmark inputs, just like the Go version, framework configuration, warmup settings, and JVM arguments supplied through JAVA_SERVICE_JVM_ARGS.

That is enough for the next engineer to ask useful questions. Did the run use a different JDK? Was the Go version different? Did the architecture change? Were JVM arguments supplied? Those are not trivia. They are benchmark inputs.

Environment capture explains where the benchmark ran. It does not make a single-machine run representative of every deployment shape.

Step 3: Validate the shared contract

Before we build confidence in any measurement, both services have to satisfy the same behavioral contract.

The contract defines GET /health, HEAD /health, and POST /work. The /work endpoint accepts application/json with four required fields: requestId, payloadSize, seed, and extraWork. Validation is intentionally strict. The decoded request body is limited to 4096 bytes, requestId must be a string with length 1..128, payloadSize must be an integer in the range 0..131072, seed must be an integer in the range 0..2147483647, and extraWork must be an integer in the range 0..100.

Successful /work responses return requestId, payloadSize, extraWork, checksum, and ok. The validators also reject unknown fields, duplicate fields, missing required fields, scalar type violations, quoted numeric strings, numeric bound violations, invalid JSON, multiple JSON documents, and decoded bodies larger than 4096 bytes.

Run the contract wrapper from the package root:

./scripts/validate-contract.sh

A successful run should show the Go service checks, the Java Helidon SE service checks, and the equivalence cases all passing:

[validate-contract] loaded shared contract fixtures
[validate-contract] Go service contract checks: PASS
[validate-contract] Java Helidon SE service contract checks: PASS
[validate-contract] error response equivalence: PASS
[equivalence] case work-small: PASS
[equivalence] case work-medium: PASS
[equivalence] case work-maximum: PASS
[equivalence] case missing-request-id: PASS
[equivalence] case payload-size-too-large: PASS
[equivalence] case duplicate-field: PASS
[equivalence] comparing success responses: PASS
[equivalence] comparing error responses: PASS
[equivalence] result: PASS
[validate-contract] complete shared validation: PASS

This wrapper does more than inspect source files. It runs the package’s unit and tool checks, builds the services as needed, starts both services on loopback, and uses the package-local contracttest command to compare responses across the shared fixtures.

That is the important part: equivalence is tested against the package contract, not assumed.

The Go and Java implementations are allowed to be different internally. They use different languages, different runtime models, and different libraries. But before the benchmark matrix runs, they need to agree on the external behavior defined by the contract.

Because this validation starts live services on loopback, make sure the package’s validation ports are available before running it. If validation fails with a bind or connection error, stop any stale service process first, then rerun the validation command. Use alternate port variables only where the package scripts explicitly document support for them.

If this step fails, stop. Fix the implementation, fixture, or environment problem first. A downstream benchmark run after a failed contract gate is not comparable evidence.

Step 4: Build the Go and Java services

Now build both benchmark service implementations from the package source.

Start with Go:

./scripts/build-go.sh

A successful Go build runs the configured checks and writes the service binary:

[build-go] checking module
[build-go] building service
[build-go] wrote bin/go-http-server
[build-go] result: PASS

The point is not that Go can produce a binary. The point is that this package does not ask you to trust a prebuilt benchmark service. The service used by the benchmark is built through the same package-root path every time.

Now build the Java Helidon SE service:

./scripts/build-java.sh

A successful Java build resolves dependencies, compiles the service, runs tests, and packages the JAR:

[build-java] resolving Java build
[validate-java] resolving Java build
[validate-java] compiling main sources
[validate-java] compiling tests
[validate-java] running tests
[validate-java] packaging Java Helidon SE service
[validate-java] wrote services/java-helidon/target/java-helidon-benchmark.jar
[validate-java] result: PASS
[build-java] wrote services/java-helidon/target/java-helidon-benchmark.jar
[build-java] result: PASS

The Java build resolves the Helidon dependencies, compiles the service, compiles and runs the Java tests, and packages services/java-helidon/target/java-helidon-benchmark.jar.

One naming detail matters here: this is the Java Helidon SE implementation used in this benchmark package. It is not a claim about every Java framework or every JVM configuration. A Spring Boot service, Quarkus service, Micronaut service, GraalVM Native Image build, or a different Helidon configuration would be a different benchmark artifact.

This article’s command path runs the Java Helidon SE service on the configured JDK. It does not invoke a GraalVM Native Image build. If a package version adds Native Image support, that should be treated as a separate benchmark artifact and recorded separately.

Step 5: Start the services and run smoke checks

Build success is not enough. We also want a fast operational check that both services can start, bind to loopback, answer /health, answer a basic /work request, write logs, and stop cleanly.

Run the smoke script:

./scripts/run-smoke.sh

By default, the script uses 127.0.0.1:18180 for the Go service and 127.0.0.1:18181 for the Java Helidon SE service. You can override those with GO_SERVICE_PORT and JAVA_SERVICE_PORT if your local machine already has something on those ports.

A successful smoke run should show both services becoming ready, passing a request check, and stopping cleanly:

[smoke-go] ready: http://127.0.0.1:18180/health
[smoke-go] request: PASS
[smoke-java] ready: http://127.0.0.1:18181/health
[smoke-java] request: PASS
[smoke-go] stopped
[smoke-java] stopped
[smoke] result: PASS

If you need alternate ports for the smoke check, pass them explicitly:

GO_SERVICE_PORT=19180 JAVA_SERVICE_PORT=19181 ./scripts/run-smoke.sh

Smoke checks are not load tests. They are a safety gate before the benchmark matrix runs.

The logs from this step are written under artifacts/logs/, and the smoke validation file is written under artifacts/validation/smoke-validation.json. Those files matter because a later failed artifact verification should tell you whether the problem happened before, during, or after service launch.

Step 6: Run the benchmark matrix

The benchmark matrix is the configured set of benchmark executions for this package. It defines which service runs, which workload settings apply, and which artifacts the runner should capture.

Now run the matrix command, but keep the scope clear: this article still does not publish benchmark results.

./scripts/run-matrix.sh

A successful matrix run should show the matrix loading, each service being prepared and captured, the manifest being generated, and the final result passing:

[run-matrix] loaded benchmark matrix
[run-matrix] prepared Go service run
[run-matrix] captured Go service artifacts
[run-matrix] prepared Java Helidon SE service run
[run-matrix] captured Java Helidon SE service artifacts
[generate-manifest] wrote artifacts/manifests/run-manifest.json
[generate-manifest] result: PASS
[run-matrix] wrote run manifest
[run-matrix] result: PASS

The default package matrix is intentionally small. Its job in this article is to prove orchestration and artifact capture: start one service, run the package-local benchctl tool against it, capture the output, stop it, repeat symmetrically for the other service, and write a run manifest.

benchctl is package-local orchestration tooling for this demo package. It is not being presented here as a general-purpose load-testing standard or as a substitute for a production benchmark methodology. This article does not claim that benchctl avoids coordinated omission or is suitable for production-grade latency analysis; those properties would need to be documented and validated before using its output for latency conclusions.

This local matrix is useful for proving orchestration and artifact capture. It is not evidence that the same behavior will appear on another host, container runtime, kernel configuration, cloud shape, or production deployment.

That design keeps this article focused. The matrix step produces data for later analysis. Here we prove that the run can be executed and captured; we do not interpret the generated files as performance results.

If you need the matrix run to use nondefault service ports, pass the port variables to the matrix command as well:

GO_SERVICE_PORT=19180 JAVA_SERVICE_PORT=19181 ./scripts/run-matrix.sh

A partial matrix run is not comparable benchmark evidence. If the Go side completes and the Java side fails, or if the matrix runs but the manifest is missing, you do not have a valid run. You have a failed run with useful diagnostic artifacts.

Step 7: Inspect generated artifacts

After the matrix runs, verify the expected artifact set:

./scripts/verify-artifacts.sh

The verifier should report each required artifact category as present:

[verify-artifacts] environment manifest: PASS
[verify-artifacts] run manifest: PASS
[verify-artifacts] contract validation output: PASS
[verify-artifacts] Go build output: PASS
[verify-artifacts] Java build output: PASS
[verify-artifacts] smoke-test logs: PASS
[verify-artifacts] benchmark matrix artifacts: PASS
[verify-artifacts] result: PASS

Now list the generated files:

find artifacts -maxdepth 3 -type f | sort

After a successful run, the artifact tree should include files like these, with the generated run manifest acting as the source of truth for the exact artifact set:

artifacts/benchmark/go/go-summary.json
artifacts/benchmark/java/javaHelidon-summary.json
artifacts/env/go-version.txt
artifacts/env/java-version.txt
artifacts/env/javac-version.txt
artifacts/logs/go-build-output.txt
artifacts/logs/go-service.log
artifacts/logs/java-build-output.txt
artifacts/logs/java-service.log
artifacts/manifests/environment-manifest.json
artifacts/manifests/run-manifest.json
artifacts/validation/contract-validation.json
artifacts/validation/equivalence-validation.json
artifacts/validation/smoke-validation.json

The package separates source inputs from generated evidence so later analysis can trace supported claims back to contracts, scripts, validation output, logs, benchmark files, and manifests.

The first artifact to inspect is the run manifest.

cat artifacts/manifests/run-manifest.json

The run manifest is the benchmark execution record. It is intended to connect the package version, service configuration, toolchain facts, matrix configuration, and artifact locations for a specific run.

The exact manifest fields are defined by the package release you run. For this package, the generated manifest should describe the run identity, package version, service configuration, toolchain facts, matrix configuration, and artifact locations.

A shortened manifest excerpt has this general shape:

{
  "runId": "run identifier generated by the package",
  "package": {
    "version": "package version recorded during the run",
    "root": "package root recorded during the run"
  },
  "services": {
    "go": {
      "runtime": "Go version recorded during the run",
      "binary": "bin/go-http-server",
      "port": 18180
    },
    "javaHelidon": {
      "helidonVersion": "Helidon version recorded during the run",
      "jdk": "Java runtime recorded during the run",
      "javac": "Java compiler recorded during the run",
      "maven": "Maven version recorded during the run",
      "jar": "services/java-helidon/target/java-helidon-benchmark.jar",
      "jvmArgs": "JVM arguments recorded during the run",
      "port": 18181
    }
  },
  "matrix": {
    "configuration": "manifests/benchmark-template.json"
  },
  "artifacts": {
    "root": "artifacts",
    "environmentManifest": "artifacts/manifests/environment-manifest.json",
    "runManifest": "artifacts/manifests/run-manifest.json",
    "validation": "artifacts/validation",
    "benchmark": "artifacts/benchmark",
    "logs": "artifacts/logs"
  }
}

Use the generated manifest, not this abbreviated excerpt, as the source of truth for later analysis.

The manifest is the bridge between reproduction and analysis. It tells later articles which package version ran, which Go and Java runtimes were recorded, which service ports were used, where the matrix configuration came from, and where to find the validation and benchmark artifacts.

It does not say which implementation is faster. It tells you how the run was produced.

What counts as a valid run

A valid run is a benchmark execution where every required gate completes and the generated artifacts match the run manifest.

For this package, a valid run means:

• The prerequisite check passes.
• Environment facts are captured.
• Shared contract validation passes.
• Live Go-versus-Java equivalence validation passes.
• Both services build from the package source.
• Smoke checks pass for both services.
• The benchmark matrix completes for both services.
• Artifact verification passes.
• The environment and run manifests exist.
• The manifest paths match the generated artifact tree.

That definition may sound strict, but it prevents a common benchmark failure mode: treating partial output as if it were comparable evidence.

If a build fails, you do not have a benchmark. If contract validation fails, you do not have a fair comparison against this package contract. If the matrix runs but the manifest is missing, you do not have an auditable run. If artifact verification fails, later analysis should not use that run.

What to rerun when validation fails

When a gate fails, stop at the failed gate. Do not keep running commands in the hope that later output will be useful.

If the package root is wrong, go back to the directory containing scripts/, contracts/, fixtures/, services/, and tools/, then rerun the prerequisite check:

./scripts/check-prereqs.sh

If environment capture fails, fix the writable artifact path or missing tool problem, then rerun:

./scripts/capture-env.sh

If contract validation fails, fix the implementation or fixture mismatch, then rerun:

./scripts/validate-contract.sh

If smoke or matrix execution fails because generated artifacts from an earlier attempt are confusing the run boundary, the package includes a cleanup script for generated run artifacts:

./scripts/clean-run.sh

A successful cleanup prints output in this shape:

[clean-run] removed generated run artifacts
[clean-run] clean boundary ready

This cleanup step removes generated run evidence. It is not a process manager. If a service is still running or a loopback port is occupied, stop that process or choose documented alternate ports before rerunning the smoke or matrix command.

Then restart from a clean boundary:

./scripts/check-prereqs.sh
./scripts/capture-env.sh
./scripts/validate-contract.sh

The important rule is simple: do not merge artifacts from failed and successful attempts. Keep each run’s evidence coherent, or discard it and rerun.

How this connects to later analysis

This reproduction run produces the evidence path that the remaining articles need.

The next article explains how generated data can be normalized, aggregated, modeled, and visualized. The results article presents benchmark results from valid runs. The final interpretation article discusses what those results mean for runtime choice, service design, and operational tradeoffs.

The order matters. Without this reproduction step, later charts would ask readers to trust an invisible process. With it, the process is inspectable: source, scripts, contracts, manifests, logs, validation output, and benchmark artifacts all live in the package.

That is how a benchmark earns the right to be discussed.

References

• Helidon Documentation — https://helidon.io/docs/
• Helidon SE Introduction — https://helidon.io/docs/v4/se/introduction
• Helidon 4 Introduction — https://helidon.io/docs/v4/about/intro
• Go net/http Package Documentation — https://pkg.go.dev/net/http
• Java Documentation — https://docs.oracle.com/en/java/
• GraalVM Native Image Documentation — https://www.graalvm.org/latest/reference-manual/native-image/

Conclusion

A reproducible benchmark starts before the first chart. It starts with a source package that another engineer can inspect, a command path that can be rerun, validation gates that fail loudly, and artifacts that connect later analysis back to the run that produced them.

In this article, we reproduced the Go net/http and Java Helidon SE benchmark environment from the package root. We checked prerequisites, captured environment facts, validated the shared contract, built both benchmark service implementations from package source, ran smoke checks, executed the matrix, and inspected the generated manifests and artifacts.

That gives the rest of the series a stable foundation. The next step is not to declare a winner. The next step is to use the generated evidence carefully, so any later result can be traced back to a valid, auditable run.

Key Takeaways

• This article defines the measurement contract before any result charts appear. Later comparisons only count if another engineer can inspect the setup, challenge the controls, and reproduce the run.
• The real comparison unit is a benchmark cell, not a label like “Go” or “Java.” A cell includes the service shape, runtime and execution mode, payload, concurrency, CPU and memory controls, telemetry profile, and repeat identity.
• Startup, readiness, warmup, measurement, and artifact capture stay separate. We do not blend early-runtime behavior into steady-state numbers or hide discarded runs behind vague “outlier” language.
• CPU counts are only meaningful when the enforcement mechanism and host controls are visible. Governor state, boost behavior, core placement, and load-generator headroom belong in the result package, not in a footnote.

If you have ever read a benchmark post and thought, I still can’t tell whether this is a runtime result or a harness result, this article is for you.

That confusion usually starts long before the chart. A bar labeled “Java” hides multiple runtime shapes. A CPU limit is given as a number, but not the mechanism that enforced it. Warmup gets blended into steady-state measurement. A few awkward runs disappear behind a line like “we removed outliers.” The numbers may be real, but the method is too loose to support the confidence the article projects.

We want to fix that before we publish a single result chart.

This article defines the run protocol for the rest of the series. It is the measurement contract for how we compare a small Go net/http service with a Helidon SE service on a specifically identified JDK and execution mode. It defines what a benchmark cell is, how warmup and measurement are separated, what host controls must be visible, what makes a run invalid, what telemetry every accepted run needs, and what artifacts must survive for reproduction. It does not present benchmark results.

One scope note up front: this series compares specific service and runtime shapes implemented in Go and Java, not the languages in the abstract. On the Java side, the comparison unit is Helidon SE on a named JDK and execution mode, not a generic “Java” bucket. On the Go side, the comparison unit is a small net/http service, not “all Go.” If later articles add a virtual-thread configuration, a startup-optimized Java mode, or another execution mode, each one will be treated as its own benchmark condition with its own startup, warmup, artifact, and reproducibility requirements.

Why methodology is part of the result

Benchmark methodology is not background detail. It is part of the result.

That matters even more in a series framed around Go and Java because readers bring strong priors. Some expect Go to win simple HTTP work by default. Some expect virtual threads to affect Java concurrency tradeoffs in certain server designs. Some assume framework choice matters more than language choice. We also see benchmark writeups collapse JDK distribution and execution mode differences into a single “Java” label. A few of those ideas may prove partly true under some conditions. None of them should get a free pass over measurement.

The runtime shapes in this series are easy to overread if we are sloppy. A small Go net/http service is not “all Go.” A Helidon SE service on a named JDK is not “all Java.” If the benchmark protocol fails to keep those differences visible, the charts will look more certain than they deserve.

That is why we are publishing the protocol before the charts. We want you to see the controls first, then the numbers.

The benchmark contract in one pass

Here is the short version.

A benchmark cell is the real unit of comparison. It includes the runtime variant, execution mode, payload profile, concurrency, CPU allocation method, memory limit, telemetry profile, and repeat ID. A warmup window is the period used to let runtime behavior settle before measurement. A measurement window is the only period whose data is eligible for later steady-state charts. An invalid run is a run excluded because the benchmark contract was broken: preflight failed, readiness never stabilized, warmup bled into measurement, required artifacts went missing, host state drifted, or the load generator became the bottleneck. An artifact manifest is the durable record of versions, host state, telemetry, and outputs required to inspect or reproduce the run.

Host state is a prerequisite, not an afterthought, so governor mode, turbo or boost status, CPU allocation method, and service-versus-load-generator placement must be recorded before measurement starts. Run order is controlled to reduce time drift, thermal drift, and first-run bias. Invalid runs are rejected by rule rather than by taste. Telemetry is mandatory because throughput and latency alone are not enough to explain a run. Later charts aggregate accepted runs only, while excluded runs stay visible in the raw records with an explicit reason.

We separate preflight, readiness, warmup, measurement, teardown, cooldown, and artifact capture so later charts describe steady-state runs rather than blended startup behavior.

That summary is intentionally strict. The rest of this article explains why each rule exists and what it protects.

What counts as a benchmark cell

A benchmark cell is the smallest comparison unit that still means something. If two runs differ in a way that could affect the result, they belong in different cells.

That sounds obvious, but benchmark articles often violate it with labels that are too broad. “Go” is not a cell. “Java” is not a cell. Even “Helidon SE” is not a cell. A useful benchmark cell has to name the workload shape, the runtime shape, and the host-control assumptions that made the run possible.

Here is the shape we mean:

benchmark_cell:
  app_variant: hello-api-json
  language_runtime: java
  framework_mode: helidon-se
  execution_mode: hotspot-jit
  runtime_distribution: named-distribution
  runtime_version: named-version
  cpu_allocation_method: declared-method
  cpu_allocation_value: declared-value
  memory_limit: declared-limit
  payload_profile: json-small
  concurrency: 64
  warmup_duration: "30s"
  measurement_duration: "60s"
  cooldown_duration: "20s"
  telemetry_profile: baseline
  repeat_id: 03
  host_control_state:
    governor: documented
    turbo_boost: documented
    loadgen_placement: documented

This is not the final manifest format for the execution article, but it captures the important idea: a benchmark cell includes more than the application binary or the language name. It includes the execution mode, the CPU-shaping method, the payload, the telemetry profile, and the repeat identity.

That matters because later analysis depends on those boundaries being real. If we quietly average together runs that differ in CPU mechanism, warmup duration, execution mode, or telemetry overhead, we are not summarizing one condition. We are averaging unlike things and hoping the label hides the damage.

A benchmark cell also gives us a cleaner way to disagree. If you think the CPU limit was wrong, or the telemetry profile was too heavy, or the execution mode was not representative, you can point to a named dimension of the benchmark cell instead of arguing with a vague bar in a chart.

A benchmark cell is the irreducible comparison unit. Runtime, execution mode, payload, concurrency, CPU allocation, telemetry profile, and repeat identity all belong in the cell definition.

How one measured run actually works

A measured run is not “start service, fire load, record requests per second.” It is a controlled sequence with explicit phase boundaries.

For this series, one measured run moves through these stages:

• host preflight
• service startup
• readiness gate
• warmup window
• measurement window
• runtime and load artifact capture
• teardown
• cooldown

Those boundaries exist because startup, warmup, and steady state answer different questions. Startup tells us when the service becomes usable. Warmup tells us how the runtime settles. The measurement window is the only part that belongs in later steady-state charts. If we collapse those phases together, we lose interpretability first and only later notice that we also lost trust.

In compact form, the lifecycle is simple:

preflight -> startup -> readiness -> warmup -> measurement -> capture -> teardown -> cooldown

The host preflight phase records the host assumptions before the run starts. If we cannot show what CPU-limiting mechanism, governor mode, or memory shape was actually in effect, we do not have a reliable basis for comparison.

The readiness gate is not warmup. The service is either ready to receive the benchmark workload or it is not. We do not start measurement while the process is still somewhere between launch and a stable request path.

The warmup window matters for both sides of this comparison. On the JVM, interpretation, profiling, and compilation can change early behavior materially, which is why warmup is a first-class benchmark concern. On Go, allocator behavior, page faults, connection reuse, cache warming, and general runtime settling can also change the shape of the first part of a run. Warmup is not “for Java.” It is for any runtime where early execution is not yet representative of steady state.

The measurement window is the only phase that contributes to later throughput and latency aggregates. We may collect telemetry across more of the run, but we do not let startup and warmup leak into the numbers we later describe as steady state.

The post-run capture phase matters just as much as the measured window. A run that cannot explain itself is not worth much. We need the latency artifact, runtime signals, process output, and acceptance decision while the context is still intact.

One measured run includes setup and observability work around the measured window, but only the steady-state segment belongs in later performance aggregates.

A strict phase model gives us two things. It makes later analysis reviewable, and it gives us a clear basis for rejecting runs when the boundaries break.

How CPU limits and host frequency controls are handled

Single-machine benchmarks get fragile quickly when CPU shape is described loosely.

Saying “this ran with 2 CPUs” is not enough. Two CPUs enforced through a cpuset, two CPUs implied by quota, two CPUs inferred by a runtime override, and two CPUs on a noisy unrestricted host are not the same condition. They affect scheduling, thread placement, contention, and runtime heuristics differently.

That is why we record the OS-visible CPU allocation method as part of the benchmark cell rather than treating CPU count as self-explanatory. Runtime-level alignment still matters, but it does not replace host-level CPU shaping.

Host-visible CPU limits and runtime-level processor settings both matter. In modern Go and Java toolchains, effective parallelism can be influenced by the CPUs and limits visible to the process, so later articles will capture both the host mechanism and any explicit runtime overrides. On the Java side, -XX:ActiveProcessorCount gives us an explicit processor-count override when we need it. Those features are useful, but they also make it more important to record what the host actually exposed to the process.

Our rule for this series is straightforward: CPU-limited comparisons use the same host-visible allocation method across variants whenever practical. If we align a runtime with an explicit flag, that alignment belongs in the manifest, but the primary description of the benchmark cell is still the host-level mechanism.

For every measured benchmark cell, the record captures the CPU allocation mechanism, the effective CPU allocation value, governor state, turbo or boost status, whether the service and load generator share cores, and whether the runtime was additionally aligned with explicit runtime settings.

CPU counts are only comparable when we also record the control mechanism, governor state, turbo status, and where the service and load generator were placed.

Frequency behavior matters too. Even when the software stack does not change, governor mode or turbo behavior can shift the run shape. Linux CPU frequency behavior is exposed through cpufreq and platform-specific drivers such as intel_pstate, which is why we treat governor state and boost visibility as required host facts rather than optional color. We prefer performance governor mode during measurement and we document whether turbo or boost is enabled. If the host does not let us control those settings directly, that limitation belongs in the manifest and in the interpretation.

The placement of the load generator matters just as much. If the service and load generator fight for the same cores, then the result is telling us about a contested host layout, not just the service under test. We want placement assumptions to be explicit rather than silently baked into the numbers.

How repeats, run ordering, and drift are managed

A benchmark gets a false sense of certainty very quickly if you run one variant first, another later, and report whichever run shape looks neatest.

We are doing the opposite.

Every important benchmark cell gets repeated measured runs, and those runs are scheduled to reduce drift bias. That means we do not run all Go first and all Java later. We do not cluster one runtime shape in the cool part of the day and another after the machine has already been busy for hours. Time drift, thermal drift, and background variance are real enough to matter on a single host.

The policy is fixed even before the execution package appears. Every primary comparison cell needs at least three accepted measured runs before it can appear in an aggregate chart. Warmup and cooldown happen per measured run. Run order is rotated or randomized within a balanced schedule. Excluded runs remain visible with their exclusion reasons. If a cell cannot produce three accepted runs without intervention, it is treated as unstable and flagged for diagnosis rather than summarized as if it were settled.

That last point matters. We are not collecting a pile of numbers and then selecting the subset that makes the chart look cleaner. We are defining comparable conditions in advance and keeping the decisions visible.

Balanced ordering becomes even more important when two runtime shapes are close. If the observed difference is modest, protocol quality matters more, not less. A loose schedule can create a story that the runtime never earned.

We also need enough repeats to say something about stability. A single strong run is not a result. At most, it is a clue.

What makes a run invalid

This is where benchmark writing often gets evasive. You see a sentence like “we removed a few noisy runs,” and suddenly you do not know whether the team corrected a harness mistake or quietly curated the result set.

We want the acceptance rules to be simple enough to audit.

A measured run is accepted only if all of these conditions hold:

• the preflight snapshot completed and matches the declared benchmark cell
• the service reached readiness and held it before warmup began
• the measured window started only after the declared warmup phase ended
• response-validity checks passed and measured-window error rates stayed within the declared acceptance rule for that workload
• the mandatory telemetry and artifact set was captured successfully
• required host controls did not drift during the run
• the load generator stayed within its validated headroom envelope and did not saturate before the service
• post-run verification did not reveal a mismatch between the run and the declared cell definition

In plain language, the policy is this:

If preflight failed, do not measure.
If readiness failed, discard the run and fix service state.
If the measured window began before warmup completed, invalidate the run.
If mandatory artifacts are missing, invalidate the run.
If host control state changed during the run, invalidate the run.
If the load generator saturated first, do not treat the result as service capacity.
If correctness checks failed, do not accept the run as a performance result.
If the run is unusual but artifacts are intact, keep it visible and flag it for investigation rather than silently averaging it in.

The important part here is not the wording. It is that the reasons to reject a run are operational and visible rather than aesthetic.

A few definitions make the contract clearer.

Warmup bleed-through means either the harness started the measured window before the declared warmup duration elapsed, or the phase markers show readiness or startup-state transitions still crossing into the measured segment.

Host drift means that a required control changed between preflight and post-run verification, such as CPU allocation, governor mode, turbo or boost visibility, memory limit, or the intended service-versus-load-generator placement.

Load-generator limit means the client side saturated first and could no longer sustain the target workload shape. For the primary load profile in this series, that headroom rule is validated separately: client CPU must stay below the established ceiling for the load profile, and a control test against a known-fast endpoint must show throughput headroom above the service under test. If that evidence is missing, the run does not qualify as a capacity result.

Run outcomes map to explicit actions: discard before measurement, invalidate and rerun after contract failures, flag intact but unusual runs, or accept runs that pass every gate.

It also helps to separate a few different outcomes.

A discard usually means the run should not have proceeded under those conditions at all, such as failed readiness or failed preflight.

An invalidate means the run happened, but the measurement window is not comparable because part of the contract failed.

A rerun after stabilization means the issue may be transient but explainable, such as host drift or obvious startup bleed-through.

A flag for investigation means the artifacts are intact, but the run deserves diagnosis before it is allowed into any aggregate.

That distinction matters because a trustworthy benchmark does not make questionable runs disappear. It keeps them visible in raw records and excludes them for named reasons.

What telemetry we capture for every accepted run

A benchmark that reports only throughput and a couple of latency percentiles is easy to publish and hard to interpret.

For this series, every accepted run needs enough evidence to support or challenge the explanation that follows. That does not mean turning on every profiler for every measurement run. Telemetry can perturb the benchmark, so the profile has to be standardized. It does mean that each accepted run needs a minimum artifact package, and suspicious runs need a clear path to deeper diagnostic reruns.

Each accepted run must persist enough data to answer these questions:

• What throughput did the service deliver, and with what error rate?
• What did the latency distribution look like across the measured window?
• Did the service keep returning valid responses for the declared workload?
• How much CPU and memory did the service consume under the declared telemetry profile?
• What runtime-specific memory, allocation, and GC signals are needed to explain anomalies?
• Was the load generator still inside its validated headroom envelope?
• Does the artifact set prove that the run matched the declared benchmark cell?

Latency distributions matter because load-generation method can distort tail-latency interpretation. That is one reason we preserve full latency artifacts rather than only a summary number. Later articles will identify the workload model and latency artifact format explicitly rather than relying on summary percentiles alone.

On the Java side, that means enough standardized runtime visibility to explain GC and runtime behavior without guessing after the fact. The exact capture mechanism belongs in the execution package, but the methodology requirement is stable: baseline measured runs use a declared low-overhead telemetry profile, and deeper runtime captures belong to explicitly labeled diagnostic reruns.

On the Go side, the rule is the same even if the tools differ. We need enough runtime visibility to explain allocation behavior, heap movement, or scheduler shape without turning every measured run into an intrusive profiling session. That telemetry profile is part of the benchmark cell definition, so readers can see what was always on and what was diagnostic-only.

A representative accepted-run package looks like this.

Required common artifacts

• cell.yaml
• preflight.txt
• launch-summary.txt
• service.stdout
• service.stderr
• env-summary.txt
• load-summary.txt
• latency-artifact
• resource-sample-artifact
• acceptance-decision.txt

Required runtime-specific artifacts for the declared telemetry profile

• runtime version capture
• runtime logs required by the declared profile
• runtime memory or GC evidence required by the declared profile

Diagnostic-only artifacts for labeled reruns

• deeper runtime captures produced only when the rerun policy calls for them

Not every runtime-specific file exists for every benchmark cell, and that is fine. What matters is that the telemetry profile is declared and the expected artifacts for that profile are actually present.

This series requires an artifact manifest and minimum evidence package for every accepted run. The next article will publish the concrete layout and capture details used to satisfy that requirement.

How this constrains later analysis

This is not the statistics article. That comes next. But the protocol only works if the interpretation posture is visible now.

The posture is deliberately conservative.

We are not going to make ranking claims from one good repeat. We are not going to treat throughput-only charts as complete. We are not going to hide discarded runs. We are not going to treat small differences as meaningful when the measurement uncertainty is of similar size.

That means later aggregation has to preserve enough information for review: repeat counts, central tendency, spread, and benchmark-cell inclusion decisions. A clean-looking median from a mixed bag of accepted and questionable runs is not good enough. A peak-throughput chart without the variance story is not enough. A p99 chart without CPU or runtime context is not enough either.

The practical reason is simple. Throughput, latency, CPU, RSS, heap, allocation, and runtime signals describe the same run from different angles. If one moves and the others do not, that tells us something. If they all move together, that tells us something else. Reading them together is how we avoid shallow explanations like “runtime A is faster” when the real story is “runtime A used more CPU to get there” or “runtime B’s tail changed because the host drifted and that run should not have been accepted.”

So the protocol imposes three rules on later articles. First, later charts aggregate accepted runs only. Second, later claims stay scoped to the measured runtime shape and benchmark cell. We do not stretch a narrow benchmark into a broad language verdict. Third, when spread is large or the observed effect is small, the article language stays correspondingly modest.

That is not timid writing. It is what defensible benchmark writing sounds like.

What artifacts must be saved for reproduction

A reproducible benchmark needs more than a general setup description. It needs a handoff package another engineer could use to rerun the work.

This article defines the contract. The next article will publish the executable side of that contract: exact environment disclosure, exact commands, manifests, run ordering, packaging details, and rerun instructions. After that, the analysis article will define how accepted artifacts become charts, including the canonical data shape, aggregation rules, uncertainty treatment, and charting constraints.

That separation is deliberate. It keeps this article focused on the measurement contract instead of turning it into a shell-script dump, and it keeps later claims tied to a visible analysis method.

For this series, reproducibility does not mean “an experienced engineer could probably guess the missing steps.” It means every accepted run leaves behind enough durable evidence that another engineer can inspect the benchmark cell definition, host controls, launch conditions, telemetry profile, measured outputs, and acceptance decision.

That is why the artifact manifest matters as much as the chart.

It is also why runtime identity belongs in manifests instead of fuzzy labels. Later articles will keep JDK distribution identity explicit, because support model, packaging, and build provenance can matter operationally even when distributions derive from the same OpenJDK codebase. If later articles add more than one Helidon SE execution mode, each one has to be named in the benchmark cell and preserved in the artifacts. If we compare multiple JDK distributions, that identity has to stay explicit. If later articles add a non-baseline Java execution mode, its setup assumptions and reproducibility requirements must be documented as part of that runtime variant rather than folded into a generic “Java” label.

If the series later includes GraalVM Native Image, it will be reported as a separate execution mode, not grouped with HotSpot-based Java runs. If later articles include startup-optimized or ahead-of-time Java modes, they will name the exact released technology used rather than using “Project Leyden” as a catch-all label.

What this article does not claim

This methodology is narrower than “how to benchmark web services” in general, and that is intentional.

It is designed for this series: a single-machine Linux/Bash benchmark comparing specific Go and Helidon SE service shapes under controlled CPU, concurrency, and payload conditions. It is not a Kubernetes benchmark, a cluster benchmark, or a production cost model. It does not stand in for every Java framework or every Go server architecture.

It also does not claim that these controls eliminate bias or guarantee a universal winner. What they do is reduce noise, lower the chance of false confidence, and make later numbers easier to challenge and reproduce.

That narrowness is a strength. It lets us be explicit about the protocol without pretending the result answers questions it was never designed to answer.

What comes next

The next step is not a winner chart. The next step is the executable benchmark package.

In the next article, we will publish the exact environment details, manifests, version capture, run order, and rerun instructions that implement this methodology on a real host. That is where you will be able to inspect the concrete CPU-limiting mechanism, runtime identities, telemetry-profile choices, workload model, and artifact layout used for the benchmark runs.

After that, we will publish the analysis rules that govern how accepted runs become charts and how uncertainty is reported. That is where the statistical and visualization constraints become just as inspectable as the run protocol.

If this article did its job, you now know what later results are obligated to prove, what evidence they must carry, and what kinds of numbers we will reject instead of averaging away.

References and Methodology Notes

• RFC 9110: HTTP Semantics — https://www.rfc-editor.org/rfc/rfc9110
• Brendan Gregg, Systems Performance, Second Edition — https://www.brendangregg.com/systems-performance-2nd-edition-book.html
• Gil Tene, “How NOT to Measure Latency” — https://www.youtube.com/watch?v=lJ8ydIuPFeU
• OpenJDK, Java Virtual Threads — https://openjdk.org/jeps/444
• OpenJDK, Java Flight Recorder — https://docs.oracle.com/en/java/javase/26/jfapi/
• Go documentation, Managing Resources for Go Programs — https://pkg.go.dev/runtime
• Go documentation, Profiling Go Programs — https://go.dev/blog/pprof
• Linux kernel documentation, CPU Performance Scaling — https://docs.kernel.org/admin-guide/pm/cpufreq.html
• Helidon documentation — https://helidon.io/docs/latest/

Key Takeaways

• The load generator is part of the benchmark instrument, not background plumbing. If the client saturates first or changes the arrival pattern under stress, later Go-versus-Java charts stop meaning what they appear to mean.
• For this series, the primary generator is benchctl, the compact Go load driver in the benchmark package. We selected it because it can hold an explicit rate, retain per-request artifacts, and fit the series’ Linux-and-Bash workflow without hiding the method behind another framework.
• Coordinated omission is not benchmark trivia. If the client naturally slows down during a stall, tail latency can look cleaner than the arriving workload actually experienced, so our main latency evidence uses fixed-rate pressure rather than purely response-paced discovery.
• Headroom is a gate, not an assumption. The accepted validation evidence keeps the generator below the 80% CPU gate at selected target rates, proves margin with a known-fast control service, and keeps client-limited cells as failure evidence instead of quietly promoting them into the comparison.

If you want a skeptical reader to trust the later benchmark charts, you have to earn that trust before the first results article.

That starts with the client.

A load generator can flatten throughput for the wrong reason, hide a stall behind response-paced behavior, or become the busiest process on the machine before the service under test is actually in trouble. In a single-machine series, the risk is even sharper because client and service share the same CPUs, caches, memory bandwidth, scheduler behavior, and loopback path.

That is why this article exists between the implementation work and the full benchmark-method article. Before we talk about warmup, repeats, CPU shaping, or result analysis, we need to show that the driver itself deserves trust.

Figure 1. We validate the load generator as part of the measurement instrument: one host, one rate-controlled client, explicit CPU capture, and retained artifacts for both the control endpoint and the service under test.

Why the load generator gets its own article

Benchmark readers usually know the obvious failure mode: the client tops out before the server does.

The more dangerous failure mode is quieter. The client still looks plausible, the throughput line still climbs, and the latency summary still looks neat enough to publish. But once the service begins to stall, the client changes its own behavior and stops applying the pressure you thought you were measuring.

That is how a benchmark can look disciplined while still telling the wrong story.

So we treat the load generator as part of the measurement apparatus, not as setup glue. If we would not trust a server-side result without knowing the runtime flags, CPU limits, and logging state, we should not trust a client-side result without knowing the pacing model, telemetry, and headroom either.

This article’s claim is narrow on purpose. We are not trying to prove a universal winner among HTTP load tools. We are trying to prove something more useful for this series: the chosen client is unlikely to be the dominant limiter at the rates we plan to measure, and the harness tells us when that stops being true.

What the generator has to do in this series

The generator has to support the benchmark question, not just produce traffic.

For this series, that means it must drive requests at explicit target rates, preserve latency events that can be inspected later, record client-side errors, and fit into a repeatable Linux/Bash workflow. It also has to work with the same artifact discipline as the services under test: raw event streams, process samples, derived summaries, and clear pass/fail decisions.

That combination matters because the later articles compare Go net/http, Java Helidon on an Oracle-built JDK, and the Oracle JDK AOT/cache path under controlled single-machine conditions. The load tool cannot be a black box in that comparison. It must expose enough evidence for a reader to ask, “Was the client the limiter?” and get a concrete answer.

What we selected and why

For this series, the primary generator is benchctl, the compact Go load driver that ships with the benchmark package.

That choice is about fit, not prestige.

We needed a generator that could do three things well. First, it had to support explicit target rates, because the series is asking not only how fast a service can run in a closed loop, but also what happens when requests keep arriving at a planned rate. Second, it had to retain inspectable per-request artifacts rather than just printing a final aggregate line and disappearing. Third, it had to fit a Bash-first workflow cleanly so the benchmark remained easy to audit.

benchctl met those requirements directly. It supports closed-loop discovery when we want a rough ceiling, fixed-rate runs when we want evidence-bearing latency rows, and raw JSONL artifacts that preserve each measured request. That last point matters. Later analysis can inspect what the client actually did instead of trusting a single terminal summary.

Just as important, benchctl is part of the same source package as the services and the control endpoint. That makes it easier to tie request artifacts, process sampling, target-rate selection, and headroom checks into one reproducible flow.

We still keep the claim bounded. Choosing a tool does not prove the benchmark is fair. It only tells you which instrument we are putting under scrutiny.

Figure 2. We selected the generator that best fits this series’ trust requirements: explicit rate control, inspectable latency output, Linux and Bash automation fit, and reproducible artifact capture.

Why coordinated omission matters here

A response-paced client answers one question. A fixed-rate client answers another.

With response-paced load, new requests are naturally coupled to earlier responses finishing. If the service stalls, the client slows down too. That makes the worst part of the stall look cleaner than it should because fewer new requests arrive while the system is already struggling.

That is coordinated omission in practical terms: the client quietly stops applying pressure at the exact moment when arriving work would still have been feeling delay.

This does not mean closed-loop discovery is useless. We still use it to find rough ceilings and to expose where a given client shape begins to saturate. But it is not our main latency evidence. For the benchmark question in this series, the important row is the one where requests continue arriving at the configured rate and the latency record shows what that pressure looked like.

So the policy is simple. Evidence-bearing latency rows use fixed-rate pressure. Closed-loop discovery remains useful, but if it saturates the client or answers a different pacing question, it does not become the row we cite as the main latency result.

That distinction is one of the reasons the retained raw event stream matters so much. It gives us enough detail to audit the run later instead of treating the client as a black box.

Figure 3. Response-paced load can appear cleaner than it should during a stall because the client naturally slows down. Our policy uses scheduled pressure so the latency record better reflects what arriving work experienced.

The validation rule: prove headroom before a run counts

The operational rule is intentionally strict.

In this series, the load generator has to stay below a sustained 80% CPU gate at the selected target rates, and a separate known-fast control service has to prove throughput margin above the service-under-test demand.

Those are two different checks, and we need both.

The CPU gate catches the obvious problem: the client itself is running out of room. The control-service gate catches a subtler one: a client can stay under its CPU threshold and still fail to provide enough throughput margin for the real benchmark.

The control service in this package is fast-echo, a deliberately boring HTTP target. It is not part of the Go-versus-Java comparison. Its job is narrower than that. It tells us whether the client can comfortably exceed the throughput band selected for the benchmark cells.

This is also where negative evidence becomes useful instead of embarrassing. If a discovery cell or a control attempt saturates the client, we keep it. It becomes proof that the gate is working, not something we quietly discard.

What the validation showed

The important point is that the current package contains validation evidence, not just a policy.

The accepted fixed-rate probes were used to choose the final controlled-load rates. The final rate-selection note accepted 200, 400, and 800 requests per second for the long benchmark. That decision was based on fixed-rate probes, not on a closed-loop capacity leaderboard.

The evidence is compact enough to show directly:

• Lower fixed-rate probe
  • Scope: 100 and 200 req/s across the required Oracle-built-JDK comparison shape
  • Accepted evidence: 108 rows; 162,000 measured requests; all 2xx; zero errors; 108/108 saturation rows passed
  • Load-generator CPU result: Maximum reported load-generator CPU: 15.559802%
  • Why it matters: Supports 200 req/s as a clean selected rate
• Higher fixed-rate probe
  • Scope: 400 and 800 req/s across the same required variants, fixtures, CPU shapes, and concurrency levels
  • Accepted evidence: 108 rows; 648,000 measured requests; all 2xx; zero errors; 108/108 saturation rows passed
  • Load-generator CPU result: Maximum reported load-generator CPU: 40.503938%
  • Why it matters: Supports 400 and 800 req/s as clean selected rates
• Final controlled-load run
  • Scope: Selected 200, 400, and 800 req/s rates
  • Accepted evidence: 486 measured cells; 6,804,000 measured non-warmup requests; all 2xx; zero request errors
  • Load-generator CPU result: After excluding terminal shutdown sampler rows with zero CPU ticks and zero file descriptors, 486/486 cells passed; maximum adjusted load-generator CPU: 44.447241%; maximum adjusted average load-generator CPU: 23.722782%
  • Why it matters: Shows the final selected rates were driven below the load-generator CPU gate
• fast-echo control
  • Scope: Known-fast local control endpoint
  • Accepted evidence: Best observed control throughput: 38,067.4 req/s at concurrency 64; zero request errors
  • Load-generator CPU result: Control run used for client headroom, not service ranking
  • Why it matters: Demonstrates client-side headroom above the selected 800 req/s ceiling

The final controlled-load run deserves one extra explanation. The canonical saturation check flagged 51 terminal process-sampler rows where CPU tick counters decreased as the load-generator process exited. The supplemental audit ignored only terminal shutdown rows with zero CPU ticks and zero file descriptors, then found all 486 measured cells passing the adjusted saturation check. That is a narrow correction for process shutdown sampling, not a broad waiver for inconvenient data.

The fast-echo control run gives the other half of the headroom story. In the Oracle-only headroom manifest, the known-fast control service reached 38,067.4 requests per second at concurrency 64, with zero request errors, in a short local headroom run. That is not a service-under-test benchmark result. It is instrument validation: evidence that the client and local path can exceed the selected final controlled-load rates by a wide margin.

That is what we want from a benchmark harness. Not “the client always wins,” but “the harness shows where a client shape stops being safe.”

Why the control endpoint matters more than a pretty throughput number

A control endpoint is easy to misunderstand.

Readers see an echo-style endpoint and assume the article is trying to sneak in a flattering benchmark number. That is not the point. The control target is not there to rank services. It is there to calibrate the measurement instrument.

To do that well, the endpoint has to be intentionally simple: stable body, stable status, no database call, no downstream RPC, no template rendering, no application-specific hot path. The more boring it is, the more useful it becomes for this job.

fast-echo did exactly what we needed. It showed that the client could exceed the selected benchmark demand in a passing shape, and it also exposed whether a client shape was not good enough. Both outcomes matter. A passing control proves headroom. A failing control proves the gate is real.

That is why the control-service evidence belongs here even though it is not a benchmark result in the usual sense.

What this still does not prove

This validation clears an important hurdle, but it does not answer every benchmark question.

It does not turn same-host measurement into a separate load-generator host. Client and service still share CPUs, caches, memory bandwidth, scheduler behavior, and loopback networking. Those interactions can still matter, especially in the tail.

It does not mean closed-loop discovery is bad or fixed-rate pressure is perfect. Those modes answer different questions, and we use them differently on purpose.

It does not prove that the later Go or Java implementation is faster. The claim here is narrower: the current load generator, saturation gate, and control proof are strong enough that the next article can talk about benchmark methodology without leaving the client as an unexamined weak spot.

That is the right level of confidence for this stage.

Acceptance criteria before any benchmark run counts

Before a later Go-versus-Java cell can count as benchmark evidence in this series, it has to clear the same contract:

• Correct pacing mode: evidence-bearing latency rows use the series’ fixed-rate mode, not an accidental response-paced fallback.
• Generator CPU below the gate: the load generator must stay below the sustained 80% CPU threshold for the target cell.
• Control-service headroom: the known-fast control service must show throughput above the selected benchmark demand.
• No hidden client failure: rows with client saturation, unstable scaling, missing telemetry, or client-side errors are kept as failure evidence, not promoted into comparison data.
• Retained artifacts: raw request events, process samples, and derived summaries must be present so the run can be audited later.

Those rules are strict because they need to be. A series that plans to compare languages and runtime shapes needs hard acceptance criteria long before it needs opinions.

Figure 4. A benchmark run only counts after the instrument passes: right tool mode, valid control proof, acceptable CPU headroom, retained artifacts, and no client-side saturation.

What this unlocks next

With the load tool validated, the next article can focus on benchmark methodology instead of first arguing about whether the client was the real limiter.

That changes the conversation in a useful way. We can move on to warmup policy, measurement windows, repeat counts, CPU-shape rules, invalid-run handling, and reproducibility knowing that the client side already has a defined contract and bounded proof behind it.

There is still plenty left to scrutinize. There should be. But the next scrutiny can be about the method and the service behavior, not about an unproven driver hiding in the background.

Conclusion

The benchmark before the benchmark is the load generator.

For this series, we selected benchctl because it matches the method we actually need: explicit rate control, inspectable raw artifacts, and tight integration with the package’s validation flow. We then validated it the way a measurement instrument should be validated, not the way a convenience tool is usually waved through.

The important result is not that every client shape passed. It is that the harness selected final controlled-load rates only after clean fixed-rate probes, then carried the same headroom discipline into the final matrix. The final accepted rates are 200, 400, and 800 requests per second. The accepted probes reached those rates without request errors or load-generator saturation, the final controlled run stayed below the load-generator CPU gate after accounting for terminal sampler shutdown rows, and the fast-echo control showed substantial local client headroom above the selected ceiling.

That does not settle the Go-versus-Java question. It settles the more basic question first: whether later runs are being driven by a tool we have reason to trust.

References and Methodology Notes

• Result-package artifact: demos/go-java-go-benchmark/manifests/recovery-fixedprobe-20260620T0100Z.csv
• Result-package artifact: demos/go-java-go-benchmark/manifests/recovery-fixedprobe-hi-20260620T0200Z.csv
• Result-package artifact: demos/go-java-go-benchmark/manifests/recovery-final-controlled-20260620T0230Z.csv
• Result-package artifact: demos/go-java-go-benchmark/manifests/headroom-fast-echo-control-oracle-only-20260619T1103Z.json
• Result-package artifact: diagnostics/final_rate_selection_20260620.md
• Result-package artifact: diagnostics/fixedprobe_highrate_audit_20260620.md
• Result-package artifact: diagnostics/final_controlled_benchmark_audit_20260620.md
• Result-package artifact: demos/go-java-go-benchmark/docs/artifact-policy.md
• HDR Histogram: background on coordinated omission, latency recording, and percentile interpretation. https://hdrhistogram.github.io/HdrHistogram/
• Linux sched(7): background on Linux scheduling behavior relevant to same-host benchmark limitations. https://man7.org/linux/man-pages/man7/sched.7.html

Key Takeaways

• This fourth article in our ten-part single-machine Linux benchmark series defines the Go net/http contract before any Go-versus-Java results are published.

• The Go service must match the Article 02 HTTP behavior at the external boundary: endpoints, JSON rules, validation, error envelopes, deterministic payload generation, and extraWork checksum semantics.

• Go does not need to imitate the Helidon SE implementation from Article 03. It does need to prove that it performs the same declared work and reports enough runtime evidence to make later results auditable.

• Runtime tuning is part of the benchmark input. Defaults come first; any GOMAXPROCS, GOMEMLIMIT, GOGC, diagnostics, profiling, CPU, or memory variation needs a clear run label and manifest evidence.

  
  

Before we believe a future Go-versus-Java chart, we need to know exactly what the Go service is doing.

Not “it is a small Go program.” Not “it uses the standard library.” Not “it looks equivalent.” We need to know where the request is decoded, how validation fails, whether unknown JSON fields are rejected, whether duplicate fields are caught, what extraWork contributes to the response, whether one side silently enables a timeout or protocol mode, and which runtime settings were active when the run was measured.

That is the job of this article.

This is a contract article, not a results article. It defines the Go net/http service shape and the runtime evidence required before any Go result can be compared with the Java/Helidon variants from Article 03. We are not producing Go benchmark numbers here. We are not generating the benchmark source package here. We are defining what must be true before Go net/http results can be compared defensibly with the Java side.

If you build Oracle AI Database-backed services, this matters even before a database call enters the benchmark. A retrieval API, embedding gateway, reranking service, feature-enrichment layer, or vector-search facade often spends real CPU and memory in ordinary application-tier work: HTTP handling, JSON decoding, validation, enrichment, response shaping, logging boundaries, and runtime scheduling.

This article does not measure Oracle AI Database. It defines the Go service boundary so later database-backed workloads do not blur service-tier behavior with database behavior.

Short Answer: The Go Contract

The Go benchmark side must be a declared net/http service that implements the Article 2 HTTP contract and aligns with the Java/Helidon plan from Article 3 at the external behavior level.

The final Go service exposes the same GET /health, HEAD /health, and POST /work behavior; applies the same JSON media-type policy, request body limit, validation rules, duplicate-field and unknown-field rejection, deterministic payload generation, extraWork checksum semantics, success response, and JSON error envelope; and keeps primary-run logging, TLS, compression, and tracing off unless a later labeled slice enables them symmetrically.

The Go implementation does not need to imitate Helidon internals. It does need to prove that it is doing the same declared work. extraWork is a first-class equivalence gate because it affects the returned checksum and must not be skipped, reduced, duplicated, reordered, or optimized away.

The runtime side needs the same discipline. Defaults come first. Tuned runs are allowed only under clear labels, with GOMAXPROCS, CPU context, memory limits, GOMEMLIMIT, GOGC, server settings, allocation evidence, GC evidence, runtime/metrics snapshots, and pprof artifacts tied to the benchmark manifest.

Exact Go, Java, OS, kernel, CPU, memory, and tool versions will be recorded in the benchmark manifest when accepted runs are executed.

What Articles 1 to 3 Established

Article 1 set the series rule: method first, measurement later. We are not trying to make a language-war claim. We are building an evidence chain strong enough that later results can be challenged on method, source, runtime identity, load-generator validity, execution protocol, and analysis rules.

Article 2 defined the language-neutral service behavior that both implementations must follow. For the Go implementation, that means using the Article 2 endpoint list, request schema, response schema, body-limit policy, JSON media-type rules, validation rules, error envelope, deterministic payload-generation rules, and extraWork semantics without alteration.

The relevant service shape is intentionally small. GET /health returns stable JSON:

{
  "status": "ok"
}

POST /work accepts a JSON object containing requestId, payloadSize, seed, and extraWork, subject to the Article 2 media-type, body-size, field, and numeric-validation rules. The service enforces the /work request-body limit, rejects unknown fields, rejects duplicate top-level fields, validates integer bounds and types strictly, generates deterministic payload bytes from payloadSize and seed, applies deterministic repeated SHA-256 work controlled by extraWork, and returns a stable success response containing requestId, payloadSize, extraWork, checksum, and ok.

Failures that use response bodies return the agreed JSON error envelope. Framework-default HTML pages, stack traces, plain-text errors, extra runtime metadata, and implementation-specific error documents are contract failures.

Article 03 defined the Java-side implementation plan and the evidence that the Java run must provide. This article uses that plan as the continuity point for external behavior and runtime evidence; it does not require Go to mirror Java internals.

The Java-side manifest must identify the exact JDK distribution, vendor, build, version, JVM mode, and relevant flags rather than treating Java, OpenJDK, and Oracle JDK as interchangeable runtime labels.

What This Article Covers

This article covers the Go net/http service shape, Article 2 equivalence, Article 3 alignment, extraWork review, runtime knobs, allocation and GC evidence, runtime/metrics, pprof, and acceptance gates.

It does not cover benchmark results. It does not define the load generator; Article 05 owns that. It does not define the full run protocol, CPU governor handling, warmup windows, repeat rules, invalid-run policy, or statistical analysis; those belong in later methodology and analysis articles.

It also does not move the series into Kubernetes or production orchestration. This ten-part series is scoped first to a controlled single-machine Linux benchmark. Host OS, kernel, CPU topology, CPU affinity or cgroup quota, memory limits, and runtime versions must be recorded in the manifest. The primary pure-service benchmark does not introduce Oracle AI Database driver behavior, SQL, vector-search execution, or database latency.

That boundary is deliberate. If implementation, tuning, load generation, execution, and analysis collapse into one step, the later chart becomes hard to trust. The Go side needs to pass its contract gates before source generation and benchmark execution.

We define the Go side as a contract gate: Article 2 supplies the external service behavior, Article 3 supplies the Java/Helidon alignment point, and Article 4 defines the Go gates that must pass before later source generation, benchmark execution, and analysis.

Why net/http Is the Go Baseline

The Go side uses the standard-library net/http package as the baseline. That is a good fit for this benchmark because it keeps the service path visible. Handler boundaries, request decoding, response writing, server settings, and diagnostics can be reviewed without also reviewing a third-party router, middleware framework, validation package, or alternate HTTP engine.

That is a reviewability argument, not a performance claim.

Go developers have good reasons to use frameworks and routers such as Gin, chi, Echo, Fiber, or fasthttp-based stacks in real services. Some provide better ergonomics for route grouping, middleware, validation, OpenAPI integration, structured errors, or specialized performance goals. Those choices can be useful production decisions. They are simply not the primary baseline for this series.

net/http is the right first baseline here because it is familiar, standard-library-based, and inspectable. The standard library keeps the baseline small, but it does not make the benchmark fair by itself. Fairness comes from matching the Article 2 contract, controlling runtime settings, capturing evidence, and rejecting runs that violate the rules.

The Go Service Shape

The Go service should be a single explicit net/http server shape for the primary pure-service benchmark. The exact source package comes later, but the implementation contract is already clear.

The route map comes from Article 2:

• GET /health returns 200 OK and {"status":"ok"} as JSON.
• HEAD /health follows HEAD behavior for the health resource.
• POST /work is the measured work endpoint.
• Unsupported methods and unknown routes follow the Article 02 status and JSON error rules. When Article 02 requires 405 Method Not Allowed, the response must include the contract-defined Allow header.

For HEAD /health, the Go handler must return the same headers and status policy defined for the health resource while writing no response body.

The handler path for POST /work is fixed at the contract level. The service checks the route and method, validates the media type, enforces the /work request body limit, decodes JSON, rejects duplicate and unknown top-level fields, validates required fields and scalar bounds, generates the deterministic payload, applies extraWork, computes the checksum, and returns the success JSON. Failures map to the Article 2 error envelope.

Here is the shape we want the final source to preserve. This is an illustrative contract snippet, not the generated benchmark source package:

// Illustrative contract shape only.
// The benchmark source package will be generated later from the finalized contracts.
func main() {
    server := &http.Server{
        Addr:    ":8080",
        Handler: http.HandlerFunc(route),
        // Addr, timeouts, header limits, body limits, TLS, and protocol settings
        // are illustrative here. Final values must be declared in the benchmark manifest.
    }
    log.Fatal(server.ListenAndServe())
}
func route(w http.ResponseWriter, r *http.Request) {
    switch r.URL.Path {
    case "/health":
        handleHealth(w, r)
    case "/work":
        handleWork(w, r)
    default:
        writeContractError(w, contractErrorForUnknownRoute(r))
    }
}
func handleHealth(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodGet && r.Method != http.MethodHead {
        writeContractError(w, contractErrorForUnsupportedMethod(r))
        return
    }
    w.Header().Set("Content-Type", "application/json")
    if r.Method == http.MethodHead {
        w.WriteHeader(http.StatusOK)
        return
    }
    _, _ = w.Write([]byte(`{"status":"ok"}`))
}
func handleWork(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        writeContractError(w, contractErrorForUnsupportedMethod(r))
        return
    }
    req, err := decodeWorkRequest(r)
    if err != nil {
        writeContractError(w, err)
        return
    }
    if err := validateWorkRequest(req); err != nil {
        writeContractError(w, err)
        return
    }
    checksum := performDeclaredWork(req.PayloadSize, req.Seed, req.ExtraWork)
    resp := WorkResponse{
        RequestID:   req.RequestID,
        PayloadSize: req.PayloadSize,
        ExtraWork:   req.ExtraWork,
        Checksum:    checksum,
        OK:          true,
    }
    body, err := json.Marshal(resp)
    if err != nil {
        writeContractError(w, contractErrorForInternalEncoding(err))
        return
    }
    w.Header().Set("Content-Type", "application/json")
    _, _ = w.Write(body)
}

The important part is not the number of lines. It is the request path:

request -> route -> media type/body limit -> JSON decode -> validation -> extraWork -> response/error contract

The final source should set response Content-Type exactly as Article 2 specifies. The illustrative snippet uses application/json only to show that Go must not fall back to framework or net/http default error documents.

The final source may use ServeMux or another standard-library routing shape, but it cannot let default net/http error documents define the benchmark behavior. Unknown routes, unsupported methods, oversized bodies, malformed JSON, and validation failures must all map to the Article 02 status and JSON error contract. If the final source uses ServeMux method/path patterns rather than explicit method checks inside handlers, the selected Go toolchain’s ServeMux matching semantics must be recorded because those semantics are version-sensitive.

JSON handling will likely use encoding/json, because it is the standard-library JSON package and keeps dependencies low. But encoding/json defaults are not the contract. The implementation must not rely on convenient decoder behavior to define benchmark behavior.

Plain struct unmarshalling does not prove duplicate-field rejection, and unknown fields are ignored unless the decoder is configured or the implementation checks them explicitly. Article 02 rejects unknown fields and duplicate top-level request fields, so the Go source needs a token-level pass, json.RawMessage inspection, or another explicit mechanism that records top-level field names before decoding into the typed request model.

The same applies to numeric handling. Article 2 requires integer JSON numbers with no fractional part and no exponent notation for payloadSize, seed, and extraWork. Numeric validation should be explicit for the chosen decode path so strings, fractional forms, exponent notation, missing values, and out-of-range values are rejected exactly as Article 02 requires.

Server configuration also belongs in the contract. If the Go service uses http.Server timeouts, header limits, handler-level body limits such as http.MaxBytesReader, TLS, HTTP/2 settings, compression, or diagnostics listeners, those settings must be declared. The primary pure-service benchmark keeps request logging, TLS, compression, and tracing off unless a later secondary slice enables them symmetrically.

Logging has the same boundary. Build and startup logs are useful. Error diagnostics are useful. Per-request access logging is off for the primary measured path unless a later logging slice includes it deliberately. A hidden log line on every request can become part of the workload.

How the Go Contract Pairs With the Java Contract

Go does not need to imitate Helidon SE internals. Java does not need to imitate Go internals.

That sounds obvious, but it is a common benchmark trap. If we try to make Go look like Java inside the process, or Java look like Go, we stop comparing real service shapes and start building an artificial hybrid.

The Java implementation uses the Helidon SE configuration defined in Article 03. The Go implementation uses net/http. They are compared at the external contract level, not by assuming identical routing, scheduling, or runtime internals. Any Helidon/JDK threading mode, including virtual-thread-related settings if used, must be recorded in the Java run manifest.

Both implementations must satisfy the same HTTP behavior from Article 02. Both must implement the same deterministic payload generation and extraWork checksum behavior. Both must use the same primary-run posture for logging, TLS, compression, and tracing. Both must report enough runtime identity and diagnostic evidence to make later results auditable.

Article 3 defines the Java/Helidon side with its request-handling plan, runtime policy, and diagnostic evidence. This article defines the Go side with net/http, Go runtime settings, allocation and GC observability, and pprof artifacts.

They align at the contract level, not by sharing internals.

The extraWork Equivalence Review

extraWork is the part of this benchmark that most deserves suspicion.

In Article 2, extraWork is not a sleep, a delay knob, or a decorative loop. The service generates deterministic payload bytes from payloadSize and seed, computes an initial SHA-256 state, then applies a deterministic repeated operation extraWork times:

buffer = generatePayload(payloadSize, seed)
state = SHA-256(buffer)
repeat extraWork times:
    state = SHA-256(state || buffer)
checksum = lowercaseHex(state)

Because the final checksum is returned, extraWork has an observable effect for the benchmark’s conformance test vectors. If the loop is skipped, shortened, duplicated, reordered, or changed, the expected checksum for those vectors will not match.

The Go implementation must treat this as declared per-request work. It is not a hidden trick and not a place for clever local interpretation. The comparison does not require Java and Go to execute identical machine instructions. It does require both services to perform the same externally specified operation for the same accepted request.

The equivalence assertion should be this direct:

given payload P
and configured extraWork=N
when the Go service handles P
and the Java service handles an equivalent P
then both services:
- validate P before performing accepted work
- generate the same deterministic payload bytes
- execute the declared work factor N
- produce the same checksum
- reject the same invalid values
- record extraWork=N in the benchmark manifest

The ordering matters. Invalid requests should fail validation before accepted work is performed. extraWork = 0 is valid and means no repeated extra operation after the initial payload checksum. extraWork above the Article 2 maximum is invalid. A string, fractional value, exponent notation, missing value, or negative value is invalid.

A few mistakes would invalidate the comparison immediately:

The comparison is invalid if:
- Go skips extraWork for a valid request that Java processes.
- Go performs fewer iterations because of integer truncation or a loop-bound mistake.
- Go allocates large temporary buffers inside the loop when the contract does not require them.
- Java performs extra hidden request work that is not declared in the Java contract.
- One implementation accepts extraWork=0 and the other rejects it.
- The work is optimized away because it has no observable or retained effect.

The allocation example needs careful handling. Different runtimes have different allocation behavior, and the benchmark does not require identical allocator internals. But if the Go source creates avoidable temporary structures inside the extraWork loop while the Java source reuses state for the same logical operation, that may be a benchmark implementation bug rather than a runtime finding.

The fix is not to tune around it after seeing results. The fix is to inspect the implementation, decide what the contract requires, test it, and only then allow the source into the benchmark package.

Before results are published, the benchmark source or contract appendix must include canonical checksum test vectors for accepted combinations of payloadSize, seed, and extraWork. extraWork is admissible only when both implementations do the same declared work for the same accepted request and fail the same invalid requests in the same way.

Runtime Tuning Is a Controlled Hypothesis

The Go runtime settings are not incidental. They shape how the process uses CPU and memory, how garbage collection is paced, and how later profiles should be interpreted.

The default run comes first. That default run should use the selected Go toolchain’s runtime defaults under the declared Linux machine or container environment. Tuned variations can come later, but only under labels that explain the hypothesis.

For example, future run labels might distinguish:

• go-nethttp-default
• go-nethttp-gomaxprocs-2
• go-nethttp-gomemlimit-labeled
• go-nethttp-gogc-labeled
• go-nethttp-profiled

Those names are examples, not final manifest labels. The rule is the point: a tuned Go run without a manifest label is not reproducible enough to compare.

A compact future-run manifest excerpt shows the level of evidence required:

run:
  label: "go-nethttp-default"
  benchmark_scope: "single-machine-linux"
  results_in_this_article: false
service:
  language: "go"
  implementation: "net/http"
  json_library: "encoding/json"
  tls_primary_path: false
  server_settings:
    read_timeout: "required-if-set"
    write_timeout: "required-if-set"
    max_request_body_bytes: 4096
runtime:
  go_version: "required"
  effective_gomaxprocs: "required"
  gomemlimit: "required-or-unset"
  gogc: "required-or-default"
  os: "required"
  kernel: "required"
  cpu_limit: "required-if-containerized-or-limited"
  memory_limit: "required-if-containerized-or-limited"
workload_contract:
  service_contract_version: "required"
  extra_work: "required"
artifacts:
  runtime_metrics: "required"
  cpu_profile: "required-for-profiled-runs"
  heap_profile: "required-for-profiled-runs"
  alloc_profile: "required-for-profiled-runs"
  goroutine_profile: "required-for-profiled-runs"

This manifest is not evidence that a benchmark has run. It is the contract for what later accepted Go runs must record.

Because Go runtime defaults can change across Go releases and execution environments, the manifest should record both configured environment values and effective values observed by the benchmark process. Shell environment alone is not enough. Accepted runs must capture the selected Go toolchain and the effective runtime values from inside the benchmark process or from a dedicated manifest probe built with the selected Go toolchain.

CPU and Memory Controls

GOMAXPROCS controls how many operating-system threads can execute Go code simultaneously. That setting matters because a service running with one effective CPU is not the same experiment as a service running with four, eight, or sixteen. It also matters when the process runs under CPU affinity, cgroup quota, or a container CPU limit.

The Go contract should record the observed GOMAXPROCS value, host CPU model, logical CPU count, CPU affinity if constrained, container CPU quota or limit if used, whether GOMAXPROCS was set manually, and the Go toolchain version that determined default behavior.

We should avoid advice like “set GOMAXPROCS to the number of cores.” That might be right in one environment and wrong in another. The benchmark question is simpler: what CPU budget did the process actually observe, what value did the runtime use, and was that value a default or a deliberate tuned variation?

Memory needs the same discipline. A container memory limit, host memory availability, process RSS, Go heap behavior, and Go runtime memory policy are related but not identical.

GOMEMLIMIT is a soft limit for memory managed by the Go runtime and garbage collector. It is not a hard cap on process RSS. If the benchmark uses GOMEMLIMIT, that setting must be recorded and labeled. If it is unset, that fact should also be recorded. The surrounding memory context still matters because a process can be constrained by an external memory limit even when the Go runtime setting is unset.

GOGC changes the garbage collector’s target percentage and represents a CPU/memory tradeoff. Changing it can reduce memory growth at the cost of more GC work, or reduce GC frequency at the cost of more memory, depending on direction and workload. This article does not recommend a universal GOGC value. The baseline uses defaults first; changing GOGC is a labeled hypothesis.

Allocation and GC Evidence

The Go implementation should avoid accidental allocations that are not required by the workload. That is not the same as over-optimizing Go.

For example, repeatedly rebuilding constant response metadata, copying request bodies unnecessarily, allocating a fresh large buffer inside every extraWork iteration, or formatting unused strings on the hot path can turn the benchmark into a measurement of source mistakes. Those are worth fixing before execution if they violate the contract or create avoidable asymmetry.

On the other hand, the Go side should not introduce exotic pooling, unsafe tricks, custom JSON encoders, or specialized buffer reuse unless those choices are declared and accepted as part of the benchmark. If we aggressively tune Go allocation behavior while leaving the Java side at a straightforward implementation, the comparison shifts from service-shape benchmarking into asymmetric hand-optimization.

The practical sequence is straightforward: implement the Article 02 contract plainly, measure allocation rate and GC behavior under representative conformance or diagnostic load, inspect whether allocation differences are contract-required, runtime-inherent, or source-level mistakes, fix source-level mistakes before benchmark execution, and label any deliberate optimization that changes the baseline service shape.

The later Go run artifacts should include allocation and heap evidence. That does not mean every primary measured run has to run with heavy profiling turned on. It means the benchmark plan must capture enough allocation and GC information to explain later throughput, latency, CPU, and memory shapes.

Allocation reductions are hypotheses. Profiles and metrics tell us whether the hypothesis is true.

Profiling and Observability

Go gives us useful built-in observability paths: runtime/metrics, runtime/pprof, net/http/pprof, and the broader Go diagnostics guidance. This article requires a plan for those artifacts before any result exists.

runtime/metrics snapshots should be captured using the supported metric list for the selected Go version. The exact metric names belong in the later source package and manifest because the supported set can evolve. At minimum, the run should capture enough runtime data to reason about goroutines, scheduler behavior, memory, heap allocation, GC settings, GC pauses or timing summaries, and observed runtime configuration. The manifest should store both the collected metric values and the supported metric-name list reported by the selected Go version, so later analysis can distinguish “metric was zero” from “metric did not exist in this toolchain.”

pprof artifacts should be planned for profiled runs. CPU, heap, allocation, and goroutine profiles are the obvious starting point. Block or mutex profiles can be useful if the workload suggests contention, but they require explicit sampling configuration and should not be casually enabled in the primary path.

pprof is evidence capture, not free instrumentation. Profiled runs need labels, and interpretation belongs later, after accepted runs exist. If profiling changes the measurement environment, the profiled run should be separate from the primary comparison run.

A future diagnostics listener might be isolated from the measured service path. This snippet shows the idea without defining final source:

// Illustrative only.
// The final benchmark package must decide whether pprof is enabled,
// how it is isolated, and how profiled runs are labeled.
if diagnosticsEnabled {
    go func() {
        diagMux := http.NewServeMux()
        diagMux.HandleFunc("/debug/pprof/", pprof.Index)
        diagMux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
        diagMux.HandleFunc("/debug/pprof/profile", pprof.Profile)
        diagMux.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
        diagMux.HandleFunc("/debug/pprof/trace", pprof.Trace)
        log.Fatal(http.ListenAndServe("127.0.0.1:6060", diagMux))
    }()
}

The important boundary is not port 6060; that is only a familiar local example. The important boundary is that diagnostics are declared, isolated where appropriate, and tied to the run label. The final source should define diagnostics-listener startup, failure, and shutdown behavior explicitly; the snippet above is not a production-ready or benchmark-ready lifecycle policy.

For each accepted Go run, the artifact plan should include the benchmark manifest, service configuration, runtime/metrics snapshot, allocation profile when profiling is enabled, heap profile when profiling is enabled, CPU profile when profiling is enabled, goroutine profile when profiling is enabled, and GC summary or trace setting if enabled.

Those artifacts do not replace results. They make later results diagnosable.

We plan Go runtime evidence before results exist: request handling, allocation behavior, scheduler and GC signals, runtime/metrics, and pprof artifacts all tie back to the benchmark manifest.

Acceptance Gates for the Go Implementation

The Go implementation is not accepted because it compiles. It is accepted only when it satisfies the contract gates.

Before the Go service is allowed into later benchmark source generation, Article 2 endpoint behavior must be implemented exactly. Routes must match the contract methods and paths. Request validation must reject unknown fields, duplicate top-level fields, wrong numeric types, out-of-range values, malformed JSON, unsupported media types, oversized bodies, unsupported methods, and unknown routes according to the agreed status and JSON error behavior. Success responses must match the Article 2 JSON contract.

The Go and Java implementations must also satisfy the same externally visible behavior. The Article 03 alignment gate keeps the Java side in view without forcing Go to mirror Helidon internals. If Java/Helidon adds a framework behavior that affects the contract, that behavior has to be declared and either matched externally or removed from the primary comparison.

The extraWork gate proves that Go and Java compute the same checksum for the same valid request and reject the same invalid work factors. It also forces us to inspect allocation behavior around the work loop before performance results are allowed.

The runtime manifest gate rejects unlabeled tuning. If GOMAXPROCS, GOMEMLIMIT, GOGC, CPU limits, memory limits, server timeouts, TLS, diagnostics mode, or profiling settings change, the run label and manifest must say so.

The allocation, GC, and pprof gates make sure future results can be explained. A surprising Go result without allocation, GC, runtime, or profile evidence is not a strong result. It is a guess with numbers attached.

And one gate is intentionally simple: no benchmark results are published from this article.

Why There Are No Results Yet

Charts would be premature here.

A Go throughput number published before the Go service contract is locked would not answer the series question. It would answer a less useful question: “What happened when this particular Go program, with these settings, was run under a not-yet-final method?”

That is not enough.

No Go result should be compared with Java until the implementation contract, source package, validation checks, load-generator evidence, run protocol, captured artifacts, and analysis rules all line up.

That order can feel slow if you want the chart. It is faster if you want a chart you can trust.

Conclusion: Handoff to Article 5

The Go side is now a contract, not a result claim.

We have defined why net/http is the baseline, how the Go service must implement the Article 2 endpoint behavior, how it aligns with the Java/Helidon plan from Article 3, why extraWork needs its own equivalence review, and which runtime settings and observability artifacts future Go runs must capture.

The next weak point is the client side. A service benchmark is not valid if the load generator is saturated, misconfigured, or measuring latency in a way that hides pauses. Article 05 moves to load-generator validity: request scheduling, coordinated-omission risk, client headroom, backpressure, timeout accounting, and the evidence needed to prove that later results are about the service under test rather than the benchmark harness.

References

• Go net/http package documentation
• Go http.Server, ServeMux, and request body limiting APIs such as MaxBytesReader
• Go encoding/json package documentation
• Go crypto/sha256 package documentation
• Go runtime package documentation, including GOMAXPROCS, GOMEMLIMIT, and GOGC
• Go runtime/debug package documentation for memory-limit and GC-percent APIs
• Go GC guide
• Go runtime/metrics package documentation
• Go runtime/pprof and net/http/pprof package documentation
• Go diagnostics guide
• Helidon documentation for the Java/Helidon continuity point defined in Article 03
• OpenJDK project
• Oracle Java downloads
• RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format
• RFC 9110: HTTP Semantics
• Oracle AI Vector Search User’s Guide for the broader Oracle AI Database-backed application context

Key Takeaways

• This article defines what Java means for this benchmark: a Helidon SE service running on explicitly identified Oracle-built JDK variants, with a declared request-handling model, JVM policy, diagnostics, and manifest evidence.
• Virtual threads are treated as a request-handling model, not a performance claim. A later benchmark must prove which model was active and measure whether this workload benefits from it.
• Runtime labels require exact build evidence. A later result is not accepted if it only says “Java” or “Java 21” without provider, version, build, architecture, flags, and environment details.
• This article defines the Java contract only. The runnable benchmark package and benchmark results come later, after the series defines the full service contract, load generation, execution method, and analysis method.

You can make “Java” mean almost anything in a benchmark.

It might mean a plain JDK HTTP server, Spring Boot, Quarkus, Micronaut, Vert.x, Netty, or Helidon. It might mean platform threads, virtual threads, a reactive handler model, a native executable, a checkpoint/restore workflow, or an ordinary JVM run with JIT warmup.

It might mean default heap ergonomics, fixed heap settings, one garbage collector, another garbage collector, request logging on, TLS on, or a tuned command line nobody wrote down.

If a later chart labels one bar Java, that chart is already too vague.

This article defines the Java side before we measure it. For this benchmark, the Java service shape is Helidon SE. Measured Java evidence in this series is limited to explicitly identified Oracle-built JDK variants. The request-handling model must be declared. Startup and warmup boundaries must be separated. JVM tuning must be framed as a hypothesis. Diagnostics must be captured before results are accepted.

No benchmark result is published here. No winner is declared here. We are defining the Java/Helidon contract that later source code and benchmark runs must satisfy.

Where This Article Fits in the Series

This is Part 3 of a ten-part, single-machine benchmark series. Part 1 set the rule: method first, measurement later. A run is not useful because it completed; it is useful only if the service behavior, runtime identity, environment, load generator, run protocol, and analysis method are all traceable.

Part 2 defines the language-neutral service behavior. Both implementations expose the same health and work endpoints, follow the same request and response contract, and use the same posture for logging, TLS, compression, and related service settings in the primary pure-service benchmark.

Those requirements are not Java requirements or Go requirements. They are the HTTP contract both services must satisfy before either one is tuned or measured.

This article binds that neutral service contract to the Java implementation shape. It answers a practical question: when a later article says Java/Helidon, what exactly does that mean?

Part 4 applies the same discipline to the Go side. It defines the Go net/http implementation shape, runtime settings, profiling plan, and extraWork equivalence review. Parts 5 through 8 define load generation, execution, reproduction, and analysis. Results do not arrive until Part 9, with final interpretation in Part 10.

That ordering matters. If we tune Java first and define the Java contract afterward, the reader has to wonder whether the contract was shaped around the result. If we define the contract now, before source generation and before measurement, later Java results become auditable.

Later results must be generated fresh from the validated contract and preserved with the artifacts needed to audit them.

What “Java” Means in This Benchmark

For this benchmark, “Java” means a Helidon SE service that implements the finalized Article 02 endpoint contract exactly. The later implementation should make the Helidon SE WebServer routing and server configuration reviewable rather than relying on undocumented defaults.

That sentence intentionally excludes a lot. It does not say all Java services. It does not say all Java frameworks. It does not say Helidon represents Spring Boot, Quarkus, Micronaut, Vert.x, Netty, or a plain JDK server. Those are valid production choices, but they are outside the Java service shape selected for this benchmark.

Helidon SE is selected because its programmatic style can make routing and server configuration explicit in the benchmark source package. That helps reviewability for this particular contract. We want a reader to inspect the later source package and see the choices that matter, not infer them from a larger application stack.

The runnable benchmark must record the released Helidon SE version and resolved dependency set actually used for each run.

A later accepted Java run must identify the runtime shape in a manifest. This YAML is only an illustrative sketch, not the final manifest schema and not evidence that this article builds a service:

java_runtime:
  implementation_id: "java-helidon-se"
  framework: "Helidon SE"
  manifest_must_record:
    framework:
      - helidon_version
      - resolved_dependency_set
    jdk:
      - provider
      - version_output_file
      - runtime_properties_file
      - system_settings_file
      - effective_flags_file
      - architecture
    request_handling:
      - selected_threading_model
      - evidence_that_model_was_active
    execution_policy:
      - heap_policy
      - gc_policy
      - startup_boundary
      - warmup_boundary
      - measurement_boundary
    diagnostics:
      - gc_log
      - jfr_recording_when_enabled
      - process_metrics
      - request_metrics
    artifacts:
      - raw_result_files
      - summary_files
      - manifest_links

The important point is not the field names. The important point is the chain of evidence. A later Java result must tie together the framework, dependency set, JDK build, request-handling model, JVM flags, heap and GC posture, diagnostics, warmup window, measurement window, and raw artifacts.

The Java side becomes auditable only when the framework, JDK identity, request-handling model, JVM policy, diagnostics, and manifest are treated as one contract.

Java Runtime-Shape Map

The Java benchmark result is not valid until the Helidon SE service shape, JDK variant, request-handling model, JVM flags, diagnostics, and manifest evidence are captured from the validated implementation.

Figure 1. We define the Java runtime shape before measurement: Article 2’s service contract flows into Helidon SE, an exact JDK variant, a declared request-handling model, JVM policy, diagnostics, and a manifest before results are allowed.

The diagram is a reminder that the benchmark does not start at the load generator. It starts with the service contract. The Java implementation must satisfy that contract, record its runtime identity, and capture enough diagnostics to explain later measurements.

The Java Service Contract

The Helidon SE service must implement the finalized Article 2 language-neutral contract without adding framework-specific drift.

That means the Java service must expose the Article 2 endpoints and behavior exactly, and the later Go service must be held to the same finalized contract. The Java service must use the same request body limit, accept the same JSON media types, reject the same invalid request shapes, enforce the same numeric bounds, reject unknown and duplicate top-level request fields, return the same success fields, and return the same JSON error shape for body-bearing failures.

The primary benchmark posture also stays the same: request logging off, TLS off, compression off, and tracing off unless a later secondary slice explicitly enables one of them for both implementations. A Helidon default cannot become a hidden benchmark input.

This is where framework details matter. JSON parsing can differ in duplicate-field handling, numeric coercion, unknown-field policy, and error reporting. If the selected JSON library does not reject duplicate top-level fields by default, the Java implementation must add an explicit parsing or validation step that does. Routing can differ in how unsupported methods are handled. Server defaults can affect request size limits, connection behavior, response headers, exception mapping, and log output.

None of those details necessarily makes one framework better or worse. Hidden differences simply make results harder to trust.

The later Java source package must make these choices explicit enough for conformance tests to fail when the service drifts. A framework-default HTML error page is not a small cosmetic issue in this benchmark; it is a contract failure. Accepting "payloadSize": "1024" as a string when the contract requires an integer JSON number is also a contract failure. Returning an extra diagnostic field in /health is a contract failure.

Benchmarks often rush toward tuning. Here, correctness and equivalence come first. A fast service that answers a different question is not evidence.

Runtime Variants and Manifest Policy

Runtime labels are not vague labels in this series. They are benchmark inputs that must be backed by exact evidence.

For the measured Java evidence in this comparison, accepted Java runtime variants must be built by Oracle and labeled precisely. Non-Oracle JDK distributions are outside this measured comparison unless a later article explicitly defines a new scope and records them as separate, verified evidence. A result is not accepted unless the run records the exact provider, version, build string, architecture, effective JVM flags, operating system, and available processors as seen by the JVM.

These command shapes show the kind of runtime evidence the benchmark package needs to preserve:

mkdir -p artifacts
java -version > artifacts/java-version.txt 2>&1
java 
  -XshowSettings:properties 
  -version > artifacts/java-runtime-properties.txt 2>&1
java 
  -XshowSettings:system 
  -version > artifacts/java-system-settings.txt 2>&1
java 
  -XX:+PrintFlagsFinal 
  -version > artifacts/jvm-flags.txt 2>&1

Those files preserve the basic runtime identity. artifacts/java-version.txt captures the version string, runtime build, and VM name. artifacts/java-runtime-properties.txt captures properties such as java.vendor, java.version, java.runtime.version, java.vm.name, os.name, and os.arch. artifacts/java-system-settings.txt captures system and container-related settings exposed by the selected JDK. artifacts/jvm-flags.txt captures effective JVM flags, including defaults and explicitly supplied options.

These commands do not prove benchmark correctness. They only show the kind of runtime evidence the later benchmark must preserve. -XshowSettings:system should be validated for each selected JDK and Linux environment because the exact system and container fields exposed by the launcher can vary.

If the benchmark runs inside a constrained environment, the manifest must also record CPU and memory limits visible to the JVM. JVM ergonomics use available processor and memory information. If the service sees two processors but the host has many more, that is not a footnote; it is part of the benchmark input.

The series will not infer a JDK advantage from branding. Runtime variants must run the same service contract, under the same run protocol, with comparable diagnostics, before any comparison is allowed. If a later result differs, the first question will be: what exact runtime, flags, machine state, and artifacts produced that difference?

Virtual Threads Are a Request-Handling Model, Not a Result Claim

Virtual threads matter for the Java side because they can change how the service handles concurrent requests. JEP 444 defines virtual threads as a final feature in JDK 21 and describes them as lightweight threads managed by the JDK rather than the operating system.

That does not make virtual threads a magic performance switch.

Virtual threads are especially relevant when a service uses a thread-per-task style and spends time blocked on I/O. They can make blocking-style code easier to write and scale for suitable workloads.

But they do not make CPU-bound work run faster. If the handler is mostly JSON parsing, deterministic payload generation, checksum work, allocation pressure, or CPU-bound simulation, virtual threads may not improve throughput. They may still affect scheduling behavior, memory pressure, thread diagnostics, and latency distribution.

The contract treats virtual threads as a declared request-handling model. A platform-thread versus virtual-thread comparison is allowed only if the selected Helidon SE release documents both modes in a way that keeps the framework version, endpoint code, server architecture, JVM flags, and other benchmark inputs constant. Otherwise, the benchmark should declare the supported Helidon request-handling model and omit the artificial comparison.

The important rule is isolation: if platform-thread and virtual-thread variants are compared, the benchmark must prove that only the intended request-handling variable changed.

Here is the contract idea as a small sketch, not a Helidon configuration file:

request_handling:
  selected_model: "declared for the run"
  comparison_model: "included only when supported and isolated"
  result_claim_allowed_from_configuration_alone: false
  required_evidence:
    - selected_helidon_configuration
    - jdk_version_and_flags
    - threading_model_declaration
    - representative_thread_diagnostics_when_enabled

Diagnostics matter here. Blocking behavior, synchronized regions, native calls, and pinned virtual threads can affect scalability. Pinning is not automatically a correctness bug, but it can explain why a virtual-thread service behaves differently from the mental model. Representative Java diagnostic runs should capture enough evidence to inspect thread behavior when the methodology calls for it.

So the benchmark will not say “virtual threads are faster.” It will ask a narrower question: under the Article 02 workload, with the selected Helidon SE service and selected JDK build, does this request-handling model change throughput, latency distribution, CPU use, memory use, warmup behavior, or diagnosability?

That question is measurable. The slogan is not.

Startup and Warmup Boundaries

The baseline Java execution model is an ordinary JVM run with a defined warmup boundary and a defined measurement boundary.

That sounds boring, and boring is good here. A JIT runtime needs warmup time. The service may initialize framework structures, compile hot paths, settle allocation behavior, and reach a different steady-state profile than it showed in the first few requests. Later methodology articles define the warmup interval, measurement interval, repeat count, invalid-run rules, and analysis rules.

Any Project Leyden-related startup or warmup path is handled separately and only if the selected JDK exposes a supported, reproducible workflow for the benchmark. Project Leyden focuses on startup time, time to peak performance, and footprint. If the selected JDK provides a supported Leyden-related workflow, such as ahead-of-time class loading and linking for that JDK, the benchmark may include it as a separately labeled startup and warmup variant with recorded commands, generated artifacts, checksums, and training input. If it does not, the variant is omitted.

The contract is simple:

startup_warmup_policy:
  baseline:
    mode: "ordinary-jvm"
    included: true
    warmup_boundary: "defined by later benchmark methodology"
  leyden_related_variant:
    included: "only when supported by the selected JDK"
    required_evidence:
      - jdk_version_and_build
      - exact_record_create_run_commands
      - generated_artifact_filenames
      - generated_artifact_checksums
      - training_input_description
      - regeneration_rule_per_application_build
  graalvm_native_image:
    included_in_this_jvm_comparison: false
    note: "Separate execution model; not silently mixed into JVM results."

This separation prevents a common confusion. Leyden-related HotSpot and JDK work is not the same thing as GraalVM Native Image. GraalVM Native Image compiles a Java application ahead of time into a native executable. That changes the build and runtime model. It may be worth measuring in a separate slice, but it is not silently part of the ordinary JVM comparison in this article.

Checkpoint and restore workflows belong in the same category: separate execution model unless explicitly defined. They should not slip into the primary Java/Helidon comparison just because they affect startup or warmup.

If a later article includes a supported Leyden-related path, the result must identify the generated artifacts, training traffic, checksums, commands, and regeneration rules. Startup features without artifact discipline quickly become unrepeatable anecdotes.

JVM Tuning Hypotheses

JVM tuning in this series starts as a hypothesis, not a conclusion.

A fixed heap may make memory behavior easier to compare, but it can increase GC frequency under some workloads. Ergonomic heap sizing may better reflect default production behavior, but it can make memory comparisons harder if the runtime adapts differently under constraints. A pause-time target can influence latency behavior, but it is a goal, not a hard guarantee. A different collector may help some workloads and hurt others. A setting that improves one percentile can move cost into CPU, allocation rate, RSS, warmup, or throughput.

So the primary Java baseline should be intentionally plain and reproducible before variants are added. A reasonable Java run profile might fix the selected JDK build, fix the selected Helidon SE version, declare heap policy, use a documented baseline collector such as G1 when selected, enable GC logging when methodology requires it, and capture enough runtime evidence to explain what happened.

This illustrative launch shape shows diagnostic intent. It is not a final benchmark runner, and target/java-service.jar is a placeholder for a later built service artifact rather than an artifact produced by this article:

mkdir -p logs artifacts
java 
  -Xms512m 
  -Xmx512m 
  -XX:+UseG1GC 
  -Xlog:gc*,safepoint:file=logs/gc.log:time,uptime,level,tags 
  -XX:StartFlightRecording=filename=logs/run.jfr,settings=profile,dumponexit=true 
  -jar target/java-service.jar

If used later against a real service artifact, this style of run would produce a GC log and a Java Flight Recorder recording for diagnostic analysis. That does not make the command optimized. It only makes the run easier to inspect.

The later methodology decides which diagnostics are enabled during primary measurement and which are enabled only for representative diagnostic runs. Java Flight Recorder and profilers are useful, but they are not free. The benchmark has to define overhead policy rather than pretending diagnostics have no effect.

JVM tuning also needs to stay connected to the workload. An application service may spend time in HTTP handling, JSON serialization, validation, request shaping, vector-search calls, embedding calls, connection pools, or model gateways. The primary benchmark in this series isolates a pure service path first. If a later dependency slice adds a database call, the Java tuning question changes because blocking I/O, driver behavior, connection pooling, and database latency enter the picture.

That is another reason not to overfit JVM flags before the benchmark contract is complete.

Diagnostics Required Before Results

Diagnostics are not extra paperwork. They are how we avoid guessing.

If a later chart shows a latency spike, we need to know whether the service was CPU-saturated, allocating heavily, collecting frequently, warming up, blocked, pinned, hitting a request limit, logging unexpectedly, or being distorted by the load generator.

If a throughput difference appears between two runtime variants, we need to know whether the same flags, heap policy, and request-handling model were actually used. If memory differs, we need heap and RSS context, not a single number detached from GC behavior.

Every accepted Java run needs enough evidence to connect the observed HTTP behavior to the runtime state that produced it. That evidence includes runtime identity, framework identity, endpoint conformance, JVM behavior, process metrics, request metrics, and manifest links back to the raw artifacts.

Runtime identity means java -version, runtime properties, system settings, effective JVM flags, operating system, architecture, and available processors as seen by the JVM. Framework identity means the Helidon SE version, dependency set, and source revision used to build the service. Endpoint conformance means the Java service passed the Article 02 contract tests before measurement. JVM behavior means heap policy and GC evidence captured under the methodology’s diagnostics policy. Process and request metrics must cover the warmup and measurement windows, not just a single end-of-run snapshot.

Representative diagnostic runs can add more detail. Java Flight Recorder is especially useful on selected JDK builds where it is available and enabled for the benchmark methodology, because it can expose JVM events relevant to allocation, GC, I/O, threads, and runtime behavior. JDK Mission Control can later inspect Java Flight Recorder recordings when diagnostic analysis is needed.

async-profiler can add sampling views for CPU and allocation behavior, but it is a separate tool with platform and permission requirements. It should be used deliberately, not assumed to be present everywhere.

The goal is not to make every benchmark run heavy. The goal is to make every accepted result explainable. Primary measurement runs and representative diagnostic runs can have different profiles, as long as the methodology records which artifacts belong to which purpose.

The later load-generator article must also define whether load is open-loop or closed-loop, how client saturation is detected, how timing is recorded, how errors are counted, and how coordinated-omission risk is handled before any latency comparison is accepted. Java diagnostics can explain the server side only if the client side is controlled well enough to trust the measurements.

Acceptance Criteria for the Later Java Implementation

This article does not generate a Helidon project, build a JAR, run a load generator, or produce benchmark results. It defines the Java side of the contract. The runnable benchmark package is deferred until the series defines the full service contract, implementation plans, load-generator validation, execution method, reproduction requirements, and analysis method.

A later Java result is not accepted unless the Java implementation and run artifacts satisfy these gates:

• The service implements the finalized Article 02 endpoint contract exactly, including JSON request rules, success responses, error responses, media-type handling, request body limit, logging posture, and TLS posture.
• The service uses Helidon SE, and the manifest records the exact framework version and resolved dependency set used in the run.
• The run records the exact JDK provider, version, build, architecture, runtime properties, system settings, effective JVM flags, heap policy, GC policy, available processors visible to the JVM, and any CPU or memory limits imposed by the execution environment.
• The measured Java runtime variants are explicitly identified Oracle-built JDK variants unless a later scope change defines and validates additional runtime evidence separately.
• The request-handling model is declared as platform threads, virtual threads, or another explicitly documented Helidon- and JDK-supported model, with evidence that the intended model was active.
• Platform-thread and virtual-thread variants are included only when the selected Helidon SE release documents both modes in a way that allows the request-handling model to change without unrelated framework, endpoint, server-architecture, or JVM-policy changes.
• Ordinary JVM execution, Leyden-related startup and warmup behavior, GraalVM Native Image, and checkpoint and restore workflows remain separate execution models unless a later article defines a distinct variant.
• The run captures GC logs when required by methodology, process metrics, request metrics, warmup and measurement timestamps, and manifest links to artifacts.
• Representative diagnostic runs include Java Flight Recorder where the methodology enables it, and selected profiler slices where platform support and permissions allow them.
• The result does not compare against Go until the Go contract, load-generator contract, execution protocol, and analysis method are complete.

The smallest evidence-capture shape looks like this:

mkdir -p artifacts logs
java -version > artifacts/java-version.txt 2>&1
java 
  -XX:+PrintFlagsFinal 
  -version > artifacts/jvm-flags.txt 2>&1

Passing those commands is not enough to validate a benchmark. It only demonstrates the posture: runtime identity and effective flags are captured before results are reviewed.

If a later Java run is missing runtime identity, framework identity, request-handling evidence, warmup boundaries, diagnostics required by the methodology, or endpoint conformance, the run is not surprising. It is invalid.

Handoff to the Go Contract

The Java side is now specified enough to constrain later implementation work.

Java means Helidon SE. Runtime labels require exact JDK evidence. Virtual threads are a request-handling model. Leyden-related behavior is conditional and separate. JVM tuning is a hypothesis. Diagnostics and manifests gate results.

The next article applies the same discipline to Go.

That matters because Go has its own ways to drift. A compact net/http service can be easier to inspect, but it can still allocate differently, parse differently, validate differently, return different errors, or perform a different amount of work in the extraWork path. The Go implementation and tuning plan must be held against the same Article 2 service contract.

Java is specified here. Go is specified next. Results come later.

Conclusion: Make Java Auditable Before Measuring It

A defensible Go-versus-Java benchmark cannot leave Java undefined.

For this series, Java is a Helidon SE implementation of the Article 2 service contract, run on explicitly identified JDK variants, with a declared request-handling model, documented startup and warmup boundaries, testable JVM tuning hypotheses, required diagnostics, and manifest-backed evidence.

That does not make Java faster or slower. It makes the later Java result reviewable.

If you are building application services that depend on databases, model gateways, vector search, or other external systems, that is the useful habit to carry forward. Before comparing runtimes, define the service boundary. Before tuning, prove equivalence. Before publishing charts, capture the runtime evidence. Then, when results arrive, you can argue about the data instead of arguing about what was measured.

References and Methodology Notes

• Helidon documentation and Helidon SE 4 introduction for the selected Java framework shape and Helidon SE concepts.
• JEP 444: Virtual Threads for virtual-thread semantics and the important caveat that virtual threads are not faster threads.
• Java SE documentation for JDK tools, launcher options, Java Flight Recorder, and JVM behavior.
• Oracle JDK downloads for runtime distribution and build selection at benchmark time.
• Project Leyden and JEP 483: Ahead-of-Time Class Loading & Linking for Leyden-related startup and warmup context.
• GraalVM Native Image documentation for the separate native-executable model that is not silently included in ordinary JVM results.
• Java Flight Recorder API documentation and JDK Mission Control for diagnostic recording and analysis.
• async-profiler for optional representative CPU, allocation, and wall-clock profiling slices after platform compatibility is validated.

Recently, I was lucky enough to be invited to give a short presentation on blended teams, and I wanted to share some of my thoughts with you, dear reader!

What is a “blended team”? I am defining it as a team that has both human and AI workers who are collaborating on the same goals and tasks.

As you know, AI is good at some things, and not so good at others. You could say “when it’s good, it’s very, very good; but when it’s bad, it’s very, very bad” (wink to Henry Wadsworth Longfellow).

Vaccaro et al., Nature Human Behaviour, 2024 · When combinations of humans and AI are useful: A systematic review and meta-analysisSystematic review and meta-analysis synthesized 106 experiments and found that human – AI combinations underperformed the stronger standalone performer on average. The combination wins only when the task, evidence standard, reviewer, and escalation point are explicit.

The diagrams below, from arXiv:2507.09089v2 [cs.AI] Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity – Joel Becker, Nate Rush, Elizabeth Barnes, David Rein, present an interesting finding from the domain of software development: on average, experts tend to dramatically over-estimate how much AI assistance will speed up development. This study found that the estimates ranged from a 10% to 50% reduction in time – but the actual observed results were that tasks took up to 40% longer when AI assistants were used.

While developers estimated that AI assistance would enable them to reduce the time required for a task, as shown in the diagram below from the same study, the result was that they in fact took significantly longer when using AI.

Why is this the case? What’s going on here?

I’d like to offer the hypothesis that AI (and Generative AI in particular) is very good at doing things that have been done before, but much worse at doing something novel. This makes a lot of sense (to me, at least) if you consider that these models have been trained on very large corpora of existing code, and that they work by probabalistic prediction of the next token. So a model that has seen many examples of how to implement a particular type of requirement is likely to do a very good job of implementing that requirement. But, if you ask a model to implement something new, that it has not seen any, or many, examples of – it’s not going to do as good a job on that.

Another important thing to consider is the quality of the code the models were trained on. Charles Babbage called it way back in 1820 – garbage in, garbage out. And every developer knows, this is still very relevant today. If you consider that most of the code that the models were trained on is not what we would consider “good” – then you might not be surprised that the code they generate is also not “good”. Here’s some charts I pulled out of a paper showing distributions of some measurements of goodness across a wide sample of open-source code. The whole paper is very interesting and well worth a read.

Jin, Siyuan; Zhang, Mianmian; Guo, Yekai; He, Yuejiang; Li, Ziyuan; Chen, Bichao; Zhu, Bing; and Xia, Yong. “Software Code Quality Measurement: Implications from Metric Distributions.” 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS), 2023, pp. 488–496. https://doi.org/10.1109/QRS60937.2023.00054

Consider another example – AI assisted detection of security vulnerabilities. Models like Mythos and Fable can detect many more vulnerabilities that we ever did before them using human review and our pre-existing tools. But on analysis, we find that a lot of those issues detected (at least in my experience) were poor quality, false positives, or just duplicates of issues that we found with our existing tools. An enormous amount of human work is needed to review the output of these models and separate the signal from the noise.

With the widespread use of AI in this area, there also came a massive amount of work for software developers to review, categorize and address these issues. We’ve seen a spike in CVEs since the availability of these models, as shown in the diagram below, created from public CVE data reported by cve.org – note that this data set is up to end of June 2026. No one really knows if that trend will continue upwards – my educated guess is that all of the activity by security researchers and developers will result in a lot of issues being fixed, and we will possibly see that level off and start to drop again, as the industry as a whole gets more focused on fixing issues, and adapts our development, testing and release processes to catch more issues before shipping code.

There’s a lot of talk about AI replacing jobs, but I think it is much better to think about tasks that AI can do, not whole jobs. We all know that the job description rarely, if ever, actually describes the real job. There are a lot of tasks that people do that don’t make it into the job description. I don’t like to think about using AI to replace a person’s job, but instead to think about which individual tasks we could use AI for – tasks where we would get an improvement (in time, quality, etc.) and which would let the person focus more time on the more difficult (less automatable) tasks.

So what tasks are AI models good at? Here’s a synthesis from a number of studies (see the citations at the end of this article):

In my own personal experience, I’ve found that AI is really good at things like doing research, fact checking, writing code (for known patterns, and with strong guardrails), scanning for vulerabilities, and less good at things like writing prose, creating graphics, making decisions, and so on. Human verification is essential for all AI outputs, but especially whenever errors are costly, context is hidded or judgement defines success.

Here’s a finding from a study that looked at usage of Copilot for various tasks and how successful that usage was for different kinds of tasks.

arXiv:2507.07935v6 [cs.AI]  Working with AI: Measuring the Applicability of Generative AI to Occupations – Kiran Tomlinson, Sonia Jaffe, Will Wang, Scott Counts, Siddharth Suri

A very interesting study by Becker et al clearly demonstrates how the amount of time spent of different tasks changes when AI is used. Interestingly, the total amount of work is not necessarily the same, this looks only at the percentages, not the total.

arXiv:2507.09089v2 [cs.AI] Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity – Joel Becker, Nate Rush, Elizabeth Barnes, David Rein

This is supported by other studies which found that the total amount of time increased. Here are some findings from one such study in the domain of open-source software development:

And here are some findings from a study on professional writing – total time reduced but the amount of time spent of each part of the work changed:

Even if AI can reduce the amount of time it takes to write code from weeks to minutes – that does not mean I can ship the code and pass on that time gain to my end users. I still need to have people review the code, and I still need to do all the other security, compliance, and release management tasks. So while an individual task might be faster, my wall clock time is not necessarily impacted.

I love a poster I saw at a ballet school which read “Hard work is not for everyone” – I think that’s a very true statement. Using AI is easy, anyone can do it. But using AI well is a very different thing – and I think it is fair to say that most people probably won’t invest the time and effort to learn how to use AI well. This means that a lot of AI applications are going to be sub-par. You can easily find evidence for this – from the hype merchants publishing YouTube videos claiming to show you how to build an AI application in “five minutes”, to the high visibility failures like those showcased in “12 famous AI disasters”, Thor Olavsrud, CIO Magazine https://www.cio.com/article/190888/5-famous-analytics-and-ai-disasters.html and I suspect that you have personal experience of talking to an AI and finding its ability to actually “help” to be very poor.

Trust is vitally important in AI, and reputation damage and financial penalties are very real risks.

Using AI “well” means things like understanding how the models work, comparing the performance of different models and hyper parameter settings for your specific use case and possibly corpora, implementing guardrails and retries, using techniques like planning and reflection, building and using evaluations, knowing how and when to use different kinds of memory, using skills, and so on. No matter how good you are, I’d argue that you won’t get it right the first time, and you need a continuous improvement loop.

It’s important to collect inputs and outputs, and then to do human reviews, and provide feedback about what is good and what bad. This data can be used to implement improvements, so that in essence the application ‘learns’ over time how to do the task right.

You need a mechanism to route and track tasks. AI can work faster than people. You need a way to queue up tasks so that it can pick form the queue based on whatever criteria are appropirate, maybe the token quota for example. If a task is not clear, it should be able to ask questions before starting work – so you need a way to allow it to assign a task back to a human to get more information. You should use techniques like guardrails and adversial reviews (ideally with a different model) to control quality.

So when is AI going to excel?

I think that this is a good litmus test to apply when trying to identify tasks that AI will be able to excel at:

• Smaller, well-defined tasks with clear context, strong guardrails, and a strong definition of “right”
• Tasks that do not require novel thought or complex decision making
• Clean boundaries and handoffs between tasks
• Repeatable, high-volume tasks
• Continuous improvement feedback loop

We have a huge amount of opportunity ahead of us. Here’s an interesting finding from an Anthropic study that compares the opportunity in various industries to the actual observed usage today:

Labor market impacts of AI: A new measure and early evidence, Massenkoff, McCrory 2026 https://www.anthropic.com/research/labor-market-impacts

There’s lots of very active research in this area I’ve shared a number of papers that I’ve read myself as well as some of my conclusions from my own personal experience. I’d love to hear your thoughts and experience, please hit me up in the comments.

[1] Noy, Shakked, and Whitney Zhang. “Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence.” Science 381(6654), 2023. https://doi.org/10.1126/science.adh2586

[2] Brynjolfsson, Erik, Danielle Li, and Lindsey R. Raymond. “Generative AI at Work.” Quarterly Journal of Economics 140(2), 2025. First circulated as NBER Working Paper 31161, 2023. https://www.nber.org/papers/w31161

[3] Ishikane, Masahiro, Yuki Kataoka, Yasushi Tsujimoto, Yuki Moriyama, Yukimasa Matsuzawa, and Norio Ohmagari. “Efficacy of a Large Language Model Data Extraction System in Evidence Reviews for Emerging Infectious Diseases: A Randomized Crossover Trial.” Open Forum Infectious Diseases 13(7), 2026. https://pmc.ncbi.nlm.nih.gov/articles/PMC13393321/

[4] Khraisha, Qusai, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. “Can Large Language Models Replace Humans in Systematic Reviews? Evaluating GPT-4’s Efficacy in Screening and Extracting Data from Peer-Reviewed and Grey Literature in Multiple Languages.” Research Synthesis Methods 15(4), 2024. https://doi.org/10.1002/jrsm.1715

[5] Kalliamvakou, Eirini. “Research: Quantifying GitHub Copilot’s Impact on Developer Productivity and Happiness.” GitHub Blog, 2022 (updated 2024). Experiment conducted with Sida Peng and Aadharsh Kannan. https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/

[6] Becker, Joel, Nate Rush, Elizabeth “Beth” Barnes, and David Rein. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR / arXiv preprint, 2025. https://arxiv.org/abs/2507.09089

[7] Nadra, Samir, and James P. Purdy. “Is genAI a Good Editor of Academic Writing?” Computers and Composition 80, 2026. https://doi.org/10.1016/j.compcom.2026.102993

[8] Kumar, Deepak. “SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback.” arXiv preprint, 2026. https://arxiv.org/abs/2603.26130

[9] Ribalta-Albado, Maria, and Pere-Pau Vázquez. “Evaluating LLMs’ Abilities to Create Charts, a Systematic Approach.” Computers & Graphics 135, 2026. https://doi.org/10.1016/j.cag.2026.104544

[10] Collier, Zachary A., Richard J. Gruss, and Alan S. Abrahams. “How Good Are Large Language Models at Product Risk Assessment?” Risk Analysis 45(4), 2025 (published online in 2024). https://doi.org/10.1111/risa.14351

Key Takeaways

• Before asking whether Go or Java is faster, both services need to answer the same HTTP question. That means the same endpoints, request fields, validation rules, work-factor semantics, response JSON, and error behavior.
• The benchmark contract defines equivalent externally specified work, not identical CPU instructions. extraWork drives a deterministic operation whose result affects a returned checksum, so conformance tests can catch many skipped or altered implementations.
• JSON comparison should be strict about schema, values, types, and error codes, but not brittle about object member order or whitespace. Framework-default HTML, text errors, stack traces, and runtime metadata are contract failures for body-bearing responses.
• Shared acceptance tests are the gate into later implementation and benchmark work. The Java/Helidon and Go services must pass the same contract before any benchmark result is admissible.

You can make two services look almost identical and still measure different work.

One implementation accepts an unknown JSON field and ignores it. The other rejects the request. One silently clamps a payload size that is too large. The other returns a validation error. One computes a checksum from generated bytes. The other computes it from a different buffer, or runs a loop that does not affect the returned result. One returns a framework-default text error. The other returns JSON.

If those services go into a benchmark, the chart is already compromised. The result might look like Go versus Java, but part of the difference could be request parsing, validation policy, payload generation, error handling, or a work loop that does not mean the same thing in both implementations.

So before asking whether Go or Java is faster, we need to make sure both services are answering the same question.

The first article in this series set a fairness rule: no benchmark result is admissible until the method and evidence chain are strong enough to support it. This series is scoped to a later single-machine benchmark method, but the exact operating system, kernel, CPU, memory, runtime versions, and isolation settings belong in the execution-method article and benchmark manifest, not in this service-contract article. Here, we define the language-neutral service behavior that the Java service and the Go service must satisfy at the HTTP boundary before either one is tuned, load-tested, or compared.

For developers building service-tier APIs around Oracle AI Vector Search, retrieval, embedding, reranking, or Oracle AI Database workflows, this boundary matters. A small application service often sits between clients, model calls, and Oracle AI Database. If two versions of that service validate different JSON, generate different payloads, or return different correctness artifacts, later runtime measurements are not about the runtime anymore. They are about drift.

This article does not publish benchmark results. It does not choose a faster runtime. It does not implement either service. It does not design the full load generator or run the benchmark package. We are defining the service behavior that later code must satisfy.

What This Article Defines, and What It Does Not Measure

A language-neutral service contract is the externally visible behavior both implementations must provide, expressed without depending on Java or Go internals. For this series, that means the behavior a client can observe over HTTP: endpoint paths, methods, request headers, request JSON, validation rules, response JSON, status codes, and error shapes.

HTTP gives us the shared vocabulary for methods, status codes, content, and representations. JSON gives us a shared request and response format. The contract uses those pieces to define observable behavior explicitly rather than letting framework defaults decide.

This article defines:

• The endpoint set and request media type
• The JSON request and response shapes
• The request body limit
• Payload-size and deterministic payload-generation rules
• The meaning of extraWork
• Validation rules, duplicate-field behavior, and unknown-field behavior
• Success and error response schemas
• JSON equivalence rules
• Cross-implementation acceptance-test categories

It does not measure throughput, latency, memory, CPU, startup, warmup, garbage collection, scheduler behavior, Oracle AI Database query performance, model inference, or vector-search performance. Those questions belong later in the series, after the service contract, implementations, load generator, execution method, and analysis method are stable.

Later latency measurements also need their own load model, request scheduling policy, timeout policy, warmup plan, sampling method, and treatment of coordinated omission.

That distinction is not academic. It is the difference between a benchmark that can be challenged constructively and a benchmark that depends on trust.

What Endpoint Is Under Test?

For this benchmark contract, both services expose the same two endpoints: GET /health and POST /work.

GET /health returns 200 OK, Content-Type: application/json, and the exact JSON object shown below. It exists to let scripts and acceptance tests confirm that the service is up and speaking JSON. It is not part of the measured work path. The response must not include extra fields.

{
  "status": "ok"
}

No timestamp. No process ID. No framework name. No Java or Go version. Health is a readiness check for this contract, not a diagnostic endpoint.

HEAD /health follows HTTP HEAD semantics for the GET /health resource: it returns the same status and headers that GET /health would return, without a response body. The conformance suite treats that as protocol behavior, not as part of the measured work path.

POST /work is the endpoint under test. It accepts a JSON request body and returns JSON on success and on body-bearing failures.

For this benchmark contract, the request Content-Type must be application/json. For common-client compatibility, the service also accepts application/json; charset=utf-8. The JSON media type registration does not define a charset parameter; accepting that parameter here is a deliberate contract choice and does not change JSON processing. A missing Content-Type on POST /work is treated as an unsupported media type. The service returns 415 Unsupported Media Type with the agreed JSON error shape.

Successful and body-bearing error responses use Content-Type: application/json.

The request body for POST /work is also bounded. If the /work request content exceeds 4096 bytes after HTTP transfer decoding, the service returns 413 Content Too Large with the agreed JSON error shape. This limit is intentionally small because the request body contains only control fields, not the generated benchmark payload.

The rule is simple: for requests that can carry a response body, JSON in, JSON out.

That means success responses are JSON. Validation errors are JSON. Malformed request errors are JSON. Unsupported media type errors are JSON. Oversized request errors are JSON. Unsupported method errors are JSON, except where HEAD semantics require no response body. Unknown route errors are JSON. Framework-default HTML pages, plain-text errors, stack traces, redirects, or implementation-specific error documents are not acceptable for this benchmark contract.

Unsupported methods are also part of the boundary. If a client sends GET /work, the service returns 405 Method Not Allowed, includes Allow: POST, and returns the agreed JSON error shape. HEAD /work is unsupported and returns 405 Method Not Allowed with Allow: POST and no response body. If a client sends an unsupported body-bearing method to /health, the service returns 405 Method Not Allowed, includes an Allow header containing GET and HEAD, and returns the agreed JSON error shape. If a client sends POST /missing, the service returns a consistently defined 404 Not Found JSON error.

For the primary pure-service benchmark, request logging, TLS, compression, and tracing are off. Those are real production concerns, and later slices can measure them deliberately. They are not mixed into the primary service-equivalence contract. If a later scenario enables one of them, both implementations must use an equivalent configuration and pass conformance again.

HTTP status-code and method behavior in this contract follows the semantics defined by RFC 9110: HTTP Semantics. The endpoint names, response fields, and exact validation policy are benchmark contract choices.

What Does Each Request Contain?

The request body for POST /work is compact. It does not send a large payload over the wire. Instead, it tells the service how to generate deterministic bytes and how much deterministic work to apply.

The body must be a single top-level JSON object. Top-level arrays, strings, numbers, booleans, and null are invalid request shapes and return 400 Bad Request. Multiple JSON documents in one request body are also invalid.

A valid request looks like this:

{
  "requestId": "case-0001",
  "payloadSize": 1024,
  "seed": 42,
  "extraWork": 10
}

Each field has a specific job.

requestId is a stable client-provided identifier. The service echoes it in a successful work response. It must be a JSON string with length 1..128.

payloadSize is the requested deterministic payload size in bytes. For this contract, it must be an integer between 0 and 131072, inclusive. That upper bound gives us payloads up to 128 KiB without putting large payload bodies into article examples or small conformance fixtures. A value below 0, a value above the maximum, a string such as "1024", or a fractional number fails validation.

seed is the input to deterministic payload generation. It must be an integer between 0 and 2147483647, inclusive. Keeping it in a conservative integer range avoids JSON number interoperability problems and makes the same fixture easy to represent in both Java and Go.

extraWork is a bounded work factor. It must be an integer between 0 and 100, inclusive. A value of 0 is valid and means “generate the payload and compute the initial correctness artifact, but apply no extra repeated work.”

Numeric fields must be JSON integers, not quoted numeric strings or fractional values.

Unknown fields are rejected. That may feel strict, but it is useful strictness. If the Go service silently accepts "payload_size" while the Java service rejects it, the two services no longer have the same contract. If one parser ignores "debug": true and the other fails, the benchmark has hidden drift. During conformance, extra fields are a 400 Bad Request.

Duplicate member names in the top-level /work request object are also invalid for this contract. The request schema does not contain nested objects; if nested objects are added later, the duplicate-member rule must be extended explicitly.

JSON syntax and interoperability rules are defined by RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format. RFC 8259 describes JSON objects as unordered collections, says object names should be unique, and notes that behavior can be unpredictable when parsers receive duplicate names. This benchmark contract removes that ambiguity by rejecting duplicate top-level request fields before field validation.

The services should not silently accept a request body such as this:

{
  "requestId": "case-0001",
  "payloadSize": 1024,
  "payloadSize": 2048,
  "seed": 42,
  "extraWork": 10
}

Test fixtures use unique field names. Conformance tests include duplicate-field cases to make sure both services fail them consistently.

How Is Payload Size Controlled?

The request does not carry the full payload. The service generates the payload deterministically from payloadSize and seed.

That choice keeps the benchmark focused. If the client sends a large JSON string, we are measuring request-body transport, JSON string parsing, escaping behavior, memory copies, and possibly different decoder strategies. Those can be useful in another benchmark, but they are not the shape we need here.

For this contract, the input tells the service how many bytes to generate. Both implementations must use the same language-neutral payload-generation algorithm. They must not call a runtime-specific random API and assume the output will match.

The contract can be expressed this way:

state = unsigned64(seed)
for i from 0 to payloadSize - 1:
    state = (state * 6364136223846793005 + 1442695040888963407) modulo 2^64
    buffer[i] = (state >> 56) & 0xff

Both implementations must use the same constants, unsigned 64-bit wraparound behavior, byte extraction, and iteration count. The same payloadSize and seed must produce the same byte sequence in Java and Go.

That gives acceptance tests a clean target. A test case can say, “For payloadSize = 1024 and seed = 42, both services must generate the same buffer and return the same checksum after the same work factor.” We do not need to inspect the buffer over HTTP; the returned correctness artifact shows whether the service produced the specified result.

What Does the Work Factor Mean?

This is the part of the contract most likely to make or break the benchmark.

extraWork is not a delay knob. It is not “run a loop for a while.” It is not “do whatever is convenient in each language.” It is the number of times the service applies a specified deterministic operation to the generated payload buffer, and the operation must affect the returned checksum.

A language-neutral sketch looks like this:

buffer = generatePayload(payloadSize, seed)
state = SHA-256(buffer)
repeat extraWork times:
    state = SHA-256(state || buffer)
checksum = lowercaseHex(state)

The || symbol means byte concatenation: the current 32-byte hash state followed by the generated buffer. The final checksum is the lowercase hexadecimal encoding of the final SHA-256 state.

This does not claim that Java and Go execute identical machine instructions. They will not. The JIT compiler, Go compiler, libraries, memory layout, CPU intrinsics, allocator behavior, and runtime scheduling can differ. Java and Go may also use different SHA-256 implementations. The contract requires equivalent externally specified work: both services generate the same buffer, apply the same deterministic operation the same number of times, and expose the resulting checksum so tests can verify the externally specified result.

That last point is important. If extraWork does not affect the response, an implementation could accidentally skip the loop and still pass a superficial status-code test. Because extraWork affects the returned checksum, conformance tests can detect many skipped or altered implementations of the specified work. That does not prove instruction-level equivalence; it proves that the observable result matches the contract.

Checksum examples in this article are format placeholders, not test vectors. Conformance fixtures must use complete expected checksum values computed from the published payload-generation and work-factor algorithms.

What Does a Successful Response Look Like?

A successful POST /work response returns 200 OK, Content-Type: application/json, and a stable JSON body.

The success schema has this shape. The checksum value below is a format placeholder, not an expected value for the earlier request:

{
  "requestId": "case-0001",
  "payloadSize": 1024,
  "extraWork": 10,
  "checksum": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "ok": true
}

The placeholder checksum is not a test vector. A real conformance fixture must use a complete lowercase SHA-256 hex string independently computed from the published payload-generation and work-factor algorithms.

The fields are deliberately limited.

requestId must match the request. payloadSize must match the accepted request value. extraWork must match the accepted request value. checksum must be derived from the deterministic payload and the work-factor operation. ok is true for a successful work response.

The response intentionally does not echo seed; the seed is validated and used to derive checksum, but it is not part of the success response schema.

The response does not include runtime metadata. No thread ID. No goroutine ID. No hostname. No process ID. No memory statistics. No framework name. No timestamp. No JDK or Go version. Those can be useful in logs, manifests, diagnostic endpoints, or benchmark artifacts, but they do not belong in the correctness response.

Why so strict? Because correctness responses are compared across implementations. If Java returns "framework": "Helidon" and Go returns "framework": "net/http", either the comparison has to ignore fields or the services fail. Ignoring fields too broadly can hide real differences. The cleaner choice is to keep the response schema focused on the contract.

How Should Validation Failures Behave?

Validation behavior is part of the benchmark, not an implementation detail.

If invalid input changes which requests are accepted, how much work is performed, or what response body is serialized, it can contaminate later measurements. So the contract defines both status code and JSON shape for responses that carry a body.

A field validation error looks like this:

{
  "error": {
    "code": "INVALID_REQUEST",
    "message": "payloadSize must be between 0 and 131072",
    "field": "payloadSize"
  }
}

For this contract, request-shape and scalar validation failures use 400 Bad Request. That includes missing required fields, wrong JSON types, out-of-range numeric values, unknown fields, duplicate top-level request fields, and invalid requestId values.

Malformed JSON also returns 400 Bad Request, with the error code MALFORMED_JSON and a field value of null. A body containing multiple JSON documents is treated the same way.

Unsupported request media types return 415 Unsupported Media Type. Request bodies larger than 4096 bytes return 413 Content Too Large. Unsupported body-bearing methods return 405 Method Not Allowed, include an Allow header listing supported methods for that target resource, and return the agreed JSON error shape. HEAD responses follow HTTP no-body semantics. Unknown routes return 404 Not Found.

A non-field error uses the same envelope:

{
  "error": {
    "code": "UNSUPPORTED_MEDIA_TYPE",
    "message": "Content-Type must be application/json or application/json; charset=utf-8",
    "field": null
  }
}

The contract uses these machine-readable error codes:

• MALFORMED_JSON for syntactically invalid JSON or multiple JSON documents in one request body.
• INVALID_REQUEST for a valid JSON body with an invalid top-level shape, missing required field, wrong type, out-of-range value, invalid requestId, unknown field, or duplicate top-level request field.
• UNSUPPORTED_MEDIA_TYPE for missing or unsupported Content-Type on POST /work.
• CONTENT_TOO_LARGE for a /work request body larger than 4096 bytes.
• METHOD_NOT_ALLOWED for unsupported methods on known routes.
• NOT_FOUND for unknown routes.

Conformance fixtures compare HTTP status, machine-readable code, field, and JSON shape when the response carries a body. The message value must be a non-empty string, but basic conformance does not require exact message equality unless a fixture explicitly opts into it. If one service returns INVALID_REQUEST and the other returns bad_request for the same fixture, that is a contract failure.

When a request has multiple problems, conformance tests use this precedence order unless a fixture explicitly states otherwise:

• Match the route first.
• Validate the HTTP method for that route.
• Validate request Content-Type for POST /work.
• Enforce the 4096-byte request-body limit.
• Decode exactly one JSON document.
• Reject duplicate top-level request fields.
• Validate required fields, unknown fields, JSON types, scalar ranges, and requestId rules.

This order keeps routing and protocol failures separate from JSON-shape failures. Basic conformance fixtures should still prefer one failure per request, but the precedence rule prevents accidental drift when multi-fault cases appear.

The error object is a small custom JSON shape. It is not claiming RFC 9457 Problem Details for HTTP APIs compliance. If a future version adopts Problem Details, it should do so explicitly, including the appropriate media type and fields.

What Makes Two JSON Responses Equivalent?

JSON response comparison should be strict enough to catch real differences and flexible enough to avoid false failures from incidental serialization details.

A response with the same fields in a different order is equivalent:

{
  "ok": true,
  "checksum": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
  "extraWork": 10,
  "payloadSize": 1024,
  "requestId": "case-0001"
}

That object is semantically equivalent to the earlier success-response shape if all required fields, values, and JSON types match. Object member order and insignificant whitespace are not part of correctness.

But these differences must fail conformance:

• "payloadSize": "1024" instead of "payloadSize": 1024
• Missing checksum
• Different requestId
• Different extraWork
• A checksum that does not match the deterministic payload and work factor
• A different machine-readable error code for the same validation failure
• Extra runtime-specific fields when the contract forbids them
• A framework-default HTML or plain-text error body where JSON is required

Byte-for-byte JSON comparison is too brittle. A loose “does the response contain a few expected fields somewhere?” comparison is too weak. The target is structural comparison: required field names, JSON types, values, error codes, and status behavior must match.

The language-neutral contract defines the same request, validation rules, deterministic work, and JSON response before Java/Helidon and Go implementations can enter benchmark execution.

How Do Acceptance Tests Prove Equivalence?

Acceptance tests are the gate between contract and benchmark.

The test runner is not the important part yet. It could eventually be a small custom tool, a script, or a test suite in a language chosen for the benchmark harness. What matters is that the same cases run against both implementations and compare the same expected behavior.

The acceptance cases should cover the ordinary path and the annoying edges.

For the happy path, the suite needs a valid minimal request, representative small and medium payloads, the maximum allowed payloadSize, extraWork = 0, and the maximum allowed extraWork. The maximum-payload case does not need to combine maximum payloadSize with maximum extraWork in a basic conformance run. The point is to prove the boundary values and deterministic behavior, not to turn contract validation into a stress test.

For determinism, the same request sent repeatedly must return the same stable fields and checksum. The same test vector sent to Java and Go must return JSON-equivalent responses. Object member ordering must not affect the comparison.

For validation, the suite needs missing required fields, wrong JSON types, negative payloadSize, payloadSize = 131073, invalid requestId, negative extraWork, extraWork above 100, malformed JSON, multiple JSON documents, duplicate fields, unknown fields, unsupported media type, oversized body, unsupported method, and unknown route.

For error equivalence, both services must return the same status code, same machine-readable error code, same required error fields when a body is allowed, and no framework-default HTML, stack traces, or plain-text errors.

For protocol behavior, GET /health must return stable JSON, HEAD /health must follow HEAD semantics for the health resource, POST /work must require JSON, response Content-Type must be consistent with the contract, and 405 responses must include the appropriate Allow header.

Passing these tests does not prove performance fairness by itself. It proves the functional contract. Performance fairness also needs implementation review, allocation analysis, tuning discipline, load-generator validation, environment capture, run protocol, and statistical analysis. Those appear later in the series.

But this is the gate that comes first. Java and Go can differ internally. They can use different libraries, runtime strategies, allocators, diagnostics, and tuning flags. If they cannot produce the same observable behavior for the same fixtures, they do not enter benchmark execution.

The Handoff to the Implementations

The contract gives the implementation articles a narrow job: build services that satisfy the same externally visible behavior.

The Java/Helidon implementation and the Go implementation can choose idiomatic libraries, data structures, and runtime configuration where the contract leaves room. They cannot change endpoint paths, request fields, validation rules, deterministic payload generation, extraWork semantics, success response fields, error response shape, or JSON equivalence rules to make one implementation easier.

That may sound restrictive, but it is what makes the later comparison useful. Once both services pass the same acceptance suite, the benchmark can move on to the next questions: how the services are implemented, how the load generator is validated, how the environment is captured, how runs are executed, and how results are analyzed. Until then, there is no Go-versus-Java result to interpret. There is only a contract to satisfy.

References

The contract choices in this article use standards and product documentation as context, but the endpoint names and field names are benchmark design decisions.

• RFC 9110: HTTP Semantics defines the HTTP vocabulary for methods, status codes, content, and representation metadata used by the contract.
• RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format defines JSON syntax and explains why object member ordering is not a good byte-level equivalence target.
• IANA media type registration for application/json provides the media-type registration context for the request Content-Type policy.
• RFC 9457: Problem Details for HTTP APIs provides context for standardized HTTP API error responses, although this article uses a smaller contract-specific error shape.
• NIST FIPS 180-4: Secure Hash Standard provides context for SHA-256, the hash function used in the deterministic work-factor sketch.
• Oracle AI Database provides the database platform context for AI application services and database-backed workflows.
• Oracle AI Vector Search provides the high-level context for service-tier APIs around vector search and AI application workflows.

Key Takeaways

• This first article does not publish benchmark results. It defines the contract that later Go net/http and Java Helidon SE runs must satisfy before any chart is worth discussing.
• Fairness in this series means testable equivalence: the same endpoint behavior, payload rules, validation and error handling, plus declared CPU, memory, logging, TLS, and process-topology choices.
• The primary benchmark is a bounded single-machine Linux service-under-test benchmark. It is relevant to developers building AI services with Oracle AI Database, but it does not measure database execution, model inference, Kubernetes, autoscaling, service mesh behavior, or cloud-provider performance unless those appear as separately labeled slices.
• If we include multiple JDK builds, a distinct request-handling mode, a Leyden-related startup or warmup path, or GraalVM Native Image, each one will be labeled as its own runtime shape with exact versions, flags, generated artifacts, and run conditions.

You have probably seen this pattern before. A benchmark shows a compact Go service next to a compact Java service. The chart is interesting. The conclusion is tempting. Then the comments get more useful than the chart.

How many cores were assigned? Was the load generator the bottleneck? Was the Go implementation doing the same work as the Java implementation? What did memory look like? What did garbage collection do? Which JDK build was used? Did virtual threads matter, or were they just present? If the selected JDK includes startup or warmup features associated with Project Leyden work, did they change startup, warmup, first-request behavior, footprint, or steady-state behavior? Were the runs long enough? Were there enough repeats? Were the statistics strong enough to support the claim?

Those are not annoying questions. They are the right questions.

This series follows up on that kind of Go-versus-Java service comparison: a Go net/http service on one side and a Java Helidon SE service on the other. That shape is worth investigating, but only if the method is strong enough to carry the result. The follow-up needs to answer the questions that usually get buried under the chart: CPU and core disclosure, memory use, GC behavior, JDK build clarity, request-handling behavior where applicable, startup or warmup features where applicable, load-generator validity, implementation equivalence, longer runs, more repeats, and statistical treatment.

So Part 1 does not publish new numbers. No old CSV, old chart, or previous result claim is evidence for this run. This article defines the rules a later run must satisfy before any result deserves attention.

If you are building AI applications that use Oracle AI Database, that distinction still matters—but the primary benchmark measures the application-tier service boundary, not database behavior. A retrieval API, embedding gateway, reranking service, feature-enrichment service, or application-tier vector-search facade can spend meaningful time in ordinary service code before and after it touches a database or a model. Runtime efficiency, tail behavior, warmup, memory pressure, and operational visibility still affect cost and reliability.

But if we mix database latency, model inference, service mesh overhead, and runtime behavior into one chart too early, we will not know what we measured.

The first move is to narrow the question.

Why This Follow-Up Starts With a Benchmark Contract

A benchmark contract is the set of rules that determines what is measured, which variants are included, how implementation equivalence is checked, how runs are accepted or rejected, and how analysis is performed before charts appear.

That sounds formal, but the idea is simple: write down the method before you see the result.

Without a contract, benchmark discussion drifts quickly. One side argues that the Go service is smaller. Another side argues that the Java runtime has better optimization and diagnostics. Someone asks whether the request-handling model changed the result. Someone else asks whether the load generator was saturated. Eventually the conversation becomes a proxy argument about language preference instead of a concrete discussion about a specific workload.

Our thesis is deliberately narrower: a useful Go-versus-Java service benchmark starts with a contract, not a chart. We will define equivalent work, fair runtime variants, measurement rules, and acceptance gates before publishing results, so later performance claims can be judged against a declared method rather than accepted on trust.

The contract has to answer a few practical questions before any chart is allowed:

• What service behavior must the Go and Java implementations provide?
• Which runtime shapes are being compared?
• Which choices are fixed for the primary run, such as logging, TLS, CPU allocation, memory policy, and process topology?
• What correctness tests prove that both implementations do the same externally visible work?
• What evidence shows that the load generator has headroom?
• Which warmup and steady-state windows define the measurement?
• How many repeats are required, and what makes a run invalid?
• Which raw artifacts connect the final chart back to source, build, runtime, machine, command, and analysis rule?

That list is the benchmark contract in plain language. A script producing numbers is not enough. A valid result has to trace back to the source revision, build, runtime, machine, command, measurement window, raw output, and analysis rule that produced it.

That dependency chain is the whole point of the series.

We define the benchmark contract in Articles 1–8 before publishing fresh results in Article 9 and interpreting the safe conclusions in Article 10.

The benchmark manifest for accepted runs will record the exact Go, JDK, Helidon, Linux, hardware, and tool versions used, including details that affect interpretation such as kernel version, CPU policy, and whether the service is running directly on the host or inside a constrained environment.

Which Go and Java Runtime Shapes Are Included

The comparison in this series is not “Go versus Java” in the abstract. That framing is too broad to be useful.

The Go side will use Go’s standard-library net/http package. That keeps the baseline focused on the built-in HTTP server path and avoids introducing third-party router or middleware choices into the primary comparison. Real Go services often include routers, middleware, validation libraries, and observability agents, and that is fine. This benchmark starts with the standard-library service boundary because it keeps the baseline small enough to audit.

The Java side will use Helidon SE, the selected Java service framework for this series, so the benchmark compares one declared Java service shape rather than the broader Java framework ecosystem. Spring Boot, Quarkus, Micronaut, and other stacks matter in real systems, but adding them here would change the question. This series is about Go net/http and Java Helidon SE under a declared service contract.

We will not report the Java side as just “Java.” If multiple JDK builds are included, each Java result will identify the JDK distribution or build provider, version, build identifier, command line, JVM flags, and support or licensing label used for that slice where that label affects interpretation. The question is not whether a label is faster in the abstract. The question is how the same Helidon SE source behaves under the specific JDK builds declared in the manifest.

Virtual threads are part of that Java service-shape discussion, but they need careful framing. JEP 444 delivered virtual threads in JDK 21. They are most relevant when the service model uses many concurrent blocking operations, especially I/O-bound work. They let developers write straightforward blocking-style code while supporting high concurrency more efficiently in workloads that spend time waiting.

They should not be treated as a general throughput or latency improvement for CPU-bound handlers. CPU-bound handlers still need CPU. If a request handler is mostly serialization, checksum work, or allocation pressure, the virtual-thread story may look different from a service that spends much of its time waiting on blocking I/O.

So we will treat the Java request-handling model as something to verify and measure, not something to assume. We will document the selected Helidon version, JDK version, server configuration, and request-handling model before applying labels such as platform threads, virtual threads, or default Helidon behavior. If that combination provides a distinct, supported, reproducible virtual-thread service shape, that slice will be labeled separately. If virtual threads are simply part of the selected Helidon shape, the manifest will say so instead of pretending that “Java” is a single opaque runtime.

Project Leyden and related startup or warmup features get the same separate-label treatment, and only when the selected JDK provides a supported, reproducible path. Project Leyden aims to improve startup time, time to peak performance, and footprint. Related JDK work, such as JEP 483: Ahead-of-Time Class Loading & Linking, is relevant to startup and warmup questions, but availability depends on the selected JDK version and build.

Leyden-related JDK features and GraalVM Native Image are different technologies. GraalVM Native Image ahead-of-time compiles Java applications into native executables. Leyden-related JDK features such as ahead-of-time class loading and linking are a different path. If either is included, we will label it separately, record the exact toolchain, flags, generated artifacts, and measurement scope, and avoid folding those results into the default JVM result.

For any startup or warmup feature associated with Leyden work, we will look especially at startup, warmup, footprint, and first-request behavior. If it appears to affect more than startup or warmup, the analysis will need artifacts to support that claim.

What Fair Means in This Benchmark

Fairness is not a slogan. It is a set of checks.

For this series, fair means the Go and Java services perform the same externally visible work: the same endpoint semantics, payload rules, validation behavior, error behavior, response shape, logging policy for the primary run, TLS policy for the primary run, declared CPU allocation, declared memory policy, and process topology.

That last part matters. One Go process versus one Java process is a different topology from several processes per runtime. A service with request logging disabled is different from one formatting a line on every request. Plain HTTP is different from TLS. A colocated load generator is different from a load generator running with enough isolated headroom to stay out of the server’s way. None of these choices is automatically wrong, but hidden choices make results hard to interpret.

Equivalent work comes before tuning. Each implementation must provide the same externally visible behavior before runtime-specific optimization is considered: request parsing, response generation, validation, error behavior, and any configured dependency interaction.

If the Go handler accidentally performs extra allocation in a work loop that the Java handler avoids, that is not a Go runtime result; it is a benchmark implementation bug until proven otherwise. If the Java service returns a different error body, validates different parameter ranges, or serializes a different payload, that is not a runtime result either.

The contract will require correctness tests before performance runs. Those tests need to compare outputs across implementations using stable request cases, not generated numeric row identifiers or one-off local observations. If an implementation changes for equivalence, the changed source becomes the benchmark source. Old results do not carry forward.

Fairness also means symmetrical restraint. We should not tune one runtime aggressively while leaving the other at casual defaults unless that is explicitly labeled as a tuned-versus-untuned comparison. We should not enable diagnostics inside the primary measurement path for one runtime and not the other unless the overhead is measured and accepted. We should not declare a run valid if the load generator is the thing being measured.

A useful benchmark should make unfairness difficult to hide.

What the Single-Machine Benchmark Measures

The primary benchmark will measure a declared single-machine Linux service-under-test workload. The service process or processes being measured will run on one declared Linux machine with a declared CPU, memory, and process policy. The load-generator placement will also be declared and validated in the run protocol because the client side can affect the result. If a separate client host is used for headroom, it remains part of the measurement harness, not a claim about distributed deployment.

Within that boundary, the benchmark will measure service behavior under controlled workload shapes: payload size, work factor, concurrency, CPU allocation, process topology, warmup, steady state, and resource pressure. It will report throughput and latency distribution together, because average response time without distribution data hides too much. It will also report CPU use, memory use, runtime behavior, startup, warmup, and steady-state behavior.

Warmup means the period before the measurement window where the service is allowed to initialize, compile hot paths, populate runtime structures, and settle into a more representative operating shape. Steady state means the measurement window after warmup, when the benchmark records the primary service behavior for that run.

For a JIT runtime, warmup is obviously important. For a non-JIT runtime, warmup can still matter because caches, connection state, allocator behavior, and CPU-frequency behavior can change early-run measurements.

Java diagnostic runs will include runtime telemetry where appropriate, such as GC behavior, allocation pressure, thread behavior, and selected Java Flight Recorder data. JDK tools such as jcmd can help capture runtime state in diagnostic runs, with exact commands recorded in artifacts. Profiling and diagnostics can add overhead, so diagnostic runs must stay separate from primary measurement runs when that overhead matters.

Go diagnostic runs will also capture runtime and diagnostic information where appropriate, using Go tooling such as pprof, runtime statistics, and tracing. Again, the point is not to drown the reader in profiles. The point is to avoid explaining a throughput or latency shape by guessing.

The load generator is part of the benchmark, not an afterthought. A service benchmark is not valid just because the server process is busy. The client side also needs headroom. If the load generator is CPU-saturated, cannot sustain the requested rate, or distorts latency measurement, the run is evidence about the test harness, not about Go versus Java service behavior.

The selected load generator must be validated for headroom and latency-measurement behavior. A fixed-rate tool such as wrk2, or another documented tool selected later, can help address coordinated-omission risk, but the benchmark must still prove that the client is not CPU-, network-, connection-, or scheduling-limited. Latency reporting will need distribution detail, not just a single average or p99. Tools and formats such as HdrHistogram are useful here because they preserve a fuller percentile spectrum.

The primary pure-service results will stay separate from secondary slices. TLS is a useful slice. Request logging is a useful slice. A real dependency, possibly an Oracle AI Database call, can be a useful slice. But those slices answer different questions.

If we add an Oracle AI Database dependency later, the result may reflect driver behavior, connection pooling, query shape, database execution, network locality, and runtime behavior together. That can be useful, but it should not be confused with the primary pure-service benchmark.

One more boundary matters: this is an end-to-end HTTP service benchmark, not a JVM microbenchmark. JMH is useful for JVM microbenchmarks, but it is not a replacement for measuring a service path through HTTP handling, payload work, runtime behavior, load generation, and resource use.

What the Benchmark Does Not Measure

The boundary is narrow on purpose.

We keep the primary benchmark inside a single-machine service boundary, so later results are useful without pretending to answer deployment, cloud, or distributed-systems questions.

The primary benchmark will not support conclusions about Kubernetes scheduling, autoscaling, service mesh behavior, multi-node networking, cloud-provider performance, distributed reliability, rolling updates, node cold starts, or production fleet operations. It will not compare cloud providers. It will not make claims about how a full distributed system behaves.

It also does not measure all Java frameworks or all Go frameworks. It does not measure universal language superiority. It does not measure model inference. It does not measure Oracle AI Database-dependent behavior, including database-side vector search, in the primary run. It does not tell you how a full retrieval-augmented generation stack behaves under production traffic.

That does not make the benchmark small in a bad way. It makes the benchmark interpretable.

If you are building an AI application on Oracle AI Database, you often need both kinds of information. You need database and model benchmarks for database and model questions. You also need application-tier service benchmarks for runtime, concurrency, warmup, memory, and operational questions. We start with the service boundary so that later, if a dependency slice is added, we can see what changed.

A benchmark that answers one question clearly is more useful than a benchmark that vaguely gestures at ten.

How the Series Gets From Contract to Results

This is a ten-part series because the method needs to be visible before the result.

Part 1, this article, defines the benchmark contract and the scope boundary. Part 2 defines the language-neutral service requirements: endpoint behavior, payload generation, work factor, response shape, validation, errors, and cross-implementation acceptance tests. Those requirements come before Go or Java implementation detail.

Parts 3 and 4 then constrain the implementations. The Java article covers the Helidon SE service shape, request handling for the selected Helidon and JDK combination, JDK runtime inputs if multiple builds are included, startup or warmup features where applicable, and Java diagnostics. The Go article covers the net/http implementation, the equivalence review, allocation behavior, Go runtime settings, and Go diagnostics.

Part 5 focuses on load generation. That includes request model, fixed-rate behavior, coordinated-omission risk, client headroom, and the evidence needed to show the load generator is not the bottleneck. Part 6 defines benchmark methodology: run protocol, CPU and memory policy, warmup, steady-state windows, invalid-run rules, resource capture, and traceability. Part 7 turns the method into execution artifacts: environment, versions, commands, manifests, run order, and raw outputs.

Part 8 defines analysis before results: canonical data shape, aggregation, uncertainty reporting, plot rules, outlier handling, and invalid-run handling. Only after those steps does Part 9 publish fresh benchmark results. Part 10 interprets what those results mean inside the declared boundary.

That order is deliberate. If the analysis rules are written after seeing the charts, the reader has to wonder whether the rules were chosen to make the charts look better. If the implementation contract is written after tuning, the reader has to wonder whether equivalence was enforced or assumed. If the load generator is validated after a surprising result, the reader has to wonder whether the harness is being rescued.

Part 1 publishes no benchmark results. Later result articles will include or link to manifests that identify the source revision, build commands, runtime versions, machine configuration, load-generator configuration, raw outputs, and analysis scripts used for each accepted run. Each accepted run must be traceable to:

• The exact Go and Java source revision
• The build commands and generated artifacts
• The Go version, JDK distribution or build provider, JDK build identifier, and Helidon version
• The runtime flags and process topology
• The load-generator command and configuration
• The Linux environment manifest for the machine
• The warmup and measurement windows
• The raw latency, throughput, CPU, memory, and runtime diagnostic files
• The analysis script or notebook that produced the chart

This is how old charts and orphaned CSV files stay out of the evidence chain. They may explain why we are asking better questions, but they are not proof for the new run.

Acceptance Gates Before Results

A run does not count just because it completed.

The benchmark needs acceptance gates that reject bad evidence before it influences conclusions. Some gates are functional: the Go and Java services must return equivalent responses for the same valid requests and equivalent errors for invalid requests. Some gates are environmental: the machine, runtime versions, CPU topology, memory, operating-system state, and process topology must be captured. Some gates are methodological: warmup and steady-state windows must be declared before the run, repeat rules must be defined, and invalid-run reasons must be applied consistently.

A run does not count if the Go and Java services return different responses, the selected JDK build is not recorded, the load generator is saturated, the warmup period is missing, a repeat is discarded without a declared reason, resource data is missing, or raw artifacts cannot be traced back to source, build, runtime, and machine state.

That list is not exhaustive, but it shows the posture. Invalidity comes from contract violations, not from surprise. A run is not rejected because one runtime looks unexpectedly good or unexpectedly bad. It is rejected because it failed a rule that existed before the result.

Repeat aggregation also needs discipline. If multiple runs are collected, the series must say how they are combined and how uncertainty is shown. If a run is unstable, the instability is either a result to explain or a contract failure to document. It is not something to hide with a prettier chart.

Resource metrics must be interpreted with throughput and latency, not as a separate appendix nobody reads. A service that reaches high throughput by consuming much more CPU is different from one that reaches the same throughput with less CPU. A low-latency shape with high memory pressure has a different operational story from a low-latency shape with stable memory. GC pauses, allocation rate, heap behavior, and warmup matter because they explain why a service behaves the way it does under load.

This is especially relevant for AI application services. The expensive part of an AI system is not always the HTTP handler, but inefficient service code can still waste CPU that you would rather spend on database work, model calls, embedding pipelines, or concurrent users. The benchmark needs to report enough resource context to make that tradeoff visible.

The Questions This Series Will Answer

By the end of the series, you should be able to answer a tighter set of questions than “Which language is faster?”

Under this single-machine service-under-test workload, how do Go net/http and Java Helidon SE behave as payload size, work factor, concurrency, CPU allocation, and process topology change?

Does workload shape change the result? A small response with little work may stress the runtime differently from a larger payload, a more allocation-heavy path, or a higher-concurrency request pattern.

How much CPU and memory does each variant consume to produce its service behavior? Throughput without resource context is incomplete.

What are the Java runtime characteristics: GC frequency, pause behavior, heap behavior, allocation pressure, thread behavior, startup, warmup, and steady-state profile?

What are the Go runtime characteristics: allocation behavior, heap behavior, scheduler behavior, GC impact, startup, warmup, and steady-state profile?

Does the selected Java request-handling model matter for this workload? If it does, where does it show up: latency distribution, CPU use, memory use, concurrency behavior, warmup, or operational visibility?

If more than one JDK build is included, what changes, if anything, when the same Java source is tested with the same declared JVM settings and separately recorded JDK distribution, version, and build identifier?

Where do startup or warmup features associated with Leyden work help, if they are included: startup, warmup, first-request behavior, footprint, steady state, or not at all under this workload?

If GraalVM Native Image is included as a separate slice, what changes when the Java application is built and measured as a native executable rather than as the default JVM run?

Did both implementations perform the same work-factor calculation and payload handling under the language-neutral service contract? If an equivalence issue is found and corrected, which corrected source revision is used for the main run?

Does adding focused service behavior, such as TLS, request logging, or a real dependency, change the interpretation? If a dependency slice is added, which part of the result belongs to the service runtime and which part belongs to the dependency path?

Which conclusions are safe inside the single-machine service-under-test boundary, and which remain out of scope?

Those are the questions worth answering. Not because they produce a more dramatic chart, but because they make the chart harder to misunderstand.

References and Methodology Notes

These references support the terminology and scope used in this kickoff article:

• Go net/http package documentation
• Go diagnostics overview
• Helidon documentation
• OpenJDK project site
• JDK builds from OpenJDK
• Oracle JDK downloads
• JEP 444: Virtual Threads
• Java virtual threads guide
• Project Leyden
• JEP 483: Ahead-of-Time Class Loading & Linking
• GraalVM Native Image documentation
• wrk2 fixed-rate HTTP load generator
• HdrHistogram
• JMH Java Microbenchmark Harness

Conclusion: Method First, Then Measurement

The most useful thing this first article can do is refuse to rush.

Go offers a standard-library HTTP server and a common single-binary deployment style. Modern Java offers Helidon SE as one focused service framework option, virtual threads for suitable blocking-concurrency workloads, extensive JDK diagnostics, multiple JDK build choices, and ongoing work around startup, warmup, and footprint.

None of that decides the benchmark.

The benchmark has to decide the benchmark, and it can only do that if the work is equivalent, the runtime shapes are explicit, the load generator is valid, the environment is captured, warmup and steady state are separated, repeats are handled consistently, and uncertainty is reported honestly.

That is the contract. Next we define the language-neutral service requirements so Go and Java are asked to do the same work before either implementation gets tuned.